Allow

When you create a website you use a SEO plugin like Yoast, Rankmath or another one. They have a  robots.txt file. This file is made to indicate some information and action for User-Agent and bots to do or not do when they are crawling your website. Allow guideline tells User-Agent that he can do the action in opposite to Disallow

				
					Allow:[path]
this mean where the bot is allow to start crwaling the website
Lets take for exemple a website called https://www.mywebsite.com/

Allow:/
this mean the bot can crwal the entire website, Disallow: 
mean the same thing

Disallow:/ 
Allow:/personnal-blog
this mean that the bot cannot crawl the website but only specifically the directory called https://www.mywebsite/personnal-blog

				
			
Scroll to Top