1) Allow is non-standard, a normal robots.txt will only disallow access
2) If you have a private folder, use .htaccess to restrict access to it, if you put a url to a private folder in your robots.txt everyone will be able to read your robots.txt and see where your "private" files are and look at them
There are couple of errors in this video:
1) Allow is non-standard, a normal robots.txt will only disallow access
2) If you have a private folder, use .htaccess to restrict access to it, if you put a url to a private folder in your robots.txt everyone will be able to read your robots.txt and see where your "private" files are and look at them
WafflesRDelicious 7 months ago
very informative thanks
jlwqwerty 1 year ago
Hi,
what if I have multiple sitemaps.
eg: one for my blog (example.com/blog/sitemap.xml) and a 2 for my site
Great video btw, created my robot.txt, looking at this video.
nikhil612 1 year ago