In this video interview, Adam Lasnik, Google Search Evangelist, explains how their Analyze Robots.txt tool helps webmasters spot problems that could block the GoogleBot from crawling files. The Web Crawl Analyzer tools can spot other types of crawling mistakes. These are found under Google Webmaster Tools, designed to help site owners diagnose and correct website problems.
thanks that help a lot
slinkyska8 2 years ago
Hi,
I tried to find the tool Adam talks about under tools in webmaster and its not there.
There is a tool which allows you to paste in your urls and then test if they would be blocked or not but that's not quite what he is talking about. This tool also does check if you have a robot.txt file.
I'd really like to find the one that Adam is talking about though!
By the way Ralph, these are excellent videos. Keep up the great work.
mikesed 3 years ago