Google ignoring robots.txt?

Loading...

Sign in or sign up now!
Alert icon
Upgrade to the latest Flash Player for improved playback performance. Upgrade now or more info.
365 views
Loading...
Alert icon
Sign in or sign up now!
Alert icon

Uploaded by on Sep 7, 2010

Matt Cutts of Google has previously stated that whilst they may list urls in your robots.txt they will not download them. Google webmaster tools says different. http://stewartmedia.biz/myblog/

Category:

Howto & Style

Tags:

License:

Standard YouTube License

  • likes, 0 dislikes

Link to this comment:

Share to:
see all

All Comments (4)

Sign In or Sign Up now to post a comment!
  • @HueyLewisRocks Neither do I but I try to be polite in publilc :)

  • You know mate not everybody listens to MC, a lot of people think he lies, he lies!! The Dr Cutts Google propaganda machine. It's fair to say I don't agree with that view of our mate Matt and hang on every word he says.

  • @pbuntine That explains some of it. Excellent point on logged in users. Data collection could also be happening via the toolbar. Time to run some tests! Thanks :)

  • Jim

    I believe that Google Labs Site performance retrieves download time from users. They don't download the page themselves. I don't know how they do this, but the result is that even pages that are only available to authenticated administrators are included in the performance evaluation. This is really bad when some of those admin pages include slow processes.

    You can remove these pages from the evaluation by removing the Google Analytics code.

    Hope this helps.

    Phil

Loading...

Alert icon
0 / 00Unsaved Playlist Return to active list
    1. Your queue is empty. Add videos to your queue using this button:
      or sign in to load a different list.
    Loading...Loading...Saving...
    • Clear all videos from this list
    • Learn more