Change Player Size
Watch this video in a new window

Uncrawled URLs in search results

Matt Cutts explains why a page that is disallowed in robots.txt may still appear in Google's search results.  
 
Customize

More From: GoogleWebmasterHelp

Secret feature of Google Reader2:21
27,888 views
Duplicate Content & Multiple Site Issues14:38
22,587 views
My Tracks2:44
6,805 views
Can you benefit from content scraped from your site?1:26
4,286 views
Why doesn't google.com validate?2:06
11,721 views
Can I publish 100+ pages at once?1:22
5,295 views
What will happen to paid links implemented using JavaScript?2:24
5,267 views
Is Google putting more weight on brands in rankings?3:31
23,037 views
Google Mobile App4:35
6,010 views
Using a barcode scanner with Google Book Search3:33
20,544 views
Are shortened URLs treated differently than other redirects?1:04
7,504 views
Canonical Link Element20:28
38,548 views
Two questions about nofollow2:46
16,337 views
Does Google value its own links for PageRank?1:27
7,941 views
What are your views on PageRank sculpting?1:49
9,618 views
How are duplicate shopping pages with different currencies handled in Google?2:24
2,495 views
Google does not use the keywords meta tag in web ranking1:58
68,059 views
Will Google ever provide statistics on browser market share?1:08
3,006 views
What is Google doing to combat the effects of comment spam?2:16
3,050 views
Will you provide information about algorithm updates?2:12
4,201 views

QuickList(0)

Featured Videos

Upgrade to Flash Player 10 for improved playback performance. Upgrade Now or get more info.
40 ratings
Sign in to rate
16,146 views
Want to add to Favorites? Sign In or Sign Up now!
Want to add to Playlists? Sign In or Sign Up now!
Want to flag a video? Sign In or Sign Up now!

Statistics & Data

Loading...

Video Responses (0)

This video has no Responses. Be the first to Post a Video Response.
Sign in to post a Comment

Text Comments (7)   Options

Loading...
mkarakas0690 (4 weeks ago) Show Hide
 0
Marked as spam
that tip is so good for me.. thanx matt!.. :)
MichaelDadona (1 month ago) Show Hide
 0
Marked as spam
I think better maintain that way, Matt. As many bloggers can 'ferry' that anchor texts like what you said. Especially, your example for NISSAN. Many small entrepreneurs (vendors) related to the industry able to get benefits from it. Like mine, Nissan Impul.
kevinargh (1 month ago) Show Hide
+1
Marked as spam
Just make sure you remove the rule from robots.txt first or Google will never see the noindex meta tag on the page.
allison30dc (1 month ago) Show Hide
 0
Marked as spam
Cool, that explains a lot, including what happened with those Google local listings that appeared to have been crawled that were robots.txt'd
danielgayle (1 month ago) Show Hide
 0
Marked as spam
This was actually pretty interesting. I didn't know that the meta tag "noindex" would actually totally dump it from the index. Very cool.
gbmodern (1 month ago) Show Hide
 0
Marked as spam
Wow, I never knew that! I though robots.txt actually blocked Google from listing the site.
Comment(s) marked as spam Show

Would you like to comment?

Join YouTube for a free account, or sign in if you are already a member.