Uncrawled URLs in search results
Loading...
Loading...
31,246
Uploaded by GoogleWebmasterHelp on Oct 5, 2009
Matt Cutts explains why a page that is disallowed in robots.txt may still appear in Google's search results.
- 62 likes, 6 dislikes
-
Link to this comment:
Share to:
Loading...
- 0:43How can new pages get indexed quickly?by GoogleWebmasterHelp35,304 views
- 1:58Google does not use the keywords meta tag in we...by GoogleWebmasterHelp195,602 views
- 1:15Which is better: an HTML site map or XML Sitemap?by GoogleWebmasterHelp23,020 views
- 2:21Secret feature of Google Readerby GoogleWebmasterHelp39,169 views
- 2:16How can I remove old content from Google's index?by GoogleWebmasterHelp14,134 views
- 1:01Should I use pipes or dashes in my titles?by GoogleWebmasterHelp21,861 views
- 6:44Robots.txt How to Use Robots.txt filesby ClaudePelanne3,436 views
- 1:12When would someone use "noindex, follow" in a r...by GoogleWebmasterHelp20,187 views
- 2:23How do you rate links from sites like Twitter a...by GoogleWebmasterHelp44,506 views
- 1:04Are shortened URLs treated differently than oth...by GoogleWebmasterHelp14,793 views
- 2:36Matt loses a betby GoogleWebmasterHelp23,449 views
- 1:31New Google mobile search results for BlackBerry...by GoogleMobileBlog18,462 views
- 26:00State of the Index 2009by GoogleWebmasterHelp18,003 views
- 3:05Snippets and Titlesby GoogleWebmasterHelp15,652 views
- 1:31Creating Your Robots.txt Fileby NicheBloggingProfits4,037 views
- 3:19Whitehat linkbaitby GoogleWebmasterHelp20,624 views
- 1:23How do meta geo tags influence the search results?by GoogleWebmasterHelp20,758 views
- 23:46State of the Index 2008by GoogleWebmasterHelp15,486 views
- 2:13How do you protect your blog from hackers?by GoogleWebmasterHelp9,756 views
- 3:11Two questions about the link: operatorby GoogleWebmasterHelp16,248 views
BUt what if we remove from URL removal tool and then since people will be linking to those specific urls ?
oceanofweb 3 weeks ago
This one is very informative. I am learning a lot.
agapitoflores001 2 months ago
@RickettsFish:
Combining crawling with indexing / serving directives
Robots meta tags and X-Robots-Tag headers are discovered when a URL is crawled. If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving directives will not be found and will therefore be ignored. If indexing or serving directives must be followed, the URLs containing those directives cannot be disallowed from crawling.
lemannequin 3 months ago
@RickettsFish
Then, this is my guess: if it's the case that many other websites link to your blocked website, Google may have to do some very basic crawling of your website, just to decide (via presence of "noindex" meta) if it should be listed or not on SERPs.
But then, I'm just guessing.
lemannequin 3 months ago
@RickettsFish Good question, seems like a catch-22 situation.
As I understand it, if you use robots.txt to block crawlers, your website won't get crawled. And it *may* not get indexed at all, unless other websites link to it. In that case, Google will show it in its index. So far, that's what Matt clearly explained in this video.
lemannequin 3 months ago
This was very helpful, but I have a question. If we use noindex, you made it sound like we need to allow Google to crawl the page by not blocking it via robots.txt. Otherwise, if we blocked it, Googlebot wouldn't read the noindex tag. Is that right?
RickettsFish 5 months ago in playlist General Tips 2
Also robots.txt Sites dont have a Cached Version
mutchy126 11 months ago
Thanks for this tip.It still avaiable today?
chiangmaihotel 1 year ago
that tip is so good for me.. thanx matt!.. :)
mkarakas0690 2 years ago
I think better maintain that way, Matt. As many bloggers can 'ferry' that anchor texts like what you said. Especially, your example for NISSAN. Many small entrepreneurs (vendors) related to the industry able to get benefits from it. Like mine, Nissan Impul.
MichaelDadona 2 years ago