Uncrawled URLs in search results

[oceanofweb]

BUt what if we remove from URL removal tool and then since people will be linking to those specific urls ?

[agapitoflores001]

This one is very informative. I am learning a lot.

[RickettsFish]

This was very helpful, but I have a question. If we use noindex, you made it sound like we need to allow Google to crawl the page by not blocking it via robots.txt. Otherwise, if we blocked it, Googlebot wouldn't read the noindex tag. Is that right?

[lemannequin]

@RickettsFish Good question, seems like a catch-22 situation.

As I understand it, if you use robots.txt to block crawlers, your website won't get crawled. And it *may* not get indexed at all, unless other websites link to it. In that case, Google will show it in its index. So far, that's what Matt clearly explained in this video.

[lemannequin]

@RickettsFish

Then, this is my guess: if it's the case that many other websites link to your blocked website, Google may have to do some very basic crawling of your website, just to decide (via presence of "noindex" meta) if it should be listed or not on SERPs.

But then, I'm just guessing.

[lemannequin]

@RickettsFish:

Combining crawling with indexing / serving directives

Robots meta tags and X-Robots-Tag headers are discovered when a URL is crawled. If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving directives will not be found and will therefore be ignored. If indexing or serving directives must be followed, the URLs containing those directives cannot be disallowed from crawling.

[mutchy126]

Also robots.txt Sites dont have a Cached Version

[chiangmaihotel]

Thanks for this tip.It still avaiable today?

[mkarakas0690]

that tip is so good for me.. thanx matt!.. :)

[MichaelDadona]

I think better maintain that way, Matt. As many bloggers can 'ferry' that anchor texts like what you said. Especially, your example for NISSAN. Many small entrepreneurs (vendors) related to the industry able to get benefits from it. Like mine, Nissan Impul.

[kevinargh]

Just make sure you remove the rule from robots.txt first or Google will never see the noindex meta tag on the page.

[allison30dc]

Cool, that explains a lot, including what happened with those Google local listings that appeared to have been crawled that were robots.txt'd

[danielgayle]

This was actually pretty interesting. I didn't know that the meta tag "noindex" would actually totally dump it from the index. Very cool.

[gbmodern]

Wow, I never knew that! I though robots.txt actually blocked Google from listing the site.