Can I use robots.txt to optimize Googlebot's crawl?

[hellmind]

Why you dont stop answering question from that 5 year old ?, It's your cousin or something?

[agapitoflores001]

He is funny. I find it educational and entertaining as well. He lifted me up from boredom.

[jasonmorgan12390]

nice video thanks for sharing very interesting .. god bless

[macfilas]

KINDA NOOOOOOOOOO VIDEOS fan club!!!

[chiasma72]

There were NO ROBOTS txting, as suggested by the title.

You're a nice guy Matt, but I'm done with you professionally. :-)

[mhaidarhanif]

Google need the brand new Googlebot technology. Just like the new search algorithm.

[Sander33333]

How can i ask a question for a video?

[mhaidarhanif]

@Sander33333 Go to the webmaster forum. If you're lucky, Matt will answer your question in a video.

[iLovePalestineDotCom]

Something that Matt did not touch on is the fact that when you present Google with a new robots.txt file restricting certain area of your site from being indexed although that is not what you really wanted to accomplish. You are in essence telling Google, if you have already indexed certain areas REMOVE THEM because NOW they are restricted. I was so hoping that Matt touches on that but he didn't. So we do not know if this act damage the status of already indexed pages or not.

[bigal21110]

@iLovePalestineDotCom good note but I think that is OK. I think he meant doing it on weekly basis, for long period of time, could confuse the google bot.

But technically, you should go live if you aren't ready

[nationaloutlookmag]

@iLovePalestineDotCom You can always remove the already indexed pages too.

[Robbertbiz]

Good answer man!

[blopaaugusto]

Thanks for the tips Matt!

[prisontv]

Don't scare googlebot, come on man :(

[seosemarpit]

Awesome Matt you Rockkk!