Sort by time | Sort by thread (beta)
- Your queue is empty. Add videos to your queue using this button:
or sign in to load a different list.
Loading...Saving...
Loading...Saving...
I have had blank robots.txt files for a while now, simply because my logs get super full of 404 not found errors for robots.txt when it doesn't exist. With this, though, I'll just add that extra bit of "confirmation" to make sure all my content gets crawled.
LinuxForHumans 6 months ago
When are you going to stop giving out this 1st grade advice? Why don't you come out and just tell it like it is?
SEO is an art of mass manipulation and crap-link racketeering; and you are constantly shuffling things around to try and negate the barrage of garbage people throw your way.
No one who is worth their spam-based pay checks pays any attention to Matt Cutts.
david88at 6 months ago
This has been flagged as spam show
You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).
Source: Google Webmaster Tools Help
Help articles › Using Webmaster Tools › Site configuration › Block or remove pages using a robots.txt file
I am confused now :)
ireshsl 6 months ago
Comment removed
ireshsl 6 months ago
Also, make the most of the robots.txt file by adding the sitemap URL in there.
adithecool 6 months ago 11