Should I block duplicate pages using robots.txt?

[kristinedwards58]

great clip keep it up =)

[agapitoflores001]

I think this has already been answered on previous videos. But anyway, like other videos, it helps a lot.

[Webnauts]

Using 304 If-Modified-Since in combination of a meta robots directives "noindex,nosnippet,noarchive,f­ollow" would be the best way to go. Everything else is simply BS.

[krishthewiz]

It sounded like the answer at the end was that we should not block. I also think that depending on the case you have to do a combination of these techniques, meta-robots, robots.txt and canonical, especially if re-architect-ing the site is not an easy option (in some CMSes' its never an option). Tried the parameter filter that Google provided, and it doesn't work as fast as I wanted it to. leaving Google to identify the dupe increase the dupes cache count 10x and resulted in a rank fall.

[palbertus]

"We can figure out the dups on our own".

Looks like Google would prefer to crawl all your site and take the filtering job on their own !

[dyoungprod]

Surprised canonical isn't mentioned as a solution here

[ericofranco]

agreed. He should mention canonical as the best choice here.

[iceveiled]

Actually web users have made google popular and the most used search engine, so if you want to point fingers, blame the collective world using the internet. I don't know about you but I don't want to go back to 1998 when search query results were filled with pages with ridiculous keyword spamming, hidden text, and the like. I'm not saying that people aren't gaming the google algorhytms as they are and forever will. But search has improved, thanks to google.

[SEOMofo]

Matt,

At the beginning of the video, it sounds like your answer is we SHOULD NOT block the URLs, because Google needs to crawl everything and figure out the duplicates for itself. But then at about 0:57 you seem to reverse your stance by saying we SHOULD block them.

Can you please clarify?

Thanks,

SEOmofo

[bigal21110]

so does it mean that if we have dups on our page, that doesnt hurt our ranking?

[PropellerBusted]

Buddha knows many things about the Internet.

[MsIleaneSpeaks]

I'm not a webmaster and after hearing this answer, I don't think most people that "think" they are webmaster really are either. LOL

Thanks for answering a question that I'm sure many people have.

@Ileane

[jschroeffel]

No mention of canonical with this question? odd

[infiltrator7777]

Isn't it SAD how we have to jump through hoops now because of Google rules??

Who says Google makes the internet rules?

[wicked4u2c]

They are not rules by "Google". They are standards for their Search Engine, just like Bing and others have theirs. At least Google has a great channel to help developers make sure their sites are optimized well for Google Search Engine. After all, Google is the most used Search Engine out there.

[infiltrator7777]

That still doesn't answer the question.

There are no rules here. He's only making assumptions.

Do you think you know?

[BIGELLOW]

@infiltrator7777 You only have to jump through the hoops if you want Google to index your site and if you want to rank highly. If you aren't concerned about search engines or ranking of your site, then you can completely ignore the "hoops".

[damianemanuel]

thx Matt, i love your videos.

[carterblizz]

what about canonical link tag?

[jazz0900]

doesn't creating duplicate content in the first place cause you to get ranked down?

[fferrero2008]

..and leaves the train 4 hours... LOL

Nice video, nice answer!

Thanks!