MapReduce for the Masses: Zero to Hadoop in Five Minutes with CommonCrawl

In this screencast, we'll show you how to go from having no prior experience with scale data analysis to being able to play with 40TB of web crawl information, and we'll do it in five minutes.

Category:

Howto & Style

Tags:

• Hadoop
• CommonCrawl
• MapReduce
• Howto
• Big
• Data

[daveccampbell]

5 minutes? going on 2 days. eclipse for windows is just different enough from your videos to make a difference. JAVA_HOME issues...guess that never happens in 'real life', huh?

[ssalevan]

@daveccampbell Ooh, that doesn't sound like fun. Shoot me a message over YouTube and we'll debug it together.

[MySchizoBuddy]

in your example you created the data set on the fly. What if you already have the 40TB of dataset you can't possibly upload that to Amazon.

[JoelDReid]

@PaulMdx I don't care enough about this to be arguing about it, but I'll discuss it I suppose. Apple, using "oh es ex" out loud or "OS 10" in print you say? Where? Can ya find an Apple webpage or recent keynote or commercial or press release or support document or white paper where they say the letter X or wrote 10 for me? I don't doubt you, I've just never seen it and think of Apple as the just about the most careful self-marketing company ever.

[PaulMdx]

@JoelDReid Even Apple use OS 10 vs. OS X interchangeably.

[someman7]

@JoelDReid Since they program, I'd say they're probably informed enough about the things that are important. Pronounciation of the name of a proprietary OS is irrelevant.

[JoelDReid]

@someman7 Ha. Well for what it's worth, my um, "level of care" is zero as well, but I do think Common Crawl should probably try hard to sound informed on oh, things like, say... the OS of the largest publicly-traded company on Earth. You don't speak for the internet (neither do I), but yeah... Apple is kind of a big deal now.

[someman7]

@JoelDReid Level of care: None.