More on information theory

A correction regarding what entropy is, an explanation of why bits are the unit of choice, and a demonstration of how information theory helps us.

More information on information:
http://tinyurl.com/zozlx
http://tinyurl.com/8bueub
http://tinyurl.com/dcsgt2
http://tinyurl.com/as2txv

The music is Isaac Albéniz's Leyenda, as performed by Gordon Rowland. http://www.musopen.com/music.php?type=piece&id=224

http://myspace.com/zjemptv
http://emptv.com/

Category:

Education

Tags:

• information
• theory
• entropy
• uncertainty
• and
• self-information
• measurement
• quantifying
• bits
• binary
• digits
• decimal
• base
• logarithms
• mathematics
• probability
• source
• symbols
• fair
• coin
• die
• communications
• claude
• shannon

[takezaspirit]

In your first video you stated:

I(X) = log2 (M)

Now you are using the values for I(X) in place of 'U before' (let's say Ub) which means that:

I(X) = Ub

You are apparently doing this to demonstrate that information is not equivalent to uncertainty but your proof requires equivocation the first place. Am I missing something?

[ZJemptv]

None of these videos include proofs; these are definitions used in the field of information theory (because they work).

[TheLawnmower17]

What's the name of that music in the background? I'm almost certain it's used in 6 Days a Sacrifice.

[ZJemptv]

It's Leyenda by Isaac Albéniz.

[faber939]

Thanks for the follow-up. ^^

Although what I was specifically curious about was the greater ammount of bits needed to represent uneven outcomes. A representation of heads or tails still only requires one bit of information regardless of the probability of it landing on either side, no? Am I missinterpreting what you've said?

[ZJemptv]

Actually, fewer bits are needed, on average, to represent uneven outcomes. A truly random information source, with equal probabilities for everything, has maximum entropy -- the most uncertainty that is possible. Unbalancing the probabilities actually decreases the entropy. As for outcomes (even and uneven) and how many bits are needed to represent them, I'll be addressing that in the next video. I promise it'll be pretty interesting stuff.

[Ninwa]

Please keep these coming. Great work and good editing.

[SuperFlyNB]

Feels like I am taking my networking admin course again. Good Job

[TalynCo]

M-My brain...

[MeAndTheOtherMeAndMe]

Just 2 amazing things happend:1st: On my 1st attempt to post a positive comment to this video my computer hung up, maybe because of a divine intervention which wanted to prevent me from doing so, or maybe because i forgot to plug in the waterpump...(what's your guess on my uncertainty?) 2nd: You managed to explain this basis of information theory in a (to me) foreign language within 5 minutes, which a mathematic teacher couldn't in my (and his) native language within 1,5 hours. You are great!

[Engravingsful]

So axe with big but rare critical need better computer

[puncheex]

@holguint123: What you are seeing is a pair of applications (thermo and info theory) that use the same math framework. When you compare the results of max entropy (in thermo, a system with homogenious matter and energy, a totally bland world) with info theory (a state in which every outcome possible is equally probable) it begins to fit. While I was introduced to both of them way back in college I never saw the connection, but ZJ has made it plain. Never stop learning. Thanks, ZJ.

[mediantiba]

I have a question. Doesn't the amount of uncertainty after you receive an outcome always equal zero?

Ub - Ua = I

Ua is always zero!

[DeluxeWarPlaya]

Good work on these videos dude, keep it up.

[DeluxeWarPlaya]

@SuperFlyNB Sounds like you missed the point of the video.

[Chrisguy42O]

please stop talking like such a smart ass

[comecra85]

you are awesome.

keep videos going.

i love you

[usename80]

what are u? i mean what do u do?

good work