More on information theory

[TalynCo]

M-My brain...

[MeAndTheOtherMeAndMe]

Just 2 amazing things happend:1st: On my 1st attempt to post a positive comment to this video my computer hung up, maybe because of a divine intervention which wanted to prevent me from doing so, or maybe because i forgot to plug in the waterpump...(what's your guess on my uncertainty?) 2nd: You managed to explain this basis of information theory in a (to me) foreign language within 5 minutes, which a mathematic teacher couldn't in my (and his) native language within 1,5 hours. You are great!

[Engravingsful]

So axe with big but rare critical need better computer

[mediantiba]

I have a question. Doesn't the amount of uncertainty after you receive an outcome always equal zero?

Ub - Ua = I

Ua is always zero!

[DeluxeWarPlaya]

Good work on these videos dude, keep it up.

[Chrisguy42O]

please stop talking like such a smart ass

[comecra85]

you are awesome.

keep videos going.

i love you

[usename80]

what are u? i mean what do u do?

good work

[osnek77]

the very reason for using bits is because its the smallest amount of information, because information is based on DIFFERENCE: "white - black" , 0 -1 on -off ect.

anyway thx for video

[brudswick]

teach me algebra! :D. I have a feeling youd do well at it.

[demonicgrinch]

This was uploaded a day after my birthday. :)

[GomerDose]

Entropy is a song by Bad Religion

[holguint123]

So then how does entropy in info theory relate back to entropy in thermodynamics? I mean I realize they are both the same thing...but the definition I have seen from stat mech is:

S = k*ln(mult. of the macrostate/outcome)

I guess I'm just trying to see how the transition is made from pure mathematics to physics.

[FiverBeyond]

@holguint123

From what I understand of information theory, the two are actually not quite the same (different applications, and, in fact, different ways of calculating).

However, maybe here's an example that helps clear things up:

If you have a ten-sided die where every side has an equal chance of being landed on, then the system is in a state of maximum entropy.

If you have a closed system where every area has an equal amount of heat this system also has maximum entropy.

[puncheex]

@holguint123: What you are seeing is a pair of applications (thermo and info theory) that use the same math framework. When you compare the results of max entropy (in thermo, a system with homogenious matter and energy, a totally bland world) with info theory (a state in which every outcome possible is equally probable) it begins to fit. While I was introduced to both of them way back in college I never saw the connection, but ZJ has made it plain. Never stop learning. Thanks, ZJ.

[takezaspirit]

In your first video you stated:

I(X) = log2 (M)

Now you are using the values for I(X) in place of 'U before' (let's say Ub) which means that:

I(X) = Ub

You are apparently doing this to demonstrate that information is not equivalent to uncertainty but your proof requires equivocation the first place. Am I missing something?

[ZJemptv]

None of these videos include proofs; these are definitions used in the field of information theory (because they work).

[takezaspirit]

You're right. I didn't mean formal proofs. Sorry about that.

However, you are not being consistent with your language from video to video. Generally, you are demonstrating that information is increases with uncertainty (which is CORRECT) but then you keep saying the opposite, such as the beginning of this video as well as your intro to your first video on information.

[takezaspirit]

I am having trouble posting a weblink. Please do a google search for...

common misconceptions information theory

and go to the first result...

Let me know what you think about that.

[takezaspirit]

btw, I really enjoyed your videos. The math was helpful for me. I am only addressing what I see as inconsistencies in definitions.

[rautenkranzmt]

So, arguably, entropy is meta-information?

[TheLawnmower17]

What's the name of that music in the background? I'm almost certain it's used in 6 Days a Sacrifice.

[ZJemptv]

It's Leyenda by Isaac Albéniz.

[Ninwa]

Please keep these coming. Great work and good editing.

[faber939]

Thanks for the follow-up. ^^

Although what I was specifically curious about was the greater ammount of bits needed to represent uneven outcomes. A representation of heads or tails still only requires one bit of information regardless of the probability of it landing on either side, no? Am I missinterpreting what you've said?

[ZJemptv]

Actually, fewer bits are needed, on average, to represent uneven outcomes. A truly random information source, with equal probabilities for everything, has maximum entropy -- the most uncertainty that is possible. Unbalancing the probabilities actually decreases the entropy. As for outcomes (even and uneven) and how many bits are needed to represent them, I'll be addressing that in the next video. I promise it'll be pretty interesting stuff.

[SuperFlyNB]

Feels like I am taking my networking admin course again. Good Job

[DeluxeWarPlaya]

@SuperFlyNB Sounds like you missed the point of the video.

[MaisonDetre]

Yay! I love your editing...its fun.