Information theory: Entropy

[bebeheuy]

I am very happy to see the vidoe after you give this An explanation of entropy in information theory and how to calculate it.

[AntoMelta]

I Really Like The Video From Your An explanation of entropy in information theory and how to calculate it.

[willamricard]

Your Video Is Very Useful Sharing An explanation of entropy in information theory and how to calculate it

[imegatrone]

after i watched this video, my insight is very open because the video An explanation of entropy in information theory and how to calculate it is very good to give information

[Jdonovanford]

The real formula has a negative symbol before the summation, man!

[xsni]

cool vid 

[SunStrokeCactus]

that dude makes me feel very stupid, and thats a very sad thing for me... :(

[MightyTokez]

i see a lot of great minds on youtube.. continue learning, finish school and lets fix this crazy planet! Thank you ZJ for this cool and informative video.

[Yallniggasbetrollin]

That's entropy or E-N-T-R-O to the P to the Y,

the reason why the sun will one day all burn out and die.

Order from disorder is a scientific rarity,

allow me to explain it with a little bit more clarity.

Did I say rarity? I meant impossibility,

at least in a closed system there will always be more entropy.

That's entropy and I hope that you're all down with it,

if you are here's your membership.

[Gahuki44]

This made my brain hurt. =(

[Anonymouz1337]

Are you a wizard?

[XHUMBITO1]

wow, you are great.



[wownov83]

lol you're explaining information entropy to people but presuming they don't know what the symbol for summation is?

[Nademox]

@wownov83 i think it's a good practice to unpack the sigma expression esp. if your audience are students?

[jjepsuomi]

nice vid =) thnx =) Still got one question ;) Why does the I(x) equal log (1/p) ? I mean, why does the function have that form, does it just suit our purposes? I think I'm asking about the intuition behind that formula:) Did the guy who invented this just sit around and then suddenly "Oh yes, it's gotta be log2 (1/p)". You understand my question? =)

[stevjen1]

I wonder how this holds true when say a building is blown up? From standing structure to a pile of rubble? What about gravitational forces?

[grungeiac]

0:26 did you say you weren't good with math? uhm. wut?

[maplebayou1]

I note that in a study published just last month (Toyabe et al. 2010) information is actually converted into energy using a Szilard's engine. Remarkable.

[ianmathwiz7]

@maplebayou1 yes, that's true. in fact, there's a reason that informational entropy has the same name as thermodynamic entropy--the two concepts are related.

[Dulipsa]

thumbs up dude!!!

[cristiantimofi]

The greater the entropy the greater the information received... :)

[oxwithoutoneeye]

as a boy you looked a lot better

a lot

[clemonsx90]

I dig your stole man, I dig your style.

[Saktoth]

'If you're like me and you're not good with math'

o_o

You arent? Oh right this is the simple stuff.

[crawhip7]

yeah i knew you were really a guy, brains 'n all. haahhaaa!

[olivermitevski]

Why explaining what sigma is? If someone doesn't already know what sigma stands for, how the hell is he going to understand entropy. LOL

[you1man]

Dude you are missing the minus "-" sign in entropy formulae. 1/2log(1/2) + 1/2log(1/2) will give -1 not 1.

[tmptsun]

@you1manno , you are wrong, she used a different formula. she used log(1/p(xi)) NOT log(p(xi)).

[FiverBeyond]

Information Theory FTW!!!

I love it, and love learning about it. Absolutely fascinating stuff...

[dirtbones]

By the end of your video I understood the thoery you were explaining. I feel smarter after reading your videos, you explain things in a clear and concise manner.

[Howie47]

Information does not arise from chaos by chance. Information coming from chance is like free energy from a perpetual motion machine. The information from a coin toss is front loaded into the coin toss mechanism. It is certain on a fair trial that information will arise. So information is not rising from uncertainty. "there is no free lunch" when it comes to the creation of information. The opposite isn't true. Info can fall into chaos. It just can't arise from it. The way I understand it.

[TheStraightFacts]

Actually information can arise by various natural processes. High information content can result from random events, such as mutation. The way that you understand it is unfortunately wrong.

[Howie47]

@TheStraightFacts You can start by learning what information is, watch this video, "Information". IF you follow all the logical reasoning you'll learn that, information past a very simple context, can only be generated by a mind.

[TheStraightFacts]

@Howie47

Information does not have to come from a mind at all. Take weather forecasting. Meteorologists gather vast amounts of 'information' from the natural world; pressure, wind direction, temperature, wind speed etc., and process this to produce accurate weather forecasts. So clearly hundreds, or thousands, of these bits of information are required to make an accurate prediction. However these bits of information do not come from a mind.

[Howie47]

@TheStraightFacts Very funny. You should have watched the video "information". Your confusing

useful facts with information. Info reduces uncertainty. Your weather facts have to be analyzed by a mind to come up with some certainty in a weather prediction. Facts so thoroughly assimilated as to have produced sagacity. That's the kind of information in DNA. It produces a certain out come. The production of an exact life form. It's written in nature, but produced by a mind.

[Howie47]

@TheStraightFacts "The process" takes a mind to analyze the "facts" you mention. Without the mental process the facts don't inform us of the future weather. You should have watched the video, "information" to get the meaning used when referring to DNA. Your def. says every thing we sense is information. DNA has specific, functional, information. A plan for the life that carries it. The kind that denotes wisdom.

[TheStraightFacts]

@Howie47

You are just talking random nonsense. I provided an obvious counter example to your claim that information only comes from mind, and showed it is demonstrably wrong. I can just as easily make the same response to DNA. Information has a technical meaning, as understood by mathematicians and information theorists, and it has nothing to do with 'wisdom' or 'mind'.

[Howie47]

@TheStraightFacts It is not random non-sense! There is a difference between specific information and random facts and figures. Seeing a face in the clouds that no one else can see, is random info. A face of Lincoln painted by an artist is specific & certain. The info in DNA reproduces an exact copy of itself. It is not just a bunch of unrelated facts that just happen to look like they were designed, because our minds are fooling us into seeing some thing that isn't really there.

[TheStraightFacts]

I am not talking about seeing a face in clouds, but 'information' as that term is understood in a technical sense can be produced naturally. Kolmogorov theory of information is well-grounded mathematically, and there are thousands of papers explaining it and its consequences. According to that interpretation any random source produces information. This is agreed upon by all information theorists, and is not a point of debate.

[Howie47]

IF you understand a theory you can put it in your own words and explain it simply. As I have. Any thing else is hype. There is no (specific) information in a cloud for the design of a human face. But an artist's portrait is information rich. Binary computer code has no specific information until it is arranged (programed) to have it. I already quoted here the experts you point too. They say specified information does not happen by chance. Not even a computer can wright a novel!

[TheStraightFacts]

Give me one information theorist that agrees with your claims. There is no such thing as 'specific information', it has no rigorous technical or mathematical defintion. I repeat your are wrong; information arises all the time, whether that is consider "by chance" or not.

[Howie47]

Believe what ever you want then. Your obviously unteachable. I already provided a quote (6 times) on this thread. From the "No Free Lunch". theorem. Which states that information is only transfered, that it doesn't arise by chance. You argue from ignorance and dubious authority. Scientist don't agree on much of any thing. Another materiast that can't think for himself.

[TheStraightFacts]

Notice that I asked you for an "information theorist." Dembski is not much of an information theorist, having published exactly 0 papers so far on the topic in the peer-reviewed scientific literature. He is flat out wrong. His bogus claims have been refuted;

talkreason . org/articles/eandsdembski . pdf

[FiverBeyond]

@Howie47

No, actually, that's one of the key points of information theory. Information can be random, or natural, or it can be intelligently generated. A good example... it doesn't matter how 'smart' the source of the bits on a hard-drive is (e.g. actual files, or just random bits)... it still takes the same amount of work to encode it.

[Howie47]

Very naive understanding of Information Theory, FiverBeyond. An actual information theorist just wrote a book. "Probability's Nature and Natures Probability. He says it is impossible for the "prescripted information" that is seen in Life, (DNA) to arise by any other cause then intelligence. Many scientist agree with him. You can watch seminar at, (scienceintegrity period Net}

[FiverBeyond]

Excuse me, but I'm studying Information Theory right now and am very surprised at some of your claims.

I've taken a look over the book you mentioned, and quite frankly, it doesn't seem to be addressing the field of Information Theory. He doesn't follow Shannon's theorems or formulae as far as I can see... it looks like he's just using the colloquial, non-Information-Theory definition of 'Information'.

As an example: does he give the equation for calculating the amount of information?

[Howie47]

You should then watch his seminar, linked on that page. "Example video" link. Your making a basic mistake. Shannon Information deals with bits of unintelligible carrier of information. The information is lacking (not coded) in random bits. I can't take your assertion to be studying "information theory" serious. OR how could you make such a basic mistake?

[FiverBeyond]

I must protest... what on earth have I stated about Information Theory that shows that I haven't studied it?

[Howie47]

Information Theorist Dr. Don Johnson, " The Information Contained in Life Lifes information origin, modification, preservation, processing, and detection as well as capacity and content are covered, proving from known science that such information has zero probability of arising by undirected processes."

[FiverBeyond]

And you quote the author of the book as support?

As I clearly just stated in my comment, this use of 'information' seems to be the every-day usage, instead of connecting to Information Theory.

You called my understand of I.T. naive... I presume, from that, that you have a basic understanding of the theory. If this is the case, then could you please tell me how Shannon's basic theorems apply to anything that Dr. Don Johnson has said, other than using some new definition of 'Information'?

[Howie47]

Your referencing "Shannon's Information theory". Information theory goes much further; and much more up to date then what Shannon"s, "to find fundamental limits on compressing and reliably storing and communicating data". At least read wiki, before you start spouting none sense.

[FiverBeyond]

You're right, information theory has explored several different areas, but Shannon's theorems have always been and still are the basic building blocks of every advance, be it the Shannon-Hartley equation, the Nyquist-Shannon equation, etc.

The more I talk to you... the more it feels like you're mixing up 'Information Theory' with 'Philosophy of Information."

Case in point: Do you believe that Information can be quantified? Could you give an example? This is very basic I.T.

[Howie47]

That all depends on what one is referring to when they say "information". Shannon was usually using the word information as a convenience. The data his theory mostly dealt with was unencode random bits. Like static. Qualified Information has explanatory properties. The difference between static noise or the first 25 prime numbers encoded in noise from outer space. One informs us of and intelligent source. Can you tell which one?

[FiverBeyond]

Yes, but that is precisely my point: the word "Information" is a highly philosophical one, but Claude Shannon's restrictive and mathematical "Information Theory" is NOT philosophy, but an actual study of a particular aspect of data.

I should point out my quest deals with quantity, which you switch to quality in your response.

I must ask again, as a very basic I.T. question: In "Information Theory", can information be quantified? If so, could you give an example?

Again, this is I.T. 101

[Howie47]

Yes, because life contains a quality of information as well as quantity. Can the quality be measured mathematically?

IF quality information is philosophical, (as you point out) which I don't disagree with, all thought it also might be scientific. Then life contains philosophical elements and is not just material.

[FiverBeyond]

Ah, but this is my point.

It is the Creationist goal to stop Information from being quantified, which is precisely what the entire field of Information Theory is supposed to do. Creationists do this because they know that if we ever actually tested to see how much information is in the genome, it would show that information is being added over time, mathematically speaking.

"Quality" lies outside Information Theory, and belongs to Philosophy of Information.

[Howie47]

Your using the same (ploy) used by beginners in the evolution debate. "abio-genesis" isn't part of evolution, "so I don't have to address that problem". Sure the number of base pairs can be added. Which tells us nothing about the quality of the information encoded there in. Do people really fall for that argument at your U.? Your drawing an artificial line between science and philosophy, that does not exist in nature. That kind of science is impudent from finding truth.

[Howie47]

I mean, "impotent from finding truth."

[FiverBeyond]

He he he... except that I haven't mentioned abiogenesis ONCE in my comments to you... have I?

But to press the point... if you believe that there's no way to measure 'information' (which contradicts everything about the whole point of Information Theory), then what on earth makes people think that genetic mutations cause a loss of information? After all... no way to measure it, right?

[Howie47]

I didn't imply that you did. You are using the same excuse. Let's be clear when we speak, as to what we mean by information. Unless your purpose is to obscure. Information is being used 2 different ways.1. Bit's that are uncoded vs. 2. encoded with prescribe information. The later articulates a meaning or purpose. If part of that is lost, it is a loss of information 2 & 1. Like tearing a page from a instruction book. One less page, or some missing base pairs. But measure the # 2 lost. NO!

[Howie47]

Part 2. We could also have a loss of no. 2

information, without a loss of no. 1. If the base pair information carriers are scrambled, (as in some mutations). their is a loss of no.2 information. But not necessarily of no. 1.

[FiverBeyond]

But again... if you can't measure the amount of information (again... the quantity. Not the 'quality'), then how can you be sure that information is lost?

Again, the method is simple. Let's measure the amount of information in the gene before a mutation, and then after mutation. If the difference is positive, then information was added. If negative, then information was taken away.

Wouldn't you agree that this is a good test?

And, of course, this is entirely plausible in Shannon's theory.

[Howie47]

Your either playing me, or your just not capable of understanding. Either way, I won't be bothered by you any more.

[FiverBeyond]

**politely tips hat**

[FiverBeyond]

Ah, but Shannon discusses this in great detail in his paper. The very inclusion of 'meaning and purpose' brings in philosophy, and breaks from I.T..

Great example: My wife asks if she should buy a cat or a dog. I'm trying to save money on a telegram, and write back only "dog".

Now... is this information?

Now suppose that instead of answering back, I decide to send back only random letters.

By sheer luck, they send back "dog".

Is this information?

[Howie47]

LOL, you really should think a little before you respond. Shannon is dead. HIs theory relates to transmitting info. "No Free Lunch theorem" also information theory, relates to the formation of prescripted information. It came about when trying to invent algorithms that would show evolution possible. Only the opposite was the case. A computer can only transfer prescripted information. It can't write it itself. Dog's are more probable, Platypuses are less. A whole book ain't happening.

[FiverBeyond]

Avoid my questions, eh? Simply laugh at my thinking abilities, and then quietly refuse to answer?

Shannon is dead (unfortunately... the man was brilliant, and a pioneer in the field) but his theory is very much alive. It's called 'Information Theory', and you can't claim that it supports you without first finding out what it is. If you briefly review the 'No Free Lunch theorem', even on Wikipedia, you'll notice that it falls into the field of algorithm analysis.

[Howie47]

The part of Information theory that deals with the rise of actual Qualified Information is called. "No Free Lunch Theory". It plainly states that actual information is only transfered, never created by random process'. Why is "no free lunch" also used in economics. Because the cost can only be transferred. Their is no free lunch or free information. Adding links to my video above.

[FiverBeyond]

Once again you try to slip in "Quality" in place of "Quantity".

Shannon's work clearly and distinctly allows us to measure and quantify information, at least, in the "Information Theory" sense. For example, in order to tell if a genetic mutation can actually add information, the math is simple: measure the information before, measure the information after, and take the difference.

Would you agree to this test?

[KarlHeinzofWpg]

Excellent.

By my calculation I've received gigabytes of information from these two videos. However, I've only reduced my original entropy score by about 75%, so I must rewatch a few more times.

Thanks for your work!

[TheReptilianAgenda]

HA HA HA HA HA you explained summation in an easy to understand way in three minutes, when i was in high-school my teacher made it seem very complex and muddled it up so it took two days for the entire class to get it.

[eiskorn12345]

i like your glasses.

[YourProfessor]

I thought entropy was the physics principle that you always get less energy out of something then what you put in. That you can't have a 100% efficient system and that there is always waste.

[TheReptilianAgenda]

gibbs free energy

[darkevilbitch666]

@Zekian: Choosing the logarithmic base 2 is only a choice of measurement unit. You could choose base 3 and have the information content counted in "trits" instead of bits, i.e. an information unit that can have three different states instead of 2. You can also calculate with the natural logarithmic base, but while making mathematical sense formally it can't be explained so straightforward as an information unit that takes 2.71... (or whatever Euler's e was) different states.

[kurliwurx]

Did you prepare these lectures?

This is seriously cool.... you can actually make it clear.

[Zekian]

Ok, But why log base 2? why not the natural log? would it still be a valid measurement if it was compared to the another, calculated with the natural log?

[ZJemptv]

Certainly, if it we use the natural log instead, the amount of information is measured in units called "nats" (by analogy to bits).

[Zekian]

interesting, is there any reason for it to measured in bits?

[Zekian]

Sorry i just began to watch the next video in the, i shall wait until seeing the rest untill questioning :P

[ZJemptv]

It's just more convenient when using digital systems, which operate with two possible states: "on" and "off", 1 and 0.

[24oscar24]

@ZJemptv

so base 2 so we can have H(x) in bits?

dem 0s and 1s?

[themikebolta]

Log base 2 is used because we are only concerned with binary {0, 1}.

[abdoanmes]

I have no idea what you just said, and now I am more confused? I forgot how I even got to this video! Where are my pants! Oh no, I've got to go. Peace =)

[notscooter952]

Interesting stuff. Thanks for sharing. Although I wonder why it's supposed to be in base two. Not sure if I understand that. Ah well. Maybe that's why my major is sociology and not calculus.

[sakruh]

it's in base-2, because we get/do not get information or we have yes/no answer to the information or there is electricity/is not electricity in the circuit..

so, it is pretty much about binary representation of information..

[faber939]

Quality stuff. I'd be interested in more in-depth analysis and also whether or not this actually has practical applications.

[werdsup]

I got banned again :P

[Ninwa]

Hey ZJ, this is a great video! I have a question for you! With no formal or really prior knowledge of information theory, but a fair shake of knowledge from programming, your example with the unbalanced coin confused me. Could you explain in a little more detail what it would mean to have .81 bits of missing information, or .81 bits of uncertainty? I suppose it's the idea of uncertainty as a measure that immediately makes me want to quantify it in terms of the possible information. Thanks!

[Ninwa]

I read an answer to my question in the video to which this was a response to. My hangup was on the .81 bits because I didn't understand exactly what you meant by that. I've realized you meant this is the average amount of information you'd have to send. Although in reality with a coin, wouldn't you be sending 1bit everytime no matter what? Bah! And then I was confused again.

[Ninwa]

As it turns out, if all communication happened over a computer network, then yes you would always be transmitting one bit (or byte for that matter.) but in the realm of information theory, it's okay to talk about information in units of fractional bits.

Cheers.

[SuperFlyNB]

This isnt just Math (general term), its calculus (specific Term) LoL.

You can be good at basic math yet suck at calculus (or trig)

[Paperxcutxsuicidex]

What in the hell are you wearing? No offense, but just o_O still..

[saib4u2c]

What are you trying to say? What is the point and why should I give a rats ass.

I get it that you are smart, but please must you? Communication theory is not what I'd like to see. If I need statistics I can go to Claude E. Shannon for all my Information Theory needs. Must you?

Do You have friends?

[Scrapheap71]

Entropy in information theory is a measure of the ammount of information MISSING.

[ZJemptv]

Duly noted at 0:18 -- I hope I got it right this time.

[Zerph21]

Now seriously, you look like Bill Hicks in this video for some reason.

or I'm just crazy, dunno.

[Zerph21]

He's a pimp (He's a pimp)

He's a pimp (He's a pimp)

He's a pimp (He's a pimp)

Just a pimp (He's a pimp)

If you ain't pimpin' for the dollar

Pimpin' for the love you ain't gettin' at home.

[EricZombie]

i've always been interested in information theory.