 So, let's talk about data compression. And we measure the amount of information in a unit called a bit. A bit corresponds to the answer to a single yes or no question. This is actually too small a unit to be very useful, so we tend to group bits into larger units. 8 bits form as a byte. 1000 bytes forms a kilobyte. Sometimes we use the fact that 2 to the 10th power is very nearly 1000, and some definitions will say that there are 1000 and 24 bytes, so the kilobyte, the difference is actually too small to be relevant. A million bytes forms a megabyte. Again, we might use the fact that 2 to the power 20 is very nearly 1 million, and so we'll have an alternate definition for what a megabyte is. A billion bytes forms a gigabyte, and so on. Well, now let's consider what a picture looks like. So any image is going to be divided into some number of pixels. This is a contraction of the term picture element. For example, if you're looking at a standard computer monitor, it might have a display size of 1024 pixels by 768, and that's described as 1024 by 768. Well, let's take a much smaller 8 by 8 screen. So maybe my screen looks something like this. So here are my picture elements 8 by 8, and if I want to specify a picture, what I'm going to do is I'm going to specify the colors of the individual pixels. So with my great artistic talent, I'll put down some details, maybe some grass, throw in some sky, maybe add a little barn here, put a roof on the barn, sun's there, cloud, a couple of cows, and there's our nice picture. So now the important question is how do we want to store this picture? And so we'll do that by playing what we call the game of data retrieval, and we're going to play this in the following way. There's going to be a bunch of things out on the table, and you can write down as many yes or no questions that you'd like, as many questions that have as their answer, either a yes or a no. I'm going to pick one of the items, and you get to ask me any of the questions that you've written down, but you're limited to just those questions. So you have to use a little bit of ingenuity to come up with the right questions to ask at the beginning. Now your goal is to identify what I've picked using as few questions as possible. So for example, let's play this game on our picture. So here I'm going to take this picture, I'm going to break it up into its pixels, I'm going to dump all the pixels onto the table, and I'm going to pick one of them up, and your goal is to try and determine the color of the pixel. So let's think about the types of questions we might write down. So what questions do I ask? Well, one possibility I might ask, well, is the pixel white? Again, this is a yes or no question. And if I write this down as one of the questions I can ask, you can ask, then this is a question that you can either get a yes or no answer to, and use this information to determine the color of the pixel that I've chose. Well, we could also ask, is the pixel green? Is the pixel brown? And again, the idea is that you're going to write these questions down beforehand, and the only questions that you'll be able to ask later on are the questions that you've written down. Now if you're looking at a typical image file, the color of each pixel would be specified by asking a sequence of 24 questions. If you've ever looked at something that says that there's 24-bit color, what it's referring to is the fact that the color of a pixel is determined by these 24 questions. And so what that means is that if I wanted to specify the colors of every pixel in this tidy image, this 8x8 image, I need to ask 8x8x241536 questions, and so the information in this image would take up 1536 bits of space. Now it's worth thinking about how big this is. So if I take a typical picture, the way that a motion picture gives the illusion of animation is that it shows a whole bunch of pictures in rapid succession. So typically what you're seeing with the motion picture, you're seeing about 24 pictures per second, and each picture is going to have around 2 million pixels or so, and each of those is going to have a color value that's specified by a 24-bit number. And so that means that one second of a movie is going to require 24 by 2 million by 24 bits. That works out to be about 144 megabytes per second of a movie. And if I look at a two-hour movie, that two-hour movie is going to require on the order of about 1,000 gigabytes, which turns out even at the high density that we can store today, that works out to be about 20 Blu-ray disks. Well, when I go down to the video store and check out the Lord of the Rings, it all fits under one disk. So the question is, how do I take this 1,000 gigabytes, these 20 Blu-ray disks, and fit it onto one disk? And the answer to that comes from the art of questioning. So here's some basic principles of asking questions. First off, don't ask questions you already know the answer to. So we might have the question, is the pixel brown? But we already know that for this particular image, the answer is definitely going to be no. So there really isn't any point in asking this question. One of the more important principles of asking questions is whatever answer you get. You should be able to eliminate many possibilities. For example, we might ask the question, is the pixel white? And we're going to get one of two answers. If we get a no answer, the problem with a no answer is that it only eliminates one possibility, namely that the pixel is white. So a no answer only eliminates one possibility. So this question, is the pixel white, is not really a good question to ask. Instead, we might consider a different question. For example, we might ask, is the pixel white, green, or yellow? And the advantage here is that if I give a no answer to this, I've eliminated three possibilities. Namely, I know the pixel is not white, not green, not yellow. So the no answer eliminates three possibilities. And also, if I get a yes answer, then I've also eliminated a bunch of possibilities, in this case four, because I know that the pixel is white, green, or yellow. Which means I know it is not going to be red, or black, or blue, or purple. So this is going to be a good question to ask. So given that, you might think about what the first question to ask should be, what question is going to eliminate as many possibilities as possible? And if you look at the picture and think about this a little bit, you realize the best first question to ask is, is the pixel blue? Because if I get a no answer to this question, I've eliminated all of these blue pixels up here. I've eliminated a whole bunch of possibilities. And that's about as good as I'm going to get for my first question. Well, what about the rest of the question? To answer that, we go back to 1951. And in 1951, Robert Fano at MIT gave his class an assignment to find an efficient way of asking this type of question. And Fano himself had invented a fairly standard method of doing so, which was known as Shannon-Fano coding after Claude Shannon, one of the founders of information theory. And one of his students, David Huffman, did something that every good teacher hopes for, just to say that his student found an even better way of doing things than Fano himself had known. The result is what's called a Huffman code. And it emerges as follows. Suppose I have a park, and imagine that I have several entrances, but only one exit. And at each entrance, what I'm going to do is I'm going to put something that represents each of the colors. And there's a crowd of people there. And the number of people in the crowd is equal to the number of pixels of that color. So if we go back to our image, we see that there's 30 pixels that are this sky blue color. There's 20 pixels that are green. There's four pixels that are red. Four that are white, three, two, and one. So there's a bunch of pixels of each of the different colors. And so all these people are going to enter the park, but there's only one exit. So all these people are going to have to exit the park in some fashion. And so we'll have everyone walk towards the park, and we're going to use the following rule. At every stage, the two smallest crowds will join up before they proceed to the exit. So what does that mean? Well, so the two smallest crowds here are this two-person crowd and this one person all by him or herself. So they'll join up and they'll form a three-person crowd. Now, again, we'll take a look at the two smallest crowds here. This is a three-person crowd. Here's a three-person crowd, and these are all larger. So we're going to take these two crowds and we're going to join them up. And again, we'll take the two smallest crowds. Here's a four and a four. So they'll join up. And again, there's a crowd of 30, a crowd of 20, a crowd of eight, a crowd of six. These two are the two smallest. They'll join up and this crowd of 20 and the crowd of 14 joins up. And then finally, this crowd of 30 and the crowd of 34, they join up and that's everybody and they all proceed out to the exit. Now, what's useful here is that this park map actually allows us to form the questions that we can ask to identify the color of the pixel. So for example, if the pixel is blue, then I can determine that by asking just one question. Is the pixel blue? And if the pixel is blue, I know that I'm going to go on this path out the blue exit. And if the pixel is blue, one question is enough to determine it. Well, what if the pixel isn't blue? Well, suppose the pixel is, for example, white. In order to determine the color of the pixel, I'll need to ask four questions. Now, my first question is, is the pixel blue? And if the answer is yes, I'm out the exit. If the answer is no, I go down this branch. So is the pixel blue? No. So then I'm going to ask, well, I have this choice, either I go down this branch or I go down the other branch. And so my question is, I'm going to ask, is the next, is the pixel green? And again, if it's green, I'm going to go down this path, but the answer in this case is no, so I'm going to go down this path. And once again, I now have a choice of two different paths I can go down. And so I want to ask, is the pixel red or white? Do I go this way or do I go this way? And I ask, is the pixel red or white? And the answer is yes. So that means I can ask the next question. So I know the pixel is red or white. I know it's going to be someplace down this branch. I come down to here and I have to ask one more question. Is the pixel red? In which case I go this way, otherwise I go this way. And so here the answer is no, if the pixel is not red. And so that tells me that I have to go out this exit. And so I'm dealing with a white pixel. Well, let's take a look at how many questions we actually need to get all the pixels in that picture. Well, those 30 blue pixels, I can get those colors by asking one question. Is the pixel red, blue? The 20 green pixels will actually need to ask two questions. Is the pixel blue? No. Is the pixel green? Yes. And I go out this way. How about the red, white and purple pixels? Well, let's see. So if I want to get a red, white or purple pixel, I ask, is it blue? There's my first question. No. Is it green? No. Is it red? Or is it red or white? Yes. Is it red? Out this way or out the white exit? On the other hand, is it red or white? No. Takes me this way. Is it purple? Yes. And so that requires a total of four questions. And then finally, the two gray and the one yellow pixels down here at the end, those I have to ask a number of questions to get there. And in those cases, I have to ask a total of five questions. Is it blue? No. Is it green? No. Is it red or white? No. Is it purple? No. Is it black? Either yes or no. And so I need five questions total to find the colors of those. And altogether, I'm going to need a total of 109 questions. And that means I can take my picture and compress it down to 109 bits. Now originally, I needed 1,536 questions, 1,536 bits, but I can get every color in that picture in just 109 questions. That's a reduction of 93%. Now, there's other methods of data compression. This is just the various mirror introduction and there's several different ways I can improve that data compression. I can use what's called psychovisual compression. And that comes from the following. If two colors look the same, they may in fact be different. But if I can't tell the difference, I can code them the same way. And this is the basis of GIF compressions. We could also use what's called lossy compression. And we use this all the time. When I write LOL, that conveys a very broad range of possible reactions, but it gets across the basic information. Lossy compression is how JPEGs work. We can also use what's called spatial and temporal redundancy. And this is based around the idea that in a movie consecutive frames are very nearly the same. So the only thing we actually need to record are the differences between successive frames. This is how MPEG compression works. And so using a variety of compression techniques, we're able to take something that should take up a huge amount of space on disk and compress it down considerably. So instead of taking 20 or so DVDs or Blu-ray disks, I can compress the Lord of the Rings down into a single three disk set.