 As we saw here, the communication problem results in this extremely complicated looking expression that we would want, we now need to analyze. This expression depends on the channel noise distribution here, which is the probability of seeing a certain output given a certain input. And in that expression has gone the channel, the input which depends on the encoding strategy and also in this expression has gone the output which depends on the decoding strategy. And our goal was to find combinations of fn and gn so as to minimize this particular error. So naturally this is not by no means an easy problem, but there are ways by which we can approach this particular problem and that is what we will talk about now. So the first subclass of problems that we can consider is what is called, so the first subclass of problems that we can consider is what are called data compression problems. So data compression refers to the situation in which the sender of information or source has to be recovered with at the destination, but it has to be done, but there is a limitation in the channel. The channel is not noisy, the channel will not add noise, but it provides us with very limited storage capacity. So there are not that many, we do not have enough possible distinct channel inputs that we can accommodate all the possible values that the source could potentially take. So the challenge in a data compression problem is to decide how to take the source that we have and optimally use it so that we are making use of the channel resources that we have. The channel resources are not constrained as I said by any disturbance or noise that they add, but rather in how many we have, each series source is perfect in the sense that the channel does not add any noise when the input goes into it, but there are not that as many inputs as the number of source symbols. So the setting is as follows, so we have here data, let us say this here is of length n, so this here is the source which we will call let us say S1 to SN, that is your data here. This has to be sent to a destination, so it has to recover, this has to be recovered at the destination, let us say whatever is recovered is denoted S1 hat to SN hat. Now the process of sending this from source to destination happens over a medium which is limited in the following sense that it can take only a certain number of, it only has a certain number of distinct inputs, which means for example the medium could be limited in the kind of fidelity it offers, it could be you know it only lets you send certain types of symbols maybe or the certain number of symbols or there is a limitation on how often you can use the medium, all of these are one and the same way of describing this particular situation. We will also consider the case, we will consider for simplicity the case where this data is actually binary, so this here is a sequence in 01 to the power n, so this is therefore a sequence of what are called bits, that is your source, so this is your source. Now when I want to talk of the limitations of the medium as I said that you cannot, the medium is not rich enough to accommodate all the possible values of the source, what I mean is that the medium can potentially take only 2 raise to k possible inputs, so it has, it can only take 2 raise to k possible inputs. So think of it this way that the, you have let us say 2 strings of length n, binary strings of length n, so these are effectively think of them as some say 2 raise to n different words or 2 raise to n different books that you want to send from the source here, from the source to the destination, but then this, the way of sending them has to go through a rack and that rack is say like a book rack or a bookshelf in the library and that bookshelf has only 2 raise to k possible slots, so it can hold only 2 raise to k possible books. So what one does is you send, you send a book on one side of the rack that is what the sender does and or the encoder would do and the decoder on the other hand gets to see, he can just pull the book out from the other end of the rack that is what the decoder at the other end would do. But you cannot potentially, you cannot use, there are only 2 raise to k possible slots, so there will always be certain books or certain sources that you will not be able to fit into this particular medium and that is essentially the problem of compression. So what you have to do is take your source which is present here, compress it in a way that into something much smaller and take and use that whatever is that there which is at the other end and to reconstruct what was sent. So this is, so since I mentioned that we need to introduce an encoder and decoder, let me just draw those entities here into the picture. So once again we have an encoder here, we will denote this encoder by F, this encoder decides which book goes in which slot, there is a decoder sitting at the other end who is able to pull the book out, that is your decoder and then he sees what is present in that slot and then reproduces the book. Now really technically we do not really need to actually send that book itself at the other end, what one only needs to do is send say an identifier for the book, say for example we can send its ISBN number or its name or URL or something like that and that is what is kept in that slot. So the decoder only looks at that particular pointer, looks at that identifier and then from that identifier reconstructs what book it was that you wanted to send. So what is being presumed here is that the encoder and decoder have to begin with agreed on a scheme that in which all our books have been given some kind of identifiers or some kind of numbers, some kind of tags are present in which both encoder and decoder, which both encoder and decoder have agreed that this is, this book will stand for this particular tag, this book will stand for this particular tag and so on. So the ISBN way of numbering or the DOI is that we assign to papers and so on, all of these are basically these agreed upon numberings or agreed upon labelings of all the possible messages that we have at the source end. So this particular, this particular labeling is our encoding strategy. So this labeling in which we basically say that in which we assign a slot to every book is our encoding strategy and what the decoder needs to do is basically just invert this particular messaging, this particular labeling. He needs to look at the slot and then from there infer what is the book that was being sent. So what we have here is that you have a labeling scheme for all values of S1 to SN that assigns S1 out of 2 raise to K inputs. Now if, so let us now think what where the difficulty lies. So what would happen if K was greater than N? So if K was greater than N, if K was greater than N, this problem would be rather easy because every possible, there are 2 raise to N possible books or 2 raise to N possible sources and there are 2 raise to K slots and the slots outnumber the books. So every book would get a unique slot. So if K is greater than N, we would never have an error, would never have an error and the reason we do not have an error is because if you remember I mentioned that this medium is noiseless, it is just a cupboard where you have these pre-decided possible shelves and all you need to do, all that the encoder needs to do is say that this shelf is being used and then the decoder at the other end just knows what book was being referred to. So if K is greater than N, there is adequate space for all the possible books that we want to communicate from one end to the other and so as a result, there would never be an error. But let us suppose in fact in the same situation would also, in fact the same situation would also arise if K is equal to N. So if K is equal to N, there would be just enough space for all the possible values of the source and you could just then get back whatever it is that was being conveyed by the sender or the encoder. So the challenge in this problem comes because as I mentioned because of compression, the effect of compression which is that you do not have, you have a resource constraint, we do not have as bigger shelf to accommodate all the possible books. So what we therefore in the domain, in the regime where K is less than N. So the number of possible inputs is smaller than the number of possible values of N. So let us see when is it that what a source will be correctly recovered at the destination. It will be correctly recovered at the destination if suppose a source value M is being sent from the source, this is mapped to an F of M here which is one of the input. The decoder should know that well if F of M is what is seen at the medium. So if F of M is what is seen here out here, then the book M is what has to be sent. In other words, the decoder should be able to invert F on this subset of inputs. So of all the possible inputs that have come here, the decoder should be able to invert and then get back and reconstruct what M is. That is when you would have no error. Now there will always be a clash as because there are fewer inputs here than the values of the source. So the way to ensure that there is no clash is to ensure that you is to sort of come up with an agreement where you can do this in one of two ways. One way is that you say well I am not going to map all the books here. I am going to take only a certain subset. I am going to really take only to a certain subset and that subset is this two, some subset of size 2 raised to k, these are going to be my high priority books. I am going to map those to I am going to give them a space in this shelf and all the others I just declare that they are not going to be recovered and that is part of my loss. The other is to say well I am going to map you know every possible book that can come up to one possible slot but the decoder will only be told one of them like the decoder will only map f of M when it sees f of M, the decoder will map it only to one of the pre-images that are there. So if there are multiple books that could have been assigned the same slot, the decoder only picks one of those particular books and then out of those books and then maps that back to use uses G to invert that f. In either case what you are doing is f here f is basically a many to one mapping but we are looking for is a many to one mapping but we are looking for a subset but we are looking for a subset of 2 raise to N we are looking for a subset of 0 1 to the N of size 2 raise to K. So we are looking for a subset of size 2 raise to K which will be will be recovered by G. So that is what will be that is the plan so that is what we want to want to look for. Now how do by what is the logic for finding such a by with what goal do we are we going to approach this particular problem. Remember the goal in all these all our communication problems is to minimize is to somehow minimize a kind of cost between the source and that is dependent on the error between the source and the destination. So let us suppose that are the cost that we are going to consider is simply the probability of error. So we are going to look for we going to seek fn and gn such that to minimize the probability that s1 to sn is not equal to s1 hat to s1 hat this is what we want to do we want to minimize this particular probability. So remember that we I have already told you that all books cannot be recovered that is you can only recover 2 raise to K because that is the way you are you are slotting you know that is the resource constraint that you have. So the goal is then to say which ones should you be prioritizing so that you get the least possible probability. Now this particular question of which gives you the least possible probability is actually a very tricky question to answer because there are many different combinations here you are you if you can if you see you are looking at a set of size 2 raise to n of which we want to look for a subset of size 2 raise to K and with whose probability is such you know with the highest possible probability so that the complement is has a very small probability. So instead of looking at this specific term which is that you are trying to minimize the actual probability the approach that Shannon takes is a far is an ingenious one is that he says well I do not really need to know what is the exact minimum probability let me not worry about that exact value instead let me just address the final question let me ask is it possible to construct f n g n for each n such that limit as n tends to infinity of this the probability this particular probability s 1 to s n not equal to s 1 hat to s n hat this limit is equal to 0. So, which means with with as n goes to infinity with probability 1 you would you would have you would recover with probability 1 you would recover recover what the source was saying. So, question that so Shannon is approaching this question by saying I do not care about what the actual minimum probability is for any particular n I am just going to ask is it really when is it possible to get let n as to make this probability of error 0 as n goes to infinity if that is if that is done then that is good enough because then I know that that is in fact the best one can do all right. Now, so now of course before one does this for as n goes to infinity it is one needs to formulate the problem for a finite n. So, so this particular so to formulate this problem for a finite n we have we can write it in the following way. So, given an epsilon greater than 0 does there exist f n g n and n large n large enough such that the probability of s 1 to s n the probability that s 1 to s n is not equal to s 1 hat to s n hat is less than epsilon. Now, this is what we want to achieve. Now, if you think about it here what we are doing is we are taking the limit as n goes to infinity. So, which means that as n increases the number of possible values of the source or you know the number of in other words the number of possible books that I was referring to that you want to store in your library the number of possible books there is itself growing and it is growing exponentially on the so now for this whole problem to make sense we also it has to also be that you have to also ensure that k itself also grows. So, we want to let therefore k to also grow with n because if k does not grow with n if k remains constant with n then the problem does not have much meaning because then the you know soon you will have a situation where the number of books becomes much larger than the number of storage amount of storage space that you have and you know you will not be able to recover anything at all. So, if you are if you are looking for this kind of objective which is a kind of a strong objective where you want the probability of error to go to 0 you know not just look for probability of error smallest probability of error we want actually the probability of error itself to go to 0 it is essential to look at a regime where k also grows with n. So, we need it is necessary that we need to indicate k to grow with n this is essential without this the problem is cannot be given any meaningful solution right. So, if so then the now the so now that we see that k must all must be growing with n we can we can talk of very many different types of cases but remember k is growing but we have to talk of cases where still k is less than n because that is the regime we are in. So, one possible way of thinking about this is to ask suppose I allow for k to be some theta times n right. So, k is suppose if a constant theta times n where theta here is less than 1. So, k is then growing at the rate theta times n where theta is the slope of or the rate of growth and that is less than 1. Now in that case notice what actually happens if you see we have 2 raise to k possible books possible slots of k is itself theta theta times n. So, this is therefore 2 raise to theta times n. Now suppose let us look at the fraction of that is that that is being let us look at the fraction of 2 raise to k to 2 raise to n this divided by 2 raise to n that is equal to 2 raise to theta times n divided by 2 raise to n. And now what is this particular value this is 2 raise to theta minus 1 times n. And since theta is strictly less than 1 this this term here is actually 2 raise to something negative right times n. And so as n goes to infinity this term actually goes to 0. So, in other words if we are looking for k growing with n but k is strictly less than n that means we are looking for something k equal to theta times n where theta is less than 1 we are effectively talking of a fraction or the number of books that are going to be recovered that are going to be correctly stored that is the number of slots that we have in the in our rack that the fraction of that relative to the number of total number of books that we actually have that fraction itself goes to 0 which means that what is going to happen in this is kind of inescapable that we would actually be recovering a very very small fraction of the total number of books. Now on the face of it this might seem like a contradiction because on the one hand I am talking I am saying here that I want the probability of recovery to be to be 1 that means or probability of error to be 0 but at the same time I am saying that we are recovering a very very tiny fraction of book tiny fraction of the value of the possible source values. What is amazing which is what Shannon really found out is that this these are not mutually contradictory and that this this can in both can in fact be done and this is in fact the secret of of the of Shannon's data compression theorem. So, I will talk about that in the next class.