 Welcome to this lecture on digital communication using GNU radio. My name is Kumar Appaya and I belong to the department of electrical engineering IIT Bombay. This lecture will be a continuation of our discussion of perfect codes and hamming codes. In the case of perfect codes let me just recollect the case of perfect codes what you must know is that whenever you receive any n-length bit sequence then a minimum number of bit flips will give you a unique code word and there will be no ambiguity. As an example if you remember a 3-1 repetition code resulted in a perfect code because all three bit sequences could be mapped back to a single code word with minimum number of bit flips while a 4-1 repetition code had an ambiguity and it was not a perfect code. We also discussed this hamming weight and distance concept in the case of binary vectors hamming weight and distance are very related let us just recollect very quickly what this is. So, if you have an n bit sequence the weight hamming weight of the sequence equal to number of ones in the sequence and if you have x 1 and x 2 as n bit sequences we define the hamming distance d h between x 1 and x 2 as the weight of x 1 plus x 2. Let us give you let us just take an example if you take 5 length sequences 0 0 1 0 1 and 0 0 0 0 1 1 where are the bits different same same different different same. So, the way the distance should be 2 the hamming distance should be 2 another way to find this is just XOR them bit wise 0 XOR 0 0 0 1 XOR 0 is 1 0 XOR 1 is 1 and this is 0 and then find the weight of this two ones resulting in a hamming distance of 2. So, the hamming distance is an easy way to kind of represent the number of bit flips that is needed to go from one n bit sequence to another and it comes in handy because we want to find the minimum number of bit flips that result in a valid code word. The other recollection that we want I want you to have is that we want to find the least hamming weight error pattern such that h x plus e equal to h e is equal to h y. If you remember in the previous class also we were discussing the fact that if y is equal to x plus e where e is an error pattern if you find h y which is called the syndrome we have h y is equal to h times x plus e which is h x plus h e this is the advantage of using a linear block code this is equal to h x is 0 h e. So, if you can somehow map h e uniquely to an e that is if h e uniquely yields an e which is the actually when I say e of minimum hamming weight then we can correct errors ok. This is what we are going for if you remember for the 4 bit parity code we could not map h e uniquely to an e if you remember if you have let us say let us take the repetition codes parity check matrix in the case of repetition code g was 1 1 1 so I am going to write this as here there are how many 1 bit patterns are there there are only 2 say there are 3 1 bit error patterns can you can you now map h e back to e yes because h e what is h e then I am going to write this as a row bear with me 0 0 1 it is 1 1 0 1 0 it is 0 1 and 1 0 0 it is 1 0 that is you have a unique connection between these and you can find out which error pattern there was, but let us say you have a code like this. Let us say that you have the 4 bit you have the 4 length error correction you know repetition code g is 1 1 1 1 so in this case you will have now in this case if you have a 4 bit repetition code if you have a single error you can correct it because a single error is always going to give you 1 1 1 1 0 0 0 1 0 0 0 1 uniquely. Let us say you want 2 bit errors we will we will we will we will know there are many more, but I am just going to kind of stop with this. So, what do each of these give us syndromes let us check. So, let us now check the syndromes in this case for 1 1 0 0 we get 0 see if you I have to multiply the 1 1 0 0 that add the first 2 columns. So, I will get 0 1 1 0 1 1 0 I get 1 1 0 then 1 0 1 0 I get 0 1 0 so far so good then 1 0 0 1 I get 1 0 0 1 I 1 1 0 ok. Now, notice that these 2 resulted in the same syndrome and therefore, you cannot correct all 2 bit error sequences. So, in that sense this is not a perfect code. So, this is actually a perfect code this is not a perfect code ok because you cannot map all the sequences by performing minimum number of bit flips to a single sequence ok fine. So, this is something just to bear in mind. Now, let us actually use this as a pretext to define the hamming code. So, this is something we just discussed we were discussing 0 0 0 0 1 0 this is the this is our you know double parity code that is you would just repeat the parity twice. So, in this case 0 1 0 0 we said that the syndromes are non unique in the sense that there are this 1 0 0 0 also yield the same syndrome. So, we cannot correct the error unambiguously here even for 1 bit error we cannot correct it unambiguously this is not a perfect code because the syndromes are non unique. The 3 repetition code perfect code syndromes are unique for 1 bit flips we are not talking about 2 bit flips that will result in an error even in the 3 repetition code, but for minimum number of bit flips are they unique then it is a perfect code. So, a perfect code is an n k linear block code where every n then sequence has a unique minimum distance code word that is the 2 power k code words perfectly partition the space of 2 power n n bit vectors of course, naturally that means into the n code words. For example, if you recall if we looked at the 3 length repetition code 0 0 0 0 0 1 0 1 0 1 0 1 0 0 will map to the sequence 0 0 0 because a single bit flip gives you those 1 1 0 1 0 1 0 1 1 1 1 1 will map to 1 1 1 because 1 1 1 of course, you do not need bit flips all the other 3 if you do 1 bit flip then you get 1 1 1 to get to 0 0 0 for these you need 2 bit flips. So, the minimum distance code words are unique that is basically if you take a n length bit vector and you perform the minimum number of bit flips you will end up getting one of the code words. The question is can we construct a family of perfect single error correcting codes and there is a specific family of perfect single error correcting codes that we are going to look at and that is called the hamming code. The hamming code as it was among the first popular error control codes that led to several other modern constructions, but it is very unique in and easy to understand and that is why we will discuss the hamming code in some detail today. So, without going into the details of how exactly they are formulated I will go with an approach where we are going to construct the hamming code using its parity check matrix. So, the hamming code is a family of perfect n k linear block codes perfect n k linear block code meaning one constraint I am imposing is that any n bit sequence is going to be you is going to have a minimum distance code word in the code which is unique that is no other code word will have the same minimum distance as this particular code word. Now, notice that this n is chosen as 2 power m minus 1 and k is 2 power m minus 1 minus m m is of course, an integer which goes from 1, 2, 3 and so on ok. Let us actually just analyze this 2 power m minus 1 we are talking about binary hamming codes ok n is 2 power m minus 1 k is 2 power m minus sorry minus 1 minus m. Now, this choice is actually a little strange ok, but it will make sense if you look at the construction of the parity check matrix. Your aim is to let us before you know going further let us just set m is equal to 1 for m is equal to 1 you get n is equal to 1 k is equal to 0 because you get 2 minus 1 minus 1 this is not a valid code. So, you will set m to be 2. So, m is 2 for m is equal to 2 we get n is equal to 3 and k is equal to 4 minus 1 minus 2 1 this is the 3 repetition code. So, the 3 repetition code is actually a hamming code according to this definition. Now, the next code that we are going to do is m is equal to n this is the code we will spend most of our time on we get n is equal to 7 k is equal to 4 because 2 power m minus 1 minus 2 1 this is the 3 is 8, 8 minus 1 is 7, 7 minus 3 is 4. This is the score called 7 4 hamming code ok. Now, discussion as to why this choice of 2 power m minus 1 2 power m minus 1 minus m well to understand that let us look at the parity check matrix. See our aim is to be able to correct exactly one error. Now, what does our error correction mean in terms of perfect code we discussed it let us look at it in terms of the parity check matrix. We want every syndrome ok we want for every minimum you know every least number of bit flips every syndrome should be unique ok. Now, we want to be able to correct one bit error that is among the 7 bits any one bit error should uniquely be able to get back the original code word without any issue. In other words the minimum distance between the code words is such that if you flip one bit still this is the closest code word right. You take any n bit vector this is a code word let us say you flip any one bit the closest hamming distance code word is always the same code word from which you started. Now, the way we are going to define it is this this is the parity check matrix that we have ok and I am going to write it over here in a specific fashion there are by the way multiple parity check matrices for the 7 4 hamming code we are taking this one specifically for just a reason ok 1 0 I believe it was 1 1 0 1 yes yeah and then we will place the identity matrix here. One reason why we place the identity here is so that we can easily write the generator matrix using that a i i a trick. So, here we have the parity check matrix of our 7 4 hamming code we are now going to claim that every one bit error sequence results in a unique syndrome ok. Now, I can of course I can go through and say what are the one bit error patterns 0 error patterns is anyway it is very simple because you get a code word directly you can just read off what the you know k bit sequence is this is of the form 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 and so on ok I am not going to write them what happens e belongs to this, but what happens if you perform he it picks out the column corresponding to the non 0 element in e that is let us say that the error occurs in the first bit then your syndrome is going to be he which is 1 1 0 error occurs in the second bit your syndrome is going to be 1 0 1 which is in the second bit error occurs in the third bit it is going to be 0 1 1. So, every one bit error results in a unique syndrome and this syndrome is just the column therefore, what do you need to do you just need to perform h times y where y is the received n bit sequence that results in h times x plus e which is h e the h e is going to be one of these columns depending on which column it is flip that bit you will get a code word that is it I mean it is it is it is that it is that simple. If you now this you see how we have designed h h has been designed in such a way that any single bit flip is going to result in a unique sequence or unique syndrome ok what do you need to do to handle this it is very simple how many single bit patterns are there it turns out that let us say that you know let us say in this case we chose n is equal to 2 power m minus 1 how many single bit pattern error patterns are there n ok and now you need your h to have that many unique entries as the columns therefore, your h is going to be of size n minus k cross n ok fine. So, now let us let us just understand this also ok you have 2 power m minus 1 single bit error patterns ok 2 power m minus 1 single bit error patterns therefore, you are going to have all m bit sequences as columns of h. So, in this case you know for example, let us say m is 3 k is 4 it is like you know your n minus k is essentially m. So, you are in general if you look at the Hamming code h is going to be of size m cross n where n is actually 2 power m minus 1 and columns of h are all 2 power m minus 1 non 0 m bit sequences. Let us try this out ok I am now going to construct the Hamming code for m is equal to 2 I am just going to construct the parity check code it is simple I will put the I am going to put the you know columns somewhat randomly, but let me just try to put the identity here 1 0 0 1 and the remaining column is 1 1 this is my h ok. What is the corresponding g for this we will use the identity trick we will put the identity here ok because this is of the form identity a an entity and a transpose not surprisingly we get the perfect code the 3 1 repetition code. And if you now do m is equal to 3 you will get this let us just for fun do m is equal to 4 ok I am going to I am going to write the columns out without thinking too much 0 0 0 1 0 0 1 0 0 0 1 1 1 0 sorry 0 I am apologize I am just writing them down in this way 0 1 0 0 0 1 0 1 0 1 1 0 0 1 1 1 and then I am going to I am going to play a trick because I do not want to write so much I am just going to copy this and then paste it move it here and then I am just going to play another trick because I am just little I do not want to have to write this much so I will just delete this oops let us just delete this and now I am just going to write 0 0 0 0 0 0 0 0 I think I missed 1 0 0 0 let me just add that it does not matter I can always swap the columns. So, this if you count has 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 15 15 and this is 4. So, the next hamming code is 15 4 as check n is equal to 2 power 4 minus 1 15 k is equal to n minus m which is so k is equal to n minus m this is correct which is 11 which means this is 4 this is correct ok constructing the generator matrix for this is an exercise I will leave to you and know it is not that you can swap the column so that you get the identity on the right and construct you get many answers it is fine. Now, coming back all the columns are unique and all the 2 power 3 minus 1 non 0 3 length vectors are present as columns. Here if you want to get a geometric idea the first thing is multiplying it by zeros gives you 0 no problem that is always true for any parity check matrix but multiplying it by any 1 bit vector always results in a always results in basically a unique element. Here is the here is the other interesting thing multiplying it by 2 you know multiplying it by any 2 length vector ok you can check for example, if you take we can take 0 0 1 and let us say 0 1 1 add them up you will not get 0 similarly if you take 1 1 1 and 0 1 0 you will not get 0. So, basically here is the thing even if you have a 2 weight kind of vector you are not going to essentially get you are not going to be able to get a 0. So, you are you know you need to flip 3 times you need to flip 3 times for you to be able to get 0 in other words again this is something I am not proving formally the code words of the hamming code have minimum distance of at least 3 this is something which you can find out. So, any unique any single bit error pattern uniquely decodable that is the key idea. So, now just using all the same tricks that we did you can always just construct the generator matrix ok I believe that over here that we can just use this approach to construct it we will copy this we will say. So, our 7 4 hamming code has this kind of pattern we will use the same approach that we use earlier this is an identity this is our a ok. So, now over here our G is going to be ok have 4 length I mean 4 rows and we will put the transpose of this over here. So, 1 1 0 1 0 1 1 0 1 1 oh sorry and 1 1 0 1 0 1 1 0 1 1 0 1 1 1 and you have your generator matrix. These are also code words of the hamming code let us check one important observation the weight is 3 meaning every code non-zero code word seems to have at least 3 ones that is you have 1 0 0 0 and then 1 1 0 you have at least 3 ones. And if you take 1 1 you know 1 triple 0 1 1 0 this corresponds to this column this column this column if you add them 1 1 0 it is this column and this column yeah 1 1 0 plus 1 0 0 plus 0 1 0 will give you 0. Similarly, you perform the inner product with this this will give you 0 and you can check that by going through all possible 0 0 0 0 0 0 1 and all multiplications with m transpose you will get all 16 code words which are 7 length. And if you now flip 1 bit of these you can easily find out that there is an error flip 2 bits you will find that you can detect the error, but you will not be able to correct it ok. So, let us just do that exercise also maybe I will just do it for 0 0 0 0 1 0 1 let us say you get this ok. If you get 0 0 0 0 1 0 1 ok you have a slight issue ok because if you know that 2 bit errors have occurred great you can always just you know say ok the 2 bit errors have occurred I have not found it, but the syndrome is going to still give you and a value. So, in this case let us say this is our e h e is going to be giving you it is going to add 1 0 0 and 0 0 1. So, if you add 1 0 0 and 0 0 1 you get 1 0 1 that means it is indicating that the error has actually occurred over here which is wrong. So, the decoded code word is going to be and you can just check I am going to write it as x hat is going to be 1 0 1 0 1 0 0 1 0 1 0 1 which is this one it is actually a code word. The hamming code is a perfect code for every syndrome it is going to give you a corrected code word, but it will always make the assumption that you have one bit error and that is when you will make start making mistakes. So, if the if you have a single bit error hamming code will do the job if you have 2 bit errors hamming code will also encounter bit errors that is something you should keep in mind. Let us now go back to our content. So, whenever you have a single bit error the weight hamming weight of e is 1 since each column of h is unique h e is the location of the error you flip the corresponding bit and that results in error correction. As I said the block error probability right you can basically start with n choose 2 p square 1 minus p whole power n minus 2 and you know like you can this is not exactly the block error probability because this is the probability of 2 bit flips and then you can add plus 3 and so on, but generally the p square term dominates. So, this is approximate whenever you have 2 bit errors the hamming code will is a perfect code it always maps to a unique element it will give you a wrong element and you will get a block error. Finding the bit error probability is little more intricate and I will leave that to some references you can go check that out and then you can always just you know find out how that is computed. Now, if you want to go further if you look at the hamming code the rate is 2 power m minus 1 minus m upon 2 power m minus 1 that is it is 4 upon 7 for the 7 4 hamming code if you start increasing m right. So, for example, we saw that over here we saw that if you go for 16 right you get 11 by 15 and so on. So, the hamming code is going to start giving you more and more you know it is it can give you you know better rates as well, but for higher block ends length the problem is single error correction is generally not sufficient that is if you have only single error correction what happens is that as the block length becomes larger the probability that you have more errors is much more from a probabilistic standpoint even if you have a coin that results in tails very rarely as you make the number of tosses larger and larger the probability of getting two tails is much much higher. So, the using a hamming code for very very large block lengths is well it may be efficient term a rate perspective, but it is generally not a good idea because the ability to correct only one bit error is not enough when the probability of error in a binary symmetric channel is reasonably high. If p is something like 10 power minus 3, 10 power minus 4, 10 power minus 5 then yes the hamming code will result in a huge benefit even if you take the m to be a high number, but the problem is that you know when you have very large number of bit errors then you are going to have issues. So, before we close the hamming code is like a motivator several modern coding techniques exist. In fact, before we go to those algebraic block codes you know it is like Reed Solomon Reed Miller are very very common. In fact, Reed Solomon codes are used in a variety of applications where uses error correction in you know disks like optical disks like you know compact disks and DVDs Reed Miller codes are also very commonly used block error you know control codes and there is also a family of modern block codes called turbo codes, LDPC codes and polar codes. These are all more of random randomness oriented or motivated codes that can perform very very well in large block lengths with excellent error performance and reasonably high rates. In fact, they are known to be very very efficient and optimal also in the capacity sense capacity is something that we will go through in the next couple of lectures. So, the other thing is there are also a group of convolutional codes and several other such applications of codes where they perform error control continuously and you can form a trellis just like your equalization to handle errors as you go. So, these are subjects for references where you can read some good error control coding related textbooks and I encourage you to do so, but in short the hamming code and linear block codes are very efficient and effective and they offer a good solution as a starting point for error correction whenever you have a practical noisy channel. In the next lecture we will be implementing the hamming code on GNU radio using a block and studying how the performance is going to be. Thank you.