 Welcome to this lecture on digital communication using GNU radio. My name is Kumar Appaya and I belong to the department of electrical engineering IIT Bombay. This lecture will be a continuation of our discussion of error control codes. If you recall, in the previous lecture we implemented the repetition code on GNU radio and we saw that it was very effective in correcting errors. To re-collect your from your memory, if you have an N1 repetition code where N is chosen as an odd number, we could reliably correct up to N minus 1 upon 2 bit errors and conclude which bit was sent and this reduced the bit error rate significantly. However, one key issue is that the rate is now 1 upon N which is very very poor and if you make the block length much larger and larger the rate suffers significantly. In this lecture, we are going to study a class of linear block codes called the Hamming code and the Hamming code is very special in that it allows you to correct a single bit error with a much higher rate than let us say a repetition code and has several advantages in terms of its geometry and the way it is constructed that make for interesting understanding. Let us proceed. So, we will go first on a discussion of perfect codes and then we will talk about Hamming codes which are a type of perfect codes. Then we will discuss syndrome based error correction. If you recall in the previous lectures also we mentioned that if you compute the syndrome that is multiplying the received bit vector by the parity check matrix, if you compare that syndrome with the various error pattern syndromes you may be able to correct errors and we will conclude with some other error control methods and a discussion on what other methods are commonly used. Just to recollect your refresh your memory linear block codes are defined as you can define it using two parameters n and k and n k linear block code maps k bits to n coded bits using a linear transformation. If you recall we preferred to implement that linear transformation using a matrix G and by convention G is a matrix that has k rows and n columns. If it since it has k rows and n columns the message k bit message is pre multiplied as a vector or row vector consisting of k bits. This results in an n bit vector which is a code word and an n minus k cross n parity check matrix is a pair to this G and this n minus k cross n pair matrix H appears and when you post multiply any of the code words the parity check matrix yields 0. In other words the parity check matrix can be used to confirm that no error has occurred typically. Syndrome are obtained by multiplication of any received bit vector with the parity check matrix. If H times the received bit vector which is the syndrome is 0 you can conclude that no errors have occurred of course. There could be many errors but let us say typically no errors have occurred and if you however get a non-zero syndrome this non-zero syndrome can not only indicate where the error is but it can also potentially correct it. It can indicate a presence of an error and maybe even tell you where the error has occurred. This we saw in the context of parity check matrices for let us say the parity check code where it indicated that an error had occurred but for the repetition code it also told you where the error was. We will now just briefly discuss this concept of perfect codes or perfect linear block codes. If you consider the 4 to double parity code which code word 0 0 0 0 1 0 1 1 0 1 1 1 and 1 1 0 0 what are the generator and parity check matrices for this? So, let us actually just do this. So, we have 0 0 0 0 let us say 4 bit parity code 0 0 0 0 and we have 1 0 1 1 and I believe then we have yeah. So, it is like this you basically take the first 2 bits find their parity repeat it twice. So, 1 0 has parity 1 so, 1 1 is repeated twice then you have because 1 0 has a parity of 1 1 plus 0 is 1. So, you repeat 1 twice and finally, 1 1 0 0. In other words this is an this is a code basically where oh I should have not repeated this here essentially this should be 0 1 yeah. So, it is like this whenever you find the parity of the first 2 elements the first 2 elements parity is just repeated twice. So, this is just like a cousin of the 3 bit parity code if you remember 3 bit parity code looked like this these are the code words you just have to append another here is oops here you append a 1 here you append a 1 you just duplicate the last element and you get this this is basically this 1 this 1 0 1 1 appears 0 1 1 1 appears and so on ok. So, now, let us now figure out the generator matrix for this code, but for that before that let us actually do one thing let us find the generator matrix for this code. We already did it once, but let us just do it once again. So, for this code the 3 bit parity code we will use the standard approach to construct the generator matrix we will basically take 2 linearly independent code words. So, in my case I am going to choose 0 sorry I am going to choose 1 0 1 0 1 1 and let us also check the find the parity check matrix. To find the parity check matrix I will actually teach you a trick this trick will come in handy if you write your generator matrix in the form i and let us say a matrix A. For example, in this case i is identity it is a 2 cross 2 identity matrix in fact, i k and the remaining is a matrix k. So, obviously, this A has to be n minus k n minus k k cross n minus k that is this A ok. Then you know that h times g transpose should be equal to 0 ok. I will give you an easy way to construct h without having to go through any hassle. If you take h to be of the form A transpose i in this case what should the size of i be ok. So, remember this A is going to be n minus k cross k which means you have taken this A it is A transpose going to be n minus k cross A size going to be i n minus k n minus k ok. And if you now take this i let me actually move this elsewhere and call this an note constructing h from g for this format ok. Now, if you now use this h and perform h g transpose that is equal to A transpose i i A transpose why because I just took transpose of g. So, now, this is equal to A transpose times i plus A transpose times i and in the modulo 2 arithmetic this is equal to the 0 matrix. Therefore, this kind of g where your original bit 0 0 0 1 1 0 1 1 appear is called a systematic basically it is called a systematic linear block code. And in a systematic linear block code I will not cover it in detail, but you can see that since the code word appears you are going to have an identity sitting in the front. If you have an identity sitting in the front you just need to stick an identity sitting in the back and whatever remaining aspect over here is present for example, here it is 1 1 that 1 1 will essentially become the transpose over here let us check ok. So, let us now do the 3 bit parity check matrix code and then we are going to write h as and let us use this approach to construct our h for this code we are going to have i k and a. So, here you are going to have a transpose my a over here is 1 1 1 1 and I am going to have an identity matrix of size n minus k in this case n minus k is 3 minus 2 and I am going to have 1 this is the parity check matrix ok. So, this is the parity check matrix for the 3 bit parity check code ok. So, now, let us use this approach to construct the parity 4 bit parity code for sorry the generator matrix and parity code for the 4 bit parity code. For the 4 bit parity check code we are going to have g is equal to and I am going to use the same approach I am going to deliberately choose the linearly independent code words in such a way that I get identity in the left side of the g matrix it is just like this I am just going to write 1 0 1 1 0 1 1 1 which are both code words of this parity check code. So, it is like 1 0 1 1 0 1 1 1 this is my g and you can now clearly see that g is a 2 row 4 column matrix and you can now split this as i and a. Let us now use the same trick that we you know used above to construct the parity check matrix we have to now if you take g as i and a we then take a transpose and i. So, I am going to just write a transpose in this case it is 1 1 1 1 and the i is going to be 4 minus 2 cross 2. So, it is going to be is this indeed the correct parity check matrix for this code let us check. So, what are our code words? Our code words are the 0 code word we do not need to check because h times 0 is going to be 0 1 0 1 because a parity check 1 0 1 1 1 and 1 1 0 and 0. Let us now just do this exercise if you choose this there is no problem if you choose 1 0 1 1 you have to add the first the third and the fourth columns that will give you a syndrome of 0. If you do 0 1 1 1 the thing is if you notice these 2 columns are the same these 2 columns are the same. So, that is giving you an indication of the slight problem not problem literally, but an issue some non uniqueness is present in this h. Of course, it is present in the g also because there is a linear dependence the column, but in the h there is it is definitely there. Similarly, h times 1 1 0 0 is also 0 0. So, now let us actually take an interesting error event let us take e to be let us take an error event where you have some bit flips which result in something non unique. Let us actually make a guess over here. So, in the slide I have mentioned 0 1 0 0 let us go with 0 1 0 0 of course, you I have written it as a string you can imagine it as a column. If you know let us say that let us not call this the error let us call this the received let us say this is my x. Now, the question is which code word was likely sent this is the question. Now, if you have 0 1 0 0 what is the syndrome 1 1 great 1 1 is the syndrome. Now, the issue with 1 1 is that you are unable to tell what the exact error happened is because this 1 1 can result in an error in the first bit the 1 1 can result in an error in the second bit. Let us also understand it in terms of the decodability of the code. Let us say these are our code words and we have 0 1 0 0. The most likely event is let us say a single bit flip a single bit flip can result in 1 1 0 0 because if you flip this or if you flip this one it can result in 0 0 0 0. Therefore, in terms of the minimum number of bit flips that are required to give you a valid code word there is a problem because two different single bit flips result in a code word. Therefore, in this particular code it is very evident that you cannot map every n bit sequence unambiguously to a single code word that is the most likely one because see what what the problem over here is this. If you have 0 1 0 0 and this is a you know 1 0 0 and 0 1 0 0 are specific bit patterns for this code. These error patterns will result in an ambiguity as to which sequence was sent. Let us actually let us actually try it. Let us say that I take 0 1 1 1 ok. Another example let us say that I take 0 1 1 1 plus the same bit pattern 0 1 0 0. What does it give me? I get 0 0 1 1. If I do h x for this again I get 1 1. This is not surprising because as we recall in the previous lectures h times x where x is a code word plus an error is the same as h times e because h times code word is 0. Now, this indicates that you get 1 1. How do you get 1 1? That means, the single bit error could have occurred here or the single bit error could have occurred here or there could be two bit errors which have occurred here. Let us ignore that two bit errors are unlikely. Now, a single bit error could have occurred here single bit error could have occurred here which means that this 0 1 1 1 by flipping this sorry by flipping this you will get sorry. Let us say we flip the second bit we get 0 0 ok. So, let me actually just correct this ok. Let me not do it for the case of 0 1 1 1. I will do it for the case of let us say 1 1 0 0 that way it becomes a little easier to analyze. So, if you take 1 1 0 0 ok. If you take 1 1 0 0 then you get 1 0 0 0 yes syndrome is still hx now if you hx is 1 1. Since hx is 1 1 this indicates that if the single bit error has occurred then that error has occurred due to this column or this column meaning the first or the second position if you take 1 1 0 0 then by flipping the first one you get 0 1 0 0 by flipping the second one you get 1 0 0 0 neither of which is a code word ok. So, in this case also there is no way you can correct for this error because see if you have found out that the bit error has occurred due to a flip of the first or the second then 0 1 0 0 1 0 0 this are the possible code words you should get no they are not. Therefore, the problem with this particular code construction is that not all n bit vectors can be uniquely mapped to a single valid code word with the minimum number of bit flips and this is because of the ambiguity present inherently in the GNH. Let us do one more example ok we have been doing this n length repetition code, but I have deliberately and carefully chosen n s an odd number of course you may say that an odd number is always a you know easy because if you have an odd number doing majority logic is very very unambiguous because if you have little 3 choosing best of 3 is easy if you have 4 choosing best of 4 if you have 2 0s and 2 1s becomes tricky. So, let us analyze this code 4 bit repetition code, code words are g is very easy for repetition code again we will use the same trick this is i this is a h is we will use a transpose i sanity check in a if you multiply 0 you get 0 if you multiply 1 1 1 1 you get 0 this is the parity check matrix for this error control code. Now, there are some problems over here as well ok let us take be intuitively let us just say that you get x is equal to 0 1 1 0 it is very easy for you to check just like in the previous case that any single bit flip any single bit flip is not going to give you a code word because if you flip this one you get triple 1 0 if you flip this one you get 0 0 1 0 0 1 0 0 0 1 1 1 none of a single bit flips is going to give you a valid code word. Let us now look at 2 bit flips if you look at 2 bit flips right then by the way when you do 2 way flips all the sequences I am going to do are equally likely how many possible 2 bit flips are there there are several, but I am going to only list the code words that are possible you will get either. Now, the problem is that 2 bit flips which are both equally likely events right are going to result in 2 different code words. Therefore, this particular code you know 4 bit repetition code it cannot be used to map all n bit sequences uniquely to a code word. So, this is also you know does not satisfy this unique decodability. So, to speak based on any n length sequence n length bit sequence that is given. So, one thing we are going to demand when we go towards the Hamming code is that for any n bit sequence that is received we want to be able to perform the minimum number of bit flips to result in a single unique code word. Let me give an example of what is a good code word in that sense 3 bit repetition ok. In a 3 bit repetition code G is 1 1 1 code words are 0 0 1 1 1 I am going to show you that minimum number of bit flips can be uniquely decoded 0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 1 1 1 1 0 1 1 1 here all of these the decoded code word is 0 0 0 all of these the decoded code word is 1 1 1 why because in all of these cases let the most likely event is 0 errors you know because it is you want 1 minus p whole cube. The next likely event is 1 bit flip which has the probability 1 p you know p into 1 minus p the whole square remember p is less than half in which case all of these uniquely give you 0 0 0 as the code word because see let us take 0 0 1 perform 1 bit flips is 1 0 1 a code word no 0 1 1 a code word no 0 0 0 a code word yes. So, performing a 1 bit flip will yield this code word uniquely similarly if you get 1 1 0 you can flip the first bit not a code word flip the second bit not a code word flip the third bit you get a code word 1 bit flip unambiguously gives you 1 1 1 as the potential code word that was sent. So, in this manner the full space of all 2 power 3 elements are mapped to a single element by the decoder without any ambiguity unlike in the case of the repetition code or even in the case of the parity check code there is always an ambiguity because single bit flip did not result in bit vector you have to do 2 flips and with 2 flips resulted in multiple. For example, you can verify that over here if you have 1 0 0 0 you get the syndrome is 1 1 that means, you have to perform 2 bit flips which 2 bit flips all those questions are difficult to answer let us now go to go further. So, what are the generator and parity check matrices for this code we just did it you can just take the 3 2 parity check codes generator matrix and you can you know it is like 1 0 I believe it is and add another this is the generator matrix and the parity check matrix also we just did there it was and you can do it for the others. So, 0 1 0 0 we were basically saying you know you have to perform all the bit flips we will not get a unique answer that is the key because if you take 0 1 0 0 flipping this bit will give you 0 0 0 0 flipping this bit will give you 1 0 0 0. So, as a small aside we are going to discuss hamming weight and distance ok. The hamming weight of a bit vector x is the number of ones or the number of non-zero elements in x as denoted as w h of x that is for example, if you have and of course, w h is the number between 0 and n. For example, let us take n is equal to 3 should take 0 1 0 the way it is 1 because it has 1 1 1 1 the way it is 3 0 0 0 the way it is 0. So, in that sense w h counts the number of ones the hamming distance th between n bit n bit sequence x 1 and n bit sequence x 2 is the number of bits in which they differ that is it is the number of bits in which they differ. In other words the find the XOR bit wise of those and count the number of ones that is an easy way of finding the number of code words in which they remember of bits in which they differ. If you for example, you have 0 1 0 and 0 1 1 these differ not in the first not in the second only in the third what this is saying is you XOR these you get 0 0 1 and then you count the number of ones you get 1. So, this is a trick that you can play and makes it easy for computer implementation also the hamming distance d h of x 1 x 2 is w h of x 1 plus x 2. Now, I will outline the decoding strategy for linear block codes. If the received bit vector is y I mean in this in this case I am changing the notation I am if x is the code word a valid code word and y is the received sequence that is x plus noise the closest in terms of hamming distance from y is typically what you want as the received code word. In other words if you get y what is the minimum number of bit flips that I have to perform to get an x which is a valid code word that is a question where you are asking why is this you know why is this the best strategy see the most likely event that has occurred for p v v in less than half in a binary symmetric channel is that no errors have occurred which means perform 0 bit flips. The next likely event is that 1 bit error has occurred. So, we will flip one of the bits and check whether we get a code word the next likely event is 2 and we keep we keep performing this and finding out which of the these you know modifications results in a code word that is minimum distance from the actual code word x. In other words we want to find out a y that has minimum hamming distance from x where x is a valid code word. An alternative formulation and this is something that we have seen in some part earlier is that we want to find the least hamming weight error pattern e such that h of x plus e which is h e is equal to h y then x is equal to y plus e is the most likely valid code word. What are we saying see we are saying perform bit flips let us express these bit flips as a vector e with the smallest number of ones why we just said that if no bit errors occurred choose e as the all zeros n bit sequence. If 1 bit errors occurred then iterate over e being 1 0 0 0 0 1 0 0 0 0 0 1 0 and find out which of the h e is is equal to h y. If you can figure this out then x is y plus e the problem is that this is not unique and if this h y being equal to h e is unambiguous like in the case of the 3 length repetition code then we can always correct the error. Now this is something that we will see in the next lecture onwards where we will continue this discussion and build on top of this and talk about the single error correcting codes called hamming codes. Thank you.