 lecture 21. So, the last thing we were talking about was this discrete time ISI model and discrete time ISI model worked out to something like, so the model worked out to something like this. The symbols you are sending out are going to go through ultimately. You can model it as if they are going through filter MC and then noise gets added to it and you get ZK. This is what I wrote as ZK and the intermediate result here I called BK. So, this MK is going to be monic causal minimum phase. All those things are true here. So, you usually think of SK as for K between 0 and L minus 1. So, we will take only L different symbols IID drawn from the constellation X. So, that is what you think of LS. So, now if this is a mutab filter, so because of the memory in the filter itself, the BK will be kind of be expected to be non-zero from what? 0 to L plus mu minus 1. There will be an extra L plus mu minus 1. So, this vector BK should think of it as non-zero from the symbol I mean the signal BK will be non-zero for K between 0 and L plus mu minus 1. Am I right? MK is a mutab filter. So, what does what did I mean when I say mutab? M0 through M mu. So, I think it is L plus mu minus 1. So, you will have to be careful about how this works out. Is that fine? Make a mistake. Is it okay? Maybe it is L plus mu. Is it L plus mu? L plus mu minus 1. Mu minus 1 plus mu. It seems to be a general disagreement. So, let us look at the last, let us look at the expression. So, BK is what? The way you write it down, it is going to be summation L equals 0 to mu ML S K minus L. Is that right? It is fine. So, if I want K equals L plus mu, if I put K equals L plus mu what happens here? So, for K equals L plus mu BL plus mu becomes what? Do the evaluation and tell me there will be only one term which is non-zero. What is that term? L equals? Is there any term that is non-zero? No, it becomes 0. See, this is very basic simple convolution. So, this becomes 0. So, I can definitely restrict BK to, well 0 in the sense it is, S does not contribute to BL plus mu basically. So, you can let this be 0 to L minus, plus mu minus. So, that is the simple check to make sure it works out. So, you get ZK. So, ZK also you have to imagine as relevant for K between 0 and L plus mu minus 1. So, that is the way it will work. So, this is BK plus NK. So, this is my model and we took some simple examples where we chose the constellation and we chose a very simple M of Z to get values for BK, where can BK lie? But once again I pointed out that is not an IID constellation. So, it is got memory, it is connected to that. So, those are few examples. Now, the real problem is you understand this model very well, but the real problem is to do detection. Given this ZK for K between 0 and L plus mu minus 1, can I find my maximum likelihood estimate or best estimate for SK, for the sequence SK, SK between 0 and L minus 1. And I cannot do it independently, I will have to do detection of the entire sequence because there is this ISI and this memory. I cannot do, for each K I cannot do it separately, I have to do for the entire sequence. So, that is the whole idea behind what is called maximum likelihood sequence detection and it can be done, can be done with some complexity and we will slowly make our way towards that sequence detection. So, basically in maximum likelihood sequence detection, so the problem becomes very simple. So, the problem is S hat is argument minimum over A in XL, norm of the vector Z minus what? I should be careful here, vector B corresponding to A. So, when this A is there, I am going to say the vector B corresponds to the signal AK convolved with MK. So, there is some notation conflict here, I am trying to be very careful with the notation. So, pay attention to this. So, AK convolved with MK is what? You actually think of it as a signal right, BK, BK between, this is BK between K 0 to L plus mu minus 1. I am going to collect all of them together and think of it as 1 vector B, B that corresponds to A. So, remember that, so this is Z minus B square. So, the reason why this is maximum likelihood is, I know N now is, N is, NK is AWGN right additive white Gaussian noise. So, you can if you want to write down the likelihood, the probability for Z given A, it will be multivariate Gaussian Z minus B. So, B will be the mean and all that. So, you can write it down. So, this you can show is maximum likelihood. So, it looks like minimum distance, but it is also maximum likelihood in this form. So, that becomes maximum likelihood detection. So, how do you solve this problem? It is the maximum likelihood sequence detection problem. So, hopefully this is clear. So, this part, the problem at least should be very clear to you. So, you can go back to this model. You want to find SK, but SK has gone through a monic causal filter with mu tabs to get BK and then noise has been added and you observe ZK. Given the observations of ZK, what is the best estimate for SK? So, how am I going to find it? I am going to collect all the ZKs into one vector Z and then look at the distance from all vectors B that are possible from different A and then find that A which gives you the smallest Z minus B. So, you have to do that extra filtering step before you can do the distance computation. You cannot compute distance from C to A directly. So, that filtering step causes problems. So, it is not as easy as before. So, how do you solve this is the maximum likelihood sequence detection problem and the solution is through what is called the, well, if you do a brute force solution, what is the complexity? Complexity goes as XL, right. So, size XL, computations required of the order of this times some constant number of computations. That is the complexity and it is difficult to implement this in practice. Nobody will want to do anything close to this, particularly when L becomes large. So, we would like to have something which is independent of L, which does not grow exponentially with L because L is the number of symbols that you are transmitting. You do not want that to grow exponentially with L. Turns out there is a nice solution which is called the Witter B algorithm which provides complexity size X to the power mu computations. So, remember this is of the order of, there will be a constant multiplying this number of both these. So, typically people write this small o big o type notation, but I am going to just say of the order of mod X power mu. So, mu is the number of tabs in your MK. So, remember that. So, this is still exponential in the number of tabs. So, Witter B is not the most implementable algorithm in the world. If you have mu equals 10 and size X being 16, it is still difficult to implement. But for small number of tabs and small size constellation like BPSK, it is okay to imagine implementing it. So, for instance, if it is BPSK, then size becomes 2. You can kind of implement Witter B up to 2 power 4 and all that very easily, 2 power 4, 2 power 6 is doable beyond that is tough. So, with, I mean with present day processor and all those things, it is possible to do these things. So, that is the situation. So, how do you go about doing it? To understand this, first we have to model this filtering as a finite state model. So, Witter B works on the state diagram for the finite state machine. So, you might remember this from your digital class long back. So, finite state machines are, they have a state and then there is an input, it produces an output, it goes to the next state. So, here one can easily do a finite state representation. So, let me see, what is my filter? My filter mk is, well, it has got a z transform which is m0 plus m1z inverse plus mu z power minus mu. So, this is my mk and my Bk is going to be summation l equals 0 to mu mlsk minus lk. For a general k, if I expand this, I get m0 sk plus m1 sk minus 1 plus so on till m mu sk minus mu. This is a simple explanation. So, to your finite state system, sk is something like an input. At the k-th instant, sk becomes the input and Bk becomes your output. So, that is how I am going to model my filtering as a finite state system. So, I am going to say this is my input at time k. So, and Bk is my output at time k. So, the past inputs into your system have to be held in a state. So, that holds the state for you. So, I am going to say all these guys come from my state, state at time k. At time k, I am in some state. So, this is the question of, this is the state before time k, after time k is always that confusion. So, when the input comes, you are defining your time k. It has got to do with, if you go back to your digital systems, this is negative edge, positive edge, you always have to obsess with those edges. So, here also there will be that minor confusion. So, I am going to settle it like this. So, I am going to say state before the k-th input occurs is this. So, that I am going to call as psi k. So, the state at, I will say state at time k, but when I say at time k, you have to interpret it as before the k-th input arrives. So, state at time k, I will call it psi k. So, this will be sk-1, a vector, remember this will be a vector sk-1 through sk-mu. So, it belongs to, if you will, chi power mu. So, the vector. So, remember once again this subtle notion. So, when I say psi k, I will say state at time k, but what I mean is that is the state before the k-th input arrives. So, when the state is stabilized to psi k, I am going to say it. So, it is a subtle thing, be careful. All right. So, given the state at time k, I can find the output corresponding to any input at time k. So, one gets this conditional independence given the state at time k. Given the state at time k, the current output is independent of all past inputs. So, that is the finite state model. So, you want to have, given the state, you want your current output to be independent of all the previous inputs. Well, the state contains the previous inputs of course, but given that there should not be anything more. So, up to k, to mu it is there, nothing more is there. So, that is the finite state model. Here it works out like this. So, what will be the next state after your input? It is very simple. You just put, instead of k here, you put k plus 1. So, it will be, it is like the D flip flop that you have in your mind. So, I will draw an implementation also. You will see what I mean. It is very easy to see this. It is going to be S k, S k minus 1, S k minus mu. So, you can imagine this filter in this finite state way, finite state model being implemented like a shift register. So, the inputs are coming from the left. They are being held in mu registers. So, to speak, they keep on getting shifted. And whatever is stored in each register is going to get multiplied by the corresponding m and the output is being formed. So, you can draw a nice picture of this. So, how do you draw this picture? So, I am holding a register. Don't think of it as a flip flop or something. So, we can think of it as several flip flops together in a practical implementation, but this is basically a register that holds the incoming value. So, you need mu of them. So, I am going to call it, what shall I call it? It is always confusing. So, I will say S k 0, no, S k 1, sorry, 2, S k mu. This should be 2. So, those are the components of my state at each time. And the input that is coming in is what? S k. How do I form the output corresponding to this input? I need a summation here. This gives me B k. How do I do the connections? Yeah, multiply each one by the corresponding m. This gets multiplied by m 0. This gets multiplied by m 1, well, m of 0. So, again, once again, my notation will get confusing here. So, let me be careful. This gets multiplied by m 0. This gets multiplied by m 1. So, until the last one, which will get multiplied by m mu. So, it is very simple. And you clock these registers every time you get an input, you can evaluate them. So, this is the simple implementation. I am sure that in your DSP course, you must have seen an implementation of FIR filters in this form. So, I am going to group all these guys together and call them my state. This is my input and this is my output. So, that is my finite state model. So, a detection problem with the finite number of states turns out ultimately, the way I have described it for the Witte-Bell algorithm is as complex as the number of states. It may not be more complex than the number of states. That makes sense. So, at the end of the day, how many ever states? What is the total number of states that you can have? Number of possible states is size x bar mu. So, it should not really get more complicated than the total number of states that you have in your system. You should not have to do more work than that. So, that is the logic way, that is kind of the intuitive reason why the Witte-Bell algorithm kind of works. So, that is the thing. So, the next thing is to do things called state diagrams. So, you put state diagrams which indicate transitions, inputs, stroke output and all that. So, what I am going to do right away, so what am I doing on time? So, there are two things here. So, state diagram you must have seen before. I am assuming in your digital systems class or something, you have seen a state diagram. So, usually the state diagram does not have a time axis. You do not distinguish different times. You have the same location for different times also. Kind of for instance, you loop back into the state and all that. So, that is not very useful for us when we do decoding and all that because we want the time to be explicitly given because the input is different for different time and you are storing all of it together. So, usually the state diagram or the modified version of the state diagram that is used for decoding in the Witte-Bell algorithm is called a trellis. That is basically the state diagram with the time axis. So, that is called the trellis and that is what we will use for these things. Trellis for a finite state machine. So, it is basically the state diagram with the time axis. So, I am going to just show examples and illustrate. It is very, very easy to do general trellis diagrams. There is no point in spending too much time with it. It is very easy stuff. So, we will do some three examples of trellis diagrams and point out a few issues, issues practically that you will have to pay attention to, but other than that, it is very easy. So, here is an example. I will start with the simplest example that we have 1 by root 2 minus 1 by root 2 z inverse and this is BPSK. So, the first issue is what is your initial state? So, before we go there, what is the state now? Psi k is what? So, the number of taps is 1. So, psi k just contains 1 element and that would be SK minus 1. So, the problem is that k equals 0, you are going to get S of 0 and you have to be worried about what your psi 0 is going to be. So, there you need an assumption. So, you can assume something. You have to assume that the previous state was either plus 1 or minus 1. Do not assume 0. Do not assume something which is not possible. Then there will be all kinds of confusion. So, I will have to assume your previous state was plus 1 or minus 1. So, in practice, it is a question of what you set your registers to be, right? In your implementation of the finite state, what do you set your initial registers to be? Is there a question? No, you have to fix it to something when you start at the encoder. That can be a prefix thing that the decoder knows. So, the question he is saying is what if you do not know what it is? So, I am assuming that the initial state is known both at the encoder and the decoder and it is said to be 1 fixed state. So, typically I will take that state to be plus 1 or if there are multiple elements, I will take all ones, right? That is a very reasonable thing, but it does not matter. You can take it to be any other state. It does not matter. So, we will take psi 0 to be plus 1. So, we will assume this is known at the receiver. So, if you say this is not known at the receiver, then the decoder becomes more complex. So, we will assume it is known at the receiver. We will work with that. So, that is the first point. So, I am going to draw my states as circles with psi k written inside it. So, basically at time 0, my state is going to be plus 1. So, it is a circle with plus 1 inside it. Now, my input can be plus 1 or minus 1. That is S of 0. If it is plus 1, what will be my output? 0 and what will be my next state plus 1. Is that clear? Basically, the input becomes the next state. Just memory 1, it is very easy to do. If my input is minus 1, my output becomes, well, you have to pay attention to plus 1 minus. You see, remember plus 1 is S k minus 1 and there is a minus there. So, it will be minus root 2 and you go to minus 1. Pay attention to this a little bit, right? So, now what is my notation? This is psi 0, this is psi 1 and here I am writing S 0 stroke b 0. Right? That is my, that is how I start my trellis diagram, so to speak. Usually, in the state diagram, I would not have drawn a separate plus 1 for psi 1, right? I would have looped back to the same state, but I want that separate time axis. Different states at different time should have different representation. So, that is why I have the trellis. Okay? So, the next state likewise, you can quickly do. Remember the same picture will have to repeat, right? Because once I am given the state, whatever happened before is irrelevant. So, this picture has to repeat and then here you will have more transitions. Okay? So, this will become plus 1 given 0, plus 1 output 0, minus 1 output, minus root 2. Here again, you will have 2 transition, minus 1 output 0, plus 1 output root 2. Right? So, this is now psi 2, this is S 1 output b 1. Okay? So, now you can repeat this. You can do one more step. Get plus 1, minus 1 and the same thing would repeat, right? You do not have to worry too much about anything else. Same thing would repeat. So, you go for a while. Okay? Plus 1, minus 1, say this is your lth stage. Okay? After a while, this is your sl minus 1, bl minus 1 and then you have, you have what? This is psi, psi l. Okay? Right? So, now, like I said, there is some memory in the system which needs to be flushed out also. Okay? Right? So, there has to be something for the next stage. Right? So, when I, so you see what I am saying. So, I have to now define bl, bl is defined. Right? In terms of the other, yes. So, you need an assumption for sl. Okay? sl really does not exist. You can either say set it to 0 or you need some assumption. So, it is good not to set sl to 0. Okay? So, you might want to set it to some fixed symbol which is known as the receiver. Okay? So, you see what I am saying. Okay? So, that is an important next step. So, that is called termination. This is at the start, you have a confusion about what initial state you set it to. At the end, when you are flushing out the memory from your encoder, you will have a confusion for what are the extra symbols that you need to shift in at the end. Okay? So, usually you might say I am shifting in 0, but 0 is not really a valid input and it will cause confusion in your trellis. It will suddenly change the structure of your trellis if you do 0. Okay? So, it is common to input some prefixed symbol. So, sl I will set as plus 1. Okay? So, I am definitely going to set as plus 1. So, if I set sl as plus 1, there will be only two possible transitions here. I will go to plus 1 here with 0 or plus 1 with root 2. And this will be my psi L plus 1. Okay? And this guy will correspond to sl. Well, sl I know is definitely plus 1. So, I do not have to write sl really, but I will write it will be the stroke BL. Okay? And this sl also we will assume is known at the receiver. So, it will be assumed that psi 0 and sl known at receiver. Okay? So, it is some prefix thing. So, you can arrange for it. Okay? So, this is the termination part of it. Okay? So, this is a complete trellis which finite state model for this simple discrete time AWGN channel. Okay? So, now we will see one more case. The next case is, so I am going to do one more example which is slightly more complicated. Then we will come back and see one more. Okay? So, I am going to take m of z to be 1 by root 2 plus half z inverse plus half z power minus 2. Okay? Is there a question? Why we had to take? Sell it all. Okay? So, question is why termination? Okay? So, why termination? Why do you have to terminate? Okay? So, you can, you need not terminate. So, you are saying I will do my maximum likelihood sequence detection only with v0 to b L minus 1. Turns out that problem is more difficult. Okay? So, you terminate, then the maximum likelihood sequence detection becomes easier. The Viterbi algorithm works well only when you start at some state and end at some state. Okay? So, this termination will be used in MLSD. The reason is when you terminate, you know that you ended at a particular state. If you do not terminate, sil you do not know what state it is. It can be anything depending on your actual sequence and that is not good for maximum likelihood sequence detection. Need to start at some state and end at some state. Okay? It is possible to work around it. I will mention a lot of practical aspects which can get over it but theoretically to be accurate MLSD, we need to terminate at a particular state. So, that is the next thing. And then you have the next model and I am going to once again say BPSK. Okay? So, the thing I want you to take a crack at is tell us diagram for this. Okay? Do for at least three stages. Okay? So, couple okay? So, do the example for three inputs S of 0, S of 1, S of 2. Okay? So, first thing to fix is what is psi k? Remember psi k is going to be S k minus 1, S k minus 2. Okay? So, how many states will you have? Four states. Okay? 2 power 2 is 4. Okay? So, and you can set the initial state to be plus 1, plus 1. Okay? I do that. It is a little bit more tricky to figure out the next state. Okay? It will be a shift and the new input will come in from the left. Right? So, that is how it is. The next state is going to come. So, if you have input S plus 1, okay? I will leave the output. Okay? The output is some linear combination of all these things. It will be a strange number. I do not care much about it. I will just draw the next state. Okay? It is going to be plus 1, plus 1. And if the input is minus 1, you will get a certain output and the next state will be minus 1, plus 1. Okay? So, fill it in if you know what the answer is. It is not too difficult. It is just that one needs to write it down. It will become strange. Okay? So, some output. Do you see how the next state became? Minus 1, plus 1. So, every time when you want to find the next state, you take the present state, shift it to the right. Do not rotate, shift it, shift it to the right. And the next, the input should come in from the left and occupy the vacant first pot. Okay? So, that is how you find the next state. And now, you can keep on proceeding. Plus 1, plus 1, minus 1, plus 1. Then you have plus 1, minus 1. You have minus 1, minus 2. Okay? So, there is a good reason why I am writing it this way. If you do not write it this way, it will get a little bit confusing. Okay? So, this is plus 1, question mark, minus 1, question mark, plus 1, question mark, minus 1. Okay? So, this is the, this is for the next input, S1. In the next input, okay? So, now we can draw the complete trellis, so to speak. Okay? So, all the transitions will be there. Okay? And you can write down all the plus 1s, minus 1s input. Okay? So, you have, this is plus 1, minus 1, this is plus 1. Okay? And then it continues for a while. Okay? Alright? So, if you want me to write down the thing, the input and output is S0, output B0, psi 1, S1, output B1, psi 2, S2 output, B2, psi 3 and so on. Okay? So, a few definitions based on this trellis. Each of these things is called a stage. Okay? So, this is called second stage of the trellis. So, basically stage 2 of the trellis. Okay? So, in every stage, you have a previous state and the next state. Okay? At every stage, you also have an input, symbol input and corresponding output. Okay? So, where should I put ZK in this trellis picture? Where does it make sense to put ZK? What does ZK know? Right? It is BK plus noise. So, I am going to associate ZK to the Kth stage of the trellis. Okay? So, I am going to say, well, there is no real place to put it at. We will put it as we go along. But I am going to say the output is associated with each stage, Z0, Z1, Z2 and so on. Okay? So, you can draw the general picture for a stage. The ith stage will have psi i, psi i plus 1, psi stroke Bi, then Zi will be associated with it and there will be all kinds of transitions. Okay? That will be the ith stage. Okay? So, you have the previous states and the next states. Each of these edges are called branches. Okay? So, these case are called branches. It is a branch. Okay? And each branch now corresponds to an input and an output possibility. Okay? So, each branch corresponds to an input and an output possibility. Okay? So, so if you notice one thing, so this, so there are not any, no, see no two states. So, there are states which are not connected by branches. Okay? So, let me put it carefully. Okay? Right? There are branches only between certain number of states. Okay? So, how many branches will be there from arising out of a particular state? If you are at a particular state, how many branches will go out? Number of possible inputs. Right? So, number of branches out of particular state will be size of X. Okay? What about coming into a particular state? How many branches will come into a particular state? So, on the right hand side, what is the answer? How many branches are coming in? Don't just say mu, go to the previous example. Same as size X, right? How do you reason that out? Can you reason that out? Well, yeah. So, but need not be, no, I mean, how do you know? So, you have to reason it out carefully. It is possible to reason out based on the next state. Given the next state, how many possible previous states can you come from? Do the reverse of what you did for going from present state to next state, right? What did you do for going from present state to next state? You shifted right and put the input on the left side. So, if we are in the next state, you have to go to the previous state, what should you do? You shift left and put something on the rightmost thing. So, how many possible things can you put there? Size X. Okay? So, size X will be the number of input coming into, input branches coming into each state. There is no problem there. There is also one more interesting question I can ask. Forgetting what that question is. Anyway, we will come back to that later. Okay? So, these are simple things to look at based on one more thing. So, the inputs corresponding to each branch that are leaving a particular state will actually be what? What will be the different inputs leaving a particular state? See, if you look at all the branches leaving a particular state, what will the inputs correspond to? Will they all be different? Will they all be same? Things like that. They will all be different, right? Different values of X. Now, if you go to the next state and look at the branches coming in, what about the inputs? They will all be the same. Okay? Right? Look at the example here to convince yourself. Okay? So, those are all subtle things about these trailers, which will be useful later. I mean, useful in case you are trying to implement it with a very small processor, it does not have too much memory, all these things you can use. Okay? So, the outputs out of a particular state, the branches correspond to different inputs. But if you go to the other side and look at all the incoming branches, they will all correspond to the same inputs. Okay? The reason is this is what is called a finite input response, non-reversive type thing. There is no feedback. Okay? If there is feedback, then more funky things can happen. Okay? All right. So, I am going to stop here for today. So, so we will