 Okay, so this is lecture 33 and the last thing we were doing was bungling this simple derivation of this a intrinsic and extrinsic information, let me do that real quick, hopefully it will work out now. Okay, well let me not say information, it's just intrinsic and extrinsic, LLRs, right, log lake-liquid ratio. So the total log likelihood ratio, suppose we write it as capital L, I said that's going to be probability for Ui equals 0 given the entire received vector, okay, so I think we split the received vector into two parts, I said R0 and R1, right, divided by probability, well log of this, looks like it's getting more, log of probability that Ui equals 1 given R0, R1, okay, so this is what we had and it was trying to look at the expression on top and try to split it into, write it in some form so that this will split nicely, the trick to doing that is to start this expression carefully and then write it differently. So the way I'm going to write it is R0 is the output vector corresponding to U, right, R1 is the output vector corresponding to the parity sequence V, okay, so I'm going to split it like that, I'm going to say R0n which is the actual received value, well I am sorry, R0i which is the actual received value corresponding to Ui and everything else, okay, so for that I'm going to say R0 not equal to I and then the entire R1, right, so actually you should take the entire R1, the parity bits are all involved, so I think the notation I had before was for simply the entire R without I, okay, so that's not quite right, you have to split it carefully so that the parity part comes out nicely, so for this, the first step, I think the first step was quite okay, I write it as a joint, joint thing, okay, so R0 not equal to I then R1, okay, so the entire parity thing divided by the conditioning events density which also I'll write as R0 not equal to I R1, okay, so this is fine, so the trick is to look at this expression Ri0 given everything else, Ui equals 0 and then all the remaining R's, okay, so once you've given that Ui equals 0, okay, you know Ri0 is going to depend on nothing else, no one, none of the other received values will play a part given Ui is 0, so all these other things while they are there, I can throw them out because they'll be independent of this Ri0, okay, so what's that remaining thing there, that's going to be Ui equals 0 comma R not equal to I 0 and R1, right, so this part is the same as this, okay, I don't have to really write there because I know it's independent, okay, the denominator is going to stay the same, so the denominator also, let me write this as f of Ri0 given R not equal to I 0, R1 multiplied by f of R 0 not equal to I R1, okay, so notice I cannot use the same argument for the first term in the denominator, right, because I'm not given Ui is 0, for overall all possible values for Ui, the Ri's are not independent clearly, we saw this example once before also, okay, so once you're given a particular value for Ui, it becomes independent, so now it's a easy task of identifying what is it that I want, okay, so I'm going to go to the next page, hopefully you can carry out, carry over, okay, okay, so you see this probability that we're evaluating becomes f of Ri0 given Ui equals 0 times, okay, what you have there, if you notice, okay, look at this term, f of Ui equals 0 comma something divided by f of that same thing, so that can be written as Ui equals 0 given, well probability that Ui equals 0 given the remaining things, right, so you think of this whole thing as some b, okay, if you think of this as some b, this is f of a comma b divided by f of b, okay, so this is the same as probability that Ui is 0 given b, okay, so once I write that down, I'll see this becomes probability that Ui is 0 given R0 not equal to i and then R1 and then the denominator, I have something which will actually not change depending on 0 or 1, okay, R0 not equal to i, R1, okay, so this is what I have for probability that C i is, no, Ui is 0 given the entire vector R, okay, I'll have a similar expression for probability that Ui is 1 given the entire vector R, what will be the only change, all these Ui equals 0s will become Ui equals 1, now when I divide that, what will happen? The denominator will cancel and then you'll simply get the two terms, so when you divide and take logarithm, you'll see you'll easily get capital L equals log of f of R i0 given Ui is 0 divided by f of R i0 given Ui is 1 plus log of these other things which I called actually extrinsic LLR given R0 not equal to i then R1 divided by probability that Ui equals 1 given it's 3 parts, R1 is the received values corresponding to the parity bits, yeah, yeah, I know but the parity corresponding to the ith stage should also be retained in this conditioning, yesterday I didn't do that in the notation itself, okay, all right, so this you'll know from BPSKWG and what will this simplify to? E power 2R i0 by sigma square, so you take log that will come out, so you get 2 by sigma square times R i0 plus that will not really simplify, I'll call that L extrinsic, this is L total, okay, so this is the this is the important boxed formula from this variation, okay, so what the BCJR algorithm will give you is, right, it will take R0 and R1 and they'll give you these L totals for each i, okay, so you'll get this vector i, so what does the BCJR algorithm give you, it will take, it will take, so for instance one can write down this bitwise MAP, okay, so let me write down the bitwise MAP description, so the bitwise MAP, MAP decoder is going to take what all as input, okay, R0, R1, anything else should I write down here as input, yeah, maybe later I'll have to write down, but now right now I'll simply say these are the inputs that it takes and then the output it would give you is this vector L1, L2, L1, okay, which is actually each Li, well I don't have i here, so I should put i here, okay, so each Li is Li total, which is actually composed of two things, which is the intrinsic information plus intrinsic LLR plus the extrinsic LLR, that's what the bitwise MAP algorithm is going to give you, okay, so this is one part and let me remind you about the other things that we talked about, we talked about puncturing, right, which was required for the turbo code and then I also give you a picture for the construction of the turbo code, which I'm going to repeat right now, so you'll see, you'll see how that thing looks, okay, remember puncturing, okay, so you remember puncturing, right, if you have several bits you simply just drop the punctured bits from transmission, so how will you deal with it in the receiver, the decoder you'll set those corresponding RA values to 0 and the BPSKWG in case, even for the bitwise MAP you would do the same thing, okay, so one can imagine doing those things quite easily, okay, all right, so let me go come back to this turbo codes, okay, so how do the encoder look, the encoder looks like again, once again I'm going to take simple examples, I'll do a very simple example instead of doing the most general case, you have a message sequence E, there's one part, the systematic part which goes out without any change, that's B0, okay, and then you have encoder 1, okay, what is this, this is the parity part of say some rate half convolutional, well recursive convolutional code, right, so that is something it would give you v1, okay, if you have k bits here you'd get, well I might have to terminate, right, so this is u plus, I don't know, did I have any notation for the terminations, so okay, we'll simply put ut, okay, maybe you terminate, maybe you don't, okay, so those are issues that we'll deal with later, but in general one can terminate, so this would be, so let's say the memory is m here, so this would be k plus m, no actually this is not really the message part, so let me just leave it here, let me not worry about the termination right now, I think it's not, it's getting too confusing, let me just write down the encoder here so that I have k bits here, okay, so I'll simply say I have k bits here for the parity, and then what do I do here, the crucial thing is this interleaver or permuter, which I call pi, okay, so again produces k input bits, which actually go into encoder 2, okay, in many cases, in fact, yeah, yeah, that is, yeah, yeah, yeah, so no, that's what, that's the definition, don't ask why you define things that, I'm going to take a recursive thing here, okay, so encoder 2, which is going to have produce v2, which is again k bits, okay, so maybe you terminate, maybe you have some termination bits built into it, so that I'll put in k itself, okay, so let's not worry about it, we'll take sufficiently large k so that all these terminations won't matter as far as rate is concerned, okay, so you put k bits, okay, so if you want a definite example here, this can be an implementation of 1 plus d power 4 divided by 1 plus d plus d squared plus d power 3 plus d power 4, okay, it can be an implementation of that, in fact, in many practical cases, people take encoder 2 and encoder 1 to be the same things, exact same implementations, okay, so this will also be, in many cases, this will also be the same thing, okay, or you can put anything else, okay, so the question was why do I want this to have feedback, people have found that having feedback is better, okay, so you'll see, if I don't have feedback, I won't get some gains because of this, some distances will not play out the way you want, okay, all right, so this is the overall picture, this is how the picture looks, and can you point out the most complicated component in this whole encoder? Pi, okay, so if you have 1000 bits, interleaving them and putting them out in a random order is one of the most complicated things that you're doing here, because everything else is what, simple delay and XOR, absolutely no complexity in anything else, except for Pi, which will require a huge memory and maybe random access, etc., etc., all those things will take a lot of complexity, okay, other than that, the encoder for turbo codes, yeah, it's one minute, so other than that, the encoder for turbo code is not terribly complicated, okay, so for instance, if your K is 100, you're not so afraid of a 100 bit permuter, okay, maybe there is some structure to the permuter, so you can do it in a different way, but in general, a random permuter is tough to implement, okay, so what else? I mean, there are other turbo code structures, this is called, as you said, the parallel concatenated structure, okay, so this is the parallel concatenated structure, okay, one can also have serial concatenated turbo codes, those have been studied, but the parallel concatenation is very popular, okay, so by design, this just becomes a rate 1 by 3 encoder, if you want higher rates, you have to puncture, for instance, the rate half turbo code is again a very classic way of obtaining a rate half turbo code, is to take alternative bits from V1 and V2, then you would get a rate half turbo code, okay, so this is also something you saw in the last class, so that's how simple turbo code is put up, okay, so quick word about termination before I move to the decoder side, okay, so suppose I terminate for encoder 1, okay, so I could always terminate for encoder 1, right, I go down the encoder after k bits or whatever k minus m bits, I see how many more bits are needed, what those bits are, okay, those are exact message bits to go back to the all zero state, okay, right, but then I have to take this entire block and permute, okay, and then when I go through encoder 2, there is no guarantee that, yeah, I mean, whatever you do, the sequence of other bits that come into encoder 2 might give you, take you to some other state, okay, right, it may not be the same, the termination bits that were used in the encoder 1 may not work for encoder 2, so there is some complication there, so it's not very clear how you terminate both encoders, maybe you want to terminate both encoders, maybe you don't want to terminate both encoders, yeah, I mean, you have to think about all those choices, yeah, usually you don't send the termination bits together, okay, yeah, so I'm going to comment about that, so you do that, right, so what, so it's possible, one can work it around so that you terminate both encoders, okay, so the point is, is it necessary, okay, so it turns out since k has become really, really large, you don't really need to terminate, see the one problem in Viterbi is what, when you do only Viterbi decoding, you need the termination, right, otherwise you don't know when to stop, what to stop and all that, so you have to go back to the all-zero state, only then you will have one survivor path, otherwise you'll keep getting 2 power m, survivor paths in each stage, okay, the advantage with Bitwise MAP is, see, Bitwise MAP is going to give you probabilities, it won't give you survivor paths, at each state, it will give you only probability of being in that state conditioned on the past and conditioned on the future, okay, so suppose you go all the way to the last stage and you don't know how to terminate or you don't know how to start, what you do is you assign equal probability for everything and start back on the reverse equation, so the Bitwise MAP decoder is reasonably immune to not terminating, but termination actually gives you some gain, it's not that it won't give you gain, but termination is not necessary for really large block length, that's what people have observed in practice, so you'd see in several times people won't really terminate, particularly turbo codes when you're running Bitwise MAP decoder, people don't usually terminate and seems to work fine, okay, so you don't terminate, maybe you terminate only one encoder at best, you don't have to terminate both encoders, okay, Bitwise MAP decoder can be done and anyway the decoding will be iterative, by the time you do several iterations it's very clear what the final status, you don't have to really know, okay, so termination is an issue, it can be solved, but usually in practice people don't really do it, okay, so that's one, one more comment I wanted to make, that's one thing, the next thing to look at in the encoder closely is, so one thing I was claiming as a drawback in the convolutional codes was what? Something about weights, right, because of being LTI the number of low weight code words is going to increase linearly with K, if you have more K you're going to keep getting proportionally more low weight code words, maybe that's not a good thing, so I was claiming that this will help you get over that, okay, so one cannot prove those things very rigorously, in fact it's possible to prove if you average over all possible interlevers, but for a specific interleaver these things are difficult to prove, so for instance in books you'll find examples of weight distributions, okay, complete weight distribution for small lengths, okay, you enumerate all the code words for small lengths, take K equals 10 or something, right, enumerate all the code words for the convolutional code, for the rate half convolutional code and for the punctured rate half turbo code, and then when you see that you see that there is a noticeable drop in the number of low weight code words, even for very small block lengths there is a small drop, particularly when you go to very long block lengths, when you go to K equals 100 and all that, right, well K equals 100 maybe it's tough, maybe K equals 30, 40 where you can actually enumerate, right, when you do the entire enumeration, run it on a computer for a long time and then see actually how the weight distribution works out, you see there is a gain, okay, number of low weight code words decreases, it doesn't become zero, in many cases if you do a one intelligent you will get the same minimum weight, it's not that the minimum weight is going to increase, but the number of minimum weight vectors will fall drastically compared to a simple rate half convolutional encode, okay, so it's a huge fall and that is one of the reasons why turbo codes work at very low SNRs, okay, so you might be familiar with this at very high SNRs, one can do something for, to compare performance of codes one can do something called the union bound, okay, the union bound goes as number of code words of the minimum weight times Q of the distance, okay, so the number of code words also plays a role, the number of code words is too large that particularly lower SNRs when the Q is not too high, not too low to drive everything down, the number of code words will play a big role and getting that down at lower SNRs makes a big difference in your performance, okay, if you SNR is so low, so high so that the Q really dominates then this outside constant doesn't really matter, okay, but for low SNRs capacity achieving SNRs, right, you really need to control everything, so at that point these kind of things give you a gain, okay, I know it was a long lecture there is no real mathematical support, if you want I will give you examples of books in which this enumeration is done, okay, that's the only thing you can do beyond that you cannot do, right, that's the whole philosophy of these capacity achieving codes, you cannot really prove that one code will work perfectly, but in practice it always works, okay, that's the justification, okay, so that kind of property is called spectral thinning and all the people have studied this in detail, so spectral thinning really works in this okay, so that will stop with the encoder side and why it works, move to the decoder side and try to see if there is a really low complex decoder, soft input decoder that can be implemented for this, okay, so that's what's important on the decoder side, right, you remember why did the LDPC codes work so much better than Reed-Solomon codes, the reasonings we had were there is some random element in the construction which makes the minimum distance grow very well and then on top of that there was an implementable soft decision decoder, okay, which was not too complex, somehow on the encoder side turbo codes seem to have all those properties, on the decoder side do we have a nice decoder, okay, so there was one question, for instance is there an ML decoder, okay, this is very unlikely that there will be an ML decoder, okay, even if it is there you will have to incorporate the complexity of the interlever, okay, previously your memory was what, only the number of flip-flops in your encoder, now we want to see a memory, the whole interlever memory needs to be taken into account, you can't do a state diagram with the interlever memory, okay, it might be very huge, okay, so you're not going to be able to do a complete trellis and there's no way you can do any VWR, it won't work, okay, so you can't have ML decoders and optimal decoders are too complex, okay, but let's look at what an optimal bitwise MAP decoder might want to compute, okay, so I want to compute probability that UA equals 0 or 1 given what all, okay, so now I'll have the entire received vector, what is this received vector now for my previous example, if I look at my previous example what all do I have, I have a set of received vectors corresponding to the systematic part, then I have a set of received values corresponding to the first encoder's parity, then I have another set of received values corresponding to the second encoder's parity, I'm going to split my received vector into these three parts and see if I can say anything with those things, okay, so you'll see with just a part of the received vector it's implementable to have a soft decision decoder becomes implementable and then you see how to work with the whole thing, okay, so I'm going to split this as R0, okay, this corresponds to U and then R1, okay, this corresponds to V1, okay, maybe the punctured V1, okay, so remember R1 need not be the same length as R0, okay, it could be half in the punctured version and then I'm going to take R2 which corresponds to V2, okay, so let me put a correspondence I'm going to split it like this, if I have all these three received vectors and condition and all these three I know my encoding, my decoding is going to be very, very complex, but suppose I throw away the conditioning on R2, suppose I only want to compute these a posteriori probabilities conditioned on R0 and R1, can I do that? Yeah, you already know you have the bitwise MAP decoder which is going to work, right, so you know that's going to work, so I can do that, okay, so what can I do, okay, so this is probably difficult to compute, so what you do is you try to compute probability that Uy is 0 given R0, R1, okay, so maybe I call this, what shall we call this, we'll call this L1i, okay, so the a posteriori LLR according using values only from encoder 1, I'm not using encoder 2 at all, clearly this is suboptimal, right, this is not the optimal solution, I don't need the other encoder and why do I need this, there's no way in which I can claim that this is going to be optimal in any way, okay, but in fact how do I know, but forgetting about all that, how do I know this will split, this will split as 2 by sigma square times R0i plus L1 extrinsic i, okay, I know this will split like this, okay, so let me just compute this and keep it aside, okay, so one of the one of the significant breakthroughs was to realize that this guy is actually the new information produced by the decoder, okay, so out of this total thing LI, the new additional information that was produced by the decoder, okay, so the first part could have been done without any decoding, right, the new additional thing that was produced was the L1 extrinsic, right, so that was the first thing that people realize, the next thing is suppose can I use this now again in some other decoder for the for encoder 2, okay, so right, notice I can do the same thing for R0 and R2 that also I can do, if I drop R1 also I can happily decode with my bitwise MAP decoder, okay, but if I do that independent of the first decoder, I am not using any information that was produced by the first decoder at all, okay, so what you do is you try to use just the new information that was produced by the decoder by your first decoder in the second bitwise MAP decoder which uses only R0 and R2 and this extrinsic LLR which is not present in R0, right, this information is not present in R0, it is something else, okay, so the question is how do you incorporate that extrinsic LLR, okay, so it is actually very easy to incorporate because notice this is actually an LLR, right, okay, it tells you the log of the probability that UA is 0 divided by log of the probability that, I mean, divided by the probability that UA is 1, okay, so that is what it tells you, okay, so this is some kind of extrinsic information produced by the first decoder which is used as a priori information by the second decoder, okay, so what is my second decoder, okay, so this is my first decoder, this is done by the first decoder, okay, I have a second decoder, okay, which would normally produce, right, it would normally produce probability that UA is 0 conditioned on R0 and R2, okay, so this I can easily do but in doing so it will also incorporate L1 X, okay, so the input to the second decoder is not only R0 and R2 but the extrinsic information that was produced by the first decoder, so how do you incorporate that, one might say how do I incorporate that, okay, so it is possible to incorporate it, you have to incorporate it and actually incorporate that, I have to dig into the Bitwise MAP algorithm a little bit, okay, so the Bitwise MAP algorithm computes probabilities in the way very similar to the branch matrix, okay, so you have a way of incorporating this extrinsic information into the branch matrix computed by the Bitwise MAP decoder, okay, it is actually very easy to do, it is not very difficult, it can be done, okay, so this is incorporated in where, incorporated in branch matrix of Bitwise MAP decoder, okay, okay, so that is why when you write down the Bitwise MAP decoder, the inputs for the Bitwise MAP decoder will be three things, first is R0, the received values corresponding to the systematic part, then it is either R1 or R2 depending on which decoder it is, for decoder 1 it is going to be R1, for decoder 2 it is going to be R2, the received values corresponding to the corresponding parity parts, okay, then you will have the third input which is the a priori information available to that decoder, where is it available from from the run of the other decoder, okay, in the very first one for the first decoder, what is the a priori information, there is no a priori information, but for the first run of the second decoder, you already have some a priori information which you have to incorporate into the branch metric of the Bitwise MAP decoder, okay, since I did not discuss the Bitwise MAP decoder in detail, maybe it is not clear to you, but what the Bitwise MAP decoder does for branch metric is, it computes some LLR for each branch, okay, so you can easily incorporate another LLR from something else, you simply add the two LLRs, right, so assuming they are independent, you add the two LLRs, how do you combine several LLRs, we did this in message passing also, you can add them once you assume they are independent, so you would add the two LLRs and you would put them in the each branch, yeah, you have to justify the independence, let us justify it by saying that k will tend to infinity and your entire lever is random enough, okay, all right, so that is what we do, that is what we do here, so if you stop here, it turns out the body coders do not do all that great, okay, so the trick is to try to repeat this, go back somehow to the first decoder once again and then use, so try to repeat the same procedure, okay, so that is the trick, so for that you need some kind of a model for the LLR that is produced by the second decoder, okay, so that model works as follows, okay, so you assume that the LLR produced by the second decoder, okay, based on the independence assumption will look like L2i equals 2 by sigma square r0i plus L1ixt plus L2ixt, okay, so remember what was the LLR produced by the by the first decoder, it was 2 by sigma square ri0 plus the extrinsic, okay, that L1 extrinsic became input to the second decoder, okay, so you say that what is produced now by the second decoder is the systematic LLR plus the a priori LLR that it got as input plus the new extrinsic information that it is added, okay, so this is the systematic part, this is actually extrinsic from the first decoder which I am going to say as a priori LLR and then this is the extrinsic, okay, so that is how, okay, I did not write this down very clearly, let me write it down once again, okay, that is extrinsic, once I split it like this, okay, I can now go back to the first decoder all over again, what will I use as a priori information there, I will use L2ixt, okay, so that is important because you notice one of the principles of message passing is what, you do not send back what you got, right, so you only send what you generated new, okay, so that is one more, so that is another principle that is being enforced here, so that helps us in maintaining the independence, okay, so once the independence is maintained, I can combine LLR in the same way I did and then hope to get better and better information, now I can keep on iterating, first decoder, second decoder, back to first decoder, then back to second decoder, first second, first second, go back and forth, back and forth, and it turns out once you do some 15 or 20 iterations, these turbo decoders really can take you very, very close to capacity, okay, under suitable designs of course, do a lot of design, but they can take you to capacity, so trick is again to do these iterations, okay, so this apparently is similar to the action of a turbo engine, I do not know how many of you know engines and motors, I do not know very well, but apparently the turbo engines and cars and all these other mechanical things do such things, okay, so they do, I do not know, there are two things which go back and forth, some heat is produced which is used by something else etc, etc, okay, so that is why it was called the turbo code and the turbo decoder, even today the thing is very popular, okay, so it is a very popular notion, actually one can see it as an instance of message passing, it is possible to see this kind of thing as an instance of message passing, okay, so what I am going to do next is to write down, do two things, first thing I am going to do is write down a step by step description of the turbo decoder, the next is to draw a block diagram, okay, just to give you an, okay, first let us draw a block diagram and then we will write down the step by step description, that is probably better, okay, so here is how a block diagram will look, let me see if I can get this out very clearly, I am going to first draw the two decoders, okay, so this is, remember when I say decoder it is the bitwise MAP decoder, okay, so there is some, one minor problem which I did not write down clearly in the previous step, the input to the second decoder should be a permuted version of the input to the first decoder, right, so that needs to be taken care of, so when I write down the block diagram, I will put those permutations in the middle, okay, so you will see there will be an invert permutation required before you go back to the first decoder again, so all these things will show up and you will see how it works, okay, so one thing that is common to both is what, one received value which is common to both is r0, right, r0 is common to both but r0 goes directly into the first decoder and after a permutation to the second decoder, okay, so that is what happens, so how do I nicely draw that, let me think, okay, so I will draw it here, hopefully, hopefully it should be okay, I will draw it here and then I will say it is a permuter pi, it goes to second decoder, this is r0, okay, well I am not able to draw the line below, so I will raise it up a little bit and draw it, okay, so that is r0, right, both of these are inputs to the first decoder, okay, so the decoder 1 also has, well let me write down decoder 1 produces two things, right, it produces the total LLR and also produces extrinsic LLR, so the extrinsic LLR, I guess let me draw it here, okay, so it is going to come out but I will have to permute the extrinsic LLRs also, right, I produce it in one sequence, they need to be permuted before they go to the second decoder, am I right or wrong, yeah, yeah, they have to be permuted, okay, so I am going to permute here, I will write down here, this is L1xed, okay, I am going to permute it, send it to the second decoder, okay, so that happened, fine, it also produces a total LLR, what will you do for the total LLR, yeah, after whenever you have to make a decision, you will use the total LLR to make a decision, right, final decision, what is the final decision in terms of LLR, if it is positive, it is bit 0, if it is negative, it is bit 1, right, that is the ratio, so that decision you have to make based on the total LLR, okay, so one can imagine how to produce these things and then what does decoder 2 do, it uses both these informations and produces two things once again, which is L2 and then L2xed, so what do you do for L2xed now, back, but you have to do the inverse of this permutation, it goes back, L2, oh I have to show R1 and R2, okay, R1 and R2 go, R1 goes to decoder 1, I will show R2 here, I mean, technically I think there is a permutation required for R2 also, no, no permutation, R2 can just go, okay, everybody happy, any other block that I missed out, okay, that is fine, so how many iterations you want to do, you do, okay, that is fine, so but remember the bitwise MAP will produce only L2 on its own, okay, to produce, what will happen is when you run the bitwise MAP decoder with R0 and R1, it will produce only L1, to get L1xed, you have to subtract 2 by sigma square R0 from L1, okay, so that is why if you see the way they write it out, the formulas will work out like this, L1xed, people will write as L1 minus, okay, the entire vector, L1 minus R0 times 2 by sigma square, okay, you have to do this computation explicitly, subtract it and then send it out roughly, okay, yeah, minus L2xed in the previous iteration, yeah, right, L2xed, okay, so I should say previous, no, it's just getting too messy, so I'll say previous here, just like that, okay, what is previous, whatever was received in the previous iteration, for instance, the first iteration, what will this be, 0, okay, you have to set that to 0, okay, so now L2xed will have a similar computation, right, but you might want to introduce the permutations if you want, okay, so you can talk about the permutation, because the input to the decoder 2 is permuted version of R0, then R1, and then permuted version of L1xed, okay, so decoder 2 produces L2, okay, so the several ways of writing it, so for instance, one can say L2 equals decoder 2 of of what, pi of R0, comma R2, comma pi of L1xed, and then you write L2xed equals L2 minus 2 by sigma square R0 minus L1xed, okay, and then you do a pi inverse of that to get back to L2xed previous, okay, L2xed previous will be pi inverse of that guy, and then once you do that, you run your decoder 1 again to give you L1, okay, so that is all, you keep on proceeding like this, you will get the whole iteration, okay, so one can write down a step-by-step description of the same thing, which maybe I think I have started off, maybe I should complete it, okay, so I will do that right now and then we will call it a day, any questions about how this is working? Okay, so typical number of iterations, in practice you might want to do 5 to 10 or something, but in theory, if you do 20, you get very, very good performance usually, okay, just you'll see a nice big difference between the first iteration and the last iteration, huge, huge gain will be there because of the before, okay, so maybe let me write down all these things carefully one after the other, I will just mark it out so that I can copy and paste it and the other thing, and then I will, I will write it, write it out for this, oops, what happened, why did this go all the way here, it didn't select, no, okay, some weird thing has happened, it has resized it, okay, let's just try pasting it again, if it doesn't work, I'll remove it, so there you go, yeah, it's fine, this works, this works, this works, this works, oh, some line I picked up from somewhere, okay, all right, so let me start and write down what all needs to be done step by step, okay, so oops, I really don't like the way it marks, okay, we'll move this, oops, it's getting very clumsy, okay, okay, it's moving, it's moving, it's moving, it's moving, okay, so that's fine, so if you, this whole thing has to be put in a loop, right, you have so many iterations, your inputs initially are what, r0, r1, r2, and you said, you said what, you said l2 x previous to 0, 0, and then you put this whole thing in a loop, what is the loop, so maybe say 4i equals 1, 2, okay, whatever, whatever number you want here, maximum iterations, right, so the first step is what, l1 equals, l1 equals decoder 1, r0, r1, comma, l2 x previous, okay, so based on this, you compute l1 x, there's no problem, then you run l2 again, then you set l2 x previous to be pi inverse of l2, okay, so I have to compute l2 x, I'm sorry, okay, so maybe I'll do this, yeah, I can just write down the computation here, l2 minus 2 by sigma square r0 minus pi of, oh pi, no, it's pi here also, minus pi of, so all these pi's are just to keep the notation consistent, if you understand that there is a permutation, I can just drop it, okay, but just to keep it consistent, this whole thing is there, and then you finish it off, okay, so you end it, okay, then make decisions based on l2, okay, decide based on it, okay, if l2i is greater than zero, you say the bit was zero, otherwise you say the bit was four, okay, sounds very simple, if you want to analyze all this, it takes, it takes, I mean can one do density evolution is the question, okay, I said it's the same as similar to, similar to message passing, can one do density evolution, density evolution becomes hopelessly difficult, because see remember why were we able to do density evolution at the end of the day for LDPC codes, operations at the bit node and check node were not complex, okay, so by the time I wrote it down, it might have seemed like log tan hyperbolic is complex, but at the end of the day, they are simple, but the bitwise MAPD code or what it does cannot be written in a statement like that, you cannot do anything analytical to figure out relationship between output PDF and input PDF, the only way you can do it is by simulation, so people would simulate it for a long time and find the histograms for the LLRs and based on that postulated relationship between output PDF and input PDF and based on Monte Carlo simulations, you can do density evolution, all that stuff is possible, people talk about thresholds and all that, but it's not as rigorous as the, not even as rigorous as the analysis for the LDPC codes, okay, so it's a little less analysis, I haven't personally done that many simulations with turbo codes, but I would imagine yes, I would imagine that even turbo codes would not converge to the wrong code word very often, I would imagine, but I could be wrong, I mean I could be wrong, but I think it's one thing that is known about turbo codes, okay, so for instance, so error floors are a bit of a problem with turbo codes, so what happens is in, if you do, okay, there should be really a way of drawing straight lines in this program, right, it's after all a computer program, but requires a lot of skill, okay, if you do a BER versus SNR plot, okay, for a turbo code, typically what happens is, so first iteration it'll be very bad and then slowly it'll improve, for instance for rate half codes, if you have say 0 dB here, 0 dB here, for a well designed rate half code, one can expect down to 10 power minus 5, one can expect the performance at somewhere around 1 dB, okay, less than 1 dB, okay, so what happens to turbo codes this, maybe this is because of wrong convergence, okay, I don't know, what happens to turbo codes this, after 10 power minus 5, typically for turbo codes, the drop does not continue, it tends to floor as in it tends to do this, okay, so this is the error floor region, okay, so one can try to push it down, one can try to push the 10 power minus 5 down a little bit, but that ends up being more complex, so what do I mean by complexity, so I should first go back and look at complexity of this decoder, right, so let's go back and see the decoder and figure out how complex it is, remember each decoder has the complexity roughly proportional to 2 power m, okay, right, 2 power m is the number of states, okay, yeah, so something like that, so roughly number of states, so what will happen is you want to do like 10 iterations, which means the number of states has to be really low, it cannot be as much as 6 or 7, okay, so if you have 64 states, then you can't run these Witter B like algorithms 10 times, 20 times, okay, so this is too complicated, so typically in practical implementations people will take only 3 state encoders, okay, so in which case each decoder has only 8 states, okay, and then you can imagine running it some 5 or 6 times, it's not too bad in today's VLSI, one can get it into the chip, okay, so it's difficult with other like even for a DSP, these things are difficult, but from VLSI you can put it in there and produce an ASIC, it will work, it will work quite well, if you have an 8 state encoder, you can do it, okay, so that's the level of complexity, okay, so if you want to keep the complexity low, have just 8 state encoders, you will see this error floor and it will be quite prominent at 10 power minus 5, okay, when you design LDPC codes, usually you don't see this error floor at 10 power minus 5, you will have to go down all the way to at least like 10 power minus 7, 10 power minus 8, I think before you see it because in practical simulations, you cannot really go way below 10 power minus 6, why can I not go below 10 power minus 6? Yeah, to get good statistics, you'll have to run like 10 power 8, 10 power 9, and with computers today it's difficult to run those kind of things, right, so it's difficult to get, but in turbo codes you can observe the error floor with simulations, so 10 power minus 5 it starts hitting an error floor, okay, which is probably slightly bad, okay, so this something happens, something that happens, yeah, just one minute, so I think to avoid this you might want to increase the number of states, but if you increase the number of states, the code becomes a little less practical because you have to do a lot of iterations, okay, so the question was about intuition for why there's an error floor here, I really don't know, maybe at some level these are less random than the LDPC codes, you still have this sequential thing, okay, so if you actually look at the distance properties, you'll see the minimum weight code words do not become really very less, okay, they'll decrease compared to the convolutional codes, but they don't go to near zero numbers, okay, in LDPC codes it'll happen like that, okay, when you generate an LDPC code and try to find its low weight code words, you'll hardly ever find any, it's very difficult to find low weight code words in an LDPC code, okay, so people have even proven that, okay, so in turbo codes it's not too bad, you can find some low weight code words without too much trouble, which means there are quite a few of them and that maybe is the reason why this error floor actually shows, okay, but there's no more intuition than that, so you can run it and see that the error floor is short, okay, so that's one problem with turbo codes, but still it's been 10 power minus five is good enough for so many applications, right, and maybe you put an outer code outside to bring it down further, okay, one great selling point is your encoder is not that complex, the only thing you have is an interleaver, but for the LDPC code you probably have a huge Gaussian elimination type things, a big matrix sitting there, it's much more complicated than what you can imagine, okay, so there are applications because of this kind of a problem, okay, so that pretty much winds up what I wanted to do with turbo codes, okay, yes, yeah, it can be L1 also if you want you can do it, for instance the question is she's asking, so can I make a decision based on L1, okay, so for the LDPC codes you had a condition for stopping, if all the parity checks are satisfied you know it was a code word, here what will you do, right, if you have a CRC or something sitting outside, a CRC I mean something called a cyclic redundancy check, which will tell you if something is valid or not, then you can stop based on somewhere in the middle, but otherwise how do you stop somewhere in the middle with the turbo code, I mean it's difficult to know just by looking at a sequence whether it's a valid path on your trellis or not, it's not so easy, okay, but for the LDPC code it's very easy because the your operations itself are based on the check node and each check node you can easily compute the check and see if it's satisfied, so that thing is a little bit difficult in turbo codes, but in practice even for LDPC code the way your architectures are today for systems, you can't just stop after one iteration if you want, okay, the rest of the system will only expect a code word after the time it takes for 20 iterations, so you better do the 20 iterations, right, you can't design a system with varying delays, okay, usually people never do, so this notion of stopping in the middle may not be that relevant, okay, yeah, but if you want to you can decide based on L1, nobody stops, yeah, yeah, those are all generalizations, see remember I am giving you a very primitive simple form of the turbo code, people have done lots of research to have so many other constructions, in fact one of the most startling constructions for turbo like codes is what's called a repeat accumulate code, if you get a chance read a little bit more about these codes, repeat accumulate codes, it's nothing, what it does is you have a bunch of code words, you repeat, what do I mean by repeat, it's just repetition, if you have k bits here, you get say some 3 k bits, then what you do is you put a permutor here, okay, you put a permutor here and then you do accumulate, what is accumulate, accumulate is basically differential encoding if you know what I mean, so you keep doing this delay sum, just a one state thing, you do this, this is an accumulator, so at every point some of all the previous things is being sent out, okay, accumulator, you get code word, you get a code word here which will again be 3 k bits, so it's roughly a 1 by 3 code, so you get 3 k bits here, and now you can have a turbo like decoder, you have a decoder for this accumulator which is just a one state, two state bitwise MAP which is very fast, and then you go through the permutor and then do a decoder for the repetition code which is again very trivial, you go back and forth, guess how well this code does, it does very well, it's not too bad, in fact there are the possibility to make this irregular, the repetition irregular, if you make the repetition irregular and optimize, you can show this gets close to capacity in some channels, okay, so this is just a simple example to show you that you can go both ways, you can take a very simple construction, what is the moral of the story here, not doing anything great, the code is essentially very simple, what is the moral of the story here, sitting some, something is sitting there which is changing the whole thing up, right, and you have a random element and you have an implementable soft decision decoder, right, for both things and you can iterate back and forth and you get good things, right, so there are so many other constructions like this, like you said, I mean why not only two encoders, why don't you put more encoders, yeah, why don't you put it in parallel, why can't you put it in serial, why can't you put part serial, part parallel, believe me, people have done this search for 15 years and they've explored every single possibility, so I think at this point I can even declare research in turbo and LDPC codes is saturating, it's finished, moved to something else, right, so that's the kind of, but this construction is very, very interesting, it's a repeated accumulate construction, there's a lot of theoretical properties which are very, very interesting, so some various things exist, somebody is reading turbo product code, who's doing it, you're doing it, that's another construction which is motivated by this turbo decoding, which again is a very simple decoder but it ends up performing very well, okay, so all these things are, things to keep in mind, okay, stop it.