 Okay, so this is, well, this is lecture 28, okay, it looks like I forgot about how to do how to get the L first time around, okay, so I think you have to select journal first and then do, okay, good, alright, so let's remind ourselves what we were seeing last time we were looking at soft message passing decoding, right? Okay, so this is with respect to BPSK over AWGN, okay, so the main idea was the message or the number that is passed from bit node to a check node is going to be LLR of that bit node at every iteration, the best estimate of the LLR is that bit node, the message that comes from the check node back to the bit node is again some LLR for that particular bit node and once one makes the independence assumption that at any given iteration all these LLRs are calculated with respect to completely independent mutually disjoint received values, so you can deal with it like in the way I wrote down, so you get two simple update equations, one for the bit node and one for the check node, so let me write down how those update equations were, so the bit to check messages were something like this, okay, so for the first iteration what do you do? Okay, iteration one you simply send yi, I think y was my notation for the received values, okay, so received values are y1 through ym, these are received LLRs, okay, received LLRs, okay, so r1 through rn is my received values, yi will be 2 by sigma square times ri, right, so that is how you find the received LLRs and in iteration one you simply send, okay, so I am not going to write down the update equation for iteration one, it is a little bit different but it is just that the notation will be different and I have to go through and do the same thing, so in iteration one step pay what do you do? Send yi, right, so bit i sends yi on all the edges, okay and step b the check node will process the yi's, right, so there is some small difference there but from iteration two onwards we had a comfortable common notation, in fact you can extend that to iteration one also, it is not a big deal but let us just write it for these iterations, what would be the, what was the notation I said? U, where did I put L? I put L on top, right, UL denotes a bit to check message, bit to check message and I had VL denoting the check to bit message, right and well to do it properly I should index all the edges and then I should subscript U with that edge number, right, that will give you the proper way of doing it but then I will have to figure out the edges that are connected to bits, just you can do it properly consistently with notation but just the notation gets more and more messy, you have to have those kind of things, so I bypassed all that by just concentrating on one bit node and telling you what the law it obeyed, just by indexing those edges from one to i, okay, so if you look at a bit node of degree i, okay, say maybe yi was what it received from the, no, no, no, what did I, what degree did I take? I must have had some reasonable notation for this, d, okay, I took degree d and the ith bit it's got degree d, then I said it could have received V1 or maybe V2, right, V2 L-1 through, did I say this, Vd L-1 in the previous iteration, from the check with check nodes, okay, so and it is going to send U1 L to its first neighbor, okay, so this is the comfortable notation I used with respect to one bit node, okay, so what was the relationship now? U1 L equals yi plus V2 L-1 plus so on till Vd L-1, okay, so likewise one can do U2 L which would be yi plus V1 L-1 then you would jump to V3, okay, so you don't use V2, you don't use that information that came from that, this was the bit to check message in iteration L, okay, for the check to bit message it's a little bit more complicated just that operations are not too complicated but it involves one nonlinear function, it gets a little bit more messy, okay, so how does that work? If we look at check node j, the j check node, let's say degree E, okay, and again we'll do a similar notation just for simplicity, I'll say I'm interested in the message it sends out in iteration L and it would have received U2 L through UE L in the L iteration, okay, so in the step A of the L iteration then what do you do for V1 L? You have to do two things, okay, the sign of V1 L is computed in one way and the magnitude of V1 L is computed in one way, okay, so the magnitude of V1 L, what do you do for that? You use this formula which is f of what? Summation, okay, so let me just, let me not write sigma, so just kind of mix, messes up things, f of U2 L plus f of U3 L plus so on till f of UE L, am I right? Is this consistent with the way f was defined? Okay, what was f of x now? Log tan hyperbolic mod x by 2, right, and remember for f of x, I'm going to only take absolute value, okay, that's understood, I'm not writing it down, maybe there's an absolute value outside just to stress that it's only the magnitude I'm interested in, okay, it's fine and what about the sign? Is the sign of U2 L product of the signs of the individual messages, okay, is that fine? Okay, so these are the two updates and so the sign, you can also think of it as, you can also evolve it with bits if you want, okay, 0 representing plus 1 and 1 representing minus 1, you can take XORs and do that, that's also fine, all right, so hopefully this is quite clear. There was a questioning, there was a question that was asked to me about conditioning and all that, so for instance, when I derived those formulas, I never said anything about conditioning, right, simply said probabilities of individual bits, then the actual usage we seem to be conditioning, but those things work out okay, so you can work it out carefully and see that it works out okay, those conditionings are not a big deal, okay, go back and think about the way we wrote it down, okay, all right, so that's the update equation and what we're going to do now in this beginning, okay, hopefully I think we'll finish it in this class is to do density evolution for this iterative process, okay, under the IID assumption, again to do it rigorously, I have to go to the tree ensemble and then start from the leaf nodes and then come up all the way up, that's the proper way of doing it, that would be nice and rigorous and proper, but we will do the hand waving IID assumption and simply do the computation very quickly and you can always go back to the tree ensemble and we'll get the same expression, okay, so I'll do it for, I'm sorry, I'll give it after class, okay, so one more thing to worry about is regular and irregular, right, for regular we had, we have the same neighborhoods, so irregular we have different neighborhoods and you have to average over all of them, all of those things still carry over, I'm doing density evolution, the only thing that has changed is the messages have now become real numbers as opposed to just being bits in the previous case, but all those other assumptions that are necessary to make density evolution possible will still hold, for instance I'll have to show that the all-zero codeword is good enough, okay, so the all-zero codeword assumption and the tree assumption neighborhood being a tree, all those things will have to hold good, only then this really makes sense, okay, so all those things carry over into this also and they're all true and I think to do with the messages being real numbers for instance, okay, so I'm going to look at the update equation for one node, okay, and fix the degree, then figure out how to track the density through that update equation and then we'll do the averaging over a particular edge, okay, so I'll do the same thing like we did before, you remember for irregular how did we do it, fixed an edge and we said what's the probability that the edge connects to a degree D bit node on the left, okay, that was what, rose lambda D, okay, so then but then you do the update only for fixing the degree to be D, given that the degree is D and then you average over the whole probability to get your average behavior over all codes in the ensemble, okay, so we'll do the same thing here, so I can conveniently start with the fixed degree, particularly say a degree D bit node and see how to track the density, okay, so one slightly different thing here is now instead of tracking just one probability, I have to track the entire PDF, okay, so that will be different, okay, so I need notation for now random variables, okay, so so far we've only dealt with bits and messages are bits, what was received is bits, everything is bits, so I can get around with without worrying about random variables and define making the distinction between the random variable and the value it takes, okay, so far I never had to do that but now since the error even I am interested in is bit error but the underlying random variables are actually continuous random variables, I will have to make that distinction, okay, so I will begin by doing that, I will say capital Yi is a random variable that denotes Yi, okay, okay, so Yi is what the LLR of the ith bit received from the channel, okay, so that's my small Yi, it's actually a random, it's a random value, right, if you do different transmissions, you'll get different Yi, you won't get the same Yi and it'll follow a distribution, that distribution is that random variable is denoted capital Yi, okay, so what will be the distribution of Yi is for instance if you have to worry about this guy what will it be? What will be the PDF? Well, I have to say two Gaussians but I am going to make my all zero codeword assumption, okay, remember that, okay, so I didn't explicitly say this but I'll assume all zero codeword, okay, so all the assumptions of the previous density evolution hold, there's nothing that changes from there, so once I assume all zero codeword what is the message that I am sending? Plus one, okay, so it's only one Gaussian, the PDF of RA is what? The received value RA is normal with mean one and variance sigma square, okay, how do I get Yi I know? I multiply by 2 by sigma square, okay, so what will be the distribution of Yi I know? Normal with mean what? 2 by sigma square, variance 4 by sigma square, do you agree, right, when I scale a normal random variable the mean will scale, the variance will scale by the square, okay, so you do 2 by sigma square whole square multiplied by sigma square again and you'll get 4 by sigma square, okay, so that's what is the PDF for Yi, okay, so this is the easiest part, okay, this is a very nice and well behaved PDF if you plot it how will it look nice, well shaped Gaussian depending on sigma the width will be different and all that, okay, so it's a very nice thing, okay, no problem, so why we know very well, okay, so now I'll say at a particular iteration L I know the PDF of a random message, of a message on a random edge averaged over, averaged over the entire ensemble of graphs, I'll say I know that, okay, once I know that I'll try to find what the PDF of the message in the next iteration, once I have a formula for that I can keep doing it over and over again, okay, so for that again let's use some notation, let's say UL is a random variable that denotes UL, okay, so I've not put a subscript here, why because I've averaged over all the codes in the ensemble, okay, so every message is supposed to follow this PDF, okay and I know it's good enough, it will closely concentrate around the average, so I know every PDF will be very close to this, it's not a problem, okay, for U1 I know what this PDF is, for the iteration 1 what is this PDF, same as that normal PDF, okay that I know, okay, I have to find for U2, okay, so I'll first begin by trying to find the PDF for V1, okay, I have to first try to find V1, right, so I have to find the PDF for capital V1 which is a random variable that denotes VL, okay, so again I've averaged over all the graphs, okay, so how do I do this, I have to use my check node update equation, so what is happening to the random variables now, if I have to write VL, how is it done, the magnitude of VL goes as f of, f of, okay, in a degree E check node, okay, for a degree E check node the magnitude of VL is going to be, I'll say U1L plus f of U2L, so on till f of what U, E minus 1, right, this is what happens to the magnitude, okay, and what happens to the sign, is that a question, what happens to the sign, so what are these U1's, U2's now, yeah, they are all IID distributed according to what, the same UL, okay, they are all independent and identically distributed and they follow the same PDF of UL, okay, so that's my assumption, it's based on the so many other things that I saw on the graph, okay, so once I do that, I can do this, okay, so now it's possible to find, okay, without too much worries, you can easily, well not easily, at least theoretically you know how to find the PDF of the left hand side node, how do you find the PDF for magnitude of VL, okay, so you know the PDF for U1L and let it go through this transformation by f, okay, and you know it's a monotonic nice transformation, so you can apply one of your standard Jacobian formulas, you'll get the PDF for f of U1L, then how do you do the summation, you convolve all those PDFs and then you run it through f again, so you'll have to do all of those, okay, and you have to do it numerically, it's tough to get an analytical formula for that, okay, so you do all that, you get the PDF for the magnitude of VL, how do you get the PMF for the sign of VL, it's only a PMF, right, okay, so you have to find the PMF for sign of U1L, what is the PMF for that, how do you find the probability here for the sign of U1L, and you know the PDF for U1L, right, how do you find the probability that it's less than 0 or greater than 0, how do you find the probability that the random variable is less than 0, integrate from minus infinity to 0 that PDF, right, so you do that, you'll get the individual probabilities and they're all independent, so you can use the 1 minus 1 minus type formula, right, so it's easy to find that, so suppose I know, suppose P is probability that U1L is less than 0, right, I can easily find probability that sign of VL will be 1, okay, what will that be, 1 minus 1 minus 2P raised to the power e minus 1 divided by 2, okay, so it's easy, this is the same as the BSC here, there's no big difference, okay, so this part is only discrete, okay, can be easily computed, okay, but for the top part above, you need to do some numerical numerical methods have to be used, okay, but even then the method simply involves transformation of random variables and convolution, convolution one can efficiently implement as FFT if you want, so there are a lot of simplifications possible, but one can do that, okay, so I am not going to write down the whole thing, one can possibly write it in notation, for instance the way this is written is, after doing all of this, you can find the PDF of VL, okay, and can be obtained by what, you do this transformation, okay, what is this capital F, okay, not CDF, sorry, yeah, so transformation of random variables, transformation of the PDF when the random variable goes through F, okay, by FFX, what is FFX, this is the log tan hyperbolic, so when random variable goes through that, your PDF will go through a transformation, that transformation, I mean, I am denoting it as capital F, okay, once I do that, what should I do, I should convolve this capital F of this with itself how many times, e minus 1 times, okay, so I will denote that as the convolution as e minus 1, okay, so all of them are the same because I have assumed u1 to ue minus 1 or iid according to the distribution of u, okay, so that is my, that is guaranteed by my tree assumption and all that, okay, so it is all independent, I have averaged over all the realizations, so it becomes independent and identically distributed, okay, so I can denote it by this and after the convolution what should I do, again I do an F, okay, so at this point you should ask me one part of it is discrete, one part of it is continuous, how can you talk about the whole thing in one go, right, so it is theoretically possible if you read probability the right way, it is possible to do all these things with the CDF itself, right, you do not have to come to the PDF, I wrote down the PDF but I can do the whole thing with the CDF and if I do it with the CDF it does not matter if partially it is discrete or continuous or whole thing, everything will work out properly, okay, so imagine that this capital F is actually going through the CDF and maybe coming back, okay, some such thing is happening, okay, okay, so if at all you happen to read very advanced probability theory you will see that it is also possible to do such things with PDF itself, okay, but it involves some strange notions which are probably not necessary but if you read the book for instance modern coding theory you will see they use some strange notation for those kind of PDFs, it is okay, we do not have to worry about it, think of this capital F as a transformation of CDFs, you do the transformation of CDF and then maybe you do some operation to it to come back to the PDF, right, but the update happens in two ways, you deal with the sign differently and the magnitude differently, okay, so that is the thing, okay, so I do not want to go into the details and show you how this works, I will illustrate, I have implemented this in MATLAB, so I will maybe illustrate this and show you how this transformation looks, it will look a little bit ugly but at least theoretically hopefully the steps are clear, okay, what you have to do is reasonably clear, right, so you know what the transformation of random variable will be, you know how to do convolution, you know how to do, how to do this computation for the sign and then you can put both these two together to get the PDF of UV, that also you have to do, right, one more step you have to do, given the magnitude and the sign you have to play around with it carefully and get the final total PDF, once you get that you know the PDF of VL, okay, so going from VL to UL plus 1 is quite easy, it is not very difficult, okay, the reason is the, for a degree D check, degree D bit note it is only summation, all you have to do is only convolution, okay, but anyway we are not done here, this is not the only thing, right, yeah, you should average this over all the row ease, okay, so let us do that also, so if you do that the PDF of VL will be obtained, okay, I am going to say equality, but remember this is all, this one needs to be careful when I say these things, but anyway I will say, finally if you put everything together summation over E, row E, capital F, okay, convolution E minus 1, what, capital F of F UL, okay, given F UL I can find F VL using this fancy formula, okay, very implicit formula which assumes you know you are doing a lot of hard work with this capital F, once you do that capital F everything else will nicely follow, okay, so I will denote this whole thing as row, okay, so whatever happens when you go from check note to bit note is this fancy function row, you know, I mean this lot of abuse of terminology here, you should know that when I put the argument of row as a function, then I am doing this complicated transformation with PDFs, okay, otherwise what did I think of row as, row of X was my simple distribution polynomial, right, row of X is summation row, okay, sister simplification, okay, so that is how the density update through the check note happens, okay, so now if I look at what happens in the L plus 1 iteration from bit note to check note, let me again look at a degree D bit note, okay, what happens here is very, very easy, right, what is happening, UL, UL plus 1 is going to be equal to what? Yi plus what? V1L plus V2L plus so on till V D minus 1L, okay, and what are these VIs, they are all IID distributed according to FVL, okay, that assumption again holds, yes, yeah, I know, I mean, but I am going to say capital F actually denotes the whole thing, you are taking care to bring in the sign also later, you have to do that also, you are right, the point is when you do this transformation only the magnitude is taken care of, okay, then you have to take care of the sign, you factor that in and I am calling that whole thing as capital F, so maybe you want to call this say F1 or something if you want, just using F, okay, so you have to do that adjustment for the sign, okay, you have to do the magnitude differently, then sign differently, then put both together, right, there will be, this F1 will be slightly different, okay, maybe we will keep it, so this F1 takes care of sign also, okay, if you want to be very rigorous about it, you can do that, okay, so maybe I will do F1 here, here, yeah, it is fine, equality is fine, no problem, if you are happy with such equalities then it is fine, yes, okay, so for here it is very easy, right, so how will I do FUL plus 1, right, I have to take FYI which I know is simply normal distribution, convolve it with something which is actually convolution of FVL D minus 1 times, I do that, try to get this, okay, so convolution, again see when I write down this function, when you implement it numerically, you cannot do continuous variable, so what will you do, you will sample it, okay, so it will actually be a discrete time convolution, okay, and you can nicely do it and you make sure you choose only two power n type samples and then you can do FFT very easily, it will work very fast, okay, it will be much faster than this painful convolution, so one can implement these things very easily, okay, so you sample it at the right points, okay, no, no, no, I should be careful here, let me be careful, I need averaging over D, right, so I forgot about the averaging over D, so let me write that down a little bit carefully here, I am sorry, okay, so I need to sum over D lambda D to get my proper averaging, so that would be FUL plus 1, sorry, okay, so this is after averaging, so these two are after averaging and you notice this FYI convolution occurs in each term, so you can pull that out if you want, okay, so once you pull it out, you will get FYI convolved with, convolved with something, okay, so that's something I'll call lambda FVL, so these two are nice ways of describing how this works, okay, so these are just expressions, anybody can write down these expressions, so the difficulty is can you actually get it implemented, okay, so the only way you learn density evolution is not by listening to me write down these expressions and explaining about them, what's the only way you learn density evolution, implement it and matter, we have to code it, okay, unless you code it, unless you understand all the intricacies, this will never get through, you will never really understand how this is working out, okay, so what about probability of error, okay, so well and well and good, I am tracking this PDF of U, okay, so I can find FU, I can find FUL plus 1 so on, what's my probability of error, yeah, it's integral from minus infinity to 0, FUL, that will be your probability of error, okay, every time you can do it, either that or in fact you can even do some more complicated things if you want, but that's a good enough measure for probability of error, okay, so that will be fine enough, okay, is that clear, I'm sorry, yeah, yeah, I'm going to come to that, okay, so if you want finally at the end of the end of everything, I can write density evolution as what, how can I write density evolution, FUL plus 1, L plus 1 is some function, I'll call it DE, okay, so density evolution, there have been too many Fs, so I'll call it DE and it's parameterized by lambda, rho and lambda and rho, right, and what's my input to this density evolution process, FY, right, remember FY and FY is parameterized by just one parameter, which is sigma, right, sigma or sigma square, sigma square would be very correct, right, so that's all, so the proper way of writing it down, I'll say FY, FY is actually normal with mean 2 by sigma square and variance 4 by sigma square, okay, so the input actually is only sigma, right, given sigma, lambda and rho, you can run density evolution and figure out probability of error at, well, I should be careful about FUL, I'm sorry, sorry, FUL, okay, so I can run this any way I want, right, so what's probability of error after iteration L, okay, so in iteration L, probability of error will be probability that ULS less than 0, okay, so this, all this works because I assume the all 0 code word, right, if I do not assume all 0 code word, probability of error even is not very easy to define, okay, so you can compute it by doing this minus 0, okay, some D whatever, okay, so now you again make the argument about what can happen to this probability of error, okay, so probability of error in iteration L, I'm sorry, okay, so as you increase L, if you start with a particular sigma and lambda and rho, as you increase L, what can happen to probability of error, again it's a bounded function and it is hopefully monotonically decreasing, okay, so it has to converge, only question is will it converge to 0 or to a non-zero value, okay, again you can define the threshold sigma star as what, yeah, the supremum over, supremum over sigma such that what, probability of error well converges to 0, okay, there will be a maximum sigma at which probability of error will tend to 0, that is my threshold, okay, so again you have to prove several properties, you have to prove monotonicity for this density evolution with respect to sigma, all those things can be proven, okay, so I have not proved all of the analytical properties, but once you prove all those things one can nicely define sigma star as the threshold, okay, so I showed you simulations for the binary symmetric channel with Gallagher A, okay, so you write down, you do the simulations for even soft decision decoding, you will see even though for calculating the threshold, you tend so many things to infinity, you tend block length to infinity, you tend L to infinity, but threshold is a very, very reasonable measure for reasonably high block lengths, when when you go to block length of 1000 with the 36 regular code, the behavior will be very, very close to threshold, okay, so you will see that waterfall behavior close to threshold in your simulations, okay, so all those things will be true, so the threshold is a very, very important parameter, okay, in fact as you will see since I have tended, since I have tended what is happening, I was told if I do this, it will come back again, but it is not going to come back, okay, maybe I should do something, apparently you can also do this, okay, anyway we will give up on that, so you will see sigma star is a function of, function of what? Lambda and rho, okay, so once you fix lambda and rho, you will have a particular sigma star, okay, and then you can do the same optimization once again, over all possible lambda and rho with a particular rate, you find that lambda and rho which has the maximum sigma star, okay which can, which has threshold to be maximum value possible or you can do the reverse optimization which is for a particular threshold, what is the maximum rate that is possible, okay, so all these optimizations one can do to optimize over lambda and rho, okay, then the comparison will be how close will the threshold be to the capacity of BPSK over it, which can also be looked at, okay, so again, I mean these things are easy to write down and when you read them, they will all make sense and they will be very wonderful to listen to and all that, but only way this will sink in is if you try to implement, okay, and it is not too hard, you can do this in MATLAB without too many complexities, so one can, one can implement this, okay, so the next thing I want to do is I want to show you, what do I want to show you, I want to show you first, let me show you an illustration of density evolution, I am going to try to open up MATLAB, but will that do something nasty to my thing or never knows, okay, so if it becomes too painful, I will close the recording, but in case it does not, let us see what happens, okay, so how can I run programs and all that, I cannot type easily, this is going to be a pain and then do that, so will you do this open, okay, I am sorry, okay, so you can see, I mean there is lots of MATLAB, if I do not know if you are not familiar with MATLAB, this may not make too much sense to you and this is in fact a badly written program, so do not think you have to write density evolution properly, so basically I am doing it for a regular code with left degree 3 and right degree 27, so what is the rate, left degree 3 and right degree 27, so what is the rate, 8 by 9, 8 by 9 is the answer, so I am doing it for, so you see there are some FFT length and all that, I am doing it in a very badly calculated way, so you will see there are a lot of plots, so what I am plotting is, I am doing something called output, basically PV is the, PV, this PV that you see here is the, so it says title says the bit to check, so V is bit to check, okay, so I have done the opposite here, I am sorry, I think I did U for the bit to check in the notation and again I am also plotting the check to bit F, so PV is the PDF of the bit to check message, P is the PDF of the bit to check message, P out is actually for the output, I am also doing that, it is not too crucial, okay, so let us see, I am going to run this, my sigma is 0.4, channel noise variance is 0.4, so which is quite low, so let us run this, what should I do, okay, okay, there you go, so we got one plot which is, can you see the plot, so it says output is basically, and do not worry about what output is, it is basically some PDF, the overall PDF with iteration 0 and the bit at a rate is 0.00621, now I have to press enter or something on the main window, how should I do that, go to the next one, now it did not go, how do I press enter here, it is paused, I have to press any key, so I do not know how to press any key, double click is not working, so I think I should do, I should do this, no, space, okay, then, yeah, it is gone, good, I got it, so let me move this here so that I can, can you see it, okay, so this is the PDF of the bit to check message in iteration 0, you see it is the same as the previous plot, right, it is the, basically this is what, this is the Gaussian with mean 2 by sigma square, what is 2 by sigma square of sigma is 0.4, yeah, so you see the mean is around 12. something which is correct, okay, so I have not made any mistake, if you want you can check with the variance also, so that is the, that is basically the input PDF, okay, so I am going to press enter, oh my goodness, oh okay, I know what I have to do, I have to run something else now, how do I run something else, or how do I load, how do I run something else, again open, but this will need some input, how do I give inputs, keyboard, no, it will not work, so if I run this, what will happen, oh, I will do this, wait, wait, wait, let me see, maybe what will happen if I run this, okay, so I have to put the, okay, so let me try this, okay, maybe I will try this, oh my goodness, no, no, no, this will not work, this will not work, this will not work, you know what I can do, I can do one more thing, let me try this, I think if this works, it will be much simpler, let us do a paste, beautiful, now I do enter, how do I do enter, enter, there was some enter here, wow, that is very nice, so but that does not mean this will work, no, okay, so I have to just cut and paste from here, oops, major experiment in, how do I do close brackets, semicolon, okay, this is wonderful, okay, okay, okay, opening it where, yeah, that will be better, why am I wasting so much time doing this, it is nice to see this happen, but looks like I cannot see my mouse, okay, there it is, oh, it is busy, stop the recorder, stop the recorder,