 Welcome back. Today, we will discuss sums of random variables. Last class, we discussed maximum and minimum. So, given n random variables, next went through x n. We figured out how to, given their joint series, we figured out how to get the distribution of the minimum and maximum. Today, we will discuss sums, sum over sum of x i's. So, first of all, so let us say that, let us just take 2 random variables. Let us say x y random variables on omega f p. So, you have 2 random variables x and y on omega f p. Just for you to remember the picture, you have omega and your x comma y is a map, is a map from omega to r 2. And given any borel set on r 2, the pre measures are f measurable, we know that. Now, the question is, if I take, so if I define z of omega equal to x of omega equal plus y of omega. So, for every omega x of omega comma y of omega is a pair of real numbers, it is a point in r 2. I am interested in x plus y, which means, I am looking at the random variable z equal to x plus y, which means for every omega, I have z of omega equal to x of omega plus y of omega. And I want to ask, what are the statistical properties of z? In particular, given the probability law or joint CDF, joint CDF of x and y on this, on r 2, I want to figure out, what the CDF of z is, that is the question at hand. However, before you go that far, you have to convince yourself that z is in factor random variable. What do I mean by that? z must be a measurable function, z must be an f measurable function. We know that x and y are f measurable functions, but we do not know if z is always a measurable function. It turns out this requires a proof. So, the proof is actually fairly simple. It exploits the fact that, it uses a fact from real analysis. It uses the fact that, the rationals are a dense countable dense subset of r. So, let me write this down. So, z is indeed a random variable on omega f. In other words, I have to show that. So, I have to show that, sets of this form, let us say I have omega such that z of omega greater than little z. So, I have to show that, these sets are f measurable or equivalently less than or equal to it does not matter. If I show this is an f measurable set, z is indeed a random variable. I only know that, x and y are random variables. So, what I can do is, I can write using the fact that, rational numbers are a dense countable subset of r. Dense by the way, do you know what a dense subset means? So, a dense subset of, let us say a dense subset of a metric space is a subset such that, the limit points of that set is the whole space itself. So, the set, if you collect the set, if you take the rationals, the set of all limit points of the rationals is, in fact, the whole real line. Every real number can be written as a limit of a sequence of rationals. So, which means that, rationals are dense in r. So, using this fact, we can write as follows. We can write, this is union q in q such that, omega for which x bigger than q and y bigger than z minus q, which is the same as saying, union q in q, x greater than q, union y greater than z minus q. And the union is over the whole thing. So, whenever, what I am saying here is, whenever x plus y is greater than z, strictly greater than z, there always exist some rational number q. See, there always exist a real number q such that, there always exist some real number such that, x greater than that real number and y greater than z minus that real number. But, since in the vicinity of any real number, I can find a rational number. I can also find a rational number, which satisfies that. Now, why am I so worried about a rational number? Because, I want to induce a countable union. So, I want to write this as a countable union of these guys. Now, if you look at this, so I have just written this as well, I think I mess this up. I think this should be an intersection, is it not? So, this be an intersection? Yes, this should be an intersection. So, this is f measurable, why? x is a random variable that is f measurable. So, because y is a random variable. So, this intersection that is f measurable and I have a countable union of f measurable sets must be f measurable. So, sum of two random variables is a random variable under no further conditions. It is always a random variable. Similarly, you can show that product of two random variables is a random variable, similar argument. Anyway, so we are not discussing products now, we are just discussing sums. Is this argument clear? It uses the fact that, the rational are dense in R. So, even just looking at this equation, you can easily figure out the CDF of Z. This expression itself gives you a, in principle a way to figure out the CDF of Z. Because, you can write probability of that, which will be the 1 minus the CDF is equal to some probability of the countable union of a bunch of this is a complementary series after all. So, you can use some probability properties to eventually figure out this. In principle, there is no problem. However, so in the special case, we will in the special cases, cases where x and y are discrete and in the case where x and y is jointly continuous, we will derive more specific formula. We have the expression simplifies a lot more. So, for everybody with me. So, if you are looking at, so you are basically looking at the probability that, let us say this is the probability you want. This is Z is equal to x plus y. So, if you are looking at, let us say you are given a CDF. So, let us say this is my x y plane and my CDF is something on this. See, joint CDF is given on this, joint CDF x and y is given. So, it is start of minus infinity. It will increase and go to, it is at minus infinity start of it is 0 and go to 1. It will be some three dimensional surface, three dimensional figure on this. Now, what I want is the probability that Z is less than or equal to little Z. So, I draw the line x plus y is equal to little Z and I am looking at the total probability mass sitting on this side of this line below the line. So, that is what you have to figure out. So, you have to look at the joint CDF and figure out how much of the probability lies below this line. If you in general this may the joint law may consist of some masses, may consist of some densities, it may consist of some crazy stuff like canter function or whatever. It can have all sorts of things lying here. You have to somehow add up those probabilities. In the case when x and y are both discrete, the job simplifies a lot more. That is because of x and y are both discrete, you will have only, it will take only a countable set of values. And for any given Z, you add up all the masses sitting on this side of the line. In that case that should be easy. So, the discrete case is very easy. And in the jointly continuous case is also very easy, because you have a joint PDF, which you can integrate over the set. So, those two cases we will cover. So, let us say that. So, this is what you want. Let us say first assume. So, this is what you want. Assume x and y are discrete. And if x and y are discrete, I know that my joint PMF is the complete specification of these two random variables with given joint PMF P x comma y. Then probability that. So, if I want f z of little z is equal to, I will simply sum over all the x's and y's, such that x plus y's are equal to z. So, I will simply write this as. So, I sum over all x and y's, such that x plus y is less than or equal to z P x comma y of x comma y. So, this I can again write as sum over P x y of x comma wait. So, let me. So, this is joint CDF. So, actually let me write this. This is easier to get just a minute. So, for the discrete case, I should just do this, because z after all will also be discrete, because x takes only countable values, y takes only countable values, x plus y will take only countable values. So, I can actually go for the PMF. So, I can instead of writing down the CDF, let me just do this. So, here it will be this will become x plus y equal to z and this will become. So, there is nothing wrong with what I was writing earlier, but this is simpler. So, this I will sum over all x. So, I am summing over all x y such that x plus y equal to z. So, I may just put y equal to z minus x and sum this PMF. And if you know the PMF of z, you will you can get the PDF P or CDF. So, you are summing x out. So, this will not depend on x, it will only depend on z, you get your PMF. And so, this is in general, this is the general formula. In particular, if x and y are independent random variables, then I can simplify this further. I can write P z of little z is equal to sum over all x. So, this will become P. So, this will product out into the marginal PMF. So, you will get P x of little x times P y of z minus x. This is only true of x and y are independent. So, this is like a discrete convolution of the two PMF. You would have seen the convolutions in your signals and systems. So, this is a. So, this is denoted by P x star P y. So, discrete convolution of the individual PMF or P y convolution P x. It is the same thing, F convolution G is same as G convolution F. So, if you are given two independent random variables and you want to find discrete random variables. And if you want to find the PMF of the sum, you will just go ahead and convolve the individual PMF. But only if they are independent, otherwise you have to do this. This is the more general formula. So far, let us do an example. Let x 1 and x 2 be Poisson random variables independent Poisson random variables with parameters lambda 1 and lambda 2. And let us say you write z is equal to x 1 plus x 2. Then P z, z is given by P x 1 convolution P x 2 of z. You know these marginal PMF because then given to be Poisson. So, P x i of x will be equal to e power minus lambda i lambda i. Let me call it k, because it is just a integer value random variable P x i of k over k factorial. And this is for k greater than or equal to 0 actually k in integers. Now, let me try and do this convolution. So, P z of z sum over k equals 0 to z e power minus lambda 1 lambda 1 power k by k factorial times e power minus lambda 2 lambda 2 power z minus k divided by z minus k z minus k factorial. Now, this looks ugly as of now, but you can simplify it. So, what you will do? So, this looks invitingly like z choose k. So, what shall I if I put a z factorial here and divide by z factorial, I will get a nice term. So, let me write this as e power minus lambda 1 plus lambda 2. So, these guys I knock off I get it out over z factorial summation k equals 0 to z, z choose k lambda 1 power k lambda 2 power z minus k. Did I leave out anything? I think I am fine. So, that is sub binomial sum is it not? So, this looks like lambda 1 plus lambda 2 power z. This is the binomial expansion of. So, z is all. So, I am just looking at z in integers. So, this will become e power minus lambda 1 plus lambda 2 lambda 1 plus lambda 2 power z over factorial z, which means you get another person. So, if you take 2 person independent person random variables, the sum is also a person random variable whose parameter is lambda 1 plus lambda 2. In fact, you can show that this holds for even some of n independent person random variables with parameters lambda i. The sum will be also pose on with parameter some of those lambdas. So, this holds for z in 0 1 2 dot. These are all for this is distribution over integers non negative integers correct. This is an element of z plus. So, let me just write let me just write 0 1 2 dot dot dot whole numbers. Any questions? Next we will consider the case where x and y are jointly continuous. So, the discrete case we are done we are going to go consider the jointly continuous case. These 2 special cases you get nice expressions. Otherwise you have no other option. So, if you do not have if you have some crazy random variables x y with some very bizarre CDF, you have to simply look at the probability mass below this line. There is no simple expression for it, but in the case when x and y are discrete or x and y are jointly continuous, you will get some nice formula. And these are the cases of maximum applicability in engineering. So, these expressions are useful to know and useful in practice as well. So, let us say x and y. So, we finish the k assume now assume next that x and y are jointly continuous. If they jointly continuous I have a joint density joint pdf with joint pdf f x y which is given to you. Now, I want the density of. So, I actually want let me say I start out by saying I just want the CDF of z right. What you will see is that z will also actually be a continuous random variable. So, not only so we can actually get its pdf as well right. So, this is what we want. So, we will write f z of little z. So, this is probability of big z less than or equal to little z. This is simply. So, you know that for any. So, you know that the probability of any borel set is given by the integral of the joint pdf. So, I am now seeking out. So, this is what I want the probability of this set is what I want right. The set that lies below the line and that is a borel set you can show. So, the probability of lying in that borel set will be the integral over that set of little f x y right. So, it will just be integral x y such that x plus y is not equal to z f x y of x y d y d x or d x d y does not matter right. This you agree this is rate on actually right. So, now this integral. So, after also if you write this as a double integral on r 2 this will become integral minus infinity integral minus infinity z minus x f x y of x y d y d x. So, what I am doing is I am first integrating y out. So, I am doing. So, if you look at this figure I am integrating from y from minus infinity to z minus x right. I am integrating along the y first and then integrating from x from minus infinity to infinity right that is what this is right this integral is. So, that is it actually I mean if this is all there is. So, you know the joint density you integrate it this is an integral you can do right and what you will be left with. So, if you do the first integral you will be left with something that is a function of z and x and then x will also get out when you integrate the second time then you will be left with some function of z. So, this is all there is, but it turns out this can be simplified a little if you put. So, conceptually it is over, but if you put let me do this right if you put t equal to x plus y then this simplifies a little bit and gives you. So, minus infinity z well let me just minus infinity infinity integral minus infinity z. So, I am just now I have put I have gotten rid of z minus x is I have put z minus x is equal to well I have put x I have put t equal to x plus y which means my y is equal to t minus x right y is equal to t minus x. So, I will have f x comma y of x comma t minus x right y is equal to t minus x and I have d y is simply d t now right. So, I will have d x d t is that fine d t d x I think. So, I am integrating this the x integral is not it yeah. So, d t d x correct this is just a substitution in the integral good. So, now this guy is non negative all right. So, I can in fact there is no harm in inter changing the order of integration that is always valid if you have a non negative argument. So, this you can write as integral minus infinity z integral minus infinity this is also these two are equal integral f x comma y of x comma t minus x this guy now I think. So, now my variable is t is not it. So, when my so I have gotten it of y right. So, y was going from minus infinity to z minus x now my t is going from minus infinity to z I think that make sense right. So, I have gotten it of y. So, this is now I have d x d t I have just flip the two integrals which I can do because my integrand is non negative. So, why did I do that if you look at this integral. So, if you look at this bit. So, that is getting rid of x right I am integrating x out. So, what remains will be a function of t right. So, I will have something that looks like a density right. So, this is this will be some non negative measurable function of t and I am saying that the c d f can be written as integral minus infinity to z some function of t d t which means this guy must be the density of z right. So, this guy must be the density of z which automatically proves that z must be a continuous random variable correct. So, which means if you give me the joint density I can easily find the density of z by evaluating this one integral right. I do not have to the integral outside simply getting the c d f from the p d f right. So, I can write thus. So, thus f z of little z is integral minus infinity f x comma y of x comma little z minus x d x. So, you give me the joint density I will find the density of the sum z this is the formula for that and it is always a continuous random variable if x and y are jointly continuous z is always continuous random variable. And then you say in particular if x and y are independent you will get f z of little z is equal to integral minus infinity infinity I know that almost everywhere my density will joint density will product out into product of the marginal densities. So, I will get f x of x f y of z minus x d x this is a convolution also right this is a convolution of. So, this is written as. So, you can write this as f z is equal to f x convolution f y right. So, if x and y are independent jointly continuous random variables then you get this formula. Now, you know that if I see x and y are independent and marginally continuous it is automatically true that they are. So, jointly continuous right. So, actually. So, if x and y are independent random variables with densities x and y right you can always use this expression is this clear. So, if you have good practice in convolution you can do a lot of examples like this right you take if x is some exponential and y is some independent Gaussian you can convolve exponential and Gaussian and to say what the sum will look like right it will not be pretty right. But, some cases you get nice answers right. So, once such example you can show as if you take x as a Gaussian random variable with parameter mu 1 and sigma 1 and x y is an independent Gaussian parameter mu 1 and mu 2 and sigma 2 then the sum will have parameters some will also be Gaussian will have and it will have parameters mu 1 plus mu 2 and the sigma will be square root of sigma 1 square plus sigma 2 square. You can show it explicitly by doing this convolution it will not be pretty in the beginning, but you can figure it out right you can convolve and figure it out. So, that is one thing you can do as a home work see eventually that is not how we will do it. So, you know that convolution is not it is not cool right it is very ugly right. So, normally what do people do when you want to convolve two things you take your favorite transform Laplace or Fourier multiply the transforms right and invert back. So, that is what we will do eventually, but for now we have not gone there. So, you can convolve for now right. So, you so if you try the Gaussian example with different parameters it will probably take you about half an hour to 40 minutes to actually do the integration. But of course, if you use transforms it is a lot easier right which we will get to later. So, let me do shall I do an example I will do an example. So, let us say x 1 and x 2 are independent exponential random variables with parameters lambda 1. So, let us say mu 1 and mu 2 and z is equal to x 1 plus x 2. Now, f z of little z is given by integral 0 to. So, this p d f is known right. So, f x i of little x will be mu i e power minus mu i x right. So, the individual c d f is known p d f is known to you. So, and for this you will take integral 0 to z because that is where both that is where both densities are non 0. You will get mu 1 e power minus mu 1 x mu 2 e power minus mu 2 z minus x d x. So, which is equal to mu 1 mu 2 e power minus mu 2 z right mu 2 z times integral 0 to z e power mu 2 minus mu 1 times x d x fine just doing the convolution. So, in this case if. So, now you have 2 cases right it may be that mu 1 and mu 2 are equal right in this case this whole thing will become e power 0 which is 1. So, the integral will simply be integral 1 d x will be z right. So, in that you will get mu square z e power minus mu z right if mu 1 mu 2 are equal to mu and if they are different this you have to integrate. So, you get 2 different forms. So, this will be equal to mu mu 1 mu 2 by mu 2 minus mu 1 e power minus mu 1 z minus e power minus mu 2 z all that this is true if mu 1 is not equal to mu 2 and this will be equal to mu square z e power minus mu z if mu 1 equal to mu 2 equal to some mu. So, if you had the parameters were the same then this will be the pdf will look like z e power minus z with mu equal to 1 right. So, this particular this is called Erlang Erlang of second order second order Erlang Erlang 2. So, if you sum 2 independent and identically distributed exponential random variables you get a what is known as an Erlang 2 distribution. And if you do this n times you will get an Erlang n distribution that will look like mu power n z power n minus 1 e power minus mu z I think there is an n factorial or n minus 1 factorial in the denominator. And the case when the parameters are different you get something right it is not some well known this is not some well known distribution this is a class this is a one of the classical well study distributions. So, Erlang has lots of applications in calculating switch blocking probabilities in telephony you know call blocking probabilities in telephony good. So, any questions on this. So, the other example. So, the homework example I was saying was if x 1 has Gaussian with parameters mu and mu 1 and sigma 1 square if we wrote down the expression for this right e power minus x minus mu whole square over 2 sigma square and so on. So, and x 2 independent with parameters n mu 2 sigma 2 square and their independent then x 1 plus x 2 is also Gaussian with parameters mu 1 plus mu 2 and sigma 1 square plus sigma 2 square. Now, I am writing squares here no yeah. So, your sigma will be the square root of sigma 1 square plus square square. So, I am writing the what is what we will interpret as the variance not the standard deviation good. So, this is this you will show actually now yeah if you try and show it now it will be a very ugly exercise, but you should probably do it just to convince yourself that when you do it in a easier way later with transforms you will have a much easier time. So, here this is a very fundamental property of the Gaussian distribution this is one of the properties it is one of the two properties that makes the Gaussian distribution in some sense the queen of all these continuous random variables right. So, if you add any number of independent Gaussian random variables you get another Gaussian random variables and the parameters also simply add. So, if even if you add n of them independent random variables you will get another Gaussian. So, it is a stable distribution under addition. So, in the discrete world you saw Poisson was stable also right. So, the continuous world this is one example of a some stable distribution this is a stable under summation. So, that is one of the reasons why this Gaussian is. So, important the other reason is central limit theorem which means that which says that if you keep adding a lot of independent random variables which are not even Gaussian you will the some will look approximately like a Gaussian. So, which this is a central limit theorem is something you will do towards the very end of the class even in proper detail before that we have a lot of ground to cover. So, this I have not proven because it is not because it is difficult because it is messy and we will do it in a more elegant way later any questions. So, if there are no more questions I will stop here.