 So today we are going to study some more properties of our random process, just so index could be time, right. So if index is time then the way we are interpreting random variable x and then we said my random process x can be thought of t omega. What is this? This is x of t of omega. So if you are going to treat this random process x, as I said it can be thought of function of two arguments. One argument is the index and other is the sample point. That index could be time itself in which case it is going to talk about that t random variable and the value it takes at the sample point omega. For the random process we have also defined mu x of t this is and so this is for all. So recall that this t could be just integers in which k it is uncountably sorry it is finite sorry infinite but still countable or it could be an interval itself in which it is uncountable, okay. So and we distinguish these two cases discrete random process and continuous random process based on whether my number of elements the number of indices in t is countable or uncountable. And then we had this function of yeah again there are two discreteness that is coming into picture here, right. One is in terms of the indices whether indices are taking discrete value or they are taking continuous values. Based on that we are classifying our random process as either discrete or random process. But each of the random variable in this process itself they could themselves be discrete or continuous further. So for example, so let us take an example, okay let me just first complete this this covariance is. So let me take a x where my t is equals to z that is all positive z plus let us say all 1, 2, 3 of infinity and my for t belonging to t let us say x of t is equals to 1 with probability half and let us say minus 1 with probability half. So this random process here, here by our definition it is discrete random process and further each of the random variables here themselves are discrete, okay. But now we can say that my x t is let us say Gaussian distributor with some mean t which depends on the index let us say and also variance some variance. Here for each index my distribution is what is taking continuous value outcome. So here even though it is a discrete random process but each of my random variable is continuous here because I know already my Gaussian random variable is continuous, right. So we said that this is a mean function, this is what correlation and this is called covariance. So yesterday in my class I think I made a one small error in saying that when these two random variables are the same we said it is going to be called as further auto covariance and auto correlation but that is not the correct. The thing is we are talking about one random process here, right. If you are just talking about one random process then in general we are going to call instead of correlation we will also call it as auto correlation and auto covariance. So if we are talking about only one random process and they are defined for this particular random process so this is auto correlation. The auto is coming here because we are talking about the same random process in that random process we are looking at the correlation between two random variables so that is why you can sometimes it is also called auto correlation. So we will see another notion called cross correlation and cross covariance when we have two different random processes. So that we will come to a bit later. Now for this random process I want to define one important notion called stationary a random process. So what we are saying is take any random process t and you take any n random variables. So just take n, n is telling okay it is telling what is the number of random variables you are interested and then take any t1, t2, tn this is going to tell you the indices which are those random variable interest you are interested. For any such n and for such any vector if you are going to look at the joint distribution of these n random variables taken at t1, t1, t2 up to tn and look at the joint distribution of their shifted versions that is t1 plus s, t2 plus s and tn plus s. If they have the same distribution then we are going to call a random process stationary. So what it means in a way is like what we are doing is this set of random variables joint distribution. Now we are looking at another set of random variables delayed by time s or shifted by time s. Then if this again have the same distribution then we are going to call our random process stationary. That means if my random process is shift invariant whatever let us say I am looking at, let us say I have a process where the each random variable correspond to one particular day. If you are going to look at the joint distribution of let us say day 2, day 5 and day 7 this joint distribution of these three and then you look at the joint distribution of day 3, day 6 and day 8 that is everybody just shifted by one. The joint distribution of first three will have the same joint distribution of the last three. In that case we are going to call it as stationary. So what is this basically means is in terms of CDF functions this is going to be same as, so if you just shift the time indices for all of them by the same amount then the joint distribution does not change. Let us now understand what is the meaning of this stationarity, what is the thing that this stationarity implies. Now let us take a second order, so what you mean by second order random process. So we say that it is second moment is going to be finite for all in possible indices. If you are going to take a second order random process, now let us apply this definition to n equals to 1. So remember this is for any n, this should happen. So let us take this n equals to 1 and then apply this definition. Then what it says is f of n equals to x of let us say x1, let us say x1 and then at some time t1 this should be same as this n is equals to 1 here, this should be same as f of x of n and then x1 that is t1 plus s and this should be true for all s because this should be also true for all s. What is this selling is you take the random variable at time t1 or you take any shift or at any other index. So if you have a t1 by choosing this different different s I should be able to get all other possible indices also. So what is this means then basically the CDF of all the random variables are the same, right? If this stationarity definition has to hold for every possible n and for every possible n, so this has to hold. So for every random variable has the same CDF, is this clear? Why this should be true? So if every random variable has the same CDF then what does that mean? In terms of the mean will they have the same mean? All of the x t's, so then mu of the process x t will be like something like mu x only, right? It does not depend on t. What was t? Here the indices but what we now said is the CDF is going to be the same at any indices we are going to look at. That is the meaning of these conditions because of that the mean at any of the random variable at all the indices is going to be the same, okay? So mean is constant. So remember that the mean of a general random process need not be constant. It is a function of indices, given index. But if it is a stationary process it does not depend on which index you are looking at. It is going to be the same for everybody. Now let us look at the covariance. Now let us apply this the same thing for n equals to 2 k's. What that means is now f of x2 x1 t1 x2 t2 is equals to f of x2 x1 t1 plus s and x2 t2 plus s, right? Now the claim is if you are going to look at any two pair of random variables their joint distribution is going to be the same for any pair of random variable. Is that true from this condition? Shifted by the same amount. So take random variable at t1 and t2 look at their CDF. Now what we are saying is you shift both t1 and t2 by the same amount s then their joint distribution remains the same, right? So that is what this condition told us. So because of that if you take the two random variables and shift by the same amount the distribution is going to be the same. Now because of this what does it say about my correlation? Are they going to be the same? If I am going to shift two random variables for the same amount then their expected value should also be the same, right? Why is that? Because these two have the same joint CDF by definition. The CDFs are going to be the same then the expectation is also going to be the same, right? So then what it means? If I am going to look at this random variable joint distribution and their shifted version of them it does not depend on what is the amount of the shift but it may still look at what is this two pair of random variables you should you would be looking at. Suppose you change t1, t2 to some other value that correlation may be different. But if you what it is just saying is you are looking at this t1, t2 and other two random variable which are just shifted versions of these two then the correlation is going to remain same. So suppose let us say this is t1 and t2 you shift both of them by the same amount, okay? So whatever this delta time difference is there here also the same delta time difference is there, right? It is just like both of them have been shifted by some amount s. But what we are just saying is this shift does not matter then what is matter then what should be the only thing that should be governing this covariance? The only thing that should be governing this covariance is this length of this interval, right? So r of x1 at t1 and t2 should be simply I should be able to write t1 minus t2 or if let us say t2 is going to be nothing we will just follow this convention. It only depends on the length of that interval rather than by what amount this random variables are shifted. So because of that here also the shift is the difference in this interval in this time t1 plus s and t2 plus s is again going to be t2 minus t1 and that is also the same here. So this correlation, autocorrelation function is only functions of the length of the interval of this, at the point where these indices are taken, okay fine. So if you go on I just did it for n equals to 1 and n equals to 2, right? You could go on doing this for any number of n, right? You can go and do it for n equals to 3. That means if you take 3 random variables and shift all of them by the same amount then they are, so then it should be the case that x of t1, x of t2 all the way up to x of tn should be same as expectation of x of t1 plus s, x of t2 plus s all the way up to x of tn plus s. So stationary it is a very very strong property, right? It makes the process kind of shift invariant and the again the joint statistics of any order is going to be independent of a shift. So we are going to say in the means or the values involving only single random variable as the first order statistics, like for example mean we already defined. So when we are going to look at two random variables, let us say x t1 and x t2, t1, t2 are two time indices, we are going to call this as a second order statistics. So here it is basically saying this is the like nth order statistics, right? nth order statistics here is shift invariant. So stationary it is basically saying that my nth order statistics are shift invariant and this should be true for n equals to 1, 2 all the way up to infinity. So this is a much stronger property. So what? So what is the significance of the second order process? The second order random process. So what is the second order statistics? We said that expectation of t is going to be finite for all t, right? Here you could as well choose x t1 equals to t2. In this case you are following that case and we need it to be finite. So as you see the stationary it is a much much stronger property which requires all order statistics to be kind of shift invariant. Instead of that one can look at a slightly weaker version of the stationarity called as wide sense stationarity. What wide sense stationarity asks is only shift in variance in the first and second order statistics. You do not care about higher order statistics, okay? That is it wants that this means to be shift invariance, mu at index t is going to be the same as mu at index x plus t that basically means that my mu x is constant, right? This is going to be the same for all possible indices and then it says that the correlation is again shift invariant. So that means as we already discussed this only depends on the length of this interval. So as long as you take any two random variables that has same, that are separated by the same amount then their cost relation is going to be the same. So basically wide sense stationarity is only restricting this condition to be hold only for n equals to 1 and n equals to 2 whereas stationary wanted it to hold for all n, all possible values of n, okay, fine. So because of this property that my co-autoprolation here only depends on the interval of the difference of the indices rather than the actual values of the indices itself. Often it is given in terms of a single random, often you can write it as a function which takes only one argument. So here it is given, it is taking two argument, right? But two argument is essentially translating to only difference in this two argument. That means I can as well think of it is a function of a single random variable, sorry single argument, okay. So if I define a process, if I say that I have a random process x with mean constant as mu and say that its auto-coalition function is r of x tau where tau is just a single variable then it should be by default we will take it as wide sense stationary random process, okay. So let me just make this more clear. So we will just say sometime to, if you say that x of t, t equals to r has same mean for all random variables with auto-corelation. So then we also already take it as like we will, we understand it as denoting a wide sense stationary process, okay. So note that even for the stationary case my r function here auto-corelation function is again only depends on a single random variable, right, sorry single argument here because it just depends on, but it also I for that I also need to ensure that for n greater than 2 also all these conditions hold. But if I just state only talk about the mean and auto-corelation function and further I say that the mean is a constant or the same for all random variables and that its auto-corelation function is just a function of a single variable then we understand that this is already indicating a wide sense stationary, this is just like convention, okay. No, it does not matter which t1 and t2 you are talking about, right. So for example as I said here you have t1 and t2 here and you have t1 plus and t2 s here. If you look at the correlation of these two random variables they are going to be the same, it just depend on this interval. Yes, it is going to change. So suppose now let us say you have another t3 here and now you are interested in correlation between so this was let us say called delta and let us call this as delta 2. If you are going to look at the auto-corelation between t1 and t3 it is going to be a function of delta 2 now, right. So and similarly so here also if you know this is like t3 s and the auto-corelation of t1 s and t3 s is going to be the same as r of delta 2 because what matters is only their separation where they are I mean with the same separation where they are it does not matter. No, it is not. So, okay let me say if you want to calculate auto-corelation between two random variables what you are going to tell me is their indices, right. All I need to know is the difference in this indices then I already have auto-corelation value for your two random variables. For example, as I said if you want to compute auto-corelation function of at index 5 and 10, okay it needed I all I need to tell you what is the auto-corelation function value at argument 5. So and if you have another set of random variable two pair of random variable one at time index 4 and another at time index 10 what is the value of the auto-corelation function I need to give at which argument 6, right that is the only thing matters and I am going to give you this value for all possible values, okay. So, this is for all possible values of indices. So, I just need to that is why this is a function of single random variable. One last thing I want to mention about this is in the last class. So, suppose if I say a process is stationary does it means already means it is wide sense stationary it is true, right because wide sense needs only a weaker requirement it only needs this CDFs to be invariant in shift only for n equals to 1 and 2, but whereas stationary it is for all n. Now is the other way direction is also true is wide sense stationary implies stationary not necessarily, right because stationary is a much stronger condition. But as now we will see that if I have a Gaussian random process even wide sense stationary implies stationary, okay let us see why is that. So, do you recall what I mean by a Gaussian random process? So, Gaussian random process is if a random process is such that if all its random variables are jointly Gaussian then we are going to call that as a Gaussian random process, right, okay. Now let us see, suppose let us say my I have a Gaussian random process which is wide sense stationary, okay let us say it is wide sense stationary, okay. Now I know that if it is wide sense stationary already so what are the other things know if it is a Gaussian random process it is necessary that each of the random variable at any of this indices is again Gaussian distributed, right. Now and now I am further assuming it is wide sense stationary that means each of this random Gaussian random variable should have the same mean, right. So, we know that X of t for all is Gaussian distributed, okay. And now we know that all this mu t has to be the same mu for some mu that is the thing that is coming from your wide sense stationary, right because this mu t is independent of mu. And now further by so another thing here if you look at this correlation what about the covariance is covariance is again a function of the difference in t2 minus t1 only, right. So, I can also write covariance to be t2 minus t1. So, just verify that. Now what about the covariance between two random variable X t1 and X t2 we have already said that this is nothing but covariance of t1 t2 by our definition and this is going to be covariance of t2 minus t1. What is that? So, by our wide sense stationary is a definition we already know the means are going to be the same for all of this Gaussian random variable and this covariance is going to be just depends on the difference in the length of their interval. Now let us look at any n set n random variables. Now I am interested in so to now I want to go from n equals to 1, n equals to 2 to any n and let us say n random variables at t1, t2 and tn. And this t1, t2, tn random variable now I want to look at their distribution. And their distribution is what we already know what is this distribution, right. A distribution was defined in terms of the mean vector and the covariance matrix. So, what is the mean vector here? The mean vector is simply mean vector is simply mu, mu right everywhere because each of this random variable has the same mu. Now what is the covariance matrix for this? The covariance matrix for this is covariance of t1, t2, covariance of t1 sorry t1, t1, t1, t2 and t1 and t2 and then covariance of tn, tn, covariance of, right. The joint distribution of this n random variable depends on the mean vector and the covariance matrix, right. Now if you look at each of this term in this covariance matrix, this covariance, each of this term only depends on the difference of these two terms, right, nothing else, okay. So now if I am going to, and what we know? We know how to define the joint distribution of the, this set of random variables, right. We in terms of the mu and k, we already did it in the last class. Now suppose if I shift each of this time indices by some amount, the mu vector is not going to change for the random variables at this time indices, right, because it is going to be the same for irrespective. Now what about the covariance matrix? Does it depend on this shift s? No, right or because the difference if you just take t1 plus s and to t2 plus s, the difference is only t2 minus t1 now. The s has no role in this case. So as you see that the joint distribution is now independent of the shift we are going to look at, right, and that is what, and this is true for any n. So then what does this mean? Our requirement of stationarity is satisfied, right. The PDF is going to be the same. The PDF in this case is invariant, that means also like CDF can argue that CDF is also invariant, right, of the shifts, then my stationarity property is satisfied. So for a Gaussian random process, white sense stationarity implies stationarity. And anyway, this other way is always true by default, right. But if you have a Gaussian random process, the weaker notion of white sense stationarity already implies the stationarity, okay. Okay, so just one last thing about this is, so far we have been talking about one random process, right. I mean so it may happen that you may have to deal with multiple random process. So let us say this is my one random process and this is my another random process where all the random variables are defined on the same probability space, right. So when we say the random process or we mean that all these x t's are defined on the same probability space, right. So now let us take another random variable, another random process where each of these y t's are defined on the still the same probability space. Now, when we have such things, maybe you may still want to define what is the correlation similarly and covariance in this case, right. So in this case, we are going to define r x y at point s of t to the expectation of the first random variable computed at time index s from the x process. And the second random variable is the coming from the second random process at time index t. And this is called cross correlation and similarly you can define cross covariance, okay. This is and one last thing if x y x t is only y then we call x and jointly we call x y t. So this is just like extension of notion from one random process to multiple random process when we have to deal with. So it may so happen that for example, let us say you are dealing with the stock exchanges at every day, the maybe Bombay stock exchange, how it is varying every day. And so you are going to model this as a one random process. And maybe you feel that whether as implication on the Bombay stock exchange. So then you may want to model another process which is like a weather on each of these days as another random process, okay. So then you may want to see like how they are correlated, what is the covariance between them. So each of this process could be corresponding to some aspects of something that you want to model like outcome of experiment which has which is like many, many indices. So that one experiment and if you have another experiment similarly which has many, many index indices. If you want to like understand how one has influence over the other or what is there like correlation or covariance, you want to look for all such properties.