 So, today we are going to start a fresh topic called Markov chains and this is this is continuing our study of random processes. So, when we have already defined what is a random process and in that so far we said that in general stochastic processes are complex to describe, if you want to describe them in generality because you need to give finite dimensional distributions of this process right, which involves you to give joint distributions of n number of random variables in that process where n could be any number right. So, a complete characterization of a stochastic process is complicated, but we saw that one simplest process we mostly dealt with was IID process independently and identically distributed and that process was very easy to describe. All you needed to describe is just give one distribution that is the common distribution that is of the random variables are following. So, and from that you could characterize all the finite dimensional distributions, but like IID process could be restrictive right, it is not necessary like every time whatever you are doing like when you have defined something as a process it is not necessary that every time what happens today is going to be totally independent of what happened the previous day or what is going to happen tomorrow is going to be totally independent of what happened today. So, IID process basically try to model something which did not have any memory in that like I can say that whatever is going to happen tomorrow is going to be independent of what happened today and what happened today is actually totally independent of what happened yesterday that means they are independent in a way also they are not carrying memory. So, what happened previously is not going to affect me or the things that is going to happen in the future, but often in reality we will see that what happened today can have some impact on tomorrow right. It is suppose let us say the weather is going to remain let us say very hot for few days you would not expect whether to suddenly go and start raining the next day like some kind of effect you expect that what happened today is going to impact tomorrow. So, then we want to bring in some aspect of memory in our stochastic process and to do that Markov chains come to our help and we are going to describe what we mean by Markov chains today. And we will be focusing on discrete time Markov chains if for all ok, so this is the definition of discrete time Markov chain. So, we are going to take a process xn which takes value in a countable set S what does that mean each of this random variable xn may outcome lie in a countable set and this countable set the elements in this countable set we are going to call 1, 2, 3 like that we index them since this is countable we can index like that. Now, this process this stochastic process we are going to call it as discrete time Markov chain if you are going to take the values I0, I1 all the way any values that belong in my set S and another J and it so happens that given that you the x0 random variable took value I0 and all the way up to xn random variable took value I that has happened already and now you ask the question in the n plus 1th round I take the value I if this probability is just equal to this that is it only depends on what is the current one xn equals to I what is the state you took and the question that you are asking that it goes to state value J in the next round is independent of all the value it took previously. So, we are going to take the value taken by this random variables as the states of this Markov chain. So, every x1, x2, x3 they are all taking values right and we are saying that each one of them are going to take a value from this countable set S and we are going to call this at a set of states and the states are basically we can index them 1, 2, 3 like that continue the call. Now, what we are saying is given any let say my process has evolved and it has taken these possible states that is my x0 it took state I0 and the x1 took state I1 and xn minus 1 took state In minus 1 and xn state took I let say this has been observed and now I want to ask the question what is the probability that in the next state sorry in the next round I am going to take the value or I am going to take the state I if this depends only on the current state you are now currently at and independent of the what has happened so far then this process is going to be called as dTmc. Now, what we are going to call this quantity let say I am in round n I am going to call it as present let say the process has evolved or I am in some particular day nth day let us call that as present. So, I am saying okay I have been observed that it has been observed that on day n or today I have taken the state I so this is like my present and previously till yesterday my states are I0 on all the way up to In minus 1 this I am going to call it as past and now this is my tomorrow and this I am going to call it as my future. So, what is basically this probability saying this future given all the observation till today only depends on my today it does not depend on my past okay. So, if this quantity holds then we are going to call this as a dTmc. So, you see that this process has some kind of memory in it right what happens tomorrow in the future it going to depend at least on today it is not like it is even independent of that. So, had it been if this process has been an IID process what this would have been yeah it will be simply probability of xn plus 1 equals to j it would have been not even dependent on my present right forget about the past. So, when this property holds at least there is kind of this kind of memory in this then we are going to call this as Markov chain. So, here discrete time should be obvious because we are indexing time that are discrete valued I am only counting 1, 2, 3 as like a time index this time indexes are not continuous okay. So, most of the courses in this course we will only focus on this kind of discrete time Markov chain and if time permits later we will touch upon discrete time sorry continuous time Markov chains okay where my time could be continuous okay. So, instead of this n to be taking only some integer value it could be taking any value in some interval. So, what it basically what this is saying is my process x0, x1 all the way up to xn minus 1 is independent of x1 plus 1 given xn. So, as I said this is my past this is my future and this is my current status. So, my past is going to be independent sorry my future is going to be independent of the past given my current. So, because of this nature if I want to understand what is going to happening in the future I only need to know where I am currently now that is the only thing that is going to impact my future. So, I do not really need to know what happened to me so far right I can ignore that because that is not going to affect what have what is going to happening how the things are going to happen in the future only depends on where I am now okay. So, x0, x1 all the way up to xn minus 1 this is my past I am going to just call this entire thing as past this all the past is going to be independent of my future given this. So, the process is discussed because you are indexing them with discrete time right and my process is discrete valued because I am saying that this is a it is taking value in a countable set that is why this is also going to be discrete here okay. So, okay now let us take xn is an IID process okay and such that this guy xn equals to 1 and we are just going to say okay this is going to probability p and where xn belongs to xn belongs to 0, 1 for all n. So, let us take the sequence of random variable where each xn is taking value to be either 0 or 1 and it is going to take value 1 with probability p. So, you can think of these is a sequence of coin tosses where whenever it comes heads you are going to take it as 1 and whenever it comes tails you can take it as 0 and probability of head coming up is p and this is obviously a Markov chain because it is we are telling it to be high ID. So, it is already going to satisfy my Markov property. Now let us define another process for all n okay yeah x i is for IID. So, if my process is IID I am saying this xn is satisfying this Markov property yeah. So, what? So, then we are saying what we want we want this property to hold right. So, if my process is IID xn plus j is anyway this like and this is going to be simply x of n plus 1 equals to j whether your condition or unconditioned does not matter right for an IID process. So, what I am saying? So, for an IID process you need to check this condition holds right for an IID process probability that xn plus 1 equals to j given this it is going to be simply xn plus 1 equals to j but this itself is again this right. How? If it is an IID process if it because xn plus 1 equals to j is independent of xn equals to i this is the meaning of independence right. So, that is why this Markov property is trivially satisfied for an IID process okay. Now let us take this yn which is nothing but summation of n this random variable. Now we want to ask the question whether this yn satisfies a discrete time Markov chain. So, first of all is this yn is going to be IID? So, okay let us see I can write it as so if I return it like I could also write it as yn minus 1 plus xn right. I just separate out the first n minus 1 by my definition that sum is yn minus 1 plus x. So, yn depends on previous yn minus 1 right. So, this feature is not independent of my current state. So, this yn plus 1 minus 1 and yn they are no more independent right. So, this sequence is definitely not independent. So, then let us see whether this satisfies Markov property. Okay, first let us first count what is the value taken by yn? What is the value taken by yn? So, each of this xs can take value 0 or 1 right. So, what is the value taken by yn? It can take value all the way up to 0 to n. So, and for n sufficiently large it can take all possible values of n right. So, as I let n go to infinity I mean each of these ys can take. So, I could that is why I could set yn belongs to s I can say where s is simply 0 1 2 all the way up to n. So, instead of just saying yn belongs to 0 to n like that I am just writing 1 value where yn is taking all the values from for all n for all n. So, now let us try to understand what is this probability. So, let us say I want to take some for some ij for some ij. So, I want to take this set of right it say I have some i1 where i n minus 1 and i that belongs to s and I will take another j belongs to s and I want to compute this that y0 equals to i0 I want to compute this. So, let us see what is this value. So, I have written yn as yn minus 1 plus xn, but I could then this is also say 2 that yn equals to yn plus n plus 1 ok. Now, let us try to understand. So, what is now let us say this is just a relation. So, now let us try to compute this what is this value is going to be suppose I know that if yn is going to be some value i what are the possible values of yn plus 1 yeah either it is going to remain at i or it can take the value i plus 1 right. Now, let us see if this guy j equals to i what is the value of this probability 1 minus p right because that means tail must have happened because of that that xn plus 1 is 0 ok. So, that must be 1 minus p and if this j equals to i plus 1 it should be p and for any other value of j which does not belong to i i plus 1 what could be the value any other value just. So, I have been told yn equals to i let us take let us take j is equals to anything other than these two values just let us take j to be i plus 2 can take can j can yn plus 1 be i plus 2 no right can it be i minus 1 no right and can it be any value other than i and i plus 1. So, then this is going to be 0. So, this value to compute this what I needed to know what is the current value of yn nothing else I needed to know right. So, because of that can I write so this so this guys are immaterial to compute this I only need to know what happened at I only need to know what is the value at yn. So, then is this discrete time Markov chain yn yes right because this exactly satisfied by condition here ok fine. Now, what you basically showed is in this yn plus 1 was basically yn minus 1 given yn. So, yn plus 1 earlier I had said that this x 1 plus 1 was independent of all this, but you can take one at a time in this this yn plus 1 was independent of yn minus 1 as long as you know yn nothing behind it are useful to me ok provided I know what is the value of yn. So, we are saying that yn plus 1 and yn minus 1 independent, but this independence is conditional condition what condition that I know what is the value of yn, but if you just try to, but this does not imply that they are independent unconditionally. So, if you just ask are yn plus 1 are independent of yn minus 1 is this true without conditioning on yn no right because if I know yn minus 1 takes value I what could be the possible values of yn plus 1. So, suppose let us say this value this realization took value I what are the values of yn plus 1 it can remain at I or jump by 1 value or jump by 2 values at most right. So, the in that way if I know this I already know about this. So, they are no more independent, but if I know what is yn then they are independent of each other ok. So, as of now we said that Markov the way we define Markov property it kinds of implied that Markov property is just talking about one step memory that the next step is conditionally independent of everything past given the present one. What about further steps in the future? So, it let us say if I know something tomorrow today I know tomorrow is going to be independent of my past given my Markov property, but does this also imply that given today my day after tomorrow is also going to be independent of past ok. Is that true? I mean do you feel that by Markov property that should imply if tomorrow is going to be independent of my past given today is day after tomorrow is also going to be independent of my today given my past yes ok let us formally verify that and that is true like all my future steps however further they are they are going to be independent of my past given my current state. So, we are going to state this formally. So, let us first try to digest this notations because we are going to use this notation again and again now in the study of Markov chains. So, let us take n n when I write n it will be usually meaning my current index my present and when I write this n 1 n 2 which are indexes in the future. So, suppose n equals to my current day let us say it is the 10th day n 1 could be 12th day n 2 could be 15th day like that they are the indexes in the future and then I would write this I 0 I 1 I n minus 1 till I to denote these are the states taken in the past that is on x 0 took the value state I 0 x 1 took state I 1 x n minus 1 took state n minus 1 sorry I n minus 1 and on my current day nth I took the state I and then these are my futures in my future date in my next n 1 n 1 indexed random variable I am going to take state J 1 and all the way let me also call this as n m and in that n mth date I am going to take state J m. So, this is the notation. So, at time 0 I took state I 0 at time n minus 1 I took state I n minus 1 in the current time I took state I and in the future this n 1 which is larger than this n I am going into state J 1 and after that I am going into state J 2 in n 2 which is larger than n 1 like that all the way up to n m. So, what I am looking is from my current time I am looking at m steps in the future I mean these steps are not immediate m step they could be at some point in the future that could be n 1 n 2 n 2 like if I am in day 10 it could be 12th 15th 16th like that if I am looking at 3 points in the future. Now, it is all it is saying is this past again does not affect this feature probabilities given my current state. So, far my DTMC by the way I defined it just said that it did not worry about all this it only worried about n which was n plus 1. Now, you I am saying if this n 1 could be larger than n sorry larger than n plus 1 and also if you look at all the things that you are going to look into the future that is going to be independent of this. So, if your stochastic process is DTMC this is also guaranteed. Okay, let us see why this is the case. So, I am doing it going to do it for but I said I am already in 2 days that is what I am saying right by why what I am saying today I know and I am asking about tomorrow. If you want to assume that today also I do not know then go back to previous day maybe you know about previous day and ask for today and the same definition applies and whatever you want also like fine if you do not know today you know about yesterday you can go and do the same thing here if you remove this part no it does not hold. This is true if you know what is today what is happened then we are saying future is independent of the past now okay. So, suppose we are saying this is not there then it should be n minus 1 this will be my latest information right this will hold we will find the that this is like what I have like he said instead of arch I only know till the previous day let us say but I have to assume that that state is I here in that case this is about future right this is still going to be independent of my previous days all the one from n minus 2 to n minus 3 till 0. So, basically what it is you have some information till some point and after that you want to ask about future now whatever information you have to know about the future you can take the latest information in that and throw away all the others that is what our DTM Markov chain is telling okay it need not be like till today and everything what. So, generally I have to see that what their information I have only the latest matters affects the future and fast I can just throw okay. So, what I am going to do is I am just going to do it for a special case and that will give you the idea how this should be done for rest of the for a how that can be generalized to this case. So, I want to know do it for this case x n plus 2 equals to j given x naught equals to I naught x n minus 1 equals to this one and x n equals to I. So, now I am basically asking tomorrow I am asking for day after tomorrow okay whether probability that my what happens day after tomorrow is conditionately independent of what happened so far given my today. So, how to show this how to show that this is equals to this okay. So, then if I can show this you could also extend that and say that okay even for the future this can be applied now how to do this. So, let us take this I am going to write it as is this correct. So, this is like conditional probability right now what I have done is I have added this x n plus 1 but now I am allowing it to take all possible values x n plus 1 can take value k and now I am allowing this k to take all possible values and then summing over it then is it this is equals to this right. So, what I am basically doing let us say I am in this where this is x n plus x n state and this is x n plus 2 okay. So, basically here it was asking okay x n I thought now I want to go to x n plus 2 equals to j this is today and this is day after tomorrow. So, before you reach day after tomorrow you have to go through tomorrow right let us say this is tomorrow and now tomorrow can be any of the possible states in s right. So, I am allowing it okay you go through tomorrow through whatever possible state and then if I do this then. So, basically I am going from here to here and I am allowing all possible roots that is different what possible values of k so that is why this probability should be equals to this okay. So, now let us now let us so after writing this let us go back to our initial things we studied how to split this probabilities. I am going to split this probability as x n plus 2 into okay maybe first this x n plus 1 equals to j k given x naught. So, I am just simplifying this into probability that x n plus 2 equals to j given x naught i naught all the way up to x n minus 1 equals to i n minus 1 x n equals to i and x n plus 1 equals to k is this correct the way I split the probabilities. So, I am just doing probability that a and b is nothing but probability of a into probability of b given a. So, I am just taking this as event a and this as event b okay. Okay now so now let us try to apply our Markov property to each one of these terms. Now I am asking probability that I am going to take state k tomorrow given I am in state i today and I am going to take state j day after tomorrow given that tomorrow I am in state k. By Markov property this should be probability that x n plus 2 equals to k given x n plus 1 equals to k x what is j. So, now what is this telling you? Okay now you go back to this probability what is saying is you from today you started in state i and you went to state k tomorrow and then tomorrow you are already in state k and from there you go to state j day after tomorrow. If that is correct like if I sum it over all these possibilities. So, this is basically asking this question right I start from i today I go to some state k tomorrow and from there further I go to state j day after tomorrow and now again I am looking into all possible roots. So, this should be exactly equal to x n plus 1 equals to j given x sorry 2 equals to j given x plus 1 equals to r x n equals to right equals to r. So, what it is saying is that what happens day after tomorrow only depends on today okay and I can forget what happens till yesterday okay fine.