 So, next we are going to focus on some property called invariant probability vector associated with a Markov chain. And this invariant probability vector has some nice properties when my Markov chain happens to be positive recurrent. So, suppose let us say I have my Markov chain that based on my communicating class relationship splits into different classes. And let us say one of the communicating class turned out to be closed. Now, can I after I do this can I ignore all the other possible other possible classes and then just focus on this class and I assume that my Markov chain is restricted to this states in this class. I can do this right because I am always circulating within this class and I am not going to visit any other states. So, I could ignore all other states and I can just think of my Markov chain is just this state. And when I do that I can now think of my Markov chain restricted to that communicating class. I can just think of my on that class my Markov chain is irreducible right because I am it is only on this states which are already closed and communicating close communicating class. So, that is why I am now henceforth going to assume irreducible Markov chains. So, that means basically I am saying that if your Markov chains has many multiple classes and if it has any of the any of these classes happen to be close communicating class then I could just focus attention on that class and on that class my Markov chain I can just think it as an irreducible Markov chain ok. So, this is main theorem. So, whenever my Markov chain and that Markov chain so that class if it happens to be positive recurrent a closed communicating class I am not making any assumption of finite noise it could be arbitrary. So, if it is finite I know it is already positive recurrent if it is not I do not know it could be anything else, but suppose let us say it is positive recurrent I know if it is finite it is already going to be positive recurrent even if it is not let us assume it is a positive recurrent. So, in that case we have this nice property. So, if let us say I have a Markov chain that is irreducible and that is positive recurrent if and only if this is both necessary and sufficient condition if there exists a probability mass function pi on my state space such that pi is a solution of this relation pi equals to pi p and all these pi's are positive and it says that further this such a pi is going to be unique. So, it is clear. So, if my Markov if my Markov chain irreducible Markov chain happens to be positive recurrent it has this nice property that there exists this probability mass function which satisfies this relation ok. So, now it is if you see that this pi has lot of nice interpretation as we go along first let us look into some of its properties. So, what is p here? It is the transition probability matrix of my DTMC ok. So, this expression basically pi equals to pi p what is this I have written in a compact form, but what is this? So, if I this is actually right. So, this is going to be what many simultaneous equations are there right in this it is a basically a collection of simultaneous equations. So, if you are going to look at state j. So, what is pi j is pi i and on s, pi is a probability mass function on s what does that mean? Can it be? So, pi is a basically probability mass function. So, it has to be vector right it is going to give probabilities to each of my element in my state space s. So, ok the way we are going to see is this is a column vector ok sorry sorry row vector and this is another row vector that is multiplied in a matrix ok. And now if I want to find the jth element in the row vector what I have to do? I have to take the jth column in my matrix p and do a dot product of that one with my pi right. So, this is basically saying. So, this is nothing but pi and my pj. So, suppose if I interpret my pj as the jth column of my transition probability matrix then my pij is nothing but this quantity and this is for all j. So, how many simultaneous equations I have here? So, this is going to be the cardinality of s and so if I am going to treat this pi as let us say variables and want to solve this. So, how many simultaneous solutions I have and in how many variables? So, this j is coming from s right. So, this so there are actually cardinality of s such equations and there are also that many variables. So, we have as many equations as the number of variables here ok fine. Now this p here we know that is a transition probability matrix right this transition probability matrix we know this is a stochastic matrix. All the nice property of all this stochastic matrices they have an eigenvalue whose value is what? So, there will have many eigenvalues but one of the eigenvalues will have a value equals to 1. So, this is for any stochastic matrix you know what is eigenvalue? So, you can check this is just a property I am stating here. So, if I have stochastic matrix that is if its rows add up to 1 all the rows and all the elements in that matrix are positive. So, you can verify that it will have one of the eigenvalue size 1. So, if one of one is an eigenvalue what can we say about this relation in for pi? So, if lambda is an eigenvalue of let us say lambda is eigenvalue of m and let us say x is associated eigenvector. So, how can I write? So, mx equals to lambda x right using that relation can you what you can say about pi equals to pi p, eigenvector what of what? So, pi is the eigenvector of eigenvalue 1 right we can say that. So, what we is trying to seek is eigenvector associated with my transition probability matrix whose eigenvalue is 1 right and further we want this pi is to be positive because that is the requirement okay. Now, we know that pi equals to pi p I can repeat this iterations and write pi equals to p pi square and like that write p equals to pi to the power n right. So, basically saying that pi equals to pi p is basically saying that this relation should also be true for any n greater than or equals to 1 okay. Now, why is the name invariant probability that we are calling it okay? Suppose you take some pi which is the value of your initial probabilities of your Markov chain. I said that Markov chain is going to be characterized by its initial distributions and a transition probability matrix right. So, you said so if we know that suppose you take this pi equals to pi p solution whatever that pi is and set the initial distribution of your Markov chain to be that pi value okay. Then it so happens that I am going to say this such then it so happens that the probability that in the nth round your Markov chain taking value i also happens to be the same probability. So, if you start your Markov chain to have initial distribution that corresponds to the solution of this relation pi equals to pi p then it so happens that in every round probability that your Markov chain is going to particular state is going to remain the same distribution. So, that means if you start your Markov chain with this initial distribution pi in every round the same distribution continues to hold they are not saying that this is probability so okay you start your Markov chain initially. So, if I start my Markov chain let us say I am going to start it in state i with probability pi i. Now let it run let us say let it run for 10 rounds. Now the probability that you are going to see it again in state i is going to be the same pi i. And now if you are going to run it for 100 rounds and then ask what is the probability that it will be in state i that is still going to remain the same pi i. If this pi i are selected such that they are the solution of this pi. So, that is why this pi is called invariant probability distribution. So, this pi is called. So, check this this is easy to verify if you are going to start from this relation just check that if I want to do it in the next round whether this relation holds and then try to see that this holds for any possible end using this relation pi equals to pi n whatever you have next property. Suppose this pi is irreducible pi is such that pi equals to pi p if I have a solution pi which is coming out of the relation pi equals to pi p then it so happens that if in this solution pi is greater than 0 if one solution is positive it must be the case that for all and for all j. So, trivially if I am going to set this pi to be all 0 vector then the relation holds. But let us say I have one solution where one of the component is not 0. If one of the component is not 0 or positive u it must be the case that the solution is such that every component there is going to be non-zero. Why is that? So, let us take a state i and j belong to s and now I am looking into the irreducible class I know that there exists some m such that p i j of m is positive because I know i and j communicate they belong to the same class and now if this relation is true then this relation is also true for the same m. I have just to recursively use for pi and then I can write this relation now p i now let us say some j is equal to this is going to be l to s and then this is like from l I am going to state j. So, this is l to j in m number of steps. So, this is over all possible states this is my definition of this is the meaning of this relation here. Now in this summation just focus on the one term here. So, this term I know that this is going to be greater than or equals to pi i and p i j m right I am only looking at the term where l takes the value j. I know that I initial my assumption is this pi i is positive and now I already shown that because if j is another state there must be some probability like this p i j of m is going to be greater strictly greater than 0. So, now I have two terms which we have both of them are strictly positive. So, then it must be the case that this guy is also going to be positive right. So, any j any pi j should also be taking positive value. So, one element in this vector pi is non-zero or a strictly positive that means all the all the elements in this vector pi should be positive. So, these are some of the properties which I think we should be comfortable in using about this p. Now let us see why this holds ok. So, what I will do is this is both this proof inverse both sufficient and necessary conditions. So, we will only show the necessary part ok. So, what we will show is so it says what is that set if and only if there exist a probability mass function on this such that this. So, if there exist some pi which holds this relation then we are going to say that it is going to be irreducible but positive recurrent Ttmc. But suppose we will start with the case that we will start with the case where we start assuming that it is a positive recurrent Ttmc. Then we will try to show that there exist a pi which satisfies this relation and further that is going to be unique and also all the elements in that are going to be non-negative ok. So, let us we will use this notation. So, for any s greater than or equals to 0 we will use this notation that ai of s is basically probability that my Markov chain tax value i is in s step. So, this one I am simply what is this? This is a probability that my Markov chain takes state s in step in round s right that I am going to denote it as a is ok. And now what we will be looking at is these two quantities. So, what is bi in looking at? Bi is looking at the average of this probability. This probability is at for the first n steps. So, I am just looking at what is the probability of that I am going to take state i in the first step. What is the probability that I am going to take state i in the second step? What is the probability that I take state i in the n step and I take add all of them and then take the average ok. This is the same quantity, but I am looking at the delayed versions one step before instead of s step I am going to start at s minus 1. So, here so when s is equals to 1 this is ai to the power 0 no which is my whatever my initial distribution of my Markov chain right when ai of 0 here is x this is probability at x 0 equals to i that is my initial distribution right. So, this guy C i n includes my initial probabilities, but this bi n does not include because it is starting from 1 to n ok. And actually I am going to denote my instead of that. So, I am going to simply denote probability of x naught equals to i as simply ai instead of writing it as ai to the power 0 I will just write it as yeah that is my initial distribution. So, now can I write my ok. So, now these are all for a particular state now I want to write it more generally instead of i I am going to now write it simply bi of n 1 by n s equals to 1 and n and then simply write a of s. So, now these are like vectors whose ith component is this. So, when I looking at the ith component of this b n I will look only ith components of this a s and get this ok. So, similarly this is also going to be c n of n. So, are you comfortable with this notation just like instead of writing component wise I am writing them in vector form now ok. Now I am going to see that. So, this one also like instead of this is component wise when I want to write it as a vector I am going to just write it as a of s. So, this is now a vector if you look in the ith component and that that is going to be ai of s that is got tell me what is the probability that I take state s in the sth round ok. So, just to be clear what a of s is is nothing, but a of s 1 a 2 s 1 like whatever like or like I can just say that this is nothing, but this is ai of s where my i belongs to where capital S set of my states. So, now let us write to see my set of so this vector what is this vector is telling me this vector is basically telling me what is the probability that in sth round I am going to take different different states and that probabilities are captured in this vector. So, this one I can always write it as simply a this is my. So, now a is a vector for this notation which is the vector of this probability. So, I can write it as a times p of s what is p my transition probability matrix. So, what is a of s? So, a of s let us focus a particular component in that let us say ai s. So, that is basically telling what is the probability that I take state s in the sth round now how can I write this I can say that you are going to start with different probabilities and in the sth round you are going to get to some state how I am going to get that is by multiplying a with p to the power s. So, what is p to the power s is giving you? S step starting probability, but now if you are going to start from whatever your initial state 1 and then multiply it with your s step probability that should give you what is your states with what probabilities are going to reach different states in the sth round. So, just I mean just try to follow these notations then what we will do is now we are just going to plug back these relations what we have so far now in this case b of n is going to be a times 1 by n. So, I have simply plugged in this relation here and I have just pulled out a outside and this is what I have here. So, if you want to write it so this is the compact notations in terms of the vector if you want to look at the particular jth component in this this is how it is going to look like. So, this is going to look like so, this is a is a vector right. So, you can pull out one outside and what is this is going to be then look like is summation s equals to 1 to n then summation i ai and then p i of j of s. Now, what we will do is we will now look in the limit of this. So, I want to now look at limit as n tends to infinity of this quantity b j n which is nothing but u tends to infinity just lightly reorganize this I will do is summation ai summation 1 by n summation p i j of s and what is s here s is going from 1 to n and this i is our state space. The same thing here I have slightly reorganized it by bringing this summation outside and taking one by n inside okay fine. I want to interchange this limits can I so see that this ai is what these are probabilities right ai is your probability that in the u start in state i. So, in that case if I look into this this is nothing but some expectation of this 1 by n summation s equals to 1 to n of i j right what is I change this i to capital I here because now this is the random variable right and that is going to be taking the probabilities as per this distribution ai because now this is the what is this what is this ai is probability on the state i and for that time this is the value I have. So, I can now think of this as an expectation term here but now the question is now if I want to change this summation and the limit it is same as asking the question can I change this limit and this expectation here now let us come back to our things we have studied if I want to interchange this limit and expectation can I do here or if at all I can do I am I can do it here. So, is that any of the three theorems we studied to interchange limit and expectation applies here. So, now what so this is my distribution and this is like the value taken by my random variable right you have to map before you apply those results we have to see what is the random variable here what is the distribution here. So, here you can say that with probability ai this is the value taken right. So, that is why this is the expected quantity now. So, the first thing we studied in our results where we wanted to interchange expectation limitation was first one was what bounded bounded converges there is bounded converges theorem applicable here. So, what is the values of the random variable taken the value the values of the random variables are this right 1 by n summation pij of s and that is changing with different values of i is this value bounded. So, pj are our probability terms this summation here cannot be more than n, but we are already dividing it by n. So, this is going to be less than 1. So, this is going to be bounded right. So, all my random variable the values here are already bounded right with by 1 with probability 1. So, I can use my bounded convergence theorem and interchange this part ok. Now, if I know that my state is recurrent in this case I know more right I know this is I have started with assuming my state is positive recurrent what I know about this quantity if my j is my state is positive recurrent greater than 0 right and we denoted it to be let us say that some this is some quantity gamma i. So, it is greater than 0 that means it is some value let us say that is gamma i which is strictly positive ok. So, now this gamma j. So, this is gamma j here and this is independent of all the index of the summation. So, I can pull out this gamma j outside and then what is the summation of A i is going to be what is the summation of A i is going to be 1 right because this A i is for the probabilities. So, this is simply going to be gamma j in this case which I know is strictly positive by my theorem. So, what I basically did is this I did for a particular j now I can do it for all possible j j states and write limit as n tends to infinity. So, b j of n is equals to gamma. So, now notice that this b n is a vector now for the j th component I have done it now I am looking at all the components put together in a vector and that limit I am going to call it as gamma j where the j th component is this gamma j. Now, what we dealt with is b n's here what we can do is whatever the way we did for this b n's the c n's almost are the same except for one delayed version right you could repeat this argument and also show that actually limit as n goes to infinity this c n of n is also gamma. So, do verify this like you have to just again go back and write a s is equals to this format if you just plug in here you are going to get a into here. So, the whole analysis will remain the same except for the fact that this p i j s will be replaced by what? s minus 1. So, because of this you see that you will also end up with the same limit in this case because we are only looking in the asymptotic region here. Now, how does this help? So, what is the relation between now c n and b n? I know in the limit they go to the same value what is the relation between b n and c n? Is that true that b n and they are just one step delayed right b n is one step delayed version of here. So, can I write b n to be c of n and p right this is one step delayed version of this. So, I take that and add the next step multiply that is means I am going to the next step by multiplying this transition probability matrix. So, that is what I am going to get b n s. Now, whatever this gamma you got this is a vector right let us take that as my pi all I want to show is there exist a pi such that pi equals to pi p right. Let us take this as my pi set pi equals to gamma. So, now in this relation this holds for any n right let n goes to infinity on both sides what is this this is gamma and this is gamma p or I have set this gamma to be pi and basically I am showing you that there exist and now we have also guaranteed that this gamma is such that all the components in this gamma are positive. So, what we have is I have a solution pi which is so I have shown that if I start assuming that my dT m c is positive recurrent there exists a solution such that pi equals to pi p and where all the components of this pi are themselves positive. Then the next question is this pi a probability vector right that is what we said right when I stated the theorem we said that there exist a probability function on s such that pi equals to pi p now is this a probability vector why. So, they will add up to 1 how you know gamma are less than 1 because gamma is this ratio this limit right we know that each of the terms here is going to be less than 1 this is because this summation is over n terms is going to be at most n you are dividing it by n. So, this every term means for each n is going to be less than 1. So, in that sense we can assume that gamma is are less than 1 that is fine, but why it is a probability vector. So, these are the issues now how to ensure that this pi is now how to argue that this pi is indeed a probability vector then how to argue that this is indeed unique I mean when I make it a probability vector that this is going to be unique how can I do that. So, is that fine. So, if I take this gamma j's to be limit of this this is going to be unique in that sense is this gamma j's is going to be unique that is fine does that prove uniqueness no yes in a way yes right because I know that limits are always going to be unique right for a if I have a limit the limit if I have a sequence its limit is going to be unique what here basically I have a sequence here which is basically deterministic sequence here which is defined in terms of your pi j's. So, this gamma j's are limits so in that way uniqueness is coming for granted for us. Now, the question is why is this a probability vector. So, for that what we need to argue is that instead of this gamma i's the way we have we can argue that we can take their normalized versions what I mean you can take this gamma j whatever vector we have add them and then divide each of this gamma j by that quantity by that summation. So, in that way it is already probability vector right. So, instead of this gamma j if I am going to look at gamma j by summation gamma j and now look at this vector. So, gamma i let us call this and if I look at this vector this is going to be probability vector right. So, that would be the but the question is then I need this summation to be finite if this summation happens to be unbounded then this division does not make sense right. So, how to ensure that this the values of gamma i's here I have if I add them they would not be blowing up they will be still less than some quantity okay. Let us see if we can quickly argue that. So, what is bi n okay. So, let me just quickly write the steps I know that pi i is going to be greater than 0 pi i I have said to be basically gamma i and I also know that this summation i equals to 1 let us take k and then if I look at bi of k this is going to be 1 okay and then we have to show that summation i equals to 1 to k pi is just equals to n this is true for all k and now if you let k equals to infinity it will be the also case that this guy i equals to 1 to k this is going to be less than that. So, I am going to leave this for yourself to verify. So, what we want to finally argue is that this summation of this pi i's or summation of the gamma i's that we are going to deal with is going to be going to be less than or equals to 1 in fact it is not going to blow up okay. So, because of that what we can do is in this case as I said we can just normalize this gamma i's by this summation of the gamma i's and then it would going to be still be a solution to this equations right. So, in this case you just gamma i's. So, you just divide both sides gamma i and summation of gamma i I have just divided both sides by this. So, this is a going to be a new vector for me and it is still going to be a solution to this equation okay. So, because of this we have a pi which is derived from this gamma which is going to satisfy this equation pi equals to pi pi and we also said that that pi is going to be probability function here and also from this argument it follows that that is going to be unique here. So, please check this yourself that I can add the pi's which happens to be strictly less than or equals to 1 that is why I could normalize okay. So, let us stop here I have been.