 For this we again need to build some conditions to check and for that we need to build some notations. So, let us start that. So, let us take as usual my xn to be dTmc with transition probability matrix P by dTmc on state space S with transition probability matrix P given to you and now take a strict subset of this set i and for any i belong to this set i and n greater than equals to 1 define T. So, let me call this as how to write this I am going to just write it as some instead of T I am just using this T. This is the same as this T. So, sorry this is a subset. Now this n is going to be. So, what is this probability is going to denote? What is this probability associated with which event? This is the probability of an event where I start with state i then in the first step I remain in my tau, second step I again remain in my tau and in the nth step also I remain in my tau. No, I am not saying anything about that I am saying that you have your big state space S take some subspace in this which is my tau, this is my T you take a particular state in this i you start from that state i and now I am asking you what is the probability that you continue to remain in this subset only. That is exactly this I start with i in the first step I remain in T, second step I remain in T till the n step also I remain in T. I do not care about what happens there but in this at least for a given n I am only talking about starting from i in this class and remaining in this class for the next n rounds. So, this is nothing but y of n is nothing but probability that basically intersection of k equals to 1 to n this is correct. So, I am asking being in the same class T in the first round, second round till the nth round that is means it is the intersection of these events. Now, let us look this for a given i as a sequence in n. So, if I increase this n what is going to happen to this probability it is going to go down because I am asking it to continue to stay in the same class again in the next round also. So, and these guys are all lower bounded by 0 because these are probabilities and also upper bounded by 1. So, what can I say about this sequence? This is monotonically decreasing 1 and this is bounded. So, it should have a limit. So, let us say this will converge to some yi and I am going to denote by i all these guys y1, y2 all the way to y of whatever the number of elements in this set. So, this T could be potentially infinite it may have countably infinite number of elements in this case. But in this case this y is going to be infinite dimension vector. Now, I am going to denote by Q. So, there is a transition probability matrix on the entire space P here when I restrict it to only the states in this tau, I am going to call it as Q. Is that fine? I can always do this restriction by removing all the states which are outside my set T. P is the transition probability matrix on the entire state space. I have written right this is a transition probability matrix P and Q is restriction of this P to the set T here. Now, we have a result which says that this y here is the maximal solution of y equals to Qi with this element equals to this y is such that all the components in that are between 0 and 1. So, when I looked at the solution pi equals to pi P, what was that? That is basically I was looking for a eigen left eigen vector, right? When I am looking for a solution y equals to Qy, what is this? Now, what is this solution y? It is a right eigen vector. So, you do not know the notion of left eigen vector. So, you let us take a matrix, any matrix. So, let us say and let us say lambda is one of the eigen value. It can be this one or it could be let us say if I am going to write. So, here what I am going to treat x as a row vector here. So, this row vector is the solution of this and here what? Here also what is this x is going to be? It is going to be a column vector here. So, if in this case what we are going to call it as? We in this case we are right eigen vector and in this case we are previously there, then we are going to call it as left eigen vector associated with my lambda, eigen value lambda. So, now we are going to say that if this y which is been defined like this is going to be the maximum solution of this relationship y equals to Qy. So, you know what is what we is a maximal solution. So, the maximal solution says that if let us say x is another solution of this, the x is another vector which satisfies this relation, then it must be the case that x is going to be less than or equals to y. So, x is a vector, y is a vector, right. What I mean by x is going to be less than or equals to y, every element of this x will be dominated by the corresponding element of y. So, there could be many solution to this y equals to Qy, but this special y which we have looked into here, it is going to be the maximal solution. That means component wise it will have the largest elements compared to any other possible solution, okay. So, we will see this, but what is of interest to us is, there is another property of this. Further, in addition, it is going to be either y equals to 0, this maximal solution will be such that either it is a just a 0 vector or its supremum will be 1. So, what does y equals to 0 means here? y equals to 0 means what? So, if suppose as y, n goes to infinity, this y, right, y equals to 0 means what? When I say y equals to 0, each component in this is going to be 0. So, y equals to 0 means what? As n increases, it is my, my Markov chain is no more remaining in my t, it is going out of this set, right. So, the solution to this, this y will be such that either it is going to be just 0 for all of them, that means you start anybody from this, it is not that eventually you are going to stay in that, you are going to get out of that. Yeah, they are just escaping, like you start with any state i in this, eventually you are no more remaining in this, you are coming out of that. No, it is as n tends to infinity, it may be happening that first step, second step, third step, you may be remaining in there. So, this is an intersection, right, it must be true for all steps. Even if this, if x k equal is not belonging to t for some k, this event is going to be this null set, right, this entire intersection and then this is going to be probability of 0. So, what we are saying is it is not necessary that if y goes to 0, that means if let us take a particular component, if I start with that particular state in this, eventually I will get out of this, I will no more going to remain in this state. Or now focus on this, I have just written soup here instead of max because this tau count could be uncountably, sorry, it could be countably infinity in our case. And it is saying that in this case, there exists some i such that starting from that state in my t, you will remain in t. So, what is this saying is soup of y i for all possible i's is 1, right. That means there exists one i which is arbitrary close to 1. The value of y is arbitrary close to 1, right. That means if you start some, some state i, you will remain in that t state with almost probability 1 or arbitrary close to 1, okay. Now, so let us quickly see this and then see how we are going to utilize this result. Now, okay, now let us see this y equals to q i, what does this basically telling you. So, the q recall, this q is nothing but it is probability but just restricted to even t. Now, let us write it for 1, y i equals to what is this, this is going to be here y is what, here y is a column vector, right. If I take a particular ith component on that, what that is going to do that is going to multiply the corresponding row of this q with my column matrix y. So, that is going to be q i j y j, right. And this j coming from, is this correct? So, instead of q, I am just going to write it as p i j. These are nothing but p i j's only, right. Because they are also all the elements of p itself. So, what this is basically saying it, this y i is what according to us, y i is the probability of staying back in, you start from i and staying back in this class itself. So, what this is just saying is, you start from i, you jump to another state j which is also in state i and after that that state continues to remain in your state. So, basically you are saying, take i another state j here, you in one step you go from this to this and once you go this j step, you continue to remain there. That is y j, right, that is the probability. And now we are looking at all possibilities of going from i to j and from there you continue to remain in the same state, in the same class. Okay, that is our interpretation. Now, we have this v i i of n by our definition is, now I am going to apply my chain rule on this and I can write, okay. So, now this you can verify that, this I can write it as. So, this is probability that x 1 is going to remain in some particular state, right. Sorry, x 1 is going to remain in my class tau only. Okay, then condition that, I condition that again in the first round I have remained in my class tau only and subsequently I am asking what the probability that I go into this. So, this one I could specifically add further, this probability I can write as x 1 equals to j where j is coming from tau and then do all these things, right. I can do just instead of looking x 1 belonging to t, that probability I can write is a summation that x 1 equals to j where j is coming from that slot. Now, this is what? This is j belongs to tau, this is now d i j and now what about this? Now, I can ask this question. I know this is a Markov chain. Once I, once you tell me what is happening at state 1, I can ignore what is happening at state 0 and now I can treat this as, yes this j is already belonging to tau, you are starting from that state. Now, you are, I am asking in the next n 1 n minus 1 steps I continue to stay in my class t. So, is this right? In that case, I can write it as p j of n minus 1. What was y i f n? This is remaining in class t starting from i. Now, this is nothing but you start from j and you are going to remain in that for the next n minus n runs. So, this is what it is. So, I have completely written this. So, now, if you are going to write it in vector notation, this is going to be like q of y of n minus 1. This is component wise, if you just write it in this, this is just in vector notation. So, when we have this, we always may, we exploit this relation to write it recursively. This is going to be q n of sorry q square of y n minus 2 and I can write it as q n of y of 0 and by definition what is going to be my y of 0? 1, right? This is starting from that state and remaining in this, I mean in the same state, what is the probability that it continues to be in the same state? That is going to be 1, right? So, this is we are going to write it as q of n and here 1 and what is this one? This is actually when I wrote it as a vector where all the components are 1. So, basically we are going to say, okay, now we are almost in a familiar territory, what we did for the previous case for when we dealt with the positive, positive recurrent matrix. We also ended up with similar equations. In this case, what we did? We took let n go to infinity on both sides. When I let go infinity, this guy is going to be y and what is this guy is going to be? q into again y, but this needed us to little carefully handle interchange of limit and expectation, but you can figure that out. This is going to be almost same arguments as that in the previous case. In the previous case, what to exchange limit and expectation we had used bounded convergence theorem. So, here also you can apply the same thing and you can actually write cost y. So, what we said is whatever the y n, the limit y which is defined in this fashion is nothing but ending up to be the right eigen vector of this matrix q here, okay. So, what I basically did is in this relation after taking n tends to infinity, after n goes to infinity, we have just written y i equals to t i j j coming from t and y i. So, this is nothing but compactly written is this relation here, okay. Now, the question is why this is the maximal solution? Can there be an x vector which satisfies this relation and still have components which dominates this y, okay. So, let us see that, can we have, suppose let us say x take a vector x which is not same as y, okay and assume that x equals to q of x. Assume that x satisfies this relation. Now, as usual I will, if this is the case then I am going to replace this x by q x which is going to give me q 2 of x and I can do this for any number of times and get q n of x, okay. And now what I will do? I am going to replace this x by, so remember when I want, I am claiming this y to be the maximal solution where y is satisfying these relations, right. I am only focusing on y's which are in this satisfying this and in this region I am claiming that y is the maximal solution. So, I also have to make similar assumption then in that case let us take this x to be also in this region and also satisfies this relation, okay. Now, what I can do? Once I get q to the power n of x, right, I am going to now replace this x by its upper bound 1 vector here, right. That is an assumption I have already made, right. So, is this true that then in this case I have, so notice that there is no issue in doing this because all this q matrix has positive elements in this. So, if you just replace this x by this 1 which component wise dominates this, this inequality still holds. But now what is q n of 1? We have already shown that this is nothing but y of n, right. So, what we have basically shown is this guy x is any x we have is going to be upper bounded by this y n. Now, just let n go to infinity, even in the limit it must be the case that. So, now you just let it go to infinity and then you will have y which is also going to dominate this x, okay. So, we have this, this x is a maximal solution. So, in the interest of time we will just skip this part now that why it is the case that either it has to be 0 or this one. So, let us try to see just next how to express this result, okay. I have this result. Now, how to use this to come up with a condition for my transience, whether my state is going to be transient or not. So, here we have a condition when is it is going to be recurrent, okay. So, now what we are doing is we are basically trying to give a classification of whether my irreducible march of chain is going to be transient or recurrent. When we know it is recurrent we also further know whether how to check whether it is a positive recurrent, okay. So, all I now need to do is whether my irreducible class is if I can classify transient or recurrent then all my classification is done, right. So, let us see why this, what this. So, take a irreducible march of chain or irreducible DTMC on state space S, take any state I belonging to this S. Now, what you do is consider all the states other than this state I, okay. So, I am just saying okay, you have this state space S. Let us say there is one element I in this you just leave this and consider everything else all the other states just leaving this I and call this tau. Now, what you do is you restrict your transition probability matrix on this Q, okay. How you are just obtaining this Q? You are just by just deleting one row in your transition probability matrix P, right because you are just getting rid of one state in this T. Now, we are going to say that DTMC is going to be recurrent if and only if the solution Y equals to QY is such that I mean the solution which is in the interval 0 and 1 is going to be Y equals to 0. If the solution happens to be something other than Y equals to 0, then it is not going to be recurrent. It is transient. If this Y is going to be 0, you know it is going to be recurrent. Then you may want to check whether pi equals to pi P relation holds and if that relation holds for some pi with all positive components in that, then you know it is positive recurrent. If not, then it is going to be null recurrent. So, now let us try to see that why this result makes sense. We are trying to show maximal solution. We are saying that X cannot be maximal only Y can be maximal. You have understood this. What we are saying that if suppose Y is a specific Y we are talking about, right. The one which is the probability that if you start from a particular state, you continue to remain in the same class forever. That was this particular distribution Y. And now this X is, I am saying this X could be something else. And this X also satisfies this. It must be the case that this X will be dominated by this Y. So, that is the meaning of this maximal solution. So, you can say that I mean for a given problem here for a given equation. So, this is my basically, so my equation is of this form. Forget about Y is what is Y, the way we have defined this. So, let us consider some arbitrary Y. I am looking at a relation Y which satisfies this. There could be many values that is going to satisfy this, right. Fine, it may be unique, but if there are multiple things, we are saying that one of the solution is going to be this Y, the one we have specifically defined. And that Y will be such that it is going to component wise dominate any other solution to this relation, coming back to this. So, if this solution is such that Y goes to 0, then it is we call it as recurrent. It could be positive recurrent or null recurrent. We are not classifying that. We are only classifying whether it is transient or recurrent. So, that I am just saying, when I said this I am not saying anything. Let us just take this state space S, take a particular subset T in this. You start with the state I and then look at the probability of this remaining in the same T as n goes to infinity. That is what my meaning of Y i and Y i n basically said that I am going to remain in the same class for n rounds, okay. So, there is nothing about whether it is going to be recurrent transient, anything it could be. I am just defining this probability of staying in this class. Now, based on that properties we have on that, we are going to now basically further classify them. Yeah, so if Y naught is not equals to 0, we are saying that it cannot be any solution like it cannot be happened that when Y i is equals to not 0, it is one possible quality could be some particular Y i can take 0.1, right. But we are saying that that is not going to happen. It has to be either this has to be one of the component has to be one close to one, right. If that happens, yes transient, okay. Now, let us consider the case when that is going to, this is not happened. That means this is going to be arbitrarily close to one, right. That means what? I start from a particular state i and then I continue to remain in that itself. Does this say anything about its why it has to be in that case? So, what I did? I have removed one state from here, okay. Now, I am focusing on the remaining states, okay. And now notice that I am talking of I have started with an irreducible state here. So, now what is happening? There will be always a path for me to go and hit some state here, right, because this is an irreducible class. I should be able to communicate some state in this part also. Now, I am saying that once I hit in that case, there will be some state where I continue to get trapped in that state. Is that clear? So, okay, either y equals to 0 or y has to be this. This means what? There is one state in tau which is probability arbitrarily close to 1, okay. So, let us say that is some state j. I do not know which is that, let us say that is some state j. Now, that state j is here and because this is a irreducible class, there is a positive probability of me going from this state to this state j. And once I hit state j, I will never come out of this class. That means I am never going to come back to i. That means basically I have escaped from i, right. I am not going to come back. That means state i is what? Transient. If one state is transient, everything else is transient. And what is this saying? It is saying that, okay, fine. If at all you go from i to some other state in this class, you getting trapped in that forever is going to happen is 0 probability. What is this y equals to 0 means? y equals to 0 means the probability that you getting trapped in the same class tau is going to be 0, right. That is the meaning. Is that clear?