 So, we will continue our discussion with first passage and first return probabilities, how to compute them and so on. So, I just recall quickly that your f i j 0 will be 0, because they cannot be any transition in 0 time. Then f i j 1 that means the first time j is reached from i in one step. So, that will simply be p i j. So, the first step transition probabilities are your first time transition probabilities also. And then for the for f i j n we wrote down this formula through which we can recursively compute the f i j. So, you see f i j n will be p i j n minus sigma k varying from 1 to n f i j k into p j j n minus k. And I had explained last time also that you see we are because p i j n simply gives you the probability of transitioning from i to j in n steps. And it does not and you in between you could number of times have visited state j. So, all that is included here because we were computing all possible path from i to j. So, that may include going to j and then staying over j and so on. And then going from j to somewhere and then coming back to j and all that. So, now we will want to subtract those probabilities from p i j n to computer f i j n. And so this says that here this is the first time in k steps you transition from i to j for the first time and then from j to j in n minus k. So, this again will include you know going from j and coming back to j, but in between also you can visit j any number of times. So, this p j j n minus k and here this is you have already reached j in k steps and then all this is happening because you are computing the transition probabilities up to n. So, therefore, we have to subtract. So, these particular paths here subtracting that means from i to j suppose you have this and then you go back here you come back to j then again you go back and come back to j and so on. So, n minus k steps you will be doing this, but in k steps you reach j for the first time from i. So, we have to subtract the possibility of traversing all such paths from this one to be able to compute the first passage probabilities. And so that is what we are doing. So, I thought that I will revisit this and explain it much better. Now, we can solve these iteratively from these equations given the initial conditions here. So, but then we need the p i j n are available for all n to compute this. So, just look at this example again the job assignment problem we have p square is this and your f 1. So, here we are using the notation that now just as p is the matrix of one step transition probabilities p square is the matrix of two step transition probabilities. We will also denote the first time first passage transition probabilities as by the matrix f 1 then it will be f 2 and f 3 and so on. So, if you want to compute for example, f 1 2 1 what am I doing here this is f 2 1 2. So, you want to compute the probability that you transition from 1 to 2 in two steps. So, this will be by the formula p 1 2 2 minus f 1 2 1 p 2 2 see here the formula here. So, therefore, you reach that means the one step in one step you go from 1 to 2 and then you have to stay with 2 because there is only one step here. And therefore, if you compute this you get 3 by 16 right. So, now what you see is that you have to. So, therefore, now using this formula you can compute you get the matrix f 2. So, you have to compute all two step transition probabilities before you can compute the three step first passage transition probabilities and for computing f 3 you would need f 1 f 2 and p 3 also. That means p 1 p p 2 p 3 f 1 f 2 you will need to compute your f 3 elements of f 3. And so this is lot of work and so I will give you now alternative method of computing these first passage probabilities without wanting to also without wanting the higher powers of p just the first transition matrix would be enough. So, the alternate way now the alternate way is see what the way we explain this is now look at it as one step. So, f i j n I am writing as sigma k varying from 1 to n p i k. So, you one step you transition right in the beginning you are starting from i. So, suppose you go to k. So, that probability is p i k and then from k you can the first step transition probability I mean first passage transition probability k to j in n minus 1. So, that means k is varying from 1 to n. So, of all possible states you starting from i and you may go to some other k and then from k you want to go j for the first time. So, here you will require n minus 1. So, you see if you have already computed your f k j n minus 1 then you can compute your f i j n by using the first step transition probabilities. And so this is a liter and more efficient way of computing your f i j n recursively. So, the argument is because I have to move out from i to some this thing and so k should not be j obviously, because I have to visit j for the first time. So, but then that means I will definitely visit some other state from i to k in one step since I want to visit j from i in n steps. So, in the first step I will definitely go somewhere which is state which is different from j and then I need f k j to n minus 1. So, from k to j I should visit j from k for the first time in n minus 1 steps. So, this is a liter way of computing your first passage probabilities. And now here we just need the one step transition matrix p and of course, we are computing these f k j all of these before I compute f i j n. And so here again just doing the same exercise. So, if you want to compute f 2 3 2. So, then this will be simply p 2 1. So, remember your k does not have to be. So, your k cannot be 3. So, k can take the value 1 and 2. So, therefore, f 2 1 f p 2 1 f 1 3 plus p 2 2 f 2 3 this is all. And so by taking the values because this is only p 1 3. So, this is p 2 1 p 1 3 plus p 2 2 p 2 3 and you can look up looking up the values in the matrix p you get this. Now, look at f 3 1 2 f 3 1 2. So, here your j is 1. So, your k can be 2 and 3. So, this will be p 3 2 into f 2 1 plus p 3 3 into f 3 1. And what is happening is that what is p 3 2? p 3 2 is 1 by 4, but then what is f 2 1? f 2 1 is p 2 1. So, p 2 1 is 0. So, this is 0 then p 3 3 is 0 into f 3 1. So, this is 0. So, but then you see remember I have not drawn the diagram here, but if you remember the path 3 2 1 does not exist because you did not have the arrow from 2 to 1 in the job transition this matrix because this is 0. So, you do not have the arc from 2 to 1. So, therefore, this path does not exist. So, anyway this you cannot transition from 3 to 1 in 2 steps first time first time from 3 to you cannot reach from 3 to 1 for the first time in 2 steps in 2 time periods. So, this is definitely a better way of computing your f i j n. So, now, once you know this then let us talk about the mean first passage times. So, remember n i j we have denoted as the number of transitions that you require for going from i to j for the first time. So, then the mean first passage time will be expectation of the random variable n i j which you will write as n from 1 to infinity n f i j n. And when you put i equal to j then it will be m i i which will be the mean recurrence time. And this we have said is equal to 1 by pi 1 by pi i. So, the steady state probability of being in. So, this is the proportion of time being in state i over the long run. So, 1 by pi i would be the mean recurrence time. So, that will be the time required for reaching. So, on the average this is the time that you will require for reaching from i to i for the first time. Now, to compute m i j's you would need the first passage time distribution. So, you need to compute you cannot apply this formula because you have to then have all f i j n. So, again we will look at nicer way of computing your mean recurrence time and mean first passage times. So, as we said that to compute m i j by the formula where you need all f i j n up to infinity is not practical. So, let us now come out with another method for computing these mean first passage times. So, you see we will condition on the state at step 1. So, see what happens is if you are computing the mean first passage time then either the transition from i to j takes in one step. So, then this is 1 into p i j. Remember you are computing the expected value of f i j n. So, m i j if it is one step then probability of transitioning from i to j in one step that is p i j. So, 1 into p i j. So, either we will transition from i to j in one step and therefore, 1 into p i j or we go from i to k in the first step because since it has to be either we transition from i to j in one step or from i i go to some other state and then from k i will transition back to j. So, then it will be once I do this then the passage time becomes 1 plus m k j because m k j is the mean time of going from k to j and so 1 plus because one transition has already taken place. So, the mean time of transitioning from k to j for the first time will be 1 plus m k j and into the probability of transitioning from i to k where k is not equal to j. So, this should be clear. Now, we just rewrite this expression because you see here you are summing with respect to k and k is not equal to j. So, p i j is missing which is available from here. So, when you add up p i j plus sigma p i k k not equal to j. So, this whole thing is the summing up the components of the i th row of the first transition matrix which must be equal to 1 because from i you have to transition to one of the states and therefore, this is 1 plus sigma m k j into p i k k not equal to j and we can rewrite this as summation p i k into m k j where k is not equal to j. So, this is now again as I said the way can be computed your formula for f i j n. You know this is also a simple way of computing your m i j without really requiring to have the whole distribution for the first passage probabilities f i j n. So, these are n equations and n unknowns. I think n unknowns means in the sense that you are asking for this m 2 j and so on m n j. So, you are asking for the first transition passage to j from any one of the states 1 to n. So, these are n variables and because your i is varying from 1 to n and then in n unknowns n equations in n unknowns. Now, similarly we can also compute i i i can also be obtained this way. So, the first mean first recurrence time can be also obtained by solving corresponding set of equations here and let us so just work out the formula this will be. So, if you want to compute m 2 3 then m 2 3 will be 1 plus and remember your k is not equal to 3. So, k can take the value 1 and 2. So, it will be p 2 1 into m 1 3 plus p 2 2 into m 2 3. So, now in order to compute m 2 3 I need m 1 3. So, I will write down the formula for m 1 3. So, m 1 3 will be so here again k is not equal to 3. So, this will be 1 plus p 1 1 m 1 3 plus p 2 3 p 1 2 into m 2 3. So, then 2 equations in 2 unknowns and we can substituting the probabilities p 2 1 p 2 2 p 1 1 and p 1 2 I get these equations. So, it is not difficult because from here you get you know like half m 2 3 is 1. So, m 2 3 is 2 immediately you get it from here and then once your m 2 3 is 2 this is half. So, this is 3 by 2 and when you bring this to this side let us see m 2 3 is 2. So, this is 1 by 2. So, this is 1 by 2 and this is 3 by 2 and this you bring here. So, this will be half m 1 3 is equal to 3 by 2. Therefore, m 1 3 is 3 it is not just 3 by 2 m 1 3 is 3 yes from here you can again substitute and make sure. So, m 1 3 is 3 this is 3 by 2 and m 2 3 is 2. So, this is what is the mistake take care. So, this is equal to 1 plus m 1 3 we are saying is coming out. So, this is 3 by 2 plus take care and this is half this is 2 plus 1 3. So, m 1 3. So, therefore, that was the right solution. So, this is m 1 3 is 3. So, working out always helps you because you can find out your. Now similarly, let us just look at the way we compute m 1 1. So, the first mean recurrence time for going from state 1 to 1 and here again your k cannot be equal to 1. So, it will be 1 plus p 1 2 m 2 1 plus p 1 3 m 3 1 and now you need m 2 1 and m 3 1. So, 3 unknowns are there. So, therefore, 3 equations m 2 1 will be 1 plus again your k is not equal to 1. So, p 2 2 m 2 1 plus p 2 3 m 3 1 and then finally, when you write down for m 3 1 it will be 1 plus p 3 2 m 2 1 plus p 3 3 m 3 and so substituting for probabilities I get these equations and again a very simple way you can solve and you can check. Let us see m 1 1 comes out to be pi by 2 and if you remember your calculations for pi 1, pi 1 was 2 by 5. So, the steady state probability of being in state 1 was 2 by 5. So, the mean of first passage time will be 5 by 2. Now, m 2 1 comes out to be 4 and m 3 1 is 2. So, I hope this is the right calculation because let us see if you want to do it here just verify. So, if m 2 1 is 4 then this side is it is 1 plus 2 and your m 3 1 is 2. So, that is also 1. So, that is 4. This is coming out to be high because remember there is no arc from 2 to 1. There is no direct arc from 2 to 1. So, you can of course, go from 2 to 3 then 3 to 1 for the first time. So, that is 2, but then other parts when you go around it will be you know. So, the mean first passage time is coming out to be 4 and m 3 1 is 2. So, now we will see of course, see the thing is that as I said when I was computing the steady state probabilities and the mean first passage times there are certain conditions under which these are valid. So, we now have to look at the situations where these things may not be valid and in fact, I have said that you know these may not even exist and so on. So, we will now start looking at. So, right now under certain conditions which we have to specify and we will do it soon. So, I want to show you that this will be true all these ways of calculating your steady state probabilities and mean passage first passage times and mean recurrence times. You will be doing it under certain conditions they are valid and when they are not valid then we have other quantities to define those states. So, we will talk about it. So, let us just recall what has been done so far. See, we said that limit probability x n equal to j given that x naught is i is actually limit p x n equal to j and then we define this as pi j. So, this was the tacit assumption that this limit exists and that it is independent of the initial state and therefore, we said that you can solve these pi j's through a system of linear equations and that is what we did. So, but this was under the assumption that this limit exists, but now it is not really true that this will always exist and this is what we need to now talk about and find out. So, therefore, that means our you know the linear equation method of solving the pi i's depends on whether this limit exists and that we said that this limit is also independent of the starting state and therefore, we could write down the system of linear equations and say that the solution will exist under the condition that sigma pi j is add up to 1 and therefore, things were simple life was easy. So, therefore, we will now look at the conditions under which this limit exists. Second is that so the recursive formula for f i j and that we wrote down this formula is valid. Here, we are not asking for any limits or anything because the powers of p can be computed and then therefore, through that you can recursively compute your first passage probabilities. So, that is valid, but here and the m i j again because remember I am saying that the m i i's the m i i will be equal to 1 upon pi i. So, there again the existence of pi i is assumed and even otherwise. So, the linear equations for solving your m i j's also needs to be examined under what conditions it will be valid, this method will be valid. So, let us start looking at examples and then we would want to talk about we start talking about the conditions. So, first suppose so now, you again consider the job assignment problem and you see that the matrix P 2 had got all entries positive. In the first transition matrix there were some zeros in the sense that you were not able to go from 2 to 1 and you did not have a loop from 3 to 3. So, you had 0 entries, but when you took the square of the matrix then all entries became positive and after that your P 3 was also all entries were positive. So, this implies that there is a path from remember because this is simply a transition probability in 2 steps from i to j. So, if it is positive that means there is a path from each i to each j and in such a case when you have a path from i to j and then you have also have a path from j to i because P i j 2 is positive and P j i 2 is also positive since all entries are positive. So, you have this you have a path here again from j to i with positive probability. Such a pair of states is said to communicate with each other. So, they said they are said to communicate and if all states communicate with each other and so now because P i j 2 is positive for all i j. So, we conclude that all states for the job assignment problem communicate with each other and such a chain or such a Markov chain or process is called ergodic and we will talk about this some more. So, but first let me say that. So, now what we will say is that we will define a closed set as a set of states which communicate with each other and for the job assignment problem it turns out that your closed set actually consists of all the states 1, 2 and 3, but it is possible that you may have more than 2 closed sets or 3 closed sets whatever it is all states may not form one closed set. That means all states may not communicate with each other and now let us look at this example here. See this is your transition diagram. So, you see that 1, 2 and 3 you can see that they all communicate with each other. There is a path from 1 to 3 from 3 to 2, 3 to 1 and so on and there is a path from 4 to 5 and 5 to 4, but there is no path from 1, 2 and 3 to 4 and 5 and this you can of course see the thing is that I have drawn this diagram with 5 states, but when your number of states is large the transition diagram will be very big and it may not be again same problem as I was talking about earlier that you know you cannot possibly enumerate all possible paths when you have a large number of states. So, our recourse is to taking higher powers of p now. So, let us start and of course here it is evident that these two. So, therefore, your two closed sets are 1, 2 and 3 and 4 and 5. So, this change is definitely not all the states do not form a closed set right. So, let us see let us start with p the transition matrix. So, you see there is no arc from 1 to 4 or 1 to 5 similarly, there is no arc from 2 to 4, 2 to 5 and from 3 to this right and same way there is no arc from 3 to 4 to 3, 4 to 1, 4 to 2, 4 to 3 and 5 to 1, 5 to 2 and 5 to 3 there is no arc. So, this is the 0's and let us see if when we take the second power that means p square then these 0's remain intact. These probabilities change right these 0's like this 0 goes away, but this and then again when you take p 3 these 0's do not go away right and here it becomes. So, therefore, now you can say that for among the states 1, 2 and 3 the closed set 1, 2, 3 now you have path from each node to the other two nodes because these are all positive right. So, they all communicate. So, it means you need at least three transition steps to see that there is a path from each of the nodes 1, 2, 3 to the other two and similarly here is of course, quite clear. So, this is also you know all the probabilities are positive here and therefore, so and you see that this structure will continue no matter what higher powers you take of p right. You will continue to have 0's here and 0's here. So, therefore, this is the situation where the all the states do not form a single closed set and in fact, if you have more than one closed set we call such a chain or a process reducible right and if all states form a closed set one single closed set then it will be irreducible. So, what we have been talking about so far has been irreducible chain and irreducible means that all states will communicate with each other and therefore, it will also be ergodic. So, but again even this classification is not enough we need to do it further and so continue the discussion. So, if a single state forms a class that means there is only one state in a class it is called absorbing because obviously, the state is not communicating with any other state it is just communicating with itself. So, then it is called absorbing that is so the once the system enters an absorbing state it will not come out of it. So, we will just look at the examples for all these all these the kinds of states that we are talking about. Now, from the 5 by 5 matrix of the 2 of the 2 closed sets we just looked at this example in which you had 2 closed sets as formed by the states 1 2 3 and the states 4 and 5 right remember there was no communication between the states 4 and 5 and 1 and 1 2 and 3 and vice versa. So, these were 2 closed sets right. So, now the if you look at the sub matrix 3 by 3 sub matrix see because otherwise it is all 0's this is 3 yeah it was 0 and 0 right because there was no communication from 4 to 5. So, in the sub matrix the entry 1 3 was by mistake written as 25 by 36 it should be 19 by 36 because all the rows the elements of any row must add up for a transition matrix must add up to 1 and so with 19 by 36 the numbers 1 by 6 plus 11 by 36 plus 19 by 36 add up to 1. So, make that correction and in p 3 also the same correction has to be made because this mistake got carried over from p 3 where it was by mistake written as 25 by 36 and not 19 by 36. So, you see that the sub matrix itself. So, now the case that we were talking about that your matrix P n in the limit will converge to where the rows are all identical sorry states k 1. So, this was a pi 1 to pi k and so on. So, all rows were identical remember we this is the case we had talked about, but now you see it has no meaning here right because the rows these 0's will can we remain forever. So, we cannot say that the rows will become identical well in the sense that these 3 rows will become identical maybe you can say, but then below here you have 0's right because again 1 2 3 4 and 5 are not communicating with 1 and 2 and 3. So, you will have 0's here and then you will have some positive entries here. So, therefore, you cannot say that the rows will become identical right these 3 rows will become identical exactly that is what I want to show you that if you this is p 3. So, p 3 this is 1 by 6 pi by 24 and 1 by 8. So, supposedly this number will go up a little and this number will also go up because you need 6 pi 24 is 1 by 4. So, you need 1 by 4. So, supposedly you can see that the numbers are closed similarly 11 by 36, 7 by 24 and 1 by 3. So, 1 by 3 is 8 by 24. So, therefore, again the numbers are getting closed and similarly here. So, you see that means these 3 closed states they themselves will satisfy the condition that the steady state probabilities you know you can compute the steady state probabilities and here it will not matter in which the system started. But to say that all the rows of the 5 by 5 transition matrix will converge to the same values would not be valid. Now, again this big example I have taken from Rabindran Phillips and Solberg and so this is a 8 by 8 transition matrix and the diagram I have drawn on the other board and see I have written down this matrix though I am not making use of it right now. But it will be a good exercise for you people also to sit down and try to if you can write a small I am sure from Matlab and so on. If you feed this matrix you can get different powers of the matrix. So, it is not too much of a problem getting higher powers and then whatever I am saying you can also conclude the same things by looking at higher powers of p. But in any case if we draw the transition diagram it will look something like this right and I am not writing down the probabilities because they are here and anyway it will clutter up the diagram. We just need to look at the connections and whatever I am concluding from that is enough I do not need the probabilities really except that this p 6 6 is 1 right. So, fine. So, now if you just glance at this transition diagram then you will see that 4 5 8 are connected 3 7 are connected and you may say that 2 and 6 are connected right. But then when you go further you see that there is no arc from 6 to 2 and you at when you reach 6 then the probability of coming back to 6 is 1. So, this is a certain event. So, that means once the system comes to state 6 it just stays there and so 6 is a absorbing state right and so this is not a close set because close set 2 and 6 should communicate and 6 and 2 communicate. So, there is an arc from 2 to 6 but there is no arc from 6 to 2 right. So, this is not a close set. So, therefore, you have 2 close sets which are 4 5 8 and 3 and 7 that for sure because you have an arc from 3 to 7 and 7 to 3 right. Now, 1 and 2 do not figure therefore, this is not a close set. So, states 1 and 2 do not figure in any of the close sets and such states we define as transient because see what will happen is that. So, first of all they are transient and since you have more than 1 state close state in the system. Therefore, this is a reducible system. So, that is another thing. Now, thirdly once the system enters a close set. So, for example, it enters state 5 4 then you see it will keep hopping among these states only. There is no way of going out. There is no arc which is going out of 4 5 or 8. So, system will then for infinite number of times go on hopping among these states 4 5 and 8 and so such states we will call as recurrent because they will keep occurring again and again. So, the moment a system enters a close set then there is no way of going out right because if this communicated with any other state then you would that will also form part of the for example, a 1 a 4 communicated in 1 then this will become a close set. So, that is not true. So, once you enter a close set you cannot get out of it and similarly here if you come to 3 then you will go on going from 3 to 7 or 7 to 3 that is all you just go round and round or because there is no loop here either there is no loop here. So, you will continue doing it and from here if you come to 2 and then you go to 6 then you just stay put in state 6 you do not get out of it. So, first of all now after defining the recurrent state, transient state and an blobbing state you have a close sets what we want to say is that each close set if you look at this itself behaves like now a reducible Markov chain. So, a sub chain of the whole system which is irreducible ergodic in the sense that if you just consider this and similarly if you consider the chain considering of states 3 and 7 then they together form a irreducible chain and this is both of them 3 and 7 communicate with each other. So, talk about further classification is needed. So, now I will give you another classification of these states in terms of first passage probabilities and that will also give you a good understanding and of course, another way of determining whether the state is recurrent or transient or whatever it is. So, now let us define f i as the probability that starting in state i the process will sometimes return to state i for the first time right because f i i n was the probability that it returns to state i from i for the first time in n steps n transitions. So, now when we add up this from 1 to infinity this will give us the probability of the system returning to starting from i returning to i for the first time in any number of steps right. So, all possible. So, therefore, this will sometimes return to state i right. So, that is important. So, this is the probability now we say that state i is recurrent if f i is 1. So, that means there is a positive probability or there is a certain event that the system will sometimes return to itself to the state i from which it started. So, if f i is 1 then it is a certain event right, but if f i is less than 1 then it is a transient state right. So, now you can interpret this as saying that you see if f i is 1 that means the system will return. So, starting from i it will return to itself after sometime and then because it is a Markov process memory less that means the past history is not to be considered then you will again the. So, the whole process starts fresh and then you will again go to other states and so on and then come back again to i and so every time you come back the system starts fresh and since your f i is 1 therefore you will keep coming back to i infinite number of times. So, therefore, the way to characterize this is a recurrent state that means is that once you come to i then you will keep coming back to i an infinite number of times. But for a transient state see what is happening is since your f i is less than 1 1 minus f i is positive. So, this is the probability that the system will not return to state i and so if this is a positive probability then we will say that this event will also occur. So, therefore, we want to interpret this as saying that a transient state will be visited only a finite number of times and so that will be the difference between a recurrent state and a transient state. So, let us just interpret this now you see if the system is in state i for exactly n periods starting in i and you are exactly for n periods you visit the state i again for n periods n times then the probability of that is f i raise to n minus 1 because f i is the probability of x returning to state i. So, that into n minus 1 and again these are independent probabilities. So, that is why I am raising it to f i raise to n minus 1 because of the Markov process the Markov property and then it does not come back to itself. So, I am computing the probability that exactly for n periods it has visited state i starting from i I mean this is I am writing down the conditional probability of starting in state i and then visiting it for exactly n times being in state for n times. So, it is already started in state i. So, now it needs to revisit state i n minus 1 times. So, this is the probability of revisiting state i n minus 1 n times starting and then 1 minus f i is the probability that it will never come back after that. Now, this is and for n greater than or equal to 1 this is now this are the probabilities from a geometric distribution if you recall. So, what we are saying is that coming to itself is a failure and then not coming to itself is a success remember. So, you are asking for the probability in n trials you are asking for the well here this is the best way to say it is that this is not coming to itself. So, that happens once and then this is coming back to itself n minus n times or coming back to itself it is n minus 1 because it is starting in i, but it has been in state i for exactly n periods that is. So, there are two different things I am saying. So, therefore, here you are saying revisiting itself n minus 1 times. So, therefore, f i raise to n minus 1. Now, when you want to compute the mean for this distribution then it will be that means n now varies from 1 to infinity and you compute the mean and it will be n times 1 minus f i into f i raise to n minus 1 and here you can take 1 minus f i outside. So, sigma n f i n minus 1 and this is a geometric distribution f i is less than 1. So, this is convergent and this is the arithmetical geometric series you should all know how to evaluate it. We have done it in the beginning of this course and so the sum is 1 upon 1 minus f i whole square and so divided multiplied by 1 minus f i. So, this is. So, there is a finite this is a finite positive number that means the average of a you see if it was to visit itself infinite number of times the mean will not be finite, but here it is the mean is finite. So, that means you can again interpret this as saying that the state will not be revisited in finite number of times only a finite number of times it will be revisited. Now, another way of characterizing a recurrent state. So, you see that we have been trying to talk about the same thing in different ways and that certainly helps you to understand the things better. So, now, let us define i n as 1 if x n is i that means if at the n th time the system is in state i and 0 otherwise. So, this is a indicator wearable and then if you add up sigma i n from 0 to infinity this will represent the number of periods that the process was in state i. So, starting from state 0 sorry from time period 0 the initial time then this will count the number of times the system was occupying state i right. And now if you want to compute this conditional expectation that is given x naught is i you want to compute sigma i n and varying from 0 to infinity the expectation. So, I will exchange the summation sign that means if this thing is finite that means if this exist then obviously I can exchange the expectation. So, this is also convergent that is all I need to change. So, therefore, this is expected value of i n given x naught is i which but i n is equal to 1 if x n is i. So, therefore, this is probability x n equal to i into 1 when you conditional probability of x n given i given that x naught is i. So, 1 times that you will write down and this we buy our definition is p i i raise to n. So, therefore, summation and varying from 0 to infinity p i i n. So, you want to say that sigma i n and varying from 0 to infinity represents the number of periods that the process is in state i. So, now if so once you have computed this then a proposition immediately follows the proposition says that if state i is recurrent that is state i is recurrent if this is infinity because this is the expected value. And if our definition is that the recurrent state will keep revisiting itself infinite number of times. So, then the expected value will also be infinite. So, therefore, this will be infinity and for transient this has to be less than infinity. So, another way of looking at it now see interesting outcome from here is that if the number of states is finite see that is important is finite. So, the number of states is finite then all states of the process cannot be cannot be transient all states cannot be transient. And why because if a transient state can be visited only a finite number of times and you have finite states let us say you have the number of states are 1, 2 and k you have k states. Now, this is a if all are transient. So, this can only be visited let us say t 1 number of times this can be visited only t 2 number of times and this can be revisited only k number of times t k number of times. And now you take max of t 1 plus sorry t 1 t 2 and t k take the maximum of this define this as capital T. And then now when you consider the time t plus 1 t plus 2 what happens because all the transient states have already been visited you cannot review the process has to go on the process has to be in some state at time t plus 1 t plus 2. So, this is a contradiction because if you have only finite number of states and all are transient they will all get visited their finite number of times. And after that beyond that time period the process is going on. So, where does it go it has to transit to one of the states. So, therefore, in a finite process finite process we mean finite number of states all states cannot be transient. So, therefore, immediately the question is asked if the number of states is not finite then is this possible that all states may be transient and yes. So, we will also look at an example and we have already done looked at such a process, but we did not really talk about this aspect of the process at that time. So, yes if the number of states are infinite then the all the states may be transient.