 So, in the last lecture I had told you that if the number of states are finite of a Markov process, then all states cannot be transient. We had argued it out that if the system has to go on, then because transient states will be visited only a finite number of times. So, if they are finite number of states, then the process must come to an end in a finite number of time. But since the process has to go on, therefore if the number of states is finite, then all states cannot be transient. But the argument does not hold when the number of states is infinite. And so I said I will give you an example that when the number of states are infinite, it is possible that all states of this process may be transient. And this is what we want to talk about today. And this is, so giving you an example when all the states are transient. Now, random walk is a very interesting and important process. And I hope that after having read about it, now you will be able to recognize situations of processes, stochastic processes which follow the behavior of a random walk. So, this is now a Markov chain when the state space consists of the integers i varying from, can take the value 0 plus minus 1 plus minus 2 and so on. So, that means I am numbering the states by minus 1 plus 1 plus 2 minus 2 0 and so on. And so this can go on for infinite number, the number of states is not finite. This is a infinite state situation. And then the transition probabilities, so if you go forward, that means if you transition from state i to i plus 1, the probabilities p and if you transition from i to i minus 1, that means if you go backwards, the probability is 1 minus p. And this is for i varying from 0 to plus minus 1 plus. So, therefore, the probability remains the same. Essentially, it is going forward, the probability is p and if you are going backwards, then the probability is 1 minus p for some any p varying between 0 and 1. So, therefore, for different values of p, you will get different random walks. This is the idea. And diagrammatically, if you look at the transition diagram, the transition diagram simply says that, so this forward probabilities are p and the backward probabilities are 1 minus p. So, from minus 1, you can go forward, then you will go to state 0 and from 0, if you go backwards, you will go transition to the state minus 1 and the corresponding probability is 1 minus p and therefore, this process can go on on either side. So, this is your transition diagram. And so, you see that from here, that all states communicate, because from anywhere to you can go anywhere. By forward movements, if you want to go from minus 1 to 2, you can go here, then you can again come here and so any possible path is there, but you can transition from any state to any state. So, all states communicate and so therefore, either all states will be recurrent or all states will be transient, because remember the recurrent states will form a class. So, if they form a class, then within a class, all states must communicate with each other, since this is already true that all the states communicate. Therefore, all the states will either be, so they are all in one class. So, they will either be recurrent or transient. And so here again, remember I had defined for you recurrent state via the probabilities of first passage probabilities. And remember, we had said that if f i is equal to 1, then the state is a recurrent state, because then there is a positive there is a positive the event that it will recur back to itself is a certain event. And if f i is less than 1, then we had said that the state is a transient state. And then using an alternate characterization of a recurrent and a transient state, we had also said that if from here, we had said that if p i i summation, this will be summation p i i n and varying from 1 to infinity. If this goes to infinity, then the state i is a recurrent and this is less than infinity, then the state i is transient. So, this was another characterization. So, that is what we will use here. And since all states are behaving in exactly the same way, because the probability is the same of going forward or backward. So, it is enough if we consider sigma n varying from 1 to infinity p 0 to 0 n. So, let me just look at the behavior of this expression. And if this is I can show that this will be a divergence series that means it will go to infinity, then I can conclude that state 0 is recurrent. And since they all the states form one single class, therefore every state is recurrent. And if sigma n varying from 1 to infinity p 0 0 n is less than infinity, then 0 is a transient state, which implies that all states are transient. So, let us now start looking at this expression. For example, if you look at p 0 0 n, that means this is the wanting to know that 0, you starting from 0, you will be back in 0 in n steps. So, since you see, you can see from here from the red diagram that you can return back to 0 only in even number of steps, in even number of transitions. If I go here, then I can come back here. So, 2, if I go from here and here and then I need another 2 steps backwards or if I go from here and then I go here back here and then or in fact, you can have any possible numbers of forwards and backwards. But they should be, so in fact if the number of transitions, if my n is odd, then I cannot come back from 0, if I start from 0. So, coming back to itself requires even number of steps and you can just try to draw number of paths and you can see that you can go from here, here, then here, then you can come back here and again here and then here. So, all possible ways are there of course, but then you will require every time even number of steps. So, therefore, if you look at p 0 0 2 n minus 1, then this will always be 0. You cannot transition back to this state starting from 0, you cannot come back to 0 in odd number of steps. So, those probabilities are 0 and for coming back even number of steps, then you require exactly n forward transitions and n backward transitions in any order, as I tried to explain from the diagram. So, this will be equal to therefore, from 2 n, you choose n forward steps, forward transitions and n backward transitions. So, therefore, the probability p 0 0 2 n will be like this. Choose n from 2 n transitions and then p transitions forward and 1 n transitions forward and n transitions backward. So, probability of backward transition is 1 minus p, probability of a forward transition is p and this is 1 minus p. Now, just open up the expression here. So, this is 2 n factorial n factorial n factorial p raise to n 1 minus p raise to n. I will now use Stirling's approximation, which I have already talked about in earlier lectures. So, factorial can be approximated by Stirling's formula and so this will be 2 n raise to 2 n plus half e raise to minus 2 n and then under root we left out the part p raise to n and 1 minus p raise to n. So, this is and therefore, similarly n factorial can also be written by Stirling's approximation formula. So, n raise to n plus half e raise to minus n and then there will be a root 2 pi. So, root 2 pi for both of them. So, therefore, it will be 2 pi n raise to plus half e raise to minus n. Now, we can cross out a few things, this and this gets cancelled out and then we can 2 raise to half we can cancel out from here. So, then I will be left with the root 2 pi and this is root 2 pi was there because this is root 2 pi and this is 2 pi. So, root 2 pi was left and then that root 2 gets cancelled by this root 2 here and then you can just see the simplification here and this is n raise to half. So, here this is n raise to 2 n that cancels out with this and then there is a n here and this is n half. So, you will left with root n. So, this is root n root pi and then you have 2 raise to 2 n which I bring inside here. So, this will be 4 into p 1 minus p raise to n. So, this is the simplification right after using Sterling's approximation formula for the factorials I get back. So, we saw that this series can be approximated by the series n varying from 1 to infinity 4 p 1 minus p raise to n upon root pi n. Now, this I can the terms I can break up as some of even terms and odd terms and here I though even though I have written the index as n does not matter it is a dummy index. So, here it could have been m also, but the idea is that I am breaking up I am writing p 0 0 n instead of this I am writing the odd I am adding up the odd terms and the even terms. Now, the odd terms do not contribute anything to this sum. So, it is only the even part and so now let us consider the case when p is equal to 1 by 2. So, in that case this will become equal to 1 and so the series will reduce to summation n varying from 1 to infinity 1 upon root pi n, which we know is a divergent series because the power root pi of course is a constant. So, this will be n raise to half 1 upon n raise to half and we know that this series 1 upon n raise to p n varying from 1 to infinity is divergent for all values of p less than or equal to 1 this we already know. So, therefore, this is the divergent series and therefore, since so the all the states we will immediately conclude that all the states are recurrent because the time to return to this is infinity. So, all states are that means sigma n varying from 1 to infinity p 0 0 n. So, this is recurrent and therefore, all other states are also recurrent. Now, we have to consider the case when p is not equal to half. So, for p not equal to half your 4 p into 1 minus p will be less than 1 remember because for p equal to half this is the maximum this as the maximum value for this term 4 p into 1 minus p is for p equal to half. So, for p less than half it will be less than 1 and so let me call this term as alpha let me denote it by alpha. Then we will show that this series converges and it is simple you just apply the ratio test. So, take the n plus 1 th term divided by the n th term. So, which will be now the n plus 1 th term will be 4 p into 1 minus p raise to n plus 1 under root pi into n plus 1 and here you are dividing by the n th term which is this. So, you have this and so here you see you left with 4 p into 1 minus p and then this root n I bring in the denominator. So, this will be 1 plus 1 by n raise to half and therefore, the limit of this ratio of the n plus 1 th term and the n th term as n goes to infinity will be you see this will go to 1 and therefore, it will just converge to 4 p into 1 minus p which is a number equal to alpha less than 1. Whatever the value of p since p is not equal to half this number it will be equal to something less than 1 and so in that case we will conclude that all states are transient states. Now, of course I had told you that you can look up for whenever now that you know this random process it is going backward forward and of course, for the transient case the probability of going forward is different from going backwards and for recurrent states it was both the probabilities are the same. So, now just on the lighter side you know you can say that if you if there is a drunken man and he is trying to walk on a along a straight line then he will take a step forward then he will take two steps backwards or he may take two steps forward and one step backward or something like this and so you know the wanderings of a drunken man you can you can sort of say that the process the walkings of a drunken man can be modeled as a random walk and of course it will depend on the p then value of p will depend on how drunk he is or something like that. So, that is the that is one of the examples and then we may be come we will come across some more in the process of this course or otherwise you can you know you can now be aware of such a process. Then we will continue with the classification of the states and after the transient states there are also states which are periodic and null states which are both of which are not of much practical use and of course the occurrence is also not that often. So, now you see we have talked of we said that a recurrent state it is possible that a recurrent state may have infinite mean recurrence time that is mathematically this sum may not converge and remember for recurrent states even you wanted to find out the first passage time and so on. Then we did it by finding the mean recurrence time the M i j and M i i and so when we did this then we assume that this is a finite that this series will converge. But so, but it is possible mathematically that this series may not converge even though the series this of course this is your condition for a state to be recurrent that is sigma f i i n n waiting for 1 to infinity which is equal to sigma which is equal to f i that is the recurrence time probability of the state coming back to itself then this is equal to 1. So, for recurrent state this probability has to be 1 because coming back to a recurrence state is a certain event. So, this series converges but this series may not converge which is your mean recurrence time and this is such a state we will define as a null state. So, not much is talked about it and so we will also not spend much time but we must complete the presentation of the states of the classification of the various states. And so for the recurrence states where we assume that this is finite because then we were solving for M i j's or M i i's and but it is possibility that this series may not converge. And this as I have already said these are of real practical use and the probability of being in null state will also go to 0. So, therefore, we will not talk about such states we will not spend much time on it but just to complete the discussion we have also considered the case when this may not converge. So, this may not be finite. Now, let us talk about periodic states. So, consider the following transition matrix and this is the corresponding transition diagram and you can see immediately from here that from 1 you will either go to 3 or to 4 and also from 2 you will go to 3 and to 4 and then again from 3 you may go to 1 or you may go to 2 and from 4 you may go to 1 and from 4 you may go to 2. So, that means there are 2 classes you can immediately see because there is no communication between the states here between among the states here and the states here. So, it is you can just I should not say classes exactly because these 2 are not communicating and these 2 are not communicating but they are communicating to the. So, you have communication between the states of this set and this set and vice versa. So, your system will alternate that means at any time either the system will be occupying states 1 and 2 or it will be occupying 3 or 4 because once if you start in 1 then you will either be in 3 or 4 and then if you are in 3 then you will either be in 2 or back to 1. So, you either communicate this way or you communicate this way. So, your system alternates between these 2 classes is that ok and that you can see by these probabilities also. Now, so let us in fact you can see from here that suppose I go to from 1 I go to 3 then I can come back to this. So, that will be in 2 transitions I can come back from 1 to itself or I can go from 1 to 3 then I can go from 3 to 2 and then 2 to 4 and then 4 to 1. So, that means it will be then that case it will be 4 transitions that will be required to go from to start from 1 and come back to 1 or and look at the other thing if you you may go to from 1 you may go to 4 then again you can come back. So, again it will be 2 transitions, but if from 1 you go to 4 and then you go to 2 then you will go to 2 to 3 and then 3 to 1. So, it will be that means you can record back to state 1 either in 2 transitions or in 4 transitions and the same story is true for state 2. That means from 2 you can come back to itself either in 2 transitions or in 4 and the same again what 3 and 4 from 3 you may go to 2 and then come back or you may go to 2 and then you may go to 4 then 4 to 1 and 1 to 3. So, all the states can be visited, revisited either in 2 transitions or 4 transitions. Revisited I mean starting from that state you will revisit it either in 2 transitions or in 4 transitions. So, you can see there is a periodicity and so let us now make a definition. So, we will say that a state which can occur at time periods m 2 m 3 m and so on where m is some integer greater than 1 is called a periodic state of period m. So, therefore, using this definition you can say that all the states of this particular chain are periodic of period 2 a state for which no such m greater than 1 exists is called a periodic. So, if I cannot find so here of course, the understanding is that actually the world should have been a state which can occur at times period m 2 m. So, that means if you are starting from that particular state and then it occurs again at period m 2 m 3 m then the period is m. So, that part is understood here. That means you are starting from a particular state and then if you can visit it at regular intervals of period m m is some integer which is greater than 1 then that state will be periodic. So, in this case of course, what is happening is that your all the states are periodic. So, this is one particular example, but of course, you can have a situation where you may have some recurrent states and. So, these are also recurrent, but not in that sense in this you see here. So, we will just see that there will be no this thing there will be you can say that this series. In fact, the periodic state you already know that the you are going to visit it at a particular time. The probability of visiting it at regular intervals is there it is a positive probability. So, for periodic states we said that the coming back number of transitions has to be factor of some integer greater than 1 that means m 2 m 3 m and so on. Then we will say that it is a periodic state of period m. Now, of course, on the transition diagram if you can see that all walks starting from i walks or paths whatever we have been saying starting from i and returning to i are of length m 2 m and so on where m is greater than 1 that i is a periodic state and the period is m. And so, for the example that we just considered I showed you that any state you start from either 1 or 2 or 3 or 4 you can come back to them in either 2 transitions 4 transitions 6 transitions. And so, we concluded that the period for each of the states was m. Now, if you can find a walk of length 1 that means if there is a loop for a state that means you can come back to it in one step or if you can find 2 walks which have relatively prime lengths. That means one walk may be of length m 1 and the other may be of m 2 and they are relatively prime they have nothing in common. Then you can conclude immediately that this state is not periodic that means it is a periodic because if 2 there can be 2 paths of length but coming starting from that state and coming back to it either in m 1 transitions or m 2 transitions and these are relative then certainly they cannot be anything common. And so, they cannot be a period to that state or if there is a loop then certainly it is not a periodic state that is one another way. So, we just trying to look at various ways in which you can characterize a periodic state. And then again if it turns out that your mth power that means the P m transition matrix raise to power m if this is greater than 0 that means all components are all entries of this matrix are positive for some m then no. So, actually it is not difficult to show that if you are given that P m is all components of the matrix P m are positive for some m then it can easily shown that your matrix P m plus 1 will also be positive that means all entries of the matrix P m plus 1 will be positive. And this you can see immediately from here see P m plus 1 can be written as P into P m. And now let alpha denote the minimum of P i j m greater than 0 and then we are saying. So, because every element of P m is positive so take the smallest one and that smallest one I am denoting by alpha and that will be positive. Now, consider the i j th element of P m plus 1 so that will be i th row of P multiplied by the j th column of P m and so this is P i into P j m. Now you see that when you multiply the i th row with the j th column and replace each entry of the j th column by alpha because all other elements are bigger. So, I am writing the smallest possible number for each of the entries of the j th column and then so it will be P i 1 plus P i 2 plus P i n times alpha because and since the rows add up to 1 so this will be equal to alpha. So, therefore the i j th entry of P i j of the matrix P m plus 1 is greater than or equal to alpha which is also positive and this holds for any elements i j of P m plus 1. So, therefore matrix P m plus 1 is also positive. Now, since all the entries of P m are positive so it seems that pick up any i then P i i m is positive that means there is a path of length m from state i to i and since P m plus 1 is also a positive matrix P i i m plus 1 is also positive which implies that there is a path of length m plus 1 from i to i and now by our definition you see m and m plus 1 for any m integer positive integer there are co-prime numbers. So, we have shown that there are two paths from i to i of co-prime lengths and hence by our definition the state i cannot be periodic it will be by definition it will be a periodic. So, and since this is true for all i therefore no state is periodic so the proof was simple. I had asked you to do it on your own, but I realized that we can show it right here. Figure it out think about it now of course periodicity is a class property as we have shown that means you will have just as we define that all states in that class would be periodic and having the same period. So, we can put together all states which are periodic of the same period. Now, consider this example that we were just we looked at the transition diagram for four states and we saw that every this thing I did not write down I think the probabilities or let us not forget. So, let us see the given that matrix P then you have P square if you now multiply P with itself then you get this matrix right. So, your P was if I yeah we just wrote down in the last time we said that one is going to 3 or 4. So, there were numbers here 1 by 3 2 by 3 and so on. So, you had positive numbers here and you had positive numbers here these were zeros and then. So, now when you take P square the numbers the non zero numbers shift here and the non zero numbers from here shift here right and each entry becomes half right and then P 3 when you now take P 3 that means you multiply P 2 with that transition matrix P then you will get. So, these entries will shift here and these entries will shift here. So, this is how so I am just wanting to through powers of the transition matrix I am just trying to show you what is happening and then finally when you take P 4 the halves will again shift here P square which you would expect because every state of the system had period 2. So, therefore, after two iterations of two transitions this will the transition matrix will be the same. So, P 2 will be P 4 will be equal to P 6 and so on right and similarly P 5 will be equal to P 3 and this is equal to P 3. This will not be equal to P that is true. So, P 5 will be P 3 and then all other powers will continue all odd powers will continue to be the same because then you see P 3 tells you a probability of going from 1 to 3 in 3 steps that is what will happen right. I can it will come back to itself and then go back and so from here it can go to 3 and 4 and again in 3 steps. So, either it happens to see you had 1 here and then you had 3 right and you had 4. So, this was it right then if you come back to 3 in one step and then if 3 you cannot go back to it you cannot come back to from 1 to 3 you want to come back in 3 steps. So, you will either go to here or you will go here right to 2 right and then you may come back to 2 3. So, it will take 3 steps to come from 1 to 3. So, starting from 1 if you want to come back to 3 then you will have to require 3 steps because in 2 steps you can either from 1 starting from 1 you can either come back to 1 or to 2 and. So, therefore, you will need odd number of steps to go from 1 to 3 or to 4. So, either in 1 step or in 3 steps or 5 steps and so on because the even transitions are reserved for coming back either to this thing. So, that means within a class and that is why we said that this is this and this is this. Of course, they all had the same period, but here what we are saying is that you can go from 1 to 3 and then 3 to back or 1 to 4 and 4 to 1 and so on. So, this is what is happening and so you see that your p i i n. So, if you see here in the first matrix your p i i. So, p 1 1 1 was 0 now it is half then again in p 3 this number is 0 and similarly all these numbers are 0 and all these numbers here in p 2 are half and then in p 4 again they are half and in p 5 when you again look at it all the numbers all these p i i's will be 0. So, this limit does not exist. So, limit p i i n and going to infinity does not exist actually it oscillates between 0 and half. So, therefore, and this for the first condition for the. So, this series will also not converge this if you want to take the summation, but anyway these are not the first step transition probabilities. So, all we are saying here is that limit p i i n as n goes to infinity does not exist this limit and also. So, this summation will also not exist this is not the convergent series p i i n n will be from 1 to infinity, but will not be a convergent because the necessary condition for this series to converge is that the n th term must go to 0 as n goes to infinity. So, here if this limit does not exist. So, therefore, I cannot say anything about this series also. So, now after having looked at all possible states of a Markov process the conclusion is that you know if the process is ergodic then and of course, finite number of states then the steady state probabilities can be found by solving system of linear equations as we saw right by solving this matrix equation pi equal to pi p and sigma pi i i varying from 1 to n is 1. So, with this condition because otherwise the solution here is not unique. So, we saw that we when we put this condition we will get a unique solution if your system is ergodic that is finite number of states and all states are recurring. And of course, therefore, this system does not contain any transient periodic or null state. So, this was one convenient way, and then we saw that there were other states also. So, for transient and null states probability of being in that state is 0 and so and of course periodic state also do not possess steady state probabilities. So, when we when the when the state possesses a steady state probability then we saw we can solve it by this linear equations adding this equation to it, but otherwise periodic states as we have seen do not possess steady state probabilities for transient and null states the probability of being in that state goes to 0 as the process continues for a long time. And so, we have finally sort of completed this argument and said that if when you know that the system is of ergodic and so on finite number then you can solve for steady state probabilities using system of linear equations. When others we have seen methods how to classify and how to decide that the state is periodic and it is null or transient. Now, the thing is that periodic and null states have are not much of practical use and they do not they are their occurrence is rare. So, we will not talk about them, but transient states you know transient processes are there that means there are lot of practical situations where the process does not continue for a long time. So, it is and therefore, we call them reduced Markov processes and so they are transient in a sense that after a while the process comes to an end. And so that we would like to look at in more detail because there are situations where your processes are not supposed to continue for forever. And so we will define them as reduced Markov processes and we will talk. So, reducible Markov chains one or more absorbing state and a number of transient states because we are saying that the process will not go on it will terminate after a short period of time. So, therefore, there will be either one or more absorbing states and remaining will be transient states. So, again by our discussion we have seen that once you reach an absorbing state you will not go out of that state. So, the process will terminate or if you are in a transient state and the number of states are then you will you know you will the number of states are finite then again after a finite period of time where the process will be over. So, this is what we are talking about. So, an interesting example and that is the gambler's ruin problem. Now, the idea is that the gambler at each play of the game has probability p of winning 1 rupee and probability q of losing 1 rupee. So, there is a game and I will tell you why it is called gambler's ruin problem. So, now successive plays of the game it should be y s plays of the game are independent. So, successive plays are independent that means it does whatever the outcome of one play the game goes on independent of what has happened and the gambler will quit playing when he wins rupees n. So, he wants to make a fortune of rupees n if he is hoping for that and so he will quit the moment he has earned n rupees. So, we want to find out the probability that starting with rupee rupees i suppose he has this much money with him the gambler's fortune will reach rupees n before reaching 0 and that is what we mean by the ruin because if we allow the process to go on then he will ultimately lose all the money and that will be the end of his gambling because then he cannot bet anymore. So, let us look at the transition diagram. So, here what we are saying is that see with 0 he cannot play because he has no money to bet. So, he stays here otherwise if he has rupee 1 then he can either lose that rupee and come back to state 0 or he will win and he will go to with probability p and then he will go to 2 that means he will have 2 rupees. So, the state is described by the amount of money he has and that is how we are using these integers to describe the situation and then of course, again if he has 2 rupees he bets and he loses that rupee then he will again revert back to having rupee 1 and so that will be the state. So, this the diagram and finally at n minus 1 when he has n minus 1 rupees he will bet again and he will if he wins he will get n rupees he will make his fortune. So, we want to compute that and of course, what we are saying is that he will stop playing. So, there was no are going back from here because he will just quit the game. So, this is the whole idea and so this is now if you look at it this is a you know the duration of the process is finite and you can see that this is an absorbing state and you may call this also an absorbing state and all other states are transient. Because the moment he has some money with him he will bet and then he will either go back to 0 for example, here he will either win and he will go to transition to this state or he loses and then he transitions here. So, again this is an absorbing state and he loses the he just loses all the money and therefore, the game is over. So, this is what we want to talk about and we are show and of course, the whole idea is to compute the probability that starting with rupees i the gamblers fortune will reach rupees n. And you can say that here of course, so the process here you can immediately see that this is a Markov process because your transition and it just depends on where you are. So, your transitioning to the next state just depends on where you are and it does not matter how you reached 1 or how you reached 2. So, you can immediately conclude that this is a Markov process and this is a you know short period duration. That means, it will terminate if the gambler ends up with rupees n otherwise of course, or if he just loses everything and he is back here. So, starting with the rupees i you want to find out the probability that the gambler will reach the fortune that he wants to make. The gamblers ruin problem I have put it as an exercise in exercise 10 which I will be discussing after some time. So, I have after explaining the problem I have now left it to you to work out the details and let us hope that you enjoy doing it and you are able to compute the probabilities. But while discussing the exercise I will also give you some hints and try to show you how to go about it.