 So, through the numerical examples, we saw that limit p n p is a transition matrix. So, the nth power as n goes to infinity, if this exists, if the limit exists, then it converges to a matrix with all rows identical. Whatever 2-3 examples we considered, we saw that the limit existed and then we saw that the rows were becoming identical as we increase the power value of n. That means, we continue taking higher powers of p. So, we can easily show that in case limit p n exists, limit p n as n goes to infinity, if this limit exists, this will always be the case. That means, whenever this limit exists, then this will converge to a matrix whose rows are all identical. So, let us show this immediately very easily. So, limit p n, suppose n goes to infinity is, so q 1 is a row q 2 q k. Suppose, these are k rows, we are considering the system when it has k states, k possible states. So, then I can write limit p n as n goes to infinity as limit p n minus 1 into p as n goes to infinity. Then as as goes to infinity, this and this have the same value. So, this is the same matrix, they will converge to the same matrix. So, this will be q 1 q 2 q k is equal to q 1 q 2 q k p. So, this reduces to the limiting case, this reduces to the system. And from here, you can say that q i, the i th row here would be the i th row multiplied by p, post-multiplied by p. So, this is it for all i. And hence, you can see that all rows of p are, I should not say of p, what I want to say is that, if converges to p is your transition matrix. So, all rows of p raise to n will converge to. So, all rows of limit p n, I should write here, limit p n as n goes to infinity are identical. Now, if you want to solve for this, we can see that immediately. See, you know that the rows of p, the rows of p have the property because it is a transition matrix. So, all rows add up to 1. And therefore, this is not a non-singular matrix. And so here, you will have infinite solutions. To this system, you will have infinite solutions. But then, if you also require the elements of q i to be non-negative and they add up to 1. That means, we are looking for a solution, where the q i is the elements of q i form probabilities. Then, this will be a unique solution. I will denote the solution by q i is equal to pi 1, pi 2, pi k. And these will be known as the steady state probabilities. So, that means, when the system has in a steady state that means, it has gone on for a long time, it settles to a steady state. Then, the probability of being in state 1 is pi 1, probability of being in state 2 is pi 2 and up to pi k. So, this is the whole idea. And now, we will come up with a method of obtaining these values pi 1, pi 2, pi k, the steady state probabilities. So, now, let us evolve a method for computing the pi i's, the steady state probabilities. See, p n can be written as p n minus 1 into p, the nth power of the transition matrix. So, then, if I take limit on both sides, then this is limit p n and n goes to infinity and this is limit p n minus 1 and goes to infinity into p. Now, as we said, since we have assumed that the pi i's exist and so, each row of p n in the limiting value would be pi 1, pi 2, pi k. So, all the rows are identical. Therefore, on this side also, you get the matrix pi 1, pi 2, pi k, pi 1, pi 2, pi k and so on. Similarly, p n minus 1 will also converge to the same matrix and this times p. So, the limiting behavior, I can just break up this in this way and then do it and so, if this is going to the limit in the limiting value to this matrix, this will also go to the same matrix and therefore, you have these equations. Now, this system, actually, since all the rows are identical, so actually, these k equations reduce to. So, now, I am, so far, I was talking about three state processes. So, now, let me just do this much in three state for the general case and then we will come back when we want to talk of specific values and examples. We will again revert back to the three state example that we have been talking about. So, let us just talk about it in general and therefore, the system reduces to k equations. That means, I can simply just equate the first row here to the first row here. So, that means, pi 1 to pi k is equal to pi 1 to pi k times the matrix p. And now, let us write out the equations in detail here. So, pi 1, the first component here will be this multiplied by the first column of p. So, the first column p is p. So, pi 1, p 1, 1 plus pi 2, p 2, 1 and so on plus pi k, p k 1. Similarly, equate the second component here element to the second element. That means, this multiplied by the second column of p. So, that gives us this. And finally, the k-th equation is pi k is equal to pi 1, p 1 k. So, the k-th column we will take when we equate pi k with this multiplied by the k-th column. So, I have these k equations. But you can immediately see that these k equations are not linearly independent since sigma p i j, j varying from 1 to k is equal to 1. So, let us just quickly check this that all the equations that see essentially what I am saying is that your first k minus 1 equations will give you the k-th equation. So, therefore, those of you who are familiar with the word rank. So, the rank of this matrix is k minus 1 or you want to show that. So, let us add the first k minus 1 equations here. So, it will be pi 1 plus pi 2 plus pi k minus 1 and this is equal to. So, when you are adding the first k. So, you will be adding p 1 1 plus p 1 2 up to p 1 k minus 1. So, it will be pi 1 into summation p 1 j, j varying from 1 to k minus 1. And similarly, pi k into summation j varying from 1 to k minus 1 p k j. Now, since the rows add up to for the transition matrix, we have put that I mean we know this that these rows of the transition matrix will always add up to 1. So, therefore, sigma j varying from 1 to k minus 1 p 1 j is actually 1 minus p 1 k. Because this plus p 1 k is 1. So, therefore, this sum is equal to 1 minus p 1 k and similarly, I substitute for all of these sums by. So, this one will be 1 minus p k k. And now you see that when you open up the bracket. So, pi 1 plus pi 2 plus pi k. So, pi 1 plus pi 2 plus pi k minus 1 cancels out, you are left with pi k. And the other things you transfer to the side, then you immediately get pi 1 p 1 k plus pi 2 p 2 k and pi k. So, this is your kth equation. So, because the probabilities of the rows sum up to 1, therefore, these k equations are not linearly independent. So, in fact, the first k or any k minus 1 will lead you to the kth one essentially. Because here, I just chose the first k minus 1, you can choose any k minus 1 and you will be able to obtain the remaining one by adding the k minus 1 equations you have chosen. So, therefore, infinite solutions because the matrix is singular, the equation matrix is singular. And therefore, but when you impose the condition, because since we are looking for these steady state probabilities and they must add up to 1, pi 1 plus pi 2 plus pi k has to be 1. Because the system will be occupying one of the states, either 1, 2 or k. So, when you impose the condition that pi 1 plus pi 2 plus pi k is 1, then you get a unique solution. And so this is therefore, we have a very neat way of computing these steady state probabilities and we know that we have a unique solution. So, you cannot say that the probability of the long run probability of being in a particular state of the system, the probabilities are more than 1. That would not be a reasonable solution. So, now let us go back to your job assignment problem and let us try to obtain. Because I was trying to get you the, have a look at the steady state probabilities by taking the powers of p, but now here let us have this seems to be a quicker way of and a neater way of solving the, trying to get the pi's. So, because in the, when you are taking the powers, you really do not know when to stop. Or in fact, you would have to go on doing it till you see that the values are really closing in. So, therefore, this would be a better way to get your steady state probabilities. And so the three equations you see, you can see from pi 1, your, this thing, p 1, 2 is, no, no, when you are writing the equation, yeah, p 2, 1 is 0. So, here you get, yeah, the matrix is there in your earlier lectures. So, these are the three equations essentially, right, for solving pi 1, pi 1, pi 2, pi 3. So, therefore, I can, from this equation I immediately get. So, the trick would be that since, you know, I do not get a unique solution to this system. So, I will solve for pi 1 and pi 2 in terms of pi 3 and then I will apply the condition that the sum is equal to 1 to get the value of pi 3 and then I will get all the values. So, from here you see, you immediately see that half of pi 1 is 3 by 4 pi 3. So, that gives you that, yeah, where is pi 1? Yeah, half of pi 1 is 3 by 4 pi 3. So, pi 1 is 3 by 2 pi 2, right, 3 by 2 pi 3, right. And then you can substitute here for pi 1 in terms of pi 3 to get your pi 2. So, pi 2 comes out to be 5 by 4 pi 3, because this is half pi 2 and this is pi 3 minus 1 by 4 into 3 by 2 pi 3. So, which makes it 3 by 8. So, 5 by 8 pi 3, therefore, pi 2 is 5 by 4 pi 3. So, now, I substitute the values pi 1 is 3 by 2 pi 3, this is 5 by 4, 5 by 4 pi 3 and this is 1. So, therefore, how much is this? I suppose I will have to redo this thing or maybe this was right, yeah, I do not know, yeah. So, what is it? This will be 6 plus 5 plus 4. So, the value was ok. This was a mistake here, but the value at computed was ok. So, this is this, right. So, therefore, your pi 3 pi 3 is 4 by 15. And so, this gives you pi 2 equal to 5 by 4 into 4 by 15, right, by the value of pi 3. So, that makes it 1 by 3. And pi 1 would be pi 1 is 3 by 2. So, 3 by 2 into 4 by 15 which is this and 3 by 5. So, pi 1 is 2 by 5. And now, let us compare these values with what we had obtained by taking powers of P. So, our pi 1 had come out to be something because 2 values are 1 0 2 upon 256 and the other one was 1 0 3 upon 256. So, you see 2 by 5 does lie between these 2 numbers. So, this was up to 4th power, right. And when you take the 5th and 6th powers, you will see that the values will get closer and you will actually reach 2 by 5, right. Similarly, your pi 1 is 1 by 3 and this is also a number lying between 85 upon 256 and 86 upon 256, right. You can compare, right. This is between. So, 1 by 3 lies between these 2. And similarly, 4 by 15 is a number which is between 68 by 256 and 69 by 256. So, the 2 things match, but certainly that is a much better quicker way of obtaining your steady state probabilities. Now, these steady state probabilities have very useful interpretations and we will continue seeing through examples drawn when we analyze the process further. So, essentially what we have said is that pi i is the probability that in distant future, one will find the system in state i. So, the probability that your process will be that means, a particular employee in the automobile manufacturing company, that particular employee will be in let us say HR, when you know after the process has gone on for let us say 4 or 4 years of 5 years, we expect the percent, the probability that the employee would be in the HR division is 1 by 3 or the probability that he will be with sales is 4 by 15, right. So, these are the long term probabilities and as we said that the initial, that means the division or the section in which he started his career is irrelevant here, right. Then you can also interpret this as the fraction of time, the system occupies state j, fraction of time, the system is occupying the state j, okay. I am writing pi i, so it should be pi i here, sorry, this is i. So, the fraction of time, the system occupies state i, right. Now, if you run many identical processes simultaneously, then you see the pi j would come out to be the fraction of processes that you would find in state j, that means if you suppose run 100 identical processes simultaneously and you find out that that is is maybe 45 of the processes are at that particular time. Of course, you let the processes run for a long time and then after certain, at particular period time, you just find out how many of these processes are occupying state j. So, if that comes out to be 45, then your pi j would be, pi j would be approximately 45 upon 100. That will be the fraction of processes that you would find in state j. And another interesting interpretation of the pi j is, you know, it is the reciprocal of the mean number of transitions between recurrence of state j. So, this recurrence I will define formally after some time, which is also very important. So, what we are saying is that this is the reciprocal of the mean number of transitions. So, on the average, how many transitions you would be required to go from state j to j. Now, recurrence means for the first time, that means you are in state j to start off and then for the first time when you revisit j. So, that number of transitions, if you take the average of such transitions, then the reciprocal of that is your pi j. And this also we will derive in another way and of course, this part we will prove later on. So, for example, what we are saying is that since pi 2 is 1 by 3 and pi 2 is our HR section. So, we are saying on the average, three transitions will be required for this particular employee to go from HR to HR. That means if he starts his career with HR, Human Resource Section, then he will after 3, on the average, he would be requiring three transitions to get back to HR. So, this is the so many interpretations that we have, we can give and then we will see how we make use of these steady state probabilities to analyze these processes further. So, let us now, I have taken this example from again Ravindran Phillips and Solberg. So, interesting physical interpretation of state probabilities. So, what he is saying is that in the job assignment problem, you would consider the three states as three reservoirs. So, node 1 and node 2 and node 3 they are reservoirs and the arcs connecting the nodes are the pipes through which liquid can flow with valves to ensure that flow goes in the direction in which the arrows are there. For example, there will be a valve here which will direct the flow from 1 to 3 only and another valve which will direct the flow from 1 to 2 and then another valve which will just direct the flow from 1 to 1. So, this is the idea. So, just think of this as representing a reservoir with these pipes connecting them with the reservoirs and then the valves to ensure that the flow goes in the direction in which the arrows are there. And then the probabilities Pijs that means for example, the probability 1, 4 associated 1, 2 would be the fraction of the liquid that is there in 1 in reservoir 1 which will be sent to 2. Similarly, if you look at 2 to 3 then it is the half the liquid which is there in reservoir 2 will be sent to from 2 to 3 and so on. So, these probabilities then can be interpreted as the fraction of a liquid in reservoir. So, Pij is the fraction of the liquid in reservoir i that will pass to reservoir j in 1 unit time. So, it will take 1 unit of time for the flow that means half the flow from here to here in 1 unit of time will go from 2 to 3 because the probability is half. So, if we think of the system made like this then what you do is you pour 1 unit of liquid into the system according to these initial probabilities that means 1 fourth of the liquid is put in reservoir 1, 1 fourth is put in reservoir 2 and half the liquid is put in reservoir 3. And then the liquid is allowed to flow according to this plan and then what we are saying is the dynamic equilibrium. So, what we have discussed that the probabilities will converge and they will become irrespective of how much liquid was initially poured into the reservoirs. So, finally, dynamic equilibrium will be attained and liquid will continue to flow, but the liquid in each reservoir will equal the steady state probabilities. So, our steady state probabilities I do not have the numbers here, but whatever we had computed pi 1, pi 2, pi 3 as. So, for example, I remember pi 2 was 1 by 3. So, pi 2 would be the that means reservoir 2 will have one third of the liquid and then pi 1 would represent the amount of liquid that is in reservoir 1 and pi 3 will be the amount of liquid present in reservoir 3. So, this is the interesting part and so what is being said is that actually equilibrium will be attained and the liquid will continue to flow according to this plan, but each reservoir will settle down to even though the starting amounts were this, each reservoir will settle down to the amount of liquid according to these steady state probabilities. And what do we mean by so equilibrium means that for each reservoir the flow out will be equal to the flow in. Then only the whatever there is the liquid is there steady state liquid that is there in the reservoir will be maintained, which is according to your pi 1, pi 2, pi 3. So, flow out from reservoir I would be so the probability that you are in reservoir I into then pi j summation over j. So, from I it can go to reservoir 1 to 2 to 3. So, this is the probability that the amount liquid flow out from reservoir I and the amount and this you can when you are summing up you are summing up with respect to j. So, pi i can come out and this will be sigma j pi j, but sigma j pi j is 1 all these probabilities and the i th row will add up to 1. So, this is pi i and the flow in from other reservoirs that the flow is coming in. So, that will be pi k into p k i the flow is coming from the k th reservoir to the i th reservoir. And so then this is the probability of being in the k th reservoir. So, this is sigma pi k p k well I am talking of probabilities, but here we are saying this is the amount that is there in the k th reservoir. And so this is the fraction which is going to i. So, I should actually interpret the whole thing in terms of this particular example. So, here also I should not refer to pi i as the probability of being in i, but this is the amount of liquid that is there in the i th reservoir and the p i j fraction of this liquid is being sent to the j th reservoir. So, therefore, flow out. So, from the i th reservoir this much is the liquid and from this these are the fractions of this liquid which are being sent to different reservoirs. So, this is the flow out and this is the flow in. So, please just interpret it this way ignore my earlier remark. So, this is sigma pi k p k i and so the 2 must be equal and therefore, you again get these if you do it for all i then you will immediately get this equation pi is equal to pi p. So, I thought this was the interesting way of looking at the steady state probabilities and some of these will fix certain ideas in your mind. Then, another example from Sheldon Ross that I want to because I really want to spend time on these steady state probabilities. So, that you get the ideas you know understand them properly. Now, here this is an example where it is a production system and these are the probabilities of transition probabilities. So, we have 4 states 1, 2 and 3, 4. Now, the states 1 and 2 are considered as acceptable or you can say when the system is up and 3, 4 are not acceptable which you have to interpret as your system is down. That means there is a breakdown the machines are not functioning. So, there are 4 states and this is your transition matrix from state i to state j and questions to be answered. Now, the questions that we want to answer are the rate at which the production process goes from up to down that means rate of breakdown. So, when it is up that means when the machines are working and then there is a breakdown. So, you want to know the rate at which the process the production process goes from up to down and another question will be the average length of time the process remains down. That is also very important because you want to know with this kind of transition matrix you want to know for how long the process will remain and of course, you always talk in terms of average length of time the process remains down when it goes down. So, when there is a breakdown for how long will it remain in that state before it comes up, but now the new thing is that you have two states which are describing the up situation and two states which are describing the down situation. So, therefore, I took up this example to again show you of course, we will compute the pi i's and then I will show you the how to compute answer this and there are many more questions that have to be answered. So, the third question you want to answer is the average length of time the process remains up when it goes up. So, these are three questions we will try to answer by computing these steady state probabilities. So, we write down the equations for finding the steady state probabilities. So, pi 1 is equal to this pi 1, pi 2, pi 3 times the first column. So, you get this equation then pi 2. So, you can just by looking at the transition matrix P then you can see that these are the four equations that we will obtain. Now, interestingly the second column here is all 1 by 4, 1 by 4, 1 by 4, 1 by 4. So, when you write the second equation this you immediately get the solution for pi 2 because all these add up to 1. The steady state probabilities have to add up to 1. So, this immediately comes out that pi 2 is equal to 1 by 4. So, I have used it already here and now since I have the value of pi 2 I should be able to immediately compute the values of pi 1, pi 2 and pi 3. So, what I do is here that is not I have to now after having got pi 2, I can now see from here even here. So, I thought that it was immediate that you could compute after you have pi 2 then yes see here you see that is where 1 has to be little clever and use inspection. So, here this is pi 1, pi 3 and pi 4 and again the coefficients are 1 by 4. So, this I can write as 1 by 4 into 1 minus pi 2 see and I have the value of pi 2 already as 1 by 4. So, this again immediately gives me pi 1 as 1 minus 1 by 4 is 3 by 4 into 1 by 4. So, 3 by 16. So, your pi 1 is 3 by 16. Now, I have the values of pi 1 and pi 2. So, then from here I can immediately get pi 3 because bring this here. So, this will be 3 by 4 pi 3 and I substitute the values of pi 1 and pi 2. So, this will be 1 by 2 into 3 by 16 plus 1 by 2 into 1 by 4 that gives me 7 by 32. So, pi 3 will be when you multiply by 4 by 3 gives you 7 by 24 and then once you have you now have pi 2 pi 3 and pi 4 is on this side when you bring it it will be half pi 4 and so again substituting for pi 2 and pi 3 you get these values and so you get pi 4. So, this was quick work. Now, this is certainly faster than computing second, third, fourth powers of P which is a 4 by 4 matrix. So, lot of multiplications if you start taking the different powers. Now, let us try to answer the questions rate of breakdown. So, rate of breakdown is a transition probability of transitioning from up to down. That means, up means when your machines are working or any one of the machines is working and then any one breakdown will mean there is a breakdown. So, that means, you are transitioning from up 1 and 2 which are up states to the down states which are 3 and 4. So, that means, you want to say that if you are in state 1 so that is a probability into your transition from 1 to 3 or 1 to 4. So, that is P 1 3 plus P 1 4 and if you are in state 2 then the transitioning from up to down is 2 3 plus 2 4 P 2 3 plus P 2 4. So, we have all these numbers for the probability of rate of breakdown will be 9 by 3 2 because any one of the breakdown say this means you are going from up to down. So, any one breakdown or two breakdown whatever it is the probability is 9 by 32 and that is your rate of breakdown. Now, you want to answer the second question which is the average length of time. The average length of time the process remains down when it goes down and the other one is the average length of time it is up when it goes up. So, both the things. So, let us define u bar as the average time the system is up and d bar is the average time the system is down. Then your rate of breakdown is now we are redefining or talking in now. So, then we will make the equations and then try to find out u bar and d bar this is the idea. So, rate of breakdown is 1 upon u bar plus d bar because you know see this is the average time it is up and average time it is down. So, one breakdown at the rate of 1 upon u bar plus d bar because this is the total time and it is up and then down average time. So, therefore, 1 upon this will give you the rate of breakdown because 1 breakdown for this much period 1 breakdown for this much period and therefore, the rate is 1 upon u bar plus d bar. So, proportion of up time is then u bar upon u bar plus d bar and similarly, you will define the proportion of down time as d bar upon u bar plus d bar. Now, let us try to find out. So, the definition of u bar will be pi 1 plus pi 2 this is because this is the you are either in state 1 or in state 2 that is when it is up. So, this is the this is the probability of being in state 1 or state 2 and then. So, this is the proportion of time and this is your rate of breakdown 1 upon u bar plus d bar which we have already computed as this from here right this is your rate of breakdown. So, that is this and so, this gives you your u bar and then we compute d bar and we will continue with the exercise. So, therefore, we saw that u bar will turn out to be 7 by 16 which is pi 1 plus pi 2 divided by u bar plus d bar which was your rate of breakdown. So, that was 9 by 32. So, then we multiply and it will be 32 by 9. So, this u bar comes out to be 14 by 9 that means, this is the average amount of time for which the system will be up right and since u bar plus d bar is 32 by 9. So, to get d bar we will simply say d bar plus u bar minus u bar which is 14 by 9. So, that comes out to be 18 by 9 which is 2 units of time right. So, therefore, what we have been able to answer the three questions that were asked the first was what is the rate of breakdown. So, this is 9 by 32 or 28 percent of the time the breakdowns occur and then the breakdowns on the average last for 2 units of time. So, that means, once the system is down then it will remain down for 2 units of time and then the third question was the average amount of time for which it is up when it is. So, then there is an average amount of time for 14 by 9. So, 14 by 9 it is up when the system is up right. So, it will remain up for 14 by 9 times. So, now the thing is certainly the system is not in a very satisfactory situation because your breakdowns on the average the breakdowns are remain for 2 units of time whereas, your actual production time is only 14 by 9 which is less than 2 units of time. So, certainly the system is not in a very healthy state and so this again gives a warning to the manufacturer to do something about it because the way the transition probabilities are given this is what your conclusions are. So, I hope you understand see the way we computed this because we had to define U bar and D bar and then of course, the other reason I took this example was that you know there were 2 states which were defining the up system and 2 states which were defining the down system and therefore, we had to the computations were not just straight forward. So, I thought that will be a good example to discuss in this course. So, once we have talked about the state probabilities pi i's which we were answering as to the know because this and of course, we have also computed this. So, for that means number of transitions required to go from i to j this was the answer and the pi i's gave you the pi i's gave you the long term probabilities of the system occupying a particular state and then of course, we also said this is the fraction of time that the system will be in state i. So, pi i's I gave you these interpretations of pi i. Now, this other kind of questions that are needed which are now the first passage and first return probabilities. So, this is also very important because you want to know how long will it take to reach a certain state and so, when you say how long will it take to reach a certain state. See the statement it means here that you know that that will be the for the first time that you reach the state. So, that means you starting from a state i and then you are wanting to say that how long will it take for you to reach state j. So, obviously the moment you reach state j you have answered that question. So, therefore, this will be what we mean by this is that the for the first time that you reach state j from i. So, that is the understanding. So, that means now for example, if n transactions occur before state j is reached from state i. So, suppose we want to just you know surmise or say that n transactions have taken place before state j is reached from state i and then we want to know the probability that n transactions will be required from going from i to j. Then you might say that would pi j raise to n be the answer for this probability because pi j n also tells you the probability of transitioning from i to j in step 1. Let us see now there is a difference because you see when you talk about pi j n then it does not say that you may reach j before and a number of times before you finally reach j in n steps. The various graphs that I drew for you earlier showed that you may know like you had you started from state 1 then you stayed in state 1 and state 1. So, this was your you know transition probability from 1 to 1 in 2 steps or in 3 steps you stayed in state 1. So, here that is fine. So, this was your p 113 p 113 of course, also included that you could go from here to here remain here and then come here or you could stay here then go here and go here. So, this is fine this particular path that you are taking no you are not coming back to state 1 before. So, starting from state 1 you are reaching here without going to state 1 beforehand. So, therefore, this is the kind of path you are looking for, but whereas when we were computing your p 113 we had all possible paths to reach from 1 to 1 in 3 steps that is what you are doing. Say for example, if you consider p i j 4 then p i j 4 I could go to 4 in step 2 also that means from i I could go to j as I said and then you could again go to j and then j this would also be there. Then you will have i to j and then you could go to k and then to j and this way. So, so many paths, but nobody is stopping you from. So, when you compute the probability p i j 4 remember we said it includes all possible paths for of going from i to j and so you could revisit j in between number of times because you have to enumerate all possible paths. So, therefore, this is not the answer what we are looking for. So, we need to make some more definitions and some more terminology has to be introduced to compute these probabilities. So, the first passage and first return probabilities we want to compute. So, essentially what we are looking for is that for the first time I reach j from i. So, in between I should not have touched state j and when the first time it occurs I want to compute the probabilities of such and so what we will do is we will make these definitions here. So, again f i j n I am defining as the first passage probability and so I need to I have written first passage here, but I will define it here. So, first passage probability of going from state i to state j and remember x n was the state in which the system is at time n. This is we have been using this notation when we were describing the transition probabilities and the Markov process. So, now what are we asking for we are saying that f i j n is equal to the probability of x n equal to j, but x n minus 1 is not j x n minus 2 is not j and x 1 is not j it is only x naught sorry x naught is x naught is i. So, starting from i in between all these n minus 1 transitions that take place you do not ever touch j, but it is only in the n th transition that you reach j. So, probability of that is what we are defining by f i j n. So, that means probability of reaching j from i in n transitions for the first time. So, if you look at f 1 f i j 0 then f i j 0 is 0 because we cannot transition then f i j 1 will be p i j your right f i j 1 will be simply p i j. So, now we want to compute f i j n that is probability of going from i to j for the first time in n steps n transitions for the first time. So, we should not have visited j in between the less than n steps and so this we will obtain by writing p i j n which is the total probability of going from i to j in n steps all possible parts which may imply revisiting j number of times and then finally, coming back to j. So, p i j n minus in sigma k varying for 1 to n minus 1 you see because you want to reach for the first time from i to j in n steps. So, your k can be allowed to vary from 1 to n minus 1 only and then this will be f i j k. So, for the first time you have visited from i to j in k steps and then once you reach j and then again you can go to many other states come back to j or stay in j whatever it is from in n minus k steps. So, for example, if you look at p 113 then what we are saying is that you can you know continue staying here in state 1 for all the 3 transitions or you can go somewhere here come back and then again revisit here and whatever possible paths you can or you can go this way this way this way. So, many other paths you can think of so we are ruling out all those paths. So, we are subtracting so that means you have visited from i to j in k steps for the first time and then in the remaining n minus k you again from j go to other places or remain in j and then come back to j. So, we subtract all these then we get the probability that f i j we subtract these from p i j n. So, we get the probability that probability of visiting j from i for the first time in n steps and then we will see lot of applications and implications of these probabilities. So, this is what we how we write down the expression for f i j n. So, let me and of course, when j is equal to i then f i i n would be the probability of going back to i from i for the first time. So, you started from state i and then when you for the first time you reach state i again in n steps of going. So, I should say from in n transitions. So, this will be f i i n and that will be the probability of recurrence of i state i in n steps. So, f i i i n and I use the word recurrence earlier and I said we will define it later on. So, that n we will of course, be talking in detail about these first passage times and first return time. So, first return probability is your recurrence probability probability of recurrence, but here of course, we are saying in terms of n transitions and then you would want to know the probability of ever returning to state i and so we will compute that also. So, now the number of transitions to go from i to j for the first time. So, transitioning from i to j for the first time is called the first passage time. So, the number of transitions required to go from i to j for the first time we will define it as a first passage time and you can see that this is also a random variable because you do not know how many transitions you will require to go from i to j for the first time. So, first passage time is a random variable and if i is equal to j then the first passage time is called the first recurrence time. So, this will be when you want to come back to the same state starting from a particular state you want to come back to it for the first time that will be your first recurrence time. So, first passage times and first recurrence times are random variables and therefore, you can redefine this again f i j n is a probability that first passage time from probability of first passage time from i to j in n steps. That means your random variable. So, f i j n is the probability of the first recurrence of the first passage time equal to n. So, f i j n will be because this is the time first time you return you go from i to j and when this value is equal to n you want to compute the probability of first passage time equal to n is your f i j n. This is how we will define. We will go through now a very interesting journey when we want to compute these f i j n and f i i n and in fact you would finally want to talk about f i j. So, that means that will be when for the first time you return from transition from i to j. So, without the n because here this is transitioning the first passage time when it is equal to n that means the probability of the first passage time equal to n. So, now you would want to then finally compute your f i j n f i i. So, we will continue with this discussion.