 So, in the last class, we started talking about hitting times and recurrence times right. So, we defined what we mean by different states like what is transient, what is recurrent state, further recurrent state we further classified as positive recurrent and null recurrent. And then we started defining what we mean by mean number of visits. So, we defined our number of visits to a state. So, we said that if I have a state j, then this quantity mj is defined as number of visits of my Markov chain to the state. And then I was interested in the question what is the expected number of visits given that I start from a particular state or I could just start it by j and we said that this is nothing but pjj of n and goes to 1 ok. We show we argued that this follows from after applying my monotone converges theorem fine. So, finding expected value of mj is straight forward here, but suppose let us say I am interested in finding probability that mj is equals to m let us say if I start from some j. So, what is mj is telling number of visits to state j. So, this mj here is a discrete valued random variable here right. It can take value 1, 2, 3 all the way up to infinity. Now, this is the expected value that is expressed in terms of n step transition probability. Now, instead of this expected value I am interested in more refined description of this mj and I want it to know exactly what is the distribution of mj itself. Now how to go about this ok. So, what is this is basically saying that if I start in state j what is the probability that I am going to hit this states say exactly m times again in the future ok. Now, how to go about this or how to find this distribution ok. Now let us try to compute this yeah x naught we want to mj is number of counts of particular state j right here. So, here I want this mj to be exactly m that means this Markov chain xn has taken this state j exactly m times ok. Now what does this mean? The probability that mj equals to m means what I hit my state once after that I hit it second time and I hit it mth time after that I never hit it again ok. Suppose I define tk to be the time of kth visit to my state j. So, when this k equals to 1 t1 denotes what? Time of first visit when k equals to 2 that means time of my second visit like that. So, now if mj has to be m what is that should happen? t1 should be some finite number right I should be busy hitting my state j in finite quantity finite time and what about t2? t2 should also finite right I am hitting it for the second time and like that t2, t3 all the way up to tm and what should be tm plus 1? It should be infinity right then only it is the case that mj is exactly equals to m. So, let us write that. So, if this is the case then it must be the case that t1 should be finite tt should be finite all the way up to tm should be finite and tm plus 1 should be finite given whatever this x not equals to j right. So, mj equals to m all it is saying is I should have hit state j m times in the future it is not telling it at what time? You may have hit it after some time for the first time subsequently after some more time you might have hit another time. So, if mj equals to m the first m visits must have happened in finite time and after that the mth m plus 1th hit might have not happened that means tm plus 1 equals to infinity. I could then write this as so is this t2 is going to be greater than t1 right that is the second visit right. So, I could write this as t1 is an infinity like this and write it all the way up to tm minus finite and tm plus 1 minus tm to infinity given x not equals to j same thing I have expressed in this format. So, now instead of looking at the time I am looking at the interval. So, t1 is the first time I visit this is the time duration more required to hit for the second time like this ok. Now, let us write this I am going to now apply my chain rule here ok. I will just apply the chain rule. Now, what is this quantity probability that t1 is less than infinity given that x not equals to j. So, what is the meaning of probability that t1 less than infinity that means I have probability of ever hitting state j starting from j right. First with it means I can just think it of ever whatever I am going to hit state j. Probability of ever hitting state j starting from x not equals to j what is this. So, did we define something in terms of the fjj's what was fjj. So, there was something called fjjn also right what was fjj of superscript n yeah starting from state j probability that you are going to hit state j for the first time in n times and we define fjj to be the sum of fjj of n's what was that interpretation we said what was the fjj then told us yeah probability of ever hitting state j. So, is it not exactly that probability that t1 less than infinity means probability of ever hitting state j. So, this is fjj now rotation and then the same thing same thing x not equals to j sorry there should be conditioning here right which are missed where is that t1 less than infinity. Now let us focus on this what it is saying you started from state j initially you hit state j in some finite time that is what meaning of t1 less than infinity and then after that you are asking probability that you are going to hit it for the next time that happens in finite time and all the way up to now can I think of this as now what I am saying you are going to hit state j at some finite time and after that conditioning on this does not going to matter because this is the Markov chain and then I am asking okay you are going to hit state j again again now I will be looking at it for how many times here m minus 1 times now I have already conditioned that you are hit it first time at some point after that you are again going to hit it m minus 1 time. So, now is it not just case that this probability is nothing but I can think of okay now I can then interpret as whenever you hit t1 less than infinity from that point again you are visiting your state j again for in the m minus m rounds okay. So, check that check that this is indeed correct like you can just write like this okay. So, the t1 has been absorbed into this right like you have been the t1 less than infinity now I am whenever that hit happened for the first time right now I can think of that is the starting point and from there subsequently visiting m minus 1 steps. So, yes x0 equals to j in some time I have hit state j again now I can think of my Markov chain from that point and then look at it hitting again m minus 1 times subsequently. So, that is what this I can this point is actually the time when I have hit for the first time but I can think that as my original point origin and write this. So, you see this this has nice recursion right in terms of f j j's. So, I can further write it as f j j again f j j and then what is this going to be m minus 2 like that. So, if you keep on writing this what will you eventually end up with is f j j just write that here will end up with this and then at the end you will end up with probability that t1 less than infinity let x0 equals to j sorry this is the last one is going to be you will make t1 equals to infinity right say here t2 minus t1 become here infinity right in the last one this will be just t1 equals to let basically this is t1 minus 0 infinity. So, that is basically t1 equals to infinity now what is this probability you started with state j and then asking the question that your first time hit is going to happen at infinity that means basically you are not hitting it what is this probability 1 minus j j j. Now, if you are going to look at this mj as a random variable which takes integer valued numbers and it has this kind of distribution can you associate it with what kind of generative distribution with what parameter with parameter f j j right that is a parameter here. So, now we have f j j here now we know something about f j j depending on whether it is transient or recurrent right suppose my state j is transient what is f j j means in that case transient case less than 1. So, in that case this probability will be non-zero probability right it will be strictly positive quantity. So, in that case we will have is going to be now suppose j is recurrent what is this quantity f j j is going to be what 1 and what is this quantity is going to be that case f j j is equals to 1 and my probability is going to be 0. So, does this make sense? So, do you expect this probability to be 0 for a recurrent case why? So, what is recurrent means I keep on hitting the state at right. So, that means what what is the value mj is going to take it is going to take infinity right because in this term here I am xn is going to take j many many times. So, it this value cannot be finite in that case. So, that is why this term is going to be 0 this probability is going to 0 and this is going to be 1 only at m equals to infinity possibly that is the only place where it is have full mass. Now we have this now let us see based on this probability we can find out the expected value of mj ok mj expression we already have here. Let us see what is the expression I get from this probability. Now what is the expected value of mj given x not equals to j how can I write this I am going to write it as m into probability right and what is the probability probability is exactly this quantity here can somebody quickly compute this and tell me what is this value is going to be expected value if this expected value correct here I take value m with this probability right. So, that is why this is the expression for my probability now what is its value you can check that this value will turn out to be fjj minus 1 minus fjj. So, again if fjj is going to be 1 this quantity is unbounded right this quantity is basically infinity that means the expected number of visits is going to be infinity. But if suppose this fjj is strictly less than 1 that is this j is a transient quantity then this expected quantity is going to be what is going to be finite. So, based on this we can make the following result. So, just let me write this we have come up with two formulas for this one is fjj divided by 1 minus fjj and another one is what this we also said that this is also same as a state j is transient if and only if expected returns. So, this statement here directly follows from this right if this if j is transient I know that fjj is less than 1 and this guy is going to be finite and if this guy is finite I know that fjj has to be less than 1. So, that is why a state is and is transient if and only if it is expected return it is our finite. Now, is this clear also I am saying that if j is transient then the limit as n goes to infinity the sequence pjj of n goes to 0. Why is this true? Because if j is transient I know that this sum is finite that means this series converges right if that is the case it must be the case that this sequence of pjjn should go to 0 the current state and we know that if state j is recurrent this guy is going to be infinity. So, that is and we also have that n goes to 1 to infinity this pjj of n is going to be infinity. So, basically we have tried to characterize what kind of state it is based on its mean return to mean return times and the mean return times we have expressed in terms of my first passage times right fjj is basically my first passage time or the what we called as a recurrence time we have already already given its definition okay fine. So, from this theorem it is clear that how I am going to check whether a state is going to be transient right all you need to do is see if expected number of visits is 0 or see if a transient j state is j then we know that also its sequence simply converges to 0 okay. Now what about the recurrence state for the recurrence state we know that recurrence state is further classified into null recurrent and positive recurrent from the here we have not given any condition we just know that when the state is going to be recurrent whenever the sum is going to be infinite that it is going to be that it is going to be a minute but what about when it is going to be null recurrent or positive recurrent for that I am just going to state a result which we will take it as granted we will prove that result a bit later okay. I know my state is recurrent to further classify that it is going to be positive recurrent or null recurrent what I will do is suppose I will look into the sequence okay there should be 1 by now I am looking at the average of the first n comes here in the sequence of pij case okay. Now I will look at this average if in the limit as n goes to infinity if this average goes to 0 starting from an i for which fij equals to 1 then I know that this j is null recurrent if this guy takes some positive value other than 0 then it is going to be sorry the first case is null recurrent then if it is going to take a positive value then we are going to call it as positive so okay what is that way how to use this result then suppose let us say you know j is recurrent that you would easily find out if you know that fjj is equals to 1 you have let us say you have already found it out now to know that my state j is positive recurrent or null recurrent one option is go with the definition we already have look at n times fjj of n right what is that the new j j we have given new j is what look at this and see that if this is going to be less than infinity then it is going to be positive recurrent and if this is going to be infinity null recurrent either go and use this definition for this you need to have this quantity is fjjn that is the first passage times of first passage distribution you need to have or if you do not want to go and but you have this sequence pijn that you have this n step transition probabilities then you can look into this average quantities and see that if this goes to 0 then you know that this j is null recurrent otherwise it is going to be positive recurrent okay so any intuition why if this goes to 0 this is going to be null recurrent and when this is greater than 0 it is positive recurrent so what would you say when it is going to be positive recurrent okay so fine see if j is recurrent you already know that fjj is equals to 1 right you can just take that i to be that particular j and then apply this but we are saying that what is fjj is equals to 1 means that means if you start from state i with probability 1 you are going to hit state j if you are going to start with such a state from where you have you know surely that you are going to hit state j then look starting from that j you compute this you can as well take i to be j okay so let us go back to this so when this is finite what would you say when this is infinity so from that at least you can connect why this could be possibly make sense this result possibly make sense okay let us focus on this particular case right so this is going to be null recurrent when we say that null recurrent then we said that it is going to come back but going to take a long time possibly to come back right so in that case so in that case we are saying that this in this when you are going to look into this n basically dominates the summation and basically kills the summation right what is the meaning of 1 by n this going to 0 so basically this means that this quantity over here this is small o of n right you understand the meaning of this small of n so if this quantity if I just divided by n that it this n dominates this and eventually kills it that means when you divide this quantity by small n and let n go to infinity then this quantity goes to 0 okay and in a similar way when this happens this means the second case this means this is like some theta n that means this quantity is eventually going to constant right that means this quantity so this n here some constant times this quantity so that is why as n goes to infinity you are going to get it to some constant value yeah for a yeah so if you recurrent means it is one of them positive recurrent and null recurrent what is not satisfied no it is like 1 by n it is not just this you are you are not just taking this summation infinity you are also dividing by n and then taking this to infinity that is the difference between this and this see recurrent means it is both here positive and null recurrent so this quantity is going to be infinity for both positive and null recurrent now I want to character I want to further classify whether it is positive recurrent to that for that instead of looking at this I start looking at this running averages so this is the running average version of this right as n goes to infinity if final j is recurrent then whether it is going to a positive recurrent or null recurrent I am going to decide based on this criteria here so just think about this like why this will kind of thing is going to translate at least just think about this intuitively okay fine