 So, today let us wind up our discussion on renewable theory whatever we started in the last class. So, in the last class we talked about this elementary reward theorem what we called as ERT. What did this theorem say if you have a process? So, just as a recap what we did? We started with the DTMC and we said let us say start with an initial state i and I am interested in some particular state j where s is my state space and I am interested in my DTMC going back to this particular state again and again. And then we have this sequence of random numbers which we called as lifetimes and we said that this lifetimes are such that these are all going to be independent. So, x n's are independent and then we said that if you look at x n's for n greater than or equals to 2 they are IID, it is an IID process. Now, if you consider this case then our ERT theorem elementary renewal theorem said that this is under the hypothesis that if probability that x j is equals to 1 and so what did we say? We said that limit as n tends to m of t by t is equals to what would you say? This limit goes to 1 by expectation of 2 almost shortly and then we also said that these also what basically said this, this result said as is the average number of renewal in the interval in any interval that converges to this rate which is given by 1 upon expectation of x 2. We discussed this last time if the expected value of x 2 is going to be large that means I am not coming back to returning to my state in small number of times, maybe I am going to take large number of rounds to come back. In that case I do not expect this quantity to be large. If this guy is going to be large, this guy is going to be small. This is because this is the average number of renewals in the interval 0 t. Fine. How is this theorem? We said we are not going to the proof of this but let us take it we understand intuitively as this should be the correct. Now what is the use of this theorem? So if you recall I had stated an earlier result which we use heavily in the study of DTMC and that result was about when is a state a j, when if I know that a state j is recurrent, when is it transient, sorry when it is a positive recurrent and when it is null recurrent. So we had state and one theorem related to this right. So what is the theorem? So in a DTMC suppose let us say f i j equals to 1 and j is recurrent. Then we said something about this limit right. What was this? j is recurrent. We said this quantity we called it something like gamma j and we said that this guy is greater than 0 if and we said that this is going to be 0 if null recurrent. Actually this theorem is a direct consequence of this elementary renewal theorem. Let us see why this is true. So remember that this theorem where all we use? We use this theorem in the proof of invariant distributions right. So when we made a claim that for an irreducible DTMC it is going to be positive recurrent if and only if pi equals to pi p. In the proof of the theorem we used it and we have also used it in many many theorems. Now why this is true? So to understand this now let us come back to our setup here again. One thing we have defined is m of t is equals to what? We have defined it to be m of k such that z of k plus term equals to z and we called this m of t as what? What? We called it actually renewal process right and this is defined for any t. So we have called it as a renewal process and this is defined for every t. You give me arbitrary time t then I am going to tell you how many renewals have happened in the interval 0 t. So what is that m of t is telling? Basically m of t is telling how many renewals have happened in the interval 0 to t. Now yes this t here is continuous time but I so if it is I may also be interested in knowing what happens at some m of n where m is my some integer value. I can ask this question right if this is going to be defined for every possible t that is real I could as well ask for a particular integer value. So then what is m of n? So that is in that case I can just write it as indicator that yk is equals to j starting k equals to 110 is this right? This is basically the number of times I have state hit state j till time n right till the nth instance. Now let us assume that for time being all this y n are such that the n they are of unit so when I say n equals to 1, n equals to 2 like this these are like separated by 1 unit. So 1 second, 2 second like the separation between any two integer corresponds to 1 unit of time okay. Now so for this also I can define I am assuming that I am starting from some initial state y equals to I right. So I can denote the expected value of this to be k equals to n k is given. So m n is nothing but expectation of this m of n earlier we have defined this small m of t which was the expectation of this m of t but now I am looking this function only at the integer value so it is fine I can define it in a similar fashion. Now let us look at the average of this quantity m of n by n as n goes to infinity. So as n goes to infinity what you expect m of n by n go to is it going to be the same limit as this or while I am doing this now instead of looking at t at all possible values I am only looking at t taking integer values. So if I am going to look at this m of t by t instead of all possible t's I only look at some integer values is this limit is going to be different from this limit or it is going to be different limit it is going to be the same limit right. So it has to be expectation of x2 okay. Now let us understand what is this quantity now let us focus on this. If I have m n by n this is going to be what so this quantity here is expectation of 1 by n summation k equals to 1 to n integer by k is equals to j given y not equals to i right I am just writing the expression for this. Now what I want I am interested in knowing the limit of this. So I am just writing this quantity in this. Now limit as n tends to infinity this expectation suppose if I interchange this expectation on limit what is this quantity is going to be when I take this expectation inside then expectation of indicator becomes what probability of given y not equals to i right. So is that same as saying probability of i j superscript k right then you see that you will end up with something very similar to what we have here okay. So before that fine if I do that I already see I am going something what I want but how can I interchange this when can I interchange expectation and limit here can I do that here. So we have been asking this question at many times right you are all many times we are facing with this problem of interchanging limit and expectation can I do it here and when I can do it. We know when we can do it we have seen several couple of cases where we can do it but the question is this corresponds to one of those cases. So which of this results like we know dominance converges theorem monotone converges theorem bounded converges theorem which one we can apply here bounded converges theorem. So if I apply I should be able to expect interchanges expectation and limit. So because of that I could as well write it as things right. So actually I really I really do not need to worry about that right because the expectation is already inside. So why I need to worry about interchanging them. So I think you guys did notice this. So I could directly so anyway this is a finite summation right I really do not need to worry about interchanging that. So I can already always plug in this and what I get is basically 1 by n p ij of k k equals to 1 by 2. Why this is i because I am conditioned that it is going to start from y naught equals to i. Now we know that this limit is equals to 1 upon expectation of x2. So we now are saying that this gamma j whatever we have we had earlier is nothing but that was 1 upon expectation of x2 okay. So now let us look at what is this expectation of x2. What is x2? x2 is the number of time it takes to visit my state j for the second time right. What will be its expected value? So okay what is this value of x2 can be? What values it takes? x2 takes what values? Integer value right and what is the probability that it takes some value n 1 by n. Why? So what is the meaning that x2 is equals to n. That means my Markov chain visited return to j again after exactly n rounds not before that right. We have used the notation for that. What was that? Fjjn right we are talking about starting from state j going back to that and that one so because of that this guy is going to be Fjj of n. Is this correct? The expected value of x2 and this also we have denoted as what? New jj right and what would we say? When this new jj is going to be finite we state this state j to be are you sure? When vj is finite is now we say that j is positive recurrent if vjj is less than infinity and we said that j is null if vjj is infinity. Now let us compare this quantity here. What we are actually showing is this gamma j here is nothing but 1 upon new jj right. So is this correct? This gamma j is nothing but 1 upon expectation of x2 that we have shown it to be 1 upon new jj. Now when this state j is positive recurrent I know that this new jj is finite. So this quantity has to be positive. When new jj is infinity that is when j is null recurrent then new jj is infinity then this quantity has to be 0 right. So basically what this result we used earlier is nothing but a simple consequence of our elementary renewal theorem okay. Now let us further go back one step. So what we are basically saying is the gamma j we have defined is nothing but new j. If you now recall the theorem we showed for positive recurrence right we said that my irreducible dt mc is positive recurrent if and only if pi equals to pi p. And how did we show that such a pi exist? We showed that that pi was nothing but this gamma j right when we actually showed we needed to show a pi exist such that pi equals to pi p and that pi was we have exactly took that pi to be this gamma j. So our pi j is nothing but the reciprocal of 1 by the it is just like 1 upon new jj right. So now how to interpret this? This expectation is what this expectation is about the number of rounds to return to state j right this is the expectation on that. Now I am going to return to that state often that means this expectation is going to be small right then what is this quantity is going to be pi j corresponding j large that means I am going to see that more and more right. So that probability that my Markov chain is visiting that state j is going to is going to happen with high probability the probability of that is going to be high that is because the number of times to visit that state again is going to be small. So I will be frequently visit coming back to that state quickly because of that I am going to see that state again and again many times. So the probability that so also recall that this pi j is the stationary probability that my Markov chain will be in state j right. So because of that I will if this guy is going to be too small I am going to see my Markov chain in that state j for a large fraction of time that is the probability of me seeing my Markov chain in state j is going to be high. Now so you should be able to bit more comfortable in applying these results like if you want to interpret how frequently I am going to visit a particular state and how that is going to be related to is number of mean visit to that is time you should be able to connect all these results and derive any properties any relevant properties about the Markov chain ok fine. Now let us come back to our renewal process the way we started defining our renewals process is by taking an underline discrete time Markov chain right we said that ok lets why n be a discrete time Markov chain starting in some state i and now I am interested in visit to particular state j and for that j I will construct a renewal process ok. So when we did like that the number of visits to number of slots it took to return to the same state j that was all integer valued right. But it is not necessary that we have to build a renewal process like this renewal process is about anything where you are interested in something happening again and again right. For example in the battery case we said that ok when the battery life ends battery life may end at any time right it need not be integer valued something. So one could be taking a renewal process as simply something where you have the lifetimes are such that they are identically distributed and all I am saying is identically distributed I am not saying these accents has to be integer valued they could be possibly continuous valued ok such that are or more generally I can just take anything. So instead of making this special case I will just say hence forth for simplicity I am I can take a sequence of RIID including the first one ok. So when I talked about my DTMC case I said ok I am interested in returning to state j particular j given that I started in a particular state I I could as well say why start with I I would start with j and then look at returning to that state in that case I do not need to make a separate distinguish between the first cycle and the subsequent cycle right. I can they are all returning to the same state starting from the state same state. So in that case I could just say that this lifetimes are all IID right. So I will not make a special distinguish distinction about the first cycle all cycles I am going to treat the same including the first one and in that case I will simply make X-sense and IID. Now actually we have already looked at a renewal process where this X-sense were exponentially distributed. What was that process we called? What is what was that called? Poison right. We have already looked into a poison process where inter arrival what we called basically inter count times are all exponentially distributed. And if this in if this X-sense are exponentially distributed with parameter lambda then actually we had said that this is nothing but a poison process with rate lambda right. So this is nothing but so in this process what we are saying so how did we describes our poison process basically we said that poison process is a counting process which is basically counting something and the time between two counts is exponentially distributed with parameter lambda right. So there count can happen at any time it is not has to happen at a some discrete time slots right. So in that way poison process is actually continuous time process. So for example one case we said that okay when you guys enter into the class one guy came that is the first count happened after sometime second guy came that is the second count happened and after sometime third guy came like that. So here you guys are not coming at some discrete times right we are coming at some some time. So poison process is about continuous time process and now but also it is basically counting something that are discrete and inter arrival times they are all exponentially distributed and this exponential distribution is again continuous random variables. Now the question is how to define okay so now let us we should be so this is basically poison process is also kind of a renewal process in that sense right it is basically looking at the counts as something happening again and again and it is just keeping tracks of how many counts has that has happened so far. Now let us define M of t for this that is number of the counts that has happened in the interval 0 to t. So this is as usual no this is the general thing this is for any renewal process so I can also define the same thing for my poison process right and then it is zk here which is nothing but x1 plus x2 all the way up to xk being less than or equals to t I can define this right for poison process also. So the poison process is characterized in terms of this inter arrival times and that is going to define my count instances that is zk's. So once I know that I already know I can define my renewal process like this. Now what is this basically M of t? M of t is the number of arrivals in the interval 0 t right. So for a poison process number of arrivals in the interval 0 t what was that poison right and what is the mean of that? 1 by lambda t lambda t right. So what do you expect this to be? What do you expect this to be? Lambda t right. So this is by connecting so if I have defined a and this is a general process I have defined right I can define this anything but if you are going to do this on a poison process you are going to get this. Now and this is by the argument saying that this is just by the interpretation okay M of t is nothing but the number of arrivals or counts in the interval 0 t and I know that is poison distributed with rate lambda t and that has to have a mean bit but you can also using this notion that zk is a sum of this you should be able to again derive the same thing okay you have to check that. Another thing for a poison process let us say these are my lifetimes right. With respect to this process lifetime process I can define a stopping time that stopping time is M of t. Let us sorry I am going to not stopping time I am going to define a random time M of t on x. So M of t is what basically M of t is always integer valued right even though it is defined for every t but M of t is giving you integer valued numbers and this is all this is all itself is a random quantity right. So now I am saying let us take this sequence and define a random time on that okay. Now I want to ask the question is M of t is a stopping time on this sequence is this random time is a stopping time on the sequence x i's or x n's. So suppose if I if I know that okay basically what I know to know if I want to answer the question is M of t is less than or equals to n is it enough to know x 1 x 2 x n is this true. So suppose let us say you have been given x 1 x 2 all the way up to x n. So if I if you know x 1 x 2 all the way up to x n and you have been also given t okay if M of t is less than or equals to n what I know this sum has to be less than or equals to t right but does it also say you that the n plus 1 lifetime has not happened within this first in within this t yeah but all we know it is x 1 plus x 2 is going to be less than or equals to t. So what is this because so this is nothing but if I sum all of them x i is going to be is going to be what is this is going to be n right this I know as z n and I know that z n has to be less than if M of t is less than or equals to n I know z n has to be less than or equals to t that we know z n plus 1 but you do not know anything about you have been not told at x n plus 1 as this x 1 plus 1 if you add to this that will go beyond t or it is still going to remain within t maybe or may not be right. So because of this without knowing x n plus 1 I cannot say that m t is going to be less than or equals to n is this clear that is what we want to answer whether m t is going to be less than or equals to n the question is can we answer this based on this information sorry I want to ask this question whether M of t is equals to n can I answer this question completely. So it depends unless I know that the x n plus 1 that is going to happen has to happen after type t that that this sum if I add x n 1 x n plus 1 to this sum that is going to be larger than t I cannot say this right. So if the only if happens that guy after adding that if that guy exceeds t then I know that till n things have been incorporated n plus 1 is going to stay outside okay. So because of this we cannot really say that this is going to be a m of t is going to be a stopping time with respect to m with my sequence x n okay stopping time okay just a minute I am I think we can also argue that. So let us try to make this more formal right okay so you are right so to make to apply for that stopping time we have to just not worry about equality we have to answer this question okay. So let us try to answer this okay what we can do is is this correct. So I am saying that any time t m of t has to either less than or equals to n or it has to be greater than or equals to n plus 1 this indeed one of this must be true and this should be equals to 1. Now I am going to write this bit n plus 1 and what is this going to be 1 minus m of t is going to be greater than or equals to n plus 1 right 1 minus of this what is this is going to be so this is going to be basically we have already said that if m of t has to be greater than or equals to n plus 1 in terms of z n plus 1 what we know we know that this guy has to be z n plus 1 has to be less than or equals to t we have argued that these two are the same events right. So this both must be equal okay okay now let us see this z n plus 1 is nothing but summation of first n events right it is correct I have just applied the definition of let us say n plus 1 here now if I want to look at this 1 minus of this this is basically 1 minus of this means the complement of this event right what is the complement of this event the complement of this event summation of i equals to 1 to n plus 1 xi is greater than or equals to t is this correct now the question that one the question that is m of t is less than or equals to n has boiled down to asking the question whether the sum of the first n plus way values has exceeded t right you see that so what we want to know that n arrivals has happened in the within the interval 0 t to ensure that I need to be confident that the nth plus 1 has not completely occurred within the interval 0 t right it must have ended after time t so that is what exactly it is capturing this is going to be the same as asking the question whether the first n plus 1 xi's their sum is going to exceed t and so answering this question is equivalent to this and to answer these questions how many xi's I need to know n plus 1 right so just knowing n random variables is not enough for me here so you can also just like that think into delay now that we have written this if you want to know that within my interval 0 to t less than or equals to n might have happened and no more that must be the case that n complete renewals might have must have been completed in that and n plus 1 must not have been completed within the interval 0 to t had n plus 1 has also been completed in the interval 0 to t then m t would have taken the value n plus 1 right it would have been greater than n so if m of t has to less than or equals to n it must be the case that n plus 1 renewal might have completed in the after time t so that is exactly it is capturing so because of that m t is not stopping now so you somehow feel by this information that if I want to know that n arrivals or renewals has happened in the in the duration 0 to t I need to know not n variables but actually n plus 1 n plus 1 renewal lives right or n plus 1 x i's so it so turns out that m of t is not a stopping time but if you just look at m of t plus 1 is a stopping time why is that this is also actually obvious so if I want to check whether m of t plus 1 is a stopping time replace m of t by m of t plus 1 right that is basically asking the question whether m of t is upper bounded by n minus 1 so in that case I have to replace n by n minus 1 so in this case it just summation happens to be i equals to 1 to only n so in that case it only depends on the n random variables but again this is the argument but convince yourself that if I want to check whether m t plus 1 is a stopping time that is the case.