 So, in the last two lectures what we have we are going to cover something called a renewal theory. So, this renewal theory as we as we will see that it is nothing but a slight more generalization of what the things we have already studied ok, but it kinds of captures the essence of many applications we seen practice and also justifies how we can model them using some of the tools like Markov chains and also what renewal theory we are going to study well ok. So, in a way we have already touched upon renewal theory when we try to study Poisson processes right, you remember when we talked about Poisson process we have a counting function which kind of try to count when the things happened and one can usually think of a count happening as basically something happening again. For example, let us say you are building a system or you are interested in a particular component of a system. Let us say specifically you are interested in the battery of some system, when you fully charge it and put it in the machine it is going to operate for some time and after that maybe it get discharged and some red light in that blinks and then you know that this guy has gone below now it is not in good shape either you need to recharge it completely or you need to replace it. If you do it then it is going to work for some time and again maybe the red light blinks and now it is time for it to replace. Now you may be interested in knowing how long this guy is going to operate in the long term. So, what will be the average lifetime of this battery that is why such questions you will be interested and here renewing is like when you have to renew the battery or when you have to replace the battery. So, such questions can arise in many cases not just like you are replacing the component it could be even thought of like when some new event happened you can think of something has happened when again the next event happened you can think that as the event has renewed and then see based on this you can try to analyze what is the rate of these things renewing what is the inter arrival time between the renewals and all these things. So, let us try to make the things formal now okay let us say I have a DTMP YN now based on this I am going to define let us say if you are going to start from a particular state Y equals to I you are going to start it from a particular state I now I can define X1 to be first visit to state I and I can also like this can also like the kth visit to so let us say some I first visit instead of that let us say let us take some other state I start from I and now I will be interested when is the first time I go to state J and then kth return to state J after that when I come to state J after that I will be looking at from J again when I go back to J for the second time and then once I hit J again when I will go back to J for the third time like that. So, this XK here denotes the kth return to state J okay now what we know probability that X1 equals to M given that Y naught equals to I if we have denoted it as FIJ to superscript M right starting from state I when I am going to hit J for the first time so this is the time of see what I am writing right now X1 is now the time taken to visit your state J again for the first time okay so when you are starting in state I you have taken let us say I am asking what is the probability that you take when I write X1 equals to M right that means you have taken M rounds to hit state J for the first time that is the meaning of tie so this should be I should be adding here time so X1 XK here I denoting the time when you start from state I how much time you are going to take to hit state J for the first time that is X1 and once you hit state J when you are going to again hit state J next time that is X2 like that so the times are denoted so is this now clear that this term is exactly equals to this and after this if I am going to say let us take any K equals to M given Y naught equals to what is this going to be so when I say let us say when I say Y naught equals to I right so I have started with something when I said the kth starting so I am going to hit to hit state J for the kth time that is the number of slots I took to return to state J is exactly to return to J for the kth time in M number of rounds what is this can be or like here I really do not need to condition upon this right because when I say XM equals to M I am asking for returning to state J for the kth time in M rounds that means before that I would have already hit state J and from state that J and here let us assume this K is greater than 2 for time being just K equals to 2 I am asking X2 equals to M so it is then going to be FJJ to the power M superscript M right because you already know it has hit state J and from there you want to go to the next state J again in M rounds and now is it does it depend on what K is or this is going to be a same for all K it is going to remain the same right once you hit state J I mean fine everything in the past I can forget and from there I want to see in how many rounds I again go to state J in M number of rounds and this is going to be that problem so because we are already saying I am going to hit take M rounds to hit state J for the kth round suppose K is going to be greater than A we are already saying K you are asking for the kth return that means it should have previously returned right otherwise K being greater than or equals to 2 does not make sense so that is why we have written these two cases separately here okay now suppose let us consider this case X1 equals to let us say M1 X2 equals to M2 and all the way up to let us say Xn equals to Mn given why not equals to R what I am asking I am asking starting from state I in the first in the initial round what is the probability that you take M1 rounds to hit state J for the first time after doing that you take another M2 rounds to again return to state J and then again like this and in the you are going to take Mn rounds to hit state J for the nth time so I can always write this as you can expand this and you can keep on doing this right so when I so this is why not equals to I comma so when I say why not equals to I and X1 equals to M1 what does this mean I started with state I in the 0th round then I hit state J for the first time in M1 rounds and then I am asking from there what is the probability that you are going to hit again state J in M2 rounds so then this is does not matter here right because this already telling that you have hit state J again so can I let us try to write this what is this quantity is going to be you have already written this and what is this it is simply going to be M2 right and here it is M1 because we are asking you took exactly M2 number of rounds to hit state J again given that you have already written to state J now like this you can keep on doing this iteration and you will end up it is going to do this so now if you look into this basically this joint distribution has split as this this probability this joint probability has split into this individual so each time is like here a probability right as we have already expressed here so now if I am going to and you see that this joint distribution has split into the individual probabilities what does this mean so this means X1, X2, XNs are all independent right and further if you forget the first one look at the others they have the same FJJ term what does that mean they are not only split the probability not only split but each of the term looks identical that means the distribution of each of this term is same rate they are all FJJ right so that means definitely this sequence if I am going to look at this process what is this process this is a process of intervisit times sorry what is this so this is the duration of the returning duration of the returning to the same states right these are all independent however if you look instead of XN starting from n greater than or equals to 2 if you look at n greater than or equals to 2 are this process what is this process is it is not only independent now if you look at only the sequence after n greater than or equals to 2 they are also further identical so they are IID this process is IID and this set of sequences are independent okay what we are going to call this X henceforth as lifetimes okay we call this as a lifetime process or we are going to call it as simply renewal lifetimes is this clear why we are calling this renewal lifetimes my process here in this case my process I am what I am interested in returning back to state J and I am looking at when I am returning I am looking at how much duration I took to return in that in the parlance of the battery example we talked about when I replace a battery and I can think of when the battery dies I am going to replace it so from starting point till the time it dies that is the time duration of that I can think at it as a lifetime of that battery right so that is why we are going to call this as renewal lifetimes okay so X1 is going to tell me what is the duration what is the lifetime of my battery when I first recharge it X2 will denote it okay once my battery was dead when I recharge it again how much time it lasted before it died again so in that way these are going to talk about the lifetimes of the renewals and now we are also going to denote ZK here I am going to define ZK as time of kth visit to state J I am hitting state interested in returning to state J again and again but I may be returning to state J first time or the second time or the third time right I am also trying to keep track of how many times I have returned to state J so JK is ZK is telling me exactly at what time I return to state J for the kth time okay for example in the battery case ZK can denote when is the when is that my battery died for the kth time right so every time it dies I recharge it and when is that it dies for the kth time okay now is there a relation between ZK's and XK's ZK equals to sigma XR I took here right so X1 denoted the time it took for the battery to die then X2 so basically X1 told me after how much time I replace the battery X2 told me next time when I replace my battery so if you just add all of them you will get time K when you ZK time ZK when you replace your battery for the kth time why equals it is a random process they are identical but their values their realizations could be different right so here what this X1 X2 X3 they are all random variables yes it is true that X1 has the different distribution than X2 X3 and others but if you take a realization maybe X the sample maybe the battery so when you started for the first time the battery let us say lasted for one week and died and then you recharged it and then what happened so so so okay let us say so let us say you have a some battery a prayer you do not know in what state it is it may be half charge maybe quarterly charge 3 4 charge whatever it is given to you and you started using it so it is going to die at some time right and once it dies you are going to completely recharge it and again look at when it is going to die again okay so clearly the first part can have a different distribution from the second and the subsequent ones right because when you initially started it the battery was in some arbitrary state but when you are going to recharge it you are always recharging it when the battery completely died and now if you even when the battery dies when you recharge it maybe it last first time for the three weeks when it dies you again recharge it maybe it again this time it went for four weeks and again you recharge it maybe this time it just lasted for one week there because this is a realization I am not talking about exact this the mean value or something right this X1 X2 are random variables right so they can take different realizations so even the X1 X2 they have identical distribution when you take a sample so let us say X1 and X2 they have the same distribution take one sample from X1 and take another sample from X2 do they need to be the same value that is what I am saying okay now based on this notations let us define what we mean by a renewal process now and the variables with Xn being then t clearly distributed we are going to just call this is event of sequence of lifetimes and we have already defined for k equals to 1 that k equals to kth renewal distance t so we have already see we have already motivated this let us suppose we have a sequence of random variables let us take them to be independent but let us take the sequence starting from n greater than or equals to be identical then we such a process we have when there is such a sequence we are going to call as sequence of lifetimes and the Zk defined in this fashion which is the sum of the first k Xis we are going to define it as the renewal instance so Zk denotes the kth renewal instance and now you take a t any real number t now you define M of t capital M of t to be supremum over k greater than or equals to 1 such that Zk is going to be less than or equals to t so let us understand what is this function is saying when you give a t it looks at all the instances when the renewal happened within the time t and takes the maximum value of that k so let us say so it is you see that Z this guy Zk is a increasing right like maybe the first renewal happens at of one week second renewal happens in the third week and the fourth renewal happens in the 10th week like that and let us say you are given some hundred today so you are going to take that as hundred today t 100 and look at all the renewals that has happened before that 100 day and see what is the latest renewal in that so if you let us say 100 day you have taken and 100 day corresponds to how many weeks let us say whatever some let us say on the 10th week or renewal happened and the next renewal happened only in the 20th week okay so then what is this M of t is going to be it is going to be so let us say let us say t equals to 100 day I am counting them here t could be continuous but let us take t to be 100 and let us say my Z 10th happened in on 89th day and and the 11th happened on 95th day and let us say and the 12th happened on 110th day so in that k what is M of t is going to be it is going to be 11 right so it will not include this because 110 is going to be larger than 100 in this case so basically what this is telling is there are number of renewals that has happened in the interval 0 to t right and now this small M of t is basically looking at the expected value of this M of t so notice that this M of t here is a random quantity and how to interpret this M of t so if you give me a realization of Z of k then I can define my M of t on that so like I have given you one realization right this realization could change and on that realization this M of t is defined and this can depending on the realization this value of M of t can also change so that is why this is a random quantity and now I will be take on this quantity as the expectation here and this M of small M of t I am going to call it as renewable functions ok. Now let us try to understand each of this now we have defined so many processes right ok anyway I started with xn based on that I have defined Z k and based on that I have defined M t and further small M of t so how do these functions look like so what kind of random variable xn is xn is discrete time but what is the value that xn is take so all these xn are defined on a given yn right you started with the dtmcyn ok so let us say dtmcyn on space some s let us say this s is countable infinity ok but still this Markov chain is discrete time ok now what we are doing is you are fixing a particular state and then you are looking at the time it takes to visit that state ok and again return to that state g. So then in that case what is the value that xn takes but this time is in terms of the discrete value right so it is going to so let us say it this xn is says ok I took 100 slots to visit state j again or I took 20 slots to visit state j again right because this now because this yn is discrete time the number of time slots you took to go back to the state again is again in terms of this discrete counts right so because of this this xn are again discrete valued random variables so like this xn can take 10, 20, 30, 31, 32 whatever but not like any value between 31 and 32 they are basically giving the count of how many slots you took to return to the state again so that is why this xn are discrete valued now our Markov chain is changing only at this discrete points right so it change in slot yeah so my Markov chain is changing but our count have we see the Markov chain only at the beginning let us say at the beginning of the slot or like we are looking it into the days day 1 this is the value day 2 this is the value and day 3 this is the value and we are looking at how many days it took to go back from this particular state to this state so because of that it is going to be discrete and now if this xn are discrete what about zk's they are also going to be discrete so this is but they are discrete time and discrete valued random variables ok so so our xn so the xn are also they are also discrete time and discrete valued random variables what about mk mt is it discrete in time or continuous in time zk's are discrete but now I am passing to this m any real number and then asking in the interval 0 to t how many renewals happened right so mt is continuous time right but it is taking discrete values ok and anyway mt is simply a real number at time t because this is expected value so for every t it is defined ok so what is the meaning of m of t this is the expected number of renewals that are going to happen in the interval 0 to t ok so is m of t increasing function if I increase t is the value of m of t increases right this is obvious because this capital M of t is itself increasing if you increase t more renewals will be included in this and m of t is going to take a larger value ok so we just saw that so m of t is non decreasing and this is because yeah and what about m of t is is non decreasing and see that I have asked z of k to be less than or equal to that means I have included t in this not the renewal just happens to key before t I am asking for the renewal that has happened till time t that includes t ok so because of this this function m of t is going to be right continuous ok so you recall this right continuity also happened when we looked at our CDF right so there we looked at probability that x is less than r equal to so because of that inequality thing that was turning out to be right and so so here also you can see similar things so non decreasing and right continuous now the question is I said a battery being renewed every time right so suppose instead of battery let us think of I am counting some events so when an event happened and then when the same event happens again I am going to take the time between these two so it may happen that sometimes instead of one event happening two events happen simultaneously so for example I can think of something like ok I am going to treat somebody entering let us say I have a one big queue that is being served and the jobs are coming into my queue so jobs could be simple customers whom I need to send sell tickets whatever so a customer can enter or it may happen that a customer himself is not entering he is entering with his family so it is not one person maybe a couple is entering so there are actually two guys entering the system and they are entering simultaneously right but how I am going to count this both are entered simultaneously I am going to treat them as two events actually I am going to give two events but going to take them that has happened the simultaneously that means suppose one guy came and with her her relative also came together let us say that guy coming was the K event and her relative is the K plus one event but the difference between them is 0 ok so because of that it may happen that when both of them happen simultaneously ZK is going to be same as ZK plus one and also notice that M of ZK plus one is equals to M of ZK because both ZK and ZK plus one so ok when I what I mean here when I wrote M of t right instead of t now I have put ZK plus one here ZK is also some number right some some basically time ok yeah so this is also going to be some time and now I am asking this function taking the value of this M at that particular time now that both of them has happened at the same time its value will be what what will going to be the value of M ZK plus one so here replace this t by ZK plus one what is going to be this value K or K plus one it is going to be K plus one right because both ZK and ZK plus one will be included in this because they have the same time so because of this these two values are going to be the same but however if you look at M ZK plus one and M ZK just before when I write ZK minus right just before the K that I will happen so what is this value we expect to be it is going to be 2 right because 2 count has happened so ok so just to understand this let us realize this ok for time being let us take this Z my Z is my ZK process right M of yeah no because ZK plus will also get included right because it will also has the same time so let us say one let us take one sample I am going to take for some particular omega ok so this is a discrete time right so let us talk so let us say K equals to 1, K equals to 2, 3, 4 and 5, 6 so let us say first arrival happened at some time so I will get some value like this the second value happens a bit later than this third value is higher than this but let us say the fourth value happens just before this and then this fifth value happens at the same as this ok let us call this this is exactly equals to x 1 let me call this this is some realization right let me just call it t 1 this is t 2 and this is like t 3 let us say t 4 and this is also t 4 it is also going to remain t 4 only ok. Now if you are going to draw its M function so I have given you one realization of my renewal instances based on this can we construct how this M of t function look like. So till t 1 let us take time to be till this point t 1 so this t 1 is going to be same as this t 1 so till t 1 what is no z does have no renewal has happened right the first renewal is happening only at the t 1 so before that there is nothing here right because but then in that case it is a supreme of an empty set because there is nothing here yeah we will include but just before t 1 ok right so just let us consider the case just before t 1 so just before t 1 there are no renewals. So let us define that come to be 0 so all this function is going to remain 0 till this point and what happens exactly at t 1 it is going to be 1 and now let us take this value t 2 let us say t 2 is somewhere here what is this way how this function is going to look till the point t 2 it is going to remain horizontal right and what happens at t 2 goes to 2 and then what happens at till t 3 horizontal and again it jumps to 3 and then what happens at t 4 t 4 till this point and how much will be the jump here at t 4 how much will be the jump it is going to be 2 units right because 2 things have happened simultaneously so that is why we wrote that k plus 1 so at this point just before this z k minus 1 in this case is just before this and it is going to include 2 events ok.