 So, in the last class, we talked about how to, we basically discussed the conditions that are transient and the recurrent states need to satisfy. So, basically we said that how to classify a irreducible discrete time Markov chain into positive recurrent, basically we said how to classify it as recurrent and transient. So, we basically said that if we have a Q matrix which is basically derived from transition probability matrix by eliminating one particular state and then we looked at equation of the form y equals to Q y and what would we say if the solution of this y equals to Q y such that if y is 0 then what would we say? So, if the solution then we said it is going to be recurrent, right. So, and also we had said that the y equals to Q y equation will have only two will be solution will be such that the y will be either all 0 vector or it will be such that supremum over yi will be equals to 1. So, these two solutions are possible when I look at y equals to Q y, but among the two solutions if I get y equals to 0 as the solution then I said we said that my reducible DTMC is recurrent otherwise it is transient. And when I know it recurrent I know further how to classify it as positive recurrent or null recurrent, right. How I do that? I look for a solution of the form pi equals to pi p if I can find a solution which is a probability vector and it has all stick positive values in it then I know it is going to be a positive recurrent otherwise it is it has to be null recurrent. So, today we are going to apply these results on a simple queuing model. So, let us consider a Q and let us say I have this periodic times here T0, T1, T2 like that and Tn. So, in each of this time slot let us assume that there is arrives a one customer with certain probability and also when a customer arrives if there are already multiple guys in this queue is just going to join the queue and when there are multiple people in the queue the guy who is in the front is going to get served. And when we will assume that in each slot one guy is going to get served and going to leave the queue. So, let us say that in each round n one guy arrive with probability lambda and there is no guy with probability and similarly I am going to say that dn equals to 1 with probability mu and 0 probability 1 plus mu. So, I am saying that. So, if you are in round Tn one going to arrive in this round is going to be probabilistic one guy arrives with probability 1, one guy arrives with probability lambda otherwise no guy arrives. Similarly, let us say there are already some guys in round del n one guy will be getting served in the server. He will complete his service and leave in the nth round with probability mu. Naturally, if there is no guys in the queue then the server is not going to serve anybody. So, we are going to assume there is a guy departing the queue only when somebody is being served or when the queue is going to be non-empty. So, in a way this is going to capture our arrival. So, this lambda is going to capture our arrival rate. So, this lambda n basically defines our arrival process in a way and then this mu i capture our. So, it is basically service, but I am adding the word virtual because if there is nobody in the queue I am going to assume there is no need to serve anybody only when there is somebody then I need to serve. So, I am just to account for this fact I am just going to call it as virtual service rate as mu. This is basically arrivals in round n this alpha n characterizes whether there is going to be arrival or not. We are going to say this is the arrival in random variable. So, arrivals are random and how that is defined? It is defined like this. It is going to arrive with probability lambda or otherwise no. So, the sequence of alpha n basically defines your arrival process. Similarly, alpha n defines your sequence of virtual service process. So, if you are going to take one slot let us say between t 1 here, we are going to say that whatever that happens the arrival here and departure in this we are going to take it as this is going to be characterized like this. So, we will try to make it more precise when we are going to write the state diagrams here. At any time slot t n we are going to denote xn as number of customers in queue. So, suppose let us say you are interested in this slot tn at the beginning of tn I am going to make it actually tn plus here. What does this mean? Just at the start of tn how many customers are there in the system you are going to denote it as xn here. Now, let us come back to this. So, one guy can arrive in this with probability alpha n. Next guy can arrive in this with probability alpha n like this. So, if and this is going to be with the same probability this lambdas are constant. The probability that a person arrives in a one slot is going to be fixed that is going to happen with probability lambda. Now, this is a Bernoulli random variable in each round somebody arriving and the entire process is like a Bernoulli process. Now, if I am interested in arrival between two customers let us take any instant when a customer arrived and the next instance or the number of slots after which the next customer arrives. How that is going to be distributed? Exponentially it is going to be what? Suppose let us say at some point one guy has come in the next slot one more guy can come with probability lambda or he may not come with 1 minus lambda. After that if he does not come he may come in the slot following that with probability lambda. If he does not come he may come in the two slots after that. So, if I am going to say that if I want let us say if I am going to denote capital T as a random variable that denotes arrival time between two customers how that T will be distributed as? It is going to be geometric distributed with what probability with what parameter? It is going to be still lambda and what about if I am going to say let us say now let us say T is the time slot between departure of two customers from my queue. How that T is going to be distributed as? Again geometric with what parameter? It is going to be distributed with mu, right? Let us say at the beginning of just at the start of Tn plus this is denotes how many customers are there in my queue and then how can I write this process now? I can write this process as Xn plus. So, one thing is definitely depends on how many people are there in the previous slot, right? And in this if the next slot if our guy is departing then I am going to remove one from this but I am going to add a plus here plus alpha n plus 1. So, if suppose let us say at any point my Xn is 0 that means there are no waiting there are no customers in the queue. Then even though I have written delta as the probability that somebody leaves with probability mu then actually there is nobody living there, right? Even if it is one living somebody then 0 minus 1 is minus 1 I will just by plus will make it 0. So, it will basically take the positive when it will any negative value it will truncate to 0, okay? So, only when this Xn is going to be get at least one then somebody can depart and then this can become 0, right? So, that is the meaning of when I said virtual because I can only serve somebody when there is somebody in the, right? Delta n plus 1 is 0 by our interpretation but the way we have written it. So, when I wrote this delta n process I did not care about whether there were some people in the queue or not I just thought about like when somebody going from this is going to happen this but somebody there is there somebody really to account for that I have to add this part only when this where it can be a departure that is defined by this process only when Xn is at least one, okay? So, the way to think about this is normally Xn, okay? If this quantity happens to be negative then I will just take it as a 0, okay? Okay, fine. So, what is this is saying in the n plus 1th round whatever if whatever I have in the Xnth round if somebody is going to leave then I will be reduced and if something is arriving then I am going to get increased. So, this is what captures the dynamics of this number of customers in the queue, is this fine? Now the question is if I have a sequence of random variables like this which have defined in this fashion does this sequence forms a DTMC? So, if I tell you what is already Xn you know what is the value possible values Xn plus 1 is taking, right? Suppose let us say Xn at any point n is i what are the possible value of Xn plus 1? So, it can be go one below let us assume this i is strictly at least one there is some one guy at least in the queue then it is going to be i minus 1, right? And can you calculate is it possible for you to calculate what is the probability that it will go to i minus 1 and or i or i plus 1, right? It can be computed based on the values of this lambda and mu. And when this i equals to 0 here what is the possible values of Xn plus 1? It can be either 0 or 1, right? So, it cannot go negative here it cannot be minus 1 that is the reason we have put plus 1 here to account for that case, okay? So, okay let us take when I said Xn plus 1, right? This is at the beginning of this instance and now how I am counting whatever Xn plus 1 whatever the value I had here and this delta n plus 1 this is the arrival that has happened in the n plus 1th round, right? That can happen anywhere here in the n plus 1th round and what about this delta n plus 1 where this departure I am actually counting? This is now our convention basically I can just take it this departure all that that departure happened just before Pn plus 1 because this we have to this we can take as our convention. So, this alpha n plus 1 can happen anywhere here but I can take this departure that has happened just before I started my this lot here, okay? Okay then I do not worry about what is the departure happening in this lot I will just take all the departure that has happened before just beginning of my Tn plus 1 slot here, okay? Okay now let us try to understand how does the states of this Markov chain look like? So, what are the possible states of this Markov chain? So, suppose let us say at this point I have Xn now when I go to this point I can take into account all the departures and arrivals that has happened in the entire interval, right? So, and that I want to get affected when I am going to count it at Xn plus 1. So, if I want to get counted, okay the way we have defined delta n plus 1 this is just all the departure as that has happened just before this that departure we have taken into account in Xn plus 1 but what about okay let us see if I do that then only it gets captured when I measured at the just beginning of Tn plus 1, right? Okay so let us write this then so Tn plus 1 is a arrival between and I am going to write it as alpha n plus 1. So, this is a departure, right? And also this is going to be arrival again this is going to be between Tn and I have excluded Tn plus 1 slot in that, okay? So, now X my state space S is going to be 0, 1, 2, all the way up to this. Now what is going to be the transition probability matrix of this Markov chain? How it is going to look like? So, instead of transition probability matrix let us try to draw transition diagram. So, my states are 0, 1, 2 let us say n and n plus 1 and then it continues. From state 0 what all the possible states I can jump to? I can go here and I can remain here. What are these probabilities? So, when I am 0 I can remain at 0 if no arrival happens, right? What is the probability of no arrival? 1 minus lambda and this is going to be lambda here. And now when I am at state 1 what are the possible transitions? What is the probability I go from 1 to 2? So, it should be the case that a new arrival happens and nobody departures, right? And then what is the probability that I go from 1 to 0 there is a departure and no arrival. And is it possible I can stay in the state 1 itself and what is that probability? So, similarly you can fill all these, okay? So, just to complete what is this going to be? So, let us take it what are the pros? So, from n minus 1 I can come to n what is this probability is going to be? It is going to be lambda into 1 minus mu and what is this going to be? Mu into 1 minus lambda and this self-loop probability is going to be again going to be this. So, we have this description, we have we know it is a Markov chain, it has this transition diagram. Now we want to see whether it is irreducible and in that case what type of class it is? Is this an irreducible Markov chain? So, can you can you reach any state from any state? Okay irreducible Markov. Now what about can you say anything about whether it is a positive recurrent, null recurrent or transient now? So, if you want to claim it is positive recurrent, how you are going to do that? So, one possibility is if your initial interest is just want to see whether it is a transient or recurrent, one thing you can do is you can pick up a state i and try to see if f i i equals to 1 or strictly less than 1. Is that f i i's are easy to compute in this case? Okay let us instead of taking that I mean of course you can compute f i i always but with lot of taking into account all the possible combinations of how I can reach i again after certain number of rounds but instead of that let us try to use our results that we have stated and proved. So, if you want to argue that irreducible DTMC is going to be positive recurrent, what is you need to verify? So, you want to check whether so check for positive recurrence okay you want to check whether there exist pi equals to pi p such that summation of pi equals to 1 and pi i equals to 0 okay. Let us verify whether this I can find such a pi for this transition matrix P the P is given by this transition diagram here okay. So, this P is given by the transition diagram now let us try to see whether I can compute this pi here and then check whether it satisfies this property. So, this pi equals to pi p means we have many equations here right let us try to write each of those equations. So, let us write the first in this what is pi 0 is going to be? So, what is the first equation is going to be? So, if you look into this what will the first row consist of? The first row will be 1 minus lambda and lambda right. So, what is then you expect this to be into lambda now what is your P is going to look like let us write it for something. So, 0 to 0 is 1 minus lambda and all 0s right and then what will be second row is going to look like okay. So, this complicated thing right let us call me I am just writing that complicated things by dash dash and what is the next thing lambda into? So, what will be the first equation when I am going to write? So, pi 0 is going to be my pi vector multiplied by this column vector right. So, what is that in that case it is going to be pi 0 into this and pi 1 into this quantity right. I have this relation which I will going to write it and after simplification you can see that if I do like this I get it as pi 0 into 1 minus mu into lambda by mu into 1 minus mu divided by 1 minus lambda just verify I have just whatever that equation there I have simplified it okay. Now, let us write the relation for pi 1. So, what is the pi 1 relation is going to look like? The pi 1 relation is now going to multiply the second column here right. So, that is going to be at least lambda into pi 0 plus that the whole thing 1 minus mu into 1 minus lambda minus lambda into 1 minus mu to pi 1 and what is this term here can anybody say is this mu into 1 minus lambda. So, what is this I want to go from second state to first state right second state to first state this is going to be lambda 1 minus mu oh sorry I want to go this one quantity right mu into 1 minus and I know after this it is all going to be 0. So, this is going to be pi 2 into mu into 1 minus lambda okay. Now, I want to plug in whatever the quantity I have for pi 1 here and write the expression for pi 2 okay I can do that right like you can again verify that after simplifying I am going to get it as pi 2 as pi 1 into lambda by mu into 1 minus mu 1 minus lambda here and as you continue to do this you will see that pi n can be written as pi n minus 1 times lambda minus mu and 1 minus mu 1 minus lambda okay. I have just simplified that you can verify all these things okay. So, now good I have this then I will do a recursion on pi n minus 1 like that I can do like this and then actually after doing this what I can get everything in terms of pi 0 into lambda by n to the power n and 1 minus mu 1 minus lambda to the power n. Is this fine? So, now actually this is going to be n minus 1 here and after this is n, but what I will exactly get is I will have 1 other 1 minus mu here because of this guy 1 minus mu here okay. So, if you just do this recursion you have this pi naught which is in this format okay pi n for all n I have written in terms of pi 0. So, I will get this equation for all n greater than or equals to 1 right. Now, I will look for this I also need to see whether. So, to find this here my pi naught is my free variable right. Once I know my pi naught I will get all the pi n's for n greater than or equals to 1 right. So, to get this pi naught what I can do I know this summation pi of n is going to be 0 right. So, this is starting from 0 right. So, now I have pi 0 plus all these quantities. So, n equals to 1 to infinity I am going to pull this pi naught out this is going to be 1 minus mu times lambda n to the power n into 1 minus mu 1 minus lambda to the power n I will just simplify the things and this I want this is equals to 1 okay. Now, let us if you simplify this quantities what you find is pi naught plus 1 upon 1 minus mu. So, if you this whole quantity if you are going to. So, let us focus on this whole quantity now what I will do is I have pulled out this 1 minus mu outside and what I will do is I will start from n equals to 0 to infinity and l m lambda minus mu and 1 minus mu 1 minus lambda n and then I am going to subtract 1 from this. So, why I did so here notice that n was running from n 1 to infinity right. Now, I have letting n go from 0 to infinity for n equals to 0 I have got 1. So, to eradicate that I have subtracted minus 1 from here okay. Is this clear how did I do this just like manipulation you can work out yourself okay. Now, let us see I want this to be equals to 1 when I can find a when I can find when the solution here makes sense for pi naught okay. Let us see suppose now right now I have taken lambda mu to be something some probabilities right which are between 0 and 1. Now, let us assume that lambda is strictly less than mu okay. So, if lambda is strictly less than mu lambda by mu is going to be less than 1 right and also 1 minus mu 1 minus lambda is also going to be less than 1. So, this product is going to be less than 1 okay. So, I have now I have a geometric series where the ratio is less than 1 right strictly less than 1. So, this guy converge right. So, for this case I can write this after simplification as. So, this simplifies a lot and just turns out to be 1 upon mu minus lambda equals to 1 because I started from n equals to 0 right. So, I have this if you just going to simplify this what you are going to get you get pi naught equals to this lambda. So, you are going to get mu mu divided by mu minus lambda. So, what you are going to get is 1 minus lambda by mu. So, suppose if that is the case lambda is less than mu you will end up with a solution pi naught which is equals to this. Now, we have a complete solution under this assumption pi naught lambda by mu is going to be less than 1. So, this is a pi naught is a quantity which is less than 1 and also positive and now if you go back and plug this pi naught here I will have all the pi n's which are strictly positive and I have ensured that those pi n's add up to 1 right. So, when this lambda is less than mu I have been able to find a relation pi equals to pi p which satisfies this condition. So, now can I claim that when lambda is going to be less than mu my irreducible DTMC is a positive recurrent right that is what my theorem said. Now, let us take the case when lambda is going to be greater than or equals to mu. When lambda is going to be greater than or equals to mu what happens to this quantity? So, lambda by mu is going to be greater than 1 this ratio is also going to be greater than 1. So, this whole quantity diverges. So, because of this the only possible solution for pi naught is going to be 0 and if you plug in this 0 all my pi n's are going to be 0. So, I will not end up with a solution here. So, the solution in that case happens to be pi equals to pi p happens to be 0 in this case right when I have lambda. So, somebody other day I was asking right when is it possible the only possible solution to pi equals to pi p can be 0. So, here you have a situation when lambda happens to be greater than mu I will end up with this. So, in that case it is not going to be positive recurrent right. So, so, now recall that the our result says that it is positive recurrent if and only if this condition happens in the case where lambda is greater than or equals to mu this condition does not happen. So, it cannot be positive recurrent. So, so maybe when lambda is going to be greater than or equals to mu either it has to be transient or null recurrent right. Now, how to verify that it is transient or null recurrent?