 So, what conditions we know to verify it is a positive, it is a transient or null recurrent. So, now that under lambda is going to be greater than or equals to mu, I know that my Markov chain is cannot be positive recurrent. It has to be either transient or null recurrent, but I know a condition if I somehow know that my DTMS is transient that is enough for me, right. If I know it is transient, if it cannot be transient, it is null recurrent. If it is transient, it cannot be null recurrent, right. So, let us try to verify my Markov chain is going to be transient or even recurrent. I know if it is a recurrent, then lambda is going to be greater than or equals to mu, it has to be necessarily null recurrent, right. It cannot be positive recurrent. So, what is the result I know in that case? So, we already know that if I have a reduced transition probability matrix Q and if I can if I have a y equals to Qy solution, if that y happens to be 0 vector, then I know it is going to be recurrent, right. So, maybe let us try to apply that. So, in this case if I just know if it is recurrent, it is enough, right. If it is no recurrent, it has to be null recurrent and if it is not recurrent, I already know it has to be transient. So, now consider Q to be TPM restricted on 1, 2, like this. So, I have just excluded state 0 from my state space and now I have my Q to be restricted to these sets here and now let us try to look at the solution of Q equals to Qi. And if my y equals to Qi solution happens to be here 0, then I know it is going to be recurrent, right. So, let us verify whether that is true. So, in this radio transition probability matrix, so again just for notation, I am going to write this P to denote lambda into 1 minus mu and Q to denote mu into 1 minus lambda because these terms come to us many times. So, just write them as P and Q. Now, if I want to write the solution for this y equals to Qi, what I am going to do, what I will get is simply, I am just going to write a simplified version, you can verify this and like after doing some manipulation over this, like the way we did when we try to solve pi equals to pi P, what we will end up is yj that is the jth term here can be written as Q by P to the power j minus 1 plus Q by P plus 1 into y1. So, I am just skipping the process like if you just write this iteratively, ok, check this anyway. What we will end up with finally, this iterative equations like where the jth component I could write in terms of y1, ok. So, if you further simplify this, it is Qi2, so let us quickly check that, why is that? So, how is my Q is going to look like? What is my first row is going to be 1 minus P minus Q. So, what is this second row is going to say? It is going to say from 1, I go back to 0, right, so I have removed S, it is going 1 to 1, right. So, this is 1 minus P minus Q and what is the, so I am interested in a row here, right, yes. So, what is the second component? It is going from 1 to state 2, it is just going to be P and after this it is going to be 0 here, right. So, what I will get and now when I multiply it with the row column vector y1, y2 here, I am going to get this into y1 into P y2, right, so that should be correct. And now if you apply the geometry, so again now I have Q by P here, so if I am going to treat it as a geometric progression here with the ratio term Q by P, what is that you are going to get here? 1 minus Q by P to the power j divided by 1 minus Q by P into y1, right. I have this, I am in the case where lambda is going to be greater than or equals to mu, so further assume, now consider the case only where lambda is equals to mu itself, then I will separately deal with the case when lambda is going to be greater than mu. So, when lambda is equals to mu what happens? When lambda and mu are equal, P and Q are going to be equal, right. So, if P and Q are going to be equal, what is this quantity is going to be? I mean this is 0, but in the, we can just look into this case here before we, it is going to be if P equals to Q equals to P, right, this is simply going to be, this sum is going to be j, j y. So, I mean what I am going to get, y j is equals to j y1 and this is true for all j, okay. Now, here y1 is my free variable and yj depends on that y1, right. Now, let us say you select some y1 with something between 0 and 1 and now my j is all possible values, right, it can be 1, 2, up to, so then in that case there exists some j large enough which will make this yj to be greater than 1, right. Whatever y you have, whatever y1 you are going to choose some positive value, it is going to make yj to be greater than 1 at some point for some j. So, because of that you will end up with the y which is between 0, 1, right. So, I am seeking a solution of this which is in this interval, right, not any y. So, if I start with any y1 positive value, I will end up with yj which are going to be larger than 1 and then in that case my y vector cannot be in this region, is that clear? So because of this, if I want to restrict my all yj's to be within the interval 0, 1, any positive value of y1 will not do. The only possible value y1 I can choose is 0. But if I choose y1 to be 0, all my yj's are going to be 0. So, then y equals to 0 is my solution. So, if y equals to 0 is my solution then what I know? It is recurrent but I already know that it cannot be positive recurrent. So, it has to be null recurrent. Now what happens for the case when lambda is strictly greater than mu? What is that? So, when lambda is going to be strictly greater than mu, so p is going to be greater than q, right. So, if p is going to be greater than q, this ratio here is going to be less than 1 and you will see that if you choose your y1 to be simply 1 minus q by p, all these guys are going to be less than 1, the solution of this. So, if you just start with this y1 equals to 1 minus q by p which is and then club like back here you get yj's. So, notice that here yj's need not sum to 1. All we need is this yj's to be between 0 and 1. So, if you start with this y1 like this, you will end up with all this yj's which are any of positive but you will also notice that they are all strictly less than or equals to 1. But as j tends to infinity, you can see that j tends to infinity, this guy is going to get 0 but 1 minus q p is same as y1 and this yj tends to 1 and we have this is also consistent with our earlier observation, right. The solution will be such that this solution y will be such that either it is going to be all 0 or it will be such that supremum of yi for all i is going to be 1 here, right. So, now the solution is such that as j tends to infinity, this is going to be 1, okay. In summary, yes that is what we said, right. In this case, we have this y vector which is not 0 like this. It is something positive here. Now, we know that if the solution to this is not 0, then it is transient. So, just summarize, so let us write my rho to be lambda by mu. So, what we showed? When rho is strictly less than 1, my DTMC is what? So, when this is, we showed that it is positive recurrent and when it is equals to, sorry, strictly less than 1, when it is equal to 1 and it is greater than 1, that is going to be, okay. Now, coming back to our q example we had. We had a q where customers are joining and they are getting served. Lambda is like arrival rate and mu is like my service rate, right. So, we can think of this ratio lambda by mu as kind of load factor. How much load I am putting into the system, right? When this load is less than 1, what I expect? What I expect like, so when this load is less than 1, what is happening is service is happening at a faster rate compared to the arrival rate, right. So, that means I can flush out the customers faster than they arrive, okay. So, because of this, I expect my q to be positive recurrent here because I am flushing out them, I can expect my state, particular state to be getting visited again and again. That is my state, the number of customers in the q. I can be revisiting those states again and again frequently. Okay, so before this let us focus on the case where lambda is going to be strictly greater than 1. When lambda is going to be strictly greater than 1, what does this imply? You are doing service at a smaller rate compared to the arrival rate, right. That means you are basically your system is low. So, when your system is low, what do you expect? So, you expect your q to blow up, right. So, that is why transient. So, what is transient here implying? So, transient is implying you start from any state, you do not come back to that state. So, you start your system in any finite state. That means you will not come back to that finite state again. That means you are going to explode. You start from any finite state, the probability of coming back to that state, there is a positive probability that you will go out of that finite state, right. And this is true for any finite state because the whole DTMC is transient, right. So, you start from any state, finite state, there is a positive probability that you do not return to that state. That means not return to that state means you are basically exploding. And when n equals to your arrival rate is just equal to a service rate, that means you may be able to flush out, but that flushing is happening very, not very often, it is happening very rarely, right. So, there is a possibility that your q can build up at some point. So, if you are a designer of some queuing system or any customer or some processing systems, so where you have to process some requirements, it could be like somebody asking for a ticket or some computer systems asking for some service or whatever jobs that are you have to deal with. So, this is basically saying that I would like to design my system such a way that my service rate has to be larger than my service, my arrival rate. So, arrival rate you can think of job arrivals or whatever jobs could be, I mean computer request or buying tickets or like whatever somebody asking for some particular service. This is what kind of also tell us when my system is going to be stable. If you define your stability to be that my q never blows up, my q will be always take some finite states and it will never blow up. Then if you want to stabilize your q or you want to stabilize your process, your operation, you want to ensure that you want to be better be in this regime positive recurrent. So, even when you just let lambda equals to mu, it is possible that you may not always be in a good state, but if you let lambda rho to be less than 1, you are going to you will keep coming to a finite state. Finite state could be one of the state could be zero state also, right? You will start with a zero state that means you have served everybody, you will keep coming back to that state again and again and that is with high frequency. So, if you do this, you may still come back to that, but that may very happen very infrequently, okay? So, fine. So, this is one example of where a priori knowing what kind of states my DTMC takes, either transient recurrent or null recurrent, I know whether my system is going to be stable or not, okay? And accordingly, if you know some arrival rate, you want to set up your service rate such that that exceeds your arrival rate, okay? Now, so let us look at other examples of where my knowledge of DTMC is going to come to our help. So, how many of you know this PageRank algorithm? Only one? So, all of you do search, right? All of you search engines, right? Now, all of us only know Google search engine because Google is so popular that it has killed all other search engines in the market. So, before that, there were quite a few. And what is the algorithm that works behind the search engines? So, one of the ideas is also derived from this in what we had this invariant probability distributions, pi equals to pi p. So, let us try to understand how is that? So, in internet, you have so many pages, right? HTML pages. And you are looking for some content which will be available at some HTML page. So, if you give that what do I expect, what you want actually? You want you to Google or any search engine to list, give the link that has the most relevant information you are searching for, right? So, maybe if you just want to talk about IIT Bombay, you want Google to first show IIT Bombay homepage rather than some blog talking about IIT Bombay, right? So, Google has to somehow rank all these pages so that it gives you the most relevant information. So, how to do that? So, obviously, you want to kind of do ranking of these pages based on that you want to list. Now, how to do the ranking? So, can anybody think of some simple way of ranking pages? So, one possibility to rank them based on the what they call it as, let me see the term, what they call as how many page, how many other HTML pages that links to this page for this information. So, if other pages are talking about IIT Bombay, how many are them referencing to this page? So, the page which is most cited, you want to bring them, right? Suppose, let us say you are searching for some research paper or some you want to understand some topic, let us say for time being DTMC. When you search on DTMC, let us say there is one paper which has been heavily cited by many researcher and that you like to read or something who had just used one DTMC word somewhere you want to use it. So, you want to definitely use the ones which is most cited, right? So, because that has been more popular and possibly that is the most informative about DTMCs. So, in that way the ones which have high citations can be preferred, right, when you want to rank. So, the one guy with the highest citation can come first, then the next one like that in that way you can order. Now, the question is how to do this or do the citation count itself? For example, as I said I created one page and I myself created some thousand other page beside my work. So, let us say I want to increase my citation. I write some crappy paper and write another thousand crappy paper which talks about this crappy paper. So, then when Google is just counting, right, how many guys are referring to this? So, that will automatically boost up this crappy paper up. So, I also do not want this. So, what I want is the guys who are more citations who are referring to this should be isolated more, right. So, suppose some thousand crappy papers are referring to this crappy, I do not want to give equal weightage to all of them. So, maybe among them maybe there are some authentic papers also good ones which are referring to this, maybe they want to be given higher preference. So, how can we do this? So, when I think these people were all thinking about how to do such kind of ranking, they possibly realize that I mean they realize that this is nothing but a solution to pi equals to pi p. Why is that? Suppose let us say I have this several loads, think of each one of them some HTML pages, they have a cross linking like they refer to each whatever manner, just think of some referencing they are happening and each one of them have some importance, okay. Suppose I am going to think of, I am going to think of each one of them as one particular state, each HTML page as a state and then pi to be the some connections basically. So, if there is a connection between one link to another based on that I can define the appropriate probabilities, we will just mention how it is. So, then we can think about that as pi equals to pi p where I am going to just again, what is this pi? This is pi i of p, this is ij or ji. So, this is going to be ij, then this is going to be j, right. So, pi j equals to summation pi i pi ij. So, here it is telling how many pages that are linking to me, right? Yeah. So, I am interested in a particular j let us say, now this is telling how many of them are connecting to me that is captured by this p and then pi is going to give me the associated weight. So, my weight pi j is going to be higher if the pi j's that are connected to me also higher, right? So, my preference is going to be higher if the other preference guide are also connecting to me. Does this capture, this equation capture that? So, suppose let us say I have given weights some preference to all the pages in the net and they are connecting to me. If the preferences are higher and they are connecting to me, my preferences will also be boosted, right? If a high preference guide connects to me, my preference will also get boosted, right? So, in that way you can try to capture the importance of the page you have in the internet and based on those importance that is now captured by this pi j, you can order the pages and then display to you, okay? So, let us just in summary now how to come up with this pi j is the question, right? Okay? So, let me write the algorithm for this. So, this is a one crude version of the algorithm. So, what you do? You take all the possible pages and draw a link between page i to page j if page i is referring to page j, okay? And then take a particular page and see that how many pages it is referring to. Suppose if it is referring to k pages, it has a k outgoing links and give a weight of 1 by k on each of these links, okay? So, that is it. Now you are not giving any importance there on each, you are just treating all the outgoing link equally and now once you have this, you have a P matrix and then solve for pi equals to pi j and on that basis you are going to rank your pages based on the values of pi you are going to get, okay? So, if you have a web page, you can just try to see what is the page rank of your web page. I do not know if Google displays it for all pages, but if you are interested just see what is the page rank of yours and you can just also find out which page has the highest page rank that tells which is the most visited web page, okay? So, this is a very crude level thing, the page rank, but it has, it is not that if you just simply do like this, it is going to give you the best solution. So, just to motivate what could be the issues, so just for the simplicity, just look into this one example. So, let us say there are three states like this and M and these are the transitions. So, I am just drawing three pages and let us say I have been assigned the weights in this fashion. So, if you solve this DTMC for its limiting distribution, what you are going to get is simply, so I am just quickly writing this, the equation you are going to get is so pi of A is going to be half pi of 1 and that is only incoming link there and then pi of M is going to be pi of M plus half pi of A. So, if you just solve this what you are going to get and now you anyway, you also have this constraint. So, if you just solve this you are going to get pi A equals to 0 and once pi A equals to 0 then you are going to get also pi N equals to 0 and then you are going to get pi M equals to 0, pi M equals to 1 this in this case. So, what is happening in this? If you just blindly take this invariant probability metric, what you are basically doing is you are basically trying to give most preference to this dead trap here. So, I am going to call this is a trap here because once you hit this link it is always self, it is a it is redirecting to itself, it has a self loop. So, because of this you will end up with giving a preference to this the most whereas ideally you do not want it possibly give a preference to this because this guy has at least link from it has more connection than this right. So, if you are just going to do like this you will not end up with a good probability distribution. So, there are to improve this there are other methods that are some are ad hoc and some are I think well developed. So, one possible thing is when you have things like this you try to redistribute the weights one possibility is whatever the. So, right now this guy is not referring to anybody right you forcefully when you get trapped here you try to come out of this how you are going to come out of this you try to deliberately add an outgoing link to this and assign some probability to this. So, one possibility is what they call it as taxing it. So, what you do is take out some percent of the tax that is some probability from each of this link and distribute that tax to all among all the states. So, one possibility is let us say I am going to tax each of this links by 70 percent. What I mean by that is all these probabilities I am going to reduce to 70 percent. So, in that case this probability is going to be 0.7 everything is going to be 0.7 this is going to be 0.7 this is also going to be 0.7 and this is going to be 0.7 and now I have said about 30 percent from each of this links right that I am going to distribute among the states. So, now, so this so I have three possible things right from here to here and also self link. So, this self I am going to add 0.1 this part I am going to add 0.1 and I am going to add a link and give a 0.1. So, whatever the 70 percent I have 30 percent I have taken out now I have just redistributed and I will do the same thing yeah. So, I have just taken out 30 percent from here right and I have just redistributed. So, it will nothing I have changed and now if you again so I have taken 30 percent out of this right. So, I add 0.1 here and this one also I will add 0.1 and also add a self loop here with 0.1 and similarly this guy also initially did not have any link to this and also this. So, I will add 0.1 here, 0.1 here and the remaining 0.1 here. Now, if you do this kind of perturbation through this taxation on this, now we will end up with another set of transition probabilities here and on that we will end up with a new set of values which I am just writing here which is at least better than what we had previously right. So, now it has kind of distributed the weights, but still as you can see that this new probabilities will depend on how much is the tax that has been put. If you are going to change this tax you are going to get a different one and the bad thing is again if you are going to tax heavy make the tax large then so suppose you are going to tax 100 percent right that means you are basically equally distributing the links all of them and you are going to make the tax 0 very low then again you are going to be coming back to a solution which is close to this. So one has to appropriately choose this taxes so that you get the correct ranking and there are many improved versions based on this. So, we will just leave it here. So, basically what we say is that in some way by looking at this equation pi equals to pi p if I am going to choose my pi matrix appropriately. So, this will the solution of this will yield somehow the preference which captures the preference of others right and in that way this is going to be a good ranking. So, let us stop here and with this we will just conclude this DTMC discussion in the next two classes I want to just talk about a little bit renewal theory. So, renewal theory is kind of you see that some of the concept we have already discussed and they are just a more generalized versions of that.