 So, what we so far discussing is just like we know, in some applications we will be interested in knowing a priori whether this state is going to be positive recurrent, null recurrent or it is going to be transient like in the insurance kind of applications I will just discuss last time we said that it is important to know whether this is a recurrent state like if I am going to go bankrupt that the state that I am going to become bankrupt is going to be recurrent I do not want to be in that business right. So, a priori if you know or if you could model all the system you want to quickly identify whether this state is going to be positive recurrent, null recurrent or whatever it is. So, we are just now stating what are these results that you can what you can say about that just by looking your Markov chain right say like all this characterization was only simply based on your Fjj values right Fjj values which you can compute from basically your transition probability matrix. So, your transition probability matrix is basically characterizing your Markov chain. So, if I tell you my Markov chain that you can quickly come up with your transition probability matrix from that you can compute this Fjjs and try to identify what are the states how they behave like. Now, based on this you can try to understand how your things are like for example, your transition probabilities you may start with on transition probability matrix and for that you observed these states are transient and these are recurrent and all. But you notice that I am going to be in recurrent state most of the time so this is a bad case I do not want to get into this. So let us change my Markov chain changing Markov chain is equivalent to saying that okay let us consider these parameters you want to look at a different set of parameters that is like you changing your design or like you changing okay let us instead of starting with this much capital maybe let us start with another set of capital or something like that and then you analyze based on that okay what are the states which are going to be positive recurrent and transient then you will see that okay in this I find that my recurrent states are less maybe this will looks like a bit safer bet and then I want to maybe go with this. So, what we are just trying to do is given this set of inputs given one description of Markov chain see Markov chain as long as somebody tells you okay these are the transitions happening you have a Markov chain you are identifying but that Markov chain it can change if you change the parameters right your transition probabilities once you have a new set you can do the same thing. So you can do this analysis for any given transition probability matrix and based on that you can say maybe like you say that okay this transition probability looks like it will lead to a situation where things are less risky and maybe as a designer or the guy who is actually going to implement them maybe he will see how he can get those parameters okay fine. But for this computation of all this set null transient and null recurrent we are really no don't need to worry about my initial trans probability I only need to worry about transition probability matrix because as you see when I was trying to do all this calculation I state given this state I kind of fix my state and from there I am trying to analyze. So the initial distribution is not affecting much right because you are already fixing a starting state and after that you are analyzing initial distribution only affects in which state you are going to start from but that you are already fixed now what is only going to decide how your future is going to have always your transition probability matrix. So that is why I am saying to analyze this all you need to know is only transition probability matrix okay. Now we know that we have this different states as of now we have classified them into two broad categories transient recurrent and further positive recurrent and null recurrent. Now is it possible that all my possible states I can group into these classes or like let us say I can group it to some classes and what is the property of each of this class. So let us try to understand that through what we call as communicating classes okay let us define something. So let us take a Markov chain where the state space is some countable set okay as we always denoted by s as I mean I am again denoting it by s which is countable. Now we are going to say that so let us take a DTMC with countable set of states now take any state j and take another state i I am going to say that j is reachable from i that means I am going to hit state j starting from state i if there exist n positive such that this n step transition is the probability of n step transition is going to be strictly positive and in that case I am going to denote that i goes to j that is from i I can reach state j. See as long as what all I need is to reach j from i is at some point at some time I should be able to reach it with some positive probability okay right now this n could be very large but that is fine as long as I can do it in some time that is fine. Now we are going to say that you take two states i and j if I can reach j from i and vice versa like also if I start from j at some point I am go back to state i if that happens then I am going to say that state i and j communicate okay. So if i is reachable from j and j is reachable from i then I am going to call i and j communicate okay and that communication symbol I am going to use it as this double arrowed line okay when I say j is reachable from i I am going to use single arrow taking i to j and when i and j are communicate I will use this one okay. Now this is a relation right what is this relation this is a communicate relation. Now we can show that this relation equivalence relation so how many of you know equivalence relation what are the properties reflexive symmetric transitive property what is reflexive property self okay. So reflexive property does this relation satisfy reflexive property so then for that what I need to check i go into itself right. So if this is the case what I need to show i is reachable from i and again i is reachable from i to show that what I need to show there exist some n positive sorry n greater than or equals to 0 so that I can go from i to i with positive probability what is that n 1 or 0 0 right because p i i equals to 1 by our definition if you are in i you go to there in 0 time with problem this that is the meaning of p i i 0 that is what we have defined like right. So that means with probability 1 if you are in the state in your state you will be there in 0 at the round so this trivially holds. Now second is symmetric what does this mean j then this implies is this true so what is sign this is so what is i let us take i communicates with j means right i is reachable from j and j is reachable from i so that means there exist some positive probability some positive finite time in which I should be able to reach one from the other right that is same as so here you start from i and go to j and then when you look at the other direction you start from j to i and just do I mean the same thing here yeah this is this is also other direction if you this is also but in this case we just need to argue that this implies this if you start with this then you can say that this implies this now what is the next so what is how to show that is this true so how you are going to show this so let us show one direction like i communicates with k. So i communicates with j means in some finite steps I go from i to j and again j communicates with k means in another finite step with positive probability I go from j and k so if you take this product I know that in this many rounds at least with positive probability I go from i to k and similarly the opposite direction so this relation is an equivalence relation so what is the property of an equivalence relation this relation partitions but the thing is it partitions your class so you have this state space S this relation if you apply it is going to partition your S into what you are going to call as equivalence classes so that means all the states within when you have partitions and if you take one particular partition in a way all the states in that partitions are going to be equivalent that is what we call as equivalent class and here all this we have so many classes so we are going to call them as equivalence classes okay fine we have some more definitions once you have a equivalence relation this is clear that it partitions now equivalence class is just like the classes we have in this partition we are going to just call them equivalence classes okay so proof for this work out yourself just ensure that you will not get any overlapping sets okay now let Xn be on S so let us say I have a DTMC with transition probability matrix P okay then so we are going to as we said S is going to be partitioned by this equivalence class and the different classes we are going to get we are going to call it as communicating classes okay because we know that each pair in that particular class or a particular partition is going to communicate with each other right so we call them as communicating class classes now you take a communicating class so if you are going to take one communicating class and you are going to take one element in that communicating class and take state which is outside this communicating class okay and if let us say I belongs to that communicating class C and J is outside this communicating class if Pij is equals to 0 then we are going to call this communicating class C as closed that means the probability of you going from any state of this to outside state of this class is going to be 0 that means you are not going basically outside of this class to any state so you are that is the kind of close class you have and if this does not hold this property then we are going to call it as open communicating class okay. Now we further say that the transition probability P is going to be irreducible if there is only one communicating class that means entire state space is just one communicating class at and in this case so and this case we say the Markov chain itself is irreducible okay again to define this communicating classes all what I need to know is this enough if I know my transition probability matrix yes right because to define this I need to know all this instep transition probability matrix but we are in the case of time homogeneous Markov chains for this this Pij's instep transition mobility can be obtained from my one step transition probability matrix so all this properties that I have defined that this relation whether it the and whether my class is going to be communicating and whether my class this communicating class is closed or it is going to be open all I need to know is just my transition probability matrix from that itself I can define all these things and now whatever the transition probability I am going to deal with I am going to call it as irreducible if that leads to me only one communicating class so that my state space is not going to be a partition it is just going to be one communicating class so once I say one communicating class I am should be reachable from one state to another state within that communicating class so that is why I am going to call it as irreducible you cannot partition into two parts that is the case so in that case we are just calling our Markov chain itself is irreducible okay a quick example on this let us say I have some states like this where can go from here go from here go from here go from here and also go from here so the points where I have put this arrows that means these transitions are possible with positive probability where the link does not exist that means that transition happening with zero probability so let us call the states 1 2 3 4 so how many states are there here 4 states and how many communicating classes are there here what are those communicating classes 1 2 because I am able to reach 1 to 2 and also 2 to 1 and what is the other communicating class 3 to 4 right this also I am able to reach 1 from each other here and now is this communicating class a close communicating class so this is one class let us call the C1 and this is called C2 is this communicating class a closed one why is that yeah we can go with positive probability from a state in this class to a outside state C2 whereas this C2 is going to be a close communicating sorry it is a closed one yes right because we are not going anywhere outside so what if if I add another states here and allow this to happen so in this case if I add this will this 5 becomes goes inside C2 but can I reach from 5 to 1 so I can go here with positive probability from there I can go here right in that case I will include 5 also in that but suppose if I remove one link here it is a that can become another itself class right with just one element in that and in this case 1 and 2 is going to be closed one it is not going to be closed one right because it can escape from this 2 to a state which is outside this class now we are saying closed this one we are going to say closed if it do not have any escape route to go outside outside a state it is some positive probability so in this case when I wrote this right if do you have any escape route to be outside go outside to state other than 1 and 2 you go 1 and 2 and then 1 and 2 you may just be like that right you will not possibility and here there is a possibility for you that you escape from being in C1 you can go to here right because there is positive probability now if I just say there is I add one and then let you go there then you are able to escape from this class so that is why it is no more closed can now comes it is called class property theorem if there is only one thing what is there to define closeness it is just going to be closed right there is only one state in that and the definition becomes kind of a vacuous here it is just one state and you have to going to that state okay so when I say it comes to this state and it is not going anywhere means it remains in this state always it is just staying in that not moving anywhere okay now the states so this theorem is stating that the communicating class we have will be such that all the states in a communicating class are going to be just one type they are all other going to be all transient are going to be all null recurrent are going to be all positive recurrent it is not possible that you have a communicating class that will have some states to be recurrent and some other states to be positive recurrent or null recurrent it is going to be either one of them okay so let us quickly look into this why this is the case suppose let us take on communicating class and let us take let us C be a communicating class and take ij belongs to C so if ij belongs to C what I know what I know there exist R and S such that pjk of R is going to be positive and pkj of S is going to be positive right this is by definition if i and j belongs to the same communicating class it must be the case that I am going to go from j to k with positive probability in some finite steps and similarly I should go from k to j in some finite step with positive probability now suppose now I want to claim that if I assume that j is transient I want to prove that then k is also transient and if I assume j to be positive recurrent then I want to show that then k is also positive recurrent and if j is null recurrent then k is also null recurrent so let us see why that is the case okay let us first look into the case where I want to go from j to j in this much steps R plus n plus S this R and S I am going to be take this whatever this value n is variable for me any n now what is this this is probability that X R plus n plus j or S is equals to j given you are going to start from X naught equals to j right this is the meaning now what I will do is I will instead of going from state j from the beginning to again state j in R plus n plus n steps I will want to reach this while going to while going to some other some states in between also so let us further condition this by saying that R n plus S equals to j what is that conditioning I want to add X R equals to k then X R plus n is equals to k given X R equals to j so what I want here is yes instead of directly going here you I also want it to be first reach state k in the first R steps subsequently in the next are n n steps that is R plus nth round I want to again hit state k and from there in the next n steps that is R plus n plus S round I want it to state j so is this probability is going to be lower bound for this yes right because I added these two extra conditions on this I basically asking this Markov chain to go through this special state at these steps not just looking at the final step here so if you just now apply the chain rule here what you will get is so already applied the chain rule and the Markov property here and by definition the first term is going to be jk R rounds then pkk in n rounds and then p going to be jj now this is k here this is going to be kj in s rounds it is correct so I have just applied chain rule and apply Markov property and this is just by definition and now what we have basically done is pjj R plus n plus S is going to be less than or equals to pjk R pkk in n steps and then pkj in steps now I want to use this property first thing you can check that pjj of n you are going to be pjj s this I am simply going to take n this is not equals to 1 here and this one here so this is because I am just taking this this guy is a lower bound sorry this guy I have lower bounded and then summing it over all possible n's and now further I am going to lower bound this by this inequality which I have obtained here so here s is the variable this R and s are fixed and I know that this pjk R and pkj s both are strictly positive quantities so now let us use the property that we have stated in a theorem suppose if I assume j is going to be transitive let us say let us assume j is transitive sorry transient so suppose if I assume j is transient what I know about this quantity as n goes to infinity we just know it is to be finite right I know that if my state j is transient this is going to be finite right and I know that this quantity here is some positive quantity and this quantity here is again some positive quantity if that is the case then what I can say about this quantity this is also finite right then what would we say in that theorem if that the theorem when the summation pijn is finite if and only if j is transient right that theorem was if and only if case so now if I assume this quantity to be finite let that is following making assumption that j is transient then I already concluding that this guy is also finite what does this imply what is transient so that implies k is also transient so what if I take any two states j and k in my class what we are concluding is if j is transient so is k so it must be the case that any states in this communicating class they must all be transient right okay so now let us look at the case where I is sorry let us look at where j is now we want to look at null recurrent let me see what I want to look at okay now let us say null recurrent now we are going to say that if j is null recurrent then we were going to show that k is also going to be null recurrent okay suppose let us say j is null recurrent and if I assume that that implies k is transient but then I am contradicting my first statement right suppose k is transient then it must be the case that j must be also transient but I am assuming that j is null recurrent so by this counter positive argument this cannot be the case so j could be either null recurrent or positive recurrent so let us see what it is so we have to then use the another result we have used where instead of looking at the sum we look at the averages okay so let us try to look at the average and then this I am going to write it as 1 by n which just we have few steps will be done okay what I am doing here is basically I am taking this 1 by n and summation of n terms here that is the average of the first n sums here and then in this I am going to drop out all the terms before r plus s and looking at the sum beyond that so basically here I am assuming that n is large it is at least r plus s right that is why it is starting from this I am only having few terms here that is why it is a lower quantity and then I am just re-indexing it so instead of starting from m r plus 1 I am going to start from n l equals to 0 and then re-indexing this quantity and also okay when l is 0 this is going to be yeah I am just re-indexing this quantities equals to this quantity and then I am going to use this relation that I have already got that is l equals to 0 to n what I have got pjj of r pkk of n and pkj what did I do so this we have already shown that this can be lower bounded like this now I am going to do a little bit manipulation I will pull out this term and this term outside so this should minus r plus s right pkk of l times n by n times pkk so there are how many terms here l goes from 0 to n minus r plus n r plus s they are just we are done now what I am going to do so is this clear I have just pulled this outside and that is the reorganized I have taken 1 by n and this quantity here but now look at this this is summation of how many terms here l equals to 0 to n minus r plus s right so what I will do is simply this n I will keep in the numerator n minus r plus s and n minus r plus s I have just multiplied and divided by this term now if you look at this quantity here this is now average of n minus r plus s terms right so there are how many terms there are n minus r plus n terms and I am also dividing by the same number of amount okay now let us now let us apply our result what would we say as n goes to infinity if I let n goes to infinity and this guy j is null recurrent what did I say where does this quantity goes to limit n tends to infinity where does this go so this goes to 0 as n goes to infinity and now as n goes to infinity you look at this there are n minus r plus s divided by n what does this ratio go to that ratio goes to 1 right because r plus n this is a constant term that does not add much and what does this ratio will go to this ratio is going to be same as limit as p k k l as sorry n as n goes to infinity right because you are just skipping some finite terms here if you look into the infinite summation of this they added by that same number here this is going to be the limit of so this quantity is going to be a limit of p k k of n so this is going to be let us say goes to 1 to m or may be m goes to infinity m goes to n equals to 1 to m okay so this quantity is just going to this so now we are what we have shown is as j goes to sorry as n goes to infinity this term is going to be 0 and this term is 1 here but this is going to be positive quantity this plus term is also going to be a positive quantity in that case what we say this limit must be 0 right because this quantity is upper bounded by 0 so then what does this mean the j is also the yeah if j is null recurrent then k is also null recurrent so now last thing suppose now let us say if j is my positive recurrent it is because it cannot be transient or null recurrent so only option you are left with is positive recurrent.