 So, let us continue with the transient solution for a Markov process. As an example, we took the case of the weather problem, the two state weather problem, the sunny state and the rainy state with the transition probabilities of sunny days becoming rainy days next day and the rainy days becoming the sunny the next day. So, with that we arrived at a solution after n days or after n steps the probability that the day will be sunny denoted here by Sn that will be given by this transient expressions where P 1 and P 2 where the probabilities of sunny turning sunny and rainy turning rainy respectively including from the initial state S naught and then of course, a generated term which will eventually be responsible for the steady state. Now, first let us see what is going to be the value of the stationary or steady state. For that we should take the limit of n tending to infinity, we obtain the then the steady state from the knowledge that since P 1 plus P 2 lies between 0 and 2 because the P 1 and P 2 individually are independent probabilities lying between 0 and 1. So, the lowest values of the sum will be 0 highest would be 2 which means P 1 plus P 2 minus 1 should be more than minus 1 and less than plus 1. So, modulus of that should always be less than 1 hence the term P 1 plus P 2 minus 1 to the power n will tend to 0 as n tends to infinity because it is a number which is less than 1 its power will tend to 0. So, that gives us a steady state probability for a sunny day eventually starting from the fact that it was sunny definitely on the first day. So, it will be 1 minus P 2 by 2 minus P 1 plus P 2 because this term also will go to 0 you will be left with this term. The correspondingly the probability for asymptotically the day to be a rainy day will be 1 minus that. So, it will be 1 minus P 1 by 2 minus P 1 plus P 2 it is just subtract and you can get it. So, it gives us the relative ratio relative strength of a day being sunny or rainy asymptotically as 1 minus P 2 by 1 minus Q 1 or using the complementary values it is either Q 2 by Q 1. So, which is the ratio of the probability of turning to a sunny day from a rainy day divided by the probability of turning to a rainy day from a sunny day. So, we can show it as a transition diagram. So, if this now a representation of states the rainy state and the sunny state then the final ratio of whether between the probabilities of sunny and rainy are just the ratio of the downward and the upward transition probabilities. So, that is about the steady state. However, the approach to steady state is also equally interesting and now the problem degenerates into 2 cases. Supposing P 1 plus P 2 is less than 1 which is possible their independent probabilities this can be let us say 0.1 and this can be 0.2 it is less than 1. Then P 1 plus P 2 minus 1 will be less than 0 because it is a it will be negative and hence P 1 plus P 2 minus 1 to the power n will oscillate from negative to positive values as n moves from odd to even. So, for n is odd negative to the power n will be negative when n is even that will be positive. As a result approach to steady state will be oscillatory. So, we should see in this picture a diagrammatic representation that if you start with the sunny state it will keep the probability will keep oscillating. The amplitude of oscillation will come down successively as n increases and eventually the steady state value that we predicted will be attained. So, this is the case when P 1 plus P 2 is less than 1. If P 1 plus P 2 is more than 1 then we have the case of more than 1 then the decay becomes monotonic not oscillatory decay or approach to steady state will be monotonic. That is because when P 1 plus P 2 is more than 1 this term is positive it will successively decrease, but all the time remain successively less every time n increases. So, that would look something like this. So, for the case P 1 plus P 2 greater than 1 we are going to have a diagrammatically saying the transition transients would look something it will reach a steady state, but it will follow a smooth monotonic decay. So, this is the probability as s n as n increases so, this is n. So, it is basically an example to show how one can obtain transient solutions. If you have for example, three state problem then it is like solving three simultaneous equations you can again in principle get it, but becomes more and more difficult as the number of states increases and one has to do it numerically by operating with the transition matrix at successive states. But in principle we have now understood how one can both obtain a transient solution as well as a steady state solution from the knowledge of transition matrix. So, this is the strength of basically the Markovian hypothesis. Most of the examples that we took they seem to be all Markovian at least we have framed it so. In order to give you a kind of a feeling for what could be a non Markovian process which are quite plenty in fact, in nature to give you a one of the practical or nearly realistic problems we take a case of some particle agglomeration modeling. So, this is a as an example of example for non Markovian processes. We take a oversimplified agglomeration model let us say it is oversimplified based on basically just to illustrate the point. So, an agglomeration model let us remind ourselves is not a rigorous model it is just highly simplified a skeletal skipping the skeletal principles just to illustrate the point so an oversimplified. So, in this model let us imagine that there is a large reservoir a box in which there are some monomer particles some molecules suspended the number is very large and they do not get depleted if one is something happens we replenish them externally. So, we keep that let us say that in this system in this reservoir there is also one seed particle of interest. So, we let us say we indicate it with another color the seed particle is let us say red. Now, we process execute the process of agglomeration in the following way. So, every in every step we allow the monomers to agglomerate and form dimers in the next step we stop that process and allow those dimers to agglomerate with the seed particle. So, this is a seed particle these are monomers. So, what do we do in the first step we allow same system we allow these monomers will be there very large number and of course, this seed particle will be there. So, in the first step at the end of step 1 with some rule with some probability we allow that one dimer particle will be formed a small collection of these two monomers it is much smaller than this seed particle. So, we allow the formation of a dimer with some probability p and with the probability 1 minus p there will be no dimer formed. So, there are there are two options. So, we talk we as we mentioned we generate a random number if the probability p is pre assigned let us say 0.4 then if the random number shows a number less than 0.4 we allow a dimer formation if it allows a number if it shows a number more than 0.4 we disallow the formation this is step 1. In the step 2 we allow these dimers only to agglomerate with the seed we assume that the monomers do not somehow is an artificial assumption, but assume that over the seed particle only the dimers if existing that will agglomerate with this and form a larger cluster. If this volume of the seed particle is V naught then here the volume will be V naught plus 2 small V naught where small V naught is the volume of a monomer. So, dimer will have a volume of 2 V naught that is the idea and so on. In the third step now we again allow only the monomer agglomeration and allow it to form a dimer. So, dimer concentration may increase if it has not been consumed already, but every simulation in steps all the odd steps will correspond to the formation of a dimer through monomer-monomer interaction and at every even step these dimers let us say will interact with the seed particle to make it grow and our interest is to construct a kind of Markovian equation for the growth of the seed particle we are only focusing on the seed particle. So, in the seed state space if you look at the seed state space its initial volume let us say is V naught it can only increase by 2 V naught because the dimer only is attaching or its volume at any time can be 4 V naught or 6 V naught. So, we are executing a kind of stochastic motion along the seed volume space. So, this is a seed volume this is my random walking and it is increasing discretely V naught is some number let us say and it will increase by 2 units. So, it has now all the character of a stochastic process and we are interested in finding out simulating how the volume continues to increase because there is no decrease we are not allowing for its evaporation main thing is to see whether like this when you proceed after some stage if it has a volume V naught plus some let us say n V naught what is the probability of transition to another volume of V naught plus n plus 2 V naught because it changes by 2 units every simulation and here n is let us say always even numbers because that is the way we have the rules suggest that our particles are the seed particles grow in only even steps. So, this transition probability P n to n plus 1 from n here n is of course, the number of steps, but we are talking of basically V n plus 1 probability that the volume V n plus 2 to V from the previous step. So, what is the transition probability is it a function only of the state or does it depend on the path. So, this is the key thing that will decide whether it is a Markovian or a non Markovian process to understand it better let us draw the whole process of whatever I described in the following fashion. So, I will now consider the states the steps along these vertical lines. So, let us say in the step 1 where I get monomer plus monomer is a dimer. So, as soon as the process is over at the end of the step 1 I have a dimer formation with the probability P. So, dimer with probability P or I may have no dimer growth or non growth non formation with probability 1 minus P need more length. So, we will write like this. So, that brings us to step 2. So, what does this say in step 1 if a dimer is formed. So, this becomes then a path 1 if dimer is not formed many realizations will actually follow path 2 all coming from the same simulation of probability P and 1 minus P. So, if path 1 is followed just before step 2 you will have a dimer formed. If path 2 is followed there will be no dimer. So, when we now seek of a transition probability and of course, here the volume was V naught because our cluster or seed was yet to grow. So, if it had followed if the whole process was coming from path 1 it will again followed to branches here because every step has probabilities let us say and in the upper one is a probability of seed growth and let us say that in order to the seed to grow there must be at least one dimer. So, if the dimer is there the transition probability will be finite. So, P of V 2 from V naught it will exist it will not be 0 in this path because it is a path where dimer is formed and hence definitely there is a probability of it growing transition probability exists. On the other hand if this path is chosen with the probability 1 minus of that then this is a non non growth for the seed in any case. So, at the end of this path when you are reading yourself for the next step say step 3 you would have seen that the transition probability for the seed to grow depended on the path because it was pre it was a prerequisite to have a monomer for to have a dimer for the seed to grow because that is the rule we have set. Although rule remains the same the fundamental rule remains the same monomers monomers interact in first step and then we allow the seed to interact with the dimer, but if the dimers are not formed then then both the paths will lead to non growth only. Hence the transition probability here will be non 0 here whereas, if the path second path is followed for the seed formation the transition probability P because there is no dimer formed will be 0 and of course, this is a non growth anyway always with some probability. Hence we will find that the transition probability the P of V 2 V naught depends on path it is non 0 only if path 1 is followed, but for all the realizations where path 2 are followed the transition probability has to be set to 0. So, this is just an illustration of an example where physical problems could be non Markovian. Although this is a very idealized problem this is indeed true in many problems in formulating the so called population balance models for aggregation and coagulation and they cannot be they have to be done by more complex non Markovian methods. What people normally do is to assume a mean field kind of a probability instead of saying assigning exactly dependent on the dimer formation number one produces an average probability for the dimer formation averaging over the 0 1 probabilities and then assigns it for the step. So, that every step there is a certain probability transition probability regardless of the path that is chosen. So, that is of course, a more simplified Markovianization of a non actually a non Markovian model. There are many examples. So, I will just I gave this specific case for example, if the transition probabilities depend on time that could also be a non Markovian process. If transition probabilities are actually dependent on another hidden variable whereas, you are interested in only formulating over let us say a state space of one another variable. Then also the system will have an apparent non Markovian nature which can which can be removed only by including the degree of freedom on which the system was hidden already. So, these are some of the interesting aspects that needs to be known before we address a transition process or before we address what approach to follow to describe that stochastic process. I wish to also at this stage to give you an advance information on how do we formulate stochastic process in the continuous time variable continuous I call it as time, but it can be stepped continuous time Markov process. Of course, time is always continuous. So, what we mean by that is we now convert step to time. So, the Chapman and Scog Chapman Kolmogorov equation specifically we derive the Chapman Kolmogorov equation takes a kind of a parallel form to the transition probability relationship for a two step process that we formulated in terms of sums basically for discrete state processes. So, for continuous state and continuous time let us say that a system it suffered motion transition or motion from a position x 1 at time t 1 x 1 at time t 1 which is denoted often by a pair x 1 a position x 2 at time t 2 and then x 3 at time t 3 here time ordered ordering is done. So, it is this way. So, basically we have a continuous space is a position x 1 and it has moved to x 3 via a transition from x 2 to x 3 and these are at times t 1 t 2 t 3. So, we say t is increasing this way t 3 is more than t 1 because there is no harm in writing, but it should not be misunderstood that t is only a parameter it is not a coordinate representation of the point. Let us see how do we write Chapman Kolmogorov equation for this problem. There are very rigorous derivations, but let us use a physical way of arriving at it. First thing is we have to merely note here that the mass probabilities which we introduced in the context of discrete state spaces probabilities become probability densities here. Probabilities they replace or they move over to probability densities. What do we mean is that now we introduce the concept of a transition probability density Px dx is the transition probability transition probability between x and d between x and x plus dx. So, probability density Px is the transition probability density, but the physical meaning of probability comes by multiplying by the element. We can formulate the continuous space and time transition problem as follows using the correct notation for transition probability density in the form Px 3 from x 1 in a time t. With this the probability of transition to a point x 3 within an interval dx 3 that is the probability can be written now as Px 3 from your point x 1 in a time interval t. So, into dx 3 is the actual probability of transition from x 1 to a point in the neighborhood of x 3. This can be written as being equal to the probability. So, probability that the particle translated to the neighborhood of a point x 2 from the starting point x 1 in some time tau. So, this will be the probability and then in the remaining time t minus tau it translated to the point x 3 from x 2 in the time t minus tau and this transition occurred between x 3 and x 3 plus dx 3 and hence dx 3 has to be multiplied. Hence x 2 is an arbitrary point it would include an integration over all accessible x 2. Now we can easily see that the dx 3 terms cancel each other. This equation can be written in a neater form by replacing t minus tau with the t itself that is t instead of t minus tau. Then actually we will have P x 3 from x 1 in a time t plus tau. So, the older t will become now the new t plus tau this will be sum over all x 2 and the product of transition probability is P x 3 from an arbitrary point x 2 in a time t and 2 x 2 from x 1 in a time tau over all accessible x 2 points. This is the Chapman Kolmogorov equation. It is a very interesting equation because one may note that the tau here is an arbitrary parameter any value of tau is allowed. Hence this forms a constraint equation for the transition probabilities. The constraint ensures that the transition probabilities occur in specific forms and one cannot have some arbitrary function for transition probabilities. So, we basically use the same concept that to have moved to x 3 from x 1 it could have passed through any of values these are all x 2 points x 2 prime x 2 double prime. So, it could have passed through any of the points. So, only the time of course, is has to continuously here change. So, it starts at any time t to have translated to point x 3 after a lapse of time t plus tau goes in 2 steps it was in at time t it is translated from x 1 to x 2 and then in the another time tau it translated from x 2 to x 3 for all x 2. So, it is intuitive, but made possible again by the Markovian approximation that this that the transition probabilities depended only on the previous states and not on the history of the paths. So, this is the Chapman Kolmogorov relationship for continuous space time variable this complements the equation for discrete spaces. With this we formal we formally introduced ourselves to the subject of Markov processes. We will understand more of this as we go along and we from now on we take up some very very important family of random work important family of stochastic processes such as random work problems and then explore how to solve these equations. Thank you.