 So, last class we are discussing about the loop transfer recovery of a LQG problems. Then just if you recall this what we have done it that LQR problem provides the excellent robust properties in the sense that gain margin is infinity and phase margin at least 60 degree. This is the robust properties excellent robust property we obtain in simply LQR problem. But when you are designing the LQG problem assuming that state equation and measurement equation is corrupted with noise and the noise static sticks is known to us a prairie. Then in that situation since the states we are estimating using a Kalman filter in that situation loop transfer function is different from LQR loop transfer function which in turn that robust properties of LQR is totally destroyed in the sense that we are not able to obtain the gain margin to infinity and the phase margin is not at least 60 degree. So, in order to recover that what is called robust properties we have adopted some technique which is called the process noise covariance matrix we can split up into a two parts. So, if just we can we are we are just what is called recall our earlier procedure the how to recover the what is called LTR agree. So, how to recover phase and gain margin of LQR and phase margin we know we know the 60 degree in LQR and gain margin is infinity, but in case of LQG these properties is totally lost. So, how to recover that one? So, that can be required by selecting the process noise that can be required by selecting the process noise covariance matrix that is gamma w gamma transpose into two parts that we have mentioned it w gamma w 0 gamma transpose plus q bit into bit transpose where this is the initial guess of noise covariance matrix initial guess of w initial guess of w this w and q is a some positive scalar quantity. So, this we have split up into two parts and we have shown in the last class how we have what is called regain the loop transformation agree as LQR problem loop. So, in this process if just following assumptions are made first assumption is made that system is minimum phase systems minimum phase system and we also made another assumption that number of inputs that means input matrix and output matrix then another matrix is input matrix B and output matrix C have the full rank with number of inputs M is equal to number of outputs. This assumption we made it to derive to modify the what is called loop transformation of LQG using the matrix identity that last class what we have derived we have briefly we are summarizing this that one using the matrix identity two identities we have used it then we have derived that linear quadratic regulator loop transformation is same as the linear quadratic regulator loop transformations. We using this that noise process noise covariance is split up into two parts and we made the assumption that minimum phase systems and B and C as a full rank with M this with this assumption and using the following matrix identity following matrix identities what is this matrix identity we have used it that is first matrix identity we have used it that phi C of S is equal to loop transformation that means closed loops transformation is equal to S i minus A minus B k whole. So, this is a closed loop transformation when there is no noise process and noise in the state equation and measurement equation when in other words when Kalman filter is not used this is this matrix this equal to we have written like this way phi of S into i plus B k phi of S inverse this matrix identity we have used it. So, where phi of S is equal to S i minus A whole inverse that we have derived last class this is one matrix identity we have used it another identity we have used it that i plus B k phi of S whole inverse B is equal to B i plus k phi of S into B whole inverse. So, that is another matrix identity we have used using this two matrix identity we finally, derived the expression finally, we derived the expression and last class if you recollect this one that expression we got it limit q tends to infinity l r phi of S this into C phi of S B bracket i plus k phi S B bracket whole inverse that bracket another bracket is there which we missed last class whole inverse agree. So, now you see this if you can see this and this is identity because A into B into C whole inverse is equal to C inverse into B inverse into A inverse provided A B C all inverse identity exist. So, if you do this process then ultimately we can k phi S into that this B then i plus k phi S B whole inverse then this is i plus k phi S B this then this whole inverse C phi S into B whole inverse agree into then C phi S into B. So, ultimately we are learned up with this identity matrix this and this identity this inversion is exist with the assumption that B and C with a number of inputs and outputs are same. So, this inversion so we are left with this one and which is nothing but a the looped trans function of L Q R that is nothing but a L S which is nothing but a looped trans function of L Q R. So, in short if you summarize if you select the process noise into this form and made the assumption that system is minimum phase system B and C have full rank with m is equal to n and following identities are applied during the derivations what we did it during the derivations of last class then we finally, learned up with this expression and which in turn we can write it this one and this is nothing but a the looped trans function of L Q R. So, which is indicates that we have regained the loop or robust properties of L Q R by selecting the process noise into this manner and where if you select like this way we will get the gain margin is infinity and the phase margin at least 60 degree. So, this is called loop transfer recovery. So, now let us go in a new topics which is we call dynamic programming of discrete time systems next is new topics is dynamic programming of for discrete time discrete time system. So, let us call and this dynamic programming is based on the principle of it is based on principle of optimality what is principle of optimality we will discuss. In other words it is based on the it is based on the based on Bellman's principle of optimality. Let us see what is this, so the optimal policy has the property with initial state and initial decision. So, optimal policy has the property with initial state and initial decision that the remaining decision that the remaining must constitute optimal cost must constitute an optimal cost of policy. Let us explain this thing suppose we have a trajectory whose initial state is x of t 0 at time t is equal to t 0 state position is x of x of t 0 here and the final terminating state is x of t f agree this one. That means x of t 0 is the if you assume this the trajectory is the optimal trajectory starting from initial state x of t 0 at time index t 0 and it is terminating at time t is equal to t f at t is equal to t f is terminating at t is equal to x f and this is x 0 at time t 0 along this optimal trajectory. So, and there is another point is there here in intermediate point let us call that is x t is equal to t m t is equal to t m that trajectory is x m. So, that point at t is equal to t m time index the state value is x m agree. So, assume x t m be an intermediate this is an this is an inter state or point on the optimal trajectory and let the cost j star x t m of x t m be the this indicates the optimal is the be the optimal cost for optimal cost for travelling the state from this point to terminal point is the j star x t m from where it is starting x t m and what is the position at time t is equal to t m x position is x t m is the this j star indicates optimal cost transferring the state from this to final state or terminal state here the cost is this and it is generated by j star from where it is starting at time t is equal to t m x what is the state position at x t m this is the call optimal cost of the trajectory moving from the intermediate point x m to terminal point x f. Similarly, if we just note j star t 0 x t 0 what is mean this indicates that optimal cost transferring the state or moving the state from initial state x t 0 or x 0 at time t is equal to 0 to final terminal star from final term in terminating state x f at t is equal to t f what is the optimal cost and that is denoted by this symbol agree. So, with the knowledge of this symbol then it you can say this one this is the intermediate point intermediate point or state then it then follows that the portion of you see from what is the optimal cost from intermediate point to terminating point optimal cost is j star t m comma x of m this is the case. So, that I am writing if then it follows the optimal that the portion of the optimal trajectory x t m to x t f is also optimal because we have considered this along this curve transferring the state from this along this time from initial state to final terminal state this path cost is optimal and this intermediate point belongs to the what is called that our optimal path in this way. So, from here to here what is the cost we obtain it that is also should be optimal. Now, let us assume that this is not optimal and this path is the optimal the dashed line from this to this if you proceed like this way I am reaching the terminal point here along this path is optimal. So, now what is the optimal cost is required optimal cost in during this process that from here to here what is the optimal cost plus from here to here what is the optimal cost is that this cost is different from this one we told that this is the optimal and which contradicts our initial assumptions. Initially we have telling along this line if you move the state from initial state x of 0 to terminal state x f at t is equal to t f then we will get the optimal cost. So, this if you move from this to this and this to this assuming that this path cost from terminal point intermediate point to the terminal point this is the minimum or the what is lesser cost compared to this. Then the whole state the state when we are transferring from x 0 to t f along this path and dotted path this will be the optimal, but that contradicts our initial assumptions. So, which cannot be done which cannot be possible. So, I am just writing these things once again assume that the dashed that the dashed curve or path represents a smaller cost than the solid path or if you just include this is a b c this path is d this path is a e path then this dashed path would provide less expensive route or path form x t 0 to x t f this contradicts our original this contradicts contradicts the optimality of the original trajectory. So, this is the optimal path. Now, this optimal control strategy is the optimal control strategy my question is that from here to here it will move or travel from this point to the initial state this to this under the control action if you think at from control point of view the states when to move from this point to this point and if you move along this line that will be give you the optimal cost and what should be the choice of control action. So, that the state will move along this path. So, the optimal control trajectory the optimal control trajectory or optimal control strategy or control law strategy can be obtained by backward pass from final stage using principle of optimality. This indicates that if you know the terminal cost of this one end point then backward you find out what is the cost is required from here to here then from here to here backward pass that means optimal control strategy can be obtained optimal control strategy when control law can be obtained backward pass from final stage to the from final stage using the optimality principle. The decision this is the thing decision is decision is nothing but a choice of alternative living path from a given node or decision what is the decision which path will move this is nothing but a alternative living path from a given node. So, let us take an example and see how we can derive the principle of optimality based on principle of optimality to find the optimal control law u in case of discrete time systems. So, discrete time dynamic systems this principle of optimality will apply in order to implement the control law based on the principle of optimality. So, let us call our discrete time model is given by that x k plus 1 is equal to f of k x of k and u of k and this dimension of this let us call number of states are n cross 1 number of inputs m cross 1 and this number of states are n cross 1 and in general this is the function of x and time index k and u. Any discrete time system in state phase form one can write it x k plus 1 k indicates the time index which is a function of state and input and the time index. So, this function can be linear or can be non-linear linear or can be non-linear agree. So, with our initial state x of t 0 is equal to x 0 and the corresponding performance index and the corresponding performance index p i performance index of the given system is given by j is equal to j 0 then phi n capital x of this plus summation of k 0 small k 0 to n minus 1 then it is a f k x k and u k this capital f is the performance index which is in generally may be quadratic form or any form. So, this is the word performing index is dependent. So, our job is to find a control law u in such a way. So, that j is minimized agree and correspondingly optimal trajectory of the state we can obtain it. So, we are mentioning that we are moving backward. So, terminal cost final terminal cost we know then we let us call x this is the call terminal cost and our terminal point t that is k is equal to n n th time index is the terminal k is equal to n is the terminal time index. So, let j k j suffix k star x of k of whole k. What is this indicates if you are considering the this indicates that we are moving the states at k th instant to final time index k is equal to n index for that one what is the minimum cost of this one. So, this we are writing is j this is the minimum cost of transferring the system state from x capital x of k at time index k to the terminal state x of n. So, this indicates from where we have studied k th instant the state is transferred to a final state that indicates which instant the state is transferred and correspondingly this is the state is x of k. We are transfer the state to a final state if k is equal to 3 and n is equal to capital n is equal to 10. That means we are transferring the state x of 2 k is equal to 2 a final state k is equal to k of 10 because capital n is 10. So, that corresponding cost we denoted by j star of 2 if k is equal to 2. So, this is the same definition what we have considered earlier that what we are talking about this. So, what is j star of what j star of t m n the state at time t is equal to t m we are transferring the state to a final state agree and what is the final state that t is equal to t f what is the cost is required. So, because x of n is the terminal cost or terminal x of this not that cost this is the terminal point because x of t is the terminal point we have the terminal cost or cost terminal cost j n star x of n. That means j n final state to final state what is the cost that is the cost of this one by defining that way that will be the terminal cost j n star and whose values is what in this expression if you see just now in this expression if you see that means n is equal to k is equal to n you see what you are getting k is equal to you see this n k is equal to n k is equal to n. So, j x of n this value will be terminal cost is that much because k is equal to n means n upper lower limit is n upper limit n minus this expression does not valid only this term is left. So, the terminal cost is phi n of x n. So, this is the terminal cost now using principle of optimality principle of optimality principle of optimality cost for transferring the state transferring the state form x n minus 1. So, our state was x n is here now x n is capital X n minus 1. So, what is the cost or transferring from this to this. So, that we are going that using pure the cost of transferring the state from x n to the terminal point x of n as what you can write it. So, j n minus 1 star of x n minus 1. So, this meaning is we are transferring the state at index time index n minus k is equal to n minus 1 to k is equal to n mean final state terminating state for that one what is our cost minimum I have to minimize that u minimize u n minus 1 and what is our performance index phi terminal cost is phi n of x n this plus phi n minus 1 x n minus 1 u n minus 1 that one bracket close this one. Now, see this is the terminal cost. So, what is our n k n minus 1 step to final step. So, final step cost is this one that k is equal to n minus 1 to n minus 1. So, we have k is n minus 1 k x of n minus 1 u of n minus 1 that is what we have written. So, now we have to minimize this one and which is a function of u. So, we have to minimize this one. So, if you recall this is nothing but as according to our notation this is nothing but a x of n terminal cost. So, now this is from n minus 1 at time index to terminal time index means n. Now, in general we can write in general we can write j k x k of this star. That means, how will you read this one we are transferring the state at k is equal to k th time index to a terminating time a terminating state at k is equal to capital N time index k is equal to k which we can write it minimize u k. You see when n minus 1 minimize that because the information is only n minus 1. So, minimize this then k 2 that is your k th state at k is equal to k th instant to a final terminal condition. Then what we can write it for this one that will be a f k u x k then u k then what is required for this one because j k plus 1 k plus 1 star k plus 1 star x k plus 1. You follow this one in this expression in this expression I will write k is equal to k n minus 1. So, last term n minus 1 of this because we are doing backward cost calculation we are doing this one. So, this will be that which I can write it minimum u of k f of k x k u k plus j k plus 1 then x k plus 1 what we can write it is nothing but a f of k x of k u of k. So, now you see u of k is a function of that f small x is a function of small f then dynamic equation is a function of u k. That means I am writing into x k plus 1 into this form. So, this then you do the minimize that thing with respect to u k. So, this is the basic principle that you calculate the what is called control strategy backward pass and ultimately you find out the optimal trajectory of the dynamic systems. So, it is clear to you because I am finding out the optimal cost from k th instant k th instant to final terminal state when the state is transform transform form k th to k th time index that means state value is x of k to final terminal time index k is equal to n then what is the optimal cost. So, this is nothing but I am writing from this one k is equal to k n minus 1. So, from k th instant that last k th instant that this k is equal to I write it and other terms I clubbed with this one and which I have written into this form. So, let us take one example and see how we can use the principle of optimality to find the control law in case of dynamic discrete time systems. So, example so our problem is we have given a discrete time system who dimension is first order difference equations. So, let us call you write it since is a first order small x for x of k is equal to 6 u of k and your initial condition is given state is x of 0. So, this is the discrete time dynamic systems subject to our this is the system and minimize not subject to minimize the performing index and what is the performing index j or j 0 from initial state to final state we have given. So, this is given to phi in general expression I am writing first phi n x of n this plus summation of k is equal to 0 to n minus 1, then your f k x of k then u of k. So, for this particular problem this is given like the x of 2 minus 20 whole square. So, what is our what is our phi n x of n is this one your phi n of x n is that one what is our capital N? The capital N is 2 plus summation of half k is equal to 0 to n minus 1 n is 2 this is your n n is 2. So, it is nothing but a 1 and which is nothing but a n minus 1 of f k that f value is given this expression is given I am just writing your 2 small x k whole square plus then your 4 u k whole square bracket closed. So, the phi capital phi x n is given this one. So, our performing index is this one our problem is now find out u k that means I have to find out u k means one in order to get x 2 terminal state x 2 that reach to the terminal state x 2 I have to find out u 0 and u 1 by applying the our optimal control strategy that this is the our performing index. So, using problem is using p o principle of optimality find u of 0 and u of 1 assume no constraints on u of k no constraint of u minimizes that one. So, now what is unknown is here you say I have to choose a control law u such that this performance index is minimized once you find out u 0 immediately you can find out x 1 then once you know x 1 find out the optimal value of u 1 you will know x 1. So, if you move along this trajectory the performance index will be minimized this one. So, how to solve that problem solution. So, we start with backward pass so we have if you recollect that j 2 star of x 2 what does it mean small x 2 this means what is the terminal cost is required and our terminal cost is that one only because k is equal to 2 if you write k is equal to 2 here instead of k is equal to 2 2 1 that this expression does not valid. So, what you are writing for this one this equal to our terminal x of 2 minus 20 whole square is equal our terminal cost. So, this next we can find out what is called j 1 of x j 1 x of small x of 1 star what does it mean I know the terminal cost x k is equal to 2 this is the terminal cost. Then I find out backward from x 2 to x 1 in other words I can form x 1 to x 2 what is the our cost is involved that is nothing, but a minimize u of 1 f of 1 x of 1 u of 1 plus j 2 star x of 2 look at this expression. So, k is equal to 1 1 to 1 so it will be f of 1 x of 1 this then this j phi 2 phi of 2 or x of 2 is nothing, but a terminal cost j of 2 this one. So, that is you minimize so now I will write expression m of u u of 1 what is the expression for this one we know already this expression is given that one agree. So, we will write it that half twice x of 1 square plus 4 u 1 of 1 square so that is our that quantity plus this quantity is that one x of 2 minus 20 whole square that things. So, this is the terminal cost I have written and this is for x is equal to 1 that 2 x of 1 square plus u y of 1 square I have written now you differentiate this with respect to u that will give you if you differentiate with respect to this and u of 1 then half twice x 1 square plus 4 u square of 1 and this I can write in terms of x 1 and x 2 then that is nothing, but a if you see this one k is equal to 1 that means 2 x of 2 is equal to 4 x 1 minus 6 u 1. So, I can write it 4 x 1 plus 6 it is 6 minus that is minus 6 u of 1 minus 20 whole square that you differentiate with respect to it then by principle of what is called how you differentiate this with respect to this then we can write it note no constraints note no constraints on u of 1. So, optimal u 1 is obtain optimal u of 1 is function of x of 1. So, this if you do a difference equation that not difference equation that optimal u of 1 and that u of 1 you will get function of x 1 only this is what we have written it here and this is nothing, but it is the optimization problems it is the problem of optimizations discrete optimizations. So, what we can write del u of 1 then whole thing what is this whole thing that this half cancel this 2. So, I will write it x 1 square of 1 plus 2 that is 2 plus 2 u square of 1 plus 4 x of 1 minus 6 u of 1 minus 20 whole square whole bracket. So, if you differentiate with respect to 1 it will be 4 u 1 then u 1 is coming here. So, this will be twice this will be a twice 4 x 1 minus 6 u of 1 minus 20 into minus 6 is equal to 0 this equal to 0. So, if you solve this one you will get the expression u of 1 is nothing, but a 12 x 1 minus 60 divided by 19 this is equation 2. So, this x 1 now look at this one in basic equation of dynamic equation k is equal to 0 means x 1 x of 0 y of 0. So, x 1 depends on u 0 and x 0 not x 1 depends on x 0 and u 0 which we do not know only x is known this is unknown. So, let us see how to do. So, we are now where we are now x 1 x 1 2 we want to shift from x 0 to x 1 what is the optimal cost required optimal path and that we have cost is j 1 star of x of 1. It indicates is what the optimal cost transferring the state from x is equal to x 1 2 x is equal to x 1 2 x 2 what is the optimal cost because we know u 1 expression is that one. If you put this value in the expression that half twice x 1 square plus 4 u of 1 square plus j 2 star of x of 2 if you put this one then j 2 star is what we got it just now. We have found out j 2 star say j 2 star of j 2 star of this is that one. So, we can write it that half twice x square of 1 plus 4 u 1 we know what is u 1 12 x 1 x of 1 minus 60 divided by 19 that square. So, that is I am writing for this part plus j 2 square that one that is this 4 x 1 minus 6 u of 1 minus 20 whole square. And if you simplify that one finally, we will get it if you simplify this one we will get it finally, that x 1 x of 1 square then twice 12 x of 1 minus 60 divided by 19 whole square plus 4 x of 1 minus 20 by 19 whole square. The 3 parts x 1 square plus twice 12 x 1 minus 60 by 19 whole square plus that term if you use the value of u 1 here then 4 x 1 minus 20 then this one. So, this is the final expression let us call this expression is 3 and except number is this is 1 and 2 we have considered that u 1 expression what is u 1 expression that is 2. So, what is our next then we want to we know the optimal cost up to backward in up to the state x of 1. Now, we are moving the state x of 0 to x of 1 what is the optimal cost of that 1. So, j 1 star x of 1 this cost we have to find out and what you how will you find out you see that our basic expression for this one. Now, this is k is equal to 0 n is equal to 2. So, k is equal to 0 n is equal to 2. So, k is equal to 0 n is equal to 2 n is equal to 1 k is equal to 0 f of 0 x of 0 u 0 then f of 1 x of 1 u 1 that cost of we have calculated backward. So, our expression will be half sorry minimum of u of 0 minimum of u of 0 then half twice x of 0 x of 0 square plus 4 u of 0 square that 1 plus j 1 star of x of 1. What is j 1 star of x 1 that optimal cost required to transferring the state of x 1 x of 1 means k is equal to 1 time index 2 x of 2 this is the time index. Now, this is we have to find out minimum value of u. So, that this what is the trajectory from x 0 to x 1 that we can find out this one. So, we will continue this thing in the next class.