 Okay so in the last lecture we were looking at linear quadratic regulator problem so we have taken the most basic model in the beginning this is the model which is free of measurement noise any state disturbance somehow the initial state is not equal to origin and the problem is to move the system from a nonzero initial state to zero initial state that is the simplest of the problem that we have been looking at well of course I will slowly relax all the assumptions and then move towards a problem which is complete in which we consider disturbances not only just simple disturbances but different disturbances we will talk about regulation at any arbitrary set point so all that we will do okay but let us begin with the simplest problem so this is right now deterministic model A, B and C are known to me okay let us assume that the system is reachable which means I can move the system from any initial condition to any final condition and now I want to find out a control law okay that will move the system from initial point to origin that is the problem right now being solved we said that this problem is now going to be posed as an optimization problem the reason for posing at an optimization problem is to fold one is that optimization framework particularly this one which we are going to develop will ensure two things it will also ensure the stability it will also ensure performance the optimization objective function allows us to specify performance so here this term here is square of the distance from the origin so this is weighted norm of x vector this weighting matrix W is typically a diagonal matrix and it is used to do scaling of the variables that is one thing second thing is also used to tell which error should go to 0 fast there are two purposes that this weighting matrix serves one is you know different states can have different numerical value so you can make them sort of on the same footing by multiplying the by a scaling function second thing is you can specify relative importance if you give higher weight okay that means that error should go to 0 fast okay so in systems where states have physical meaning you can specify that this particular you know this concentration should go to should be controlled faster some level you do not care if it is oscillating a little bit so you can actually do a trade-off by specifying this weighting matrix same is true about the second term here second term is trying to weigh the input efforts okay you do not want very large input efforts also you have trade-offs in inputs you know some inputs are costly to manipulate some inputs are cheap to manipulate for example it is quite likely that if I am manipulating in some system steam and cooling water and a coolant coolant it might be cheaper to manipulate coolant than manipulating steam because steam you know you need energy cost cooling water just pumping cost you know so you might use cooling water liberally but not use steam liberally so all those things can be actually incorporated by choosing this matrix W U this is again typically a diagonal matrix this is not going to be a so diagonal matrix diagonal elements are all positive these are all positive different matrices and they will give you relative importance of each or you know if you here it is other way around if you put large weight on a particular input that input will not change too much okay that input will not change too much because your decision variables are U so suppose U 1 first component of U has a large weight associated with it in matrix W U then the first component will not change too much because if you change first component the cost function will change by large amount okay so you can specify relative importance of each manipulate variables all these things are very difficult to do when you do pole placement you cannot say place the poles that is that input 1 is not changing too fast but input 2 is changing fast all these things become very difficult coming to pole placement design but it is possible to do this using optimization framework because optimization framework is directly relates to time domain specification you can very easily give you know translate your control requirement into a optimization problem that is not so difficult okay so the third term is of course waiting on the terminal or the final state now how is this problem solved okay so solution is found by this dynamic programming okay so this is a this is a famous method dynamic programming which was developed by Bellman and then it has been used very widely in optimization or optimization based team the idea here is very simple that you work here backward in time so you first see here the problem if you go back here to look here this problem is posed over a finite time time 0 to time n is the final time so you want to reach close to the origin in n capital N sample capital N is not equal to state dimension capital N is some number okay so this problem is posed for a finite time okay we will then we will then relax this also we will then because the real system you know let us say some chemical plant or a power plant it does not work I mean in reality it works for finite time but you cannot develop a controller which will be for you know one year and solve it so here right now even though n has been used we will first develop a method for n and then we will say n goes to infinity and derive asymptotic river okay that is what is going to be done okay so what is done here in dynamic programming you start from the end so you start from n sample okay the last sample you choose an optimal input for the last sample having chosen in optimal input for the last sample you move back in time to n-1 okay then choose an optimal input for n-1 having chosen for n and n-1 you move back in time to n-2 okay in optimal state estimation in Kalman filtering we are moving forward in time okay now here you have specified you know the final time and then you are moving backward in time so you have to have some patience before you finish okay I am sure I can see a question on your face how can you specify final time well just wait okay look at the development and then we will come back to okay now let us define this quantity this matrix S okay this is a matrix S which is defined as Wn what is Wn Wn here is this weight here on the terminal state okay so Wn is a positive definite matrix which is the weighting on the final state okay that is after n time that is the so I am starting this algorithm with Wn and then look at what I have to solve I have to solve I want to minimize this jk okay starting from time k to time n-1 okay I want to choose time k to time n-1 summation see this problem has been defined from k equal to 0 to k equal to n-1 and now I am starting at some point k okay so at k equal to n at k equal to n only this last term remains you see this see I have this summation see I have this summation here go back here I think this one index is wrong it should be k equal to not so j equal to this should be k equal to 0 so this last term is what remains see if I want to optimize this backward in time okay first I should consider the last term so last term is this term right last term is this term yeah no so the Bellman's principle is this so you start with the last term and you move backward in time you optimize the last term having optimized last term then you optimize last but one then having optimized those two you move backward in time and then you go to 0 that is Bellman's that is a method okay how it will lead to a controller design we will see that just it is a tool okay no no so you actually solve you actually formulate you break it into multiple optimization problem okay this is one giant optimization problem okay see this problem here is a giant optimization problem it is from the 0 to n-1 you want to decide everything okay now now un-1 see look at this particular system there is a time evolution thing here why we are able to use this particular concept of Bellman's principle see un-1 you just tell me if I am optimizing this particular problem un-1 is it going to affect say x0 or x1 or x2 it is not the un- term will only affect xn okay so actually choosing an optimum un-1 can be solved independently of you know u0 to u0 to un-2 okay so you fix un-1 optimally having fixed un-1 optimally you move to un-2 un-2 is not going to again affect things in the past it is going to affect only future okay so you move backward so you take advantage of this particular fact okay to break this problem one giant problem into sequence of smaller problem okay and then that will lead you to a very elegant solution to this particular problem okay so what I am going to say here is so this is starting from any arbitrary k k can be 0 if k is 0 it reduces to the original problem I have written it here from any arbitrary k why this is written from arbitrary k will become clear as we go on okay I am defining this term jk I am defining two things sn and jk jk is the problem starting from time k to time n-1 so this is a this is you can say this is a kth sub problem this is the kth sub problem so I am going to solve actually instead of one big optimization problem I am going to solve a connected series of optimization problem and that will lead me to a solution so I am defining two things sn and jk okay so according to this definition this here what you have here this is j0 this is j0 because j0 to n-1 okay this is j0 and what is jn jn is the last term okay jn will be only this term okay sorry for j yeah jn will be the last term only the last term yeah okay so what is jn-1 jn-1 is xn-1 transpose wx xn-1 plus this u term plus jn right okay so what I am going to do next so this last term I am substituting in terms of un-1 what is the decision variable here see see what is the decision variable for the last term is only un-1 okay I only have to choose un-1 because un-1 will have effect term xn okay so so my jn is now written in terms of xn-1 un-1 sn is wn you have defined this okay and this is the term that I want to minimize with respect to un-1 okay so if I minimize this now you have to expand this right what I have done here is you know there will be four terms right this is just matrix multiplication you just multiply and then rearrange and then you will get this you know in between things this is not very difficult to get so this is just multiplication and rearrangement and the first term see if you see here this first term here okay xn-1 transpose this term this is xn-1 transpose xn-1 this cannot be influenced by can this be influenced by un-1 this cannot be okay so this term is ruled out this cannot be influenced by un-1 only this term this term and this term this last three terms can be influenced by un-1 so actually when you minimize you only worry you have to worry about one two and three okay is it okay there are four terms x transpose x okay x transpose u u transpose x and u transpose u so last three terms are relevant first term is not relevant okay because first term cannot be influenced by un-1 it can be influenced by un-2 un-3 that is a different story it cannot be influenced by un-1 okay yeah so jn I have defined so I am going to define this jk look so the first to define the last term that is jn then what is jn-1 jn-1 is jn plus one more term okay what is jn-2 to jn jn-1 two I mean there will be three terms okay and when you collect from j0 will be collection of all terms from n to 0 no no no right now so this is an intermediate step in the derivation just do not try to write now interpret okay so right now n even though I am saying it is final time the system may not have reached the final state 0 at time n so you may not have become 0 just see the derivation okay so this with this jn and this jn-1 is everyone with me on this I have just combined added terms together and club the terms of un-1 and xn-1 they are separated them and then written it like this okay so this last term cannot be influenced and then how am I going to solve this problem okay I have to solve this problem of minimizing this function with respect to with respect to un-1 okay so this is done by method of completing square okay so so just keep that jn-1 business aside look at this now consider a scalar quadratic function okay just look at this set are you are you convinced here what has been done just see the just try to derive this can you derive this see this form what I have done is I have written this abstract form here see what is the abstract form x transpose something into u u transpose something into x u transpose a positive definite matrix into u so there are three terms u transpose u u transpose x x transpose u okay so I can now think of see this is like a multi-dimensional quadratic equation okay I want to I want to convert it into a complete square okay if I can write it as sum of two squares okay then I can use that to solve the problem that is the idea okay so just see whether you can derive this just see if this result can be derived just check just do the algebra so if you have a scalar function in abstract form see that a and what is a matrix and all is very very cumbersome okay leave it aside right now just see whether you can derive this result okay u transpose au plus z transpose u plus u transpose z so second step is very obvious the third step would require expansion the third step requires expansion how will you find a minimum of this function pardon me no no no see this is is this see what is a a is a positive definite matrix that is very very important so what about a inverse a inverse is also positive definite matrix okay so what is the property of a positive definite matrix x transpose any vector transpose a into that vector will always be a positive quantity okay so this is a positive quantity this is always a positive quantity for any x any u this is always a positive quantity by virtue of the fact that A is possible definite what about this guy this is also positive quantity okay okay what is the lowest value this function can take no what is the decision variable my decision variable is u my decision variable is u z some vector see actually here map what will you map you map u to u n-1 to map u to u n-1 and this entire thing x transpose phi transpose Sn into gamma that you can map to z okay that you can map to z so we will see what is that mapping okay so what is the minimum value it can take the minimum value it can take if I am if I have to minimize this function with respect to u it is a function of u z sum vector which is constant yeah so if you can put this equal to 0 see if you put this term equal to 0 this is this a is a positive definite matrix so the smallest value this term can take is 0 when will you take it 0 only when this vector is 0 okay so the minimum value is this okay that is when you choose that is when you choose u is equal to yeah – a inverse z okay so now just do this mapping choose a to be this matrix gamma transpose Sn gamma plus w u okay this is the matrix which appeared in u transpose part okay and choose z to be gamma transpose Sn phi xn-1 okay if you do this if you do this what is the solution a inverse so inverse of this matrix okay into zz do you get this a inverse a is a a is a with a always be invertible this a will it be always invertible why Sn is wn okay wn is a positive definite matrix so is a always a positive definite matrix Sn is a positive definite matrix gamma transpose Sn gamma is a positive definite matrix w u is a positive definite matrix okay so addition of two positive matrixes will give you a positive matrix okay so this is this a is a positive definite matrix so a is invertible okay and you get this so my controller my controller can be written like this this is a inverse see look at look at this here look at here a inverse okay times this whole thing into x actually what I am getting is a state feedback control law what I wanted to derive was a state feedback controller okay I got state feedback control law for one particular instant so what was this particular matrix this a-a inverse into this matrix is the gain okay is the gain and xn this is a state feedback controller okay is everyone clear about this so it gives minimum loss function to be equal to this if you actually substitute you actually substitute and find out the minimum of jn-1 jn-1 will turn out to be this where Sn-1 is actually I have just done the algebra okay you can you can go back and substitute and see how it works so what I have done here just go back and see here see I had this I wanted to minimize this function I want to minimize this function okay so I am just doing it by completing the square okay I am not worried about this part because this first term is not going to be influenced by is not going to be influenced by w okay so to find out the minimum for the remaining three part what I have done is I have expressed it as sum of two square this is one square this is another square so this is always a positive number this is always a positive number okay the smallest value this can take is zero okay and then I just chose I just chose a to be this and z to be this okay this gives me a minimum of the original objective function I go back and substitute the optimum value of u and then so this is my state feedback this is my optimum this is my optimum un-1 okay which is this matrix times xn-1 okay now this is the result which you would have got even by minimization if you are done doh by doh you would have got the same result okay why do we need Bellman's principles will come to that so this is a local result for one you can get by minimization Bellman's principle talks about the series of connected problem okay so that is where so if you actually take this solution what is the minimum value of gn-1 substitute this solution into the into the you know that gn-1 expression you will get you will get this but gn and gn-1 I combined into one because gn and gn-1 both will be influenced by un-1 see un-1 appears in two terms okay it appears in gn-1 by virtue of the fact of the definition of the objective function it appears in gn because un-1 influences xn okay so actually if you go back here see gn is only this but un-1 appears in gn-1 because the way of optimization problem is defined see you just see the optimization problem is defined with respect to see what is gn only xn okay what is gn-1 this also contains this term so I have grouped all the terms that contain un-1 together okay so that is the reason I looked at gn-1 okay and I optimized gn-1 with respect to un-1 okay and that optimum is optimum is optimum un is found to be this okay I am going to call this particular time varying matrix as gn-1 okay this is my notation and when I substitute this particular solution into gn-1 I get this quantity okay this is pure algebra I just took this solution I just took this solution substitute it there whatever term I get okay that I am calling as sn-1 and I am calling as sn-1 my intention here is to derive a recurrent relationship between sn-1, gn-1, sn-1, gn-1 remember what we did in Kalman filtering we had covariance and we had Kalman gain and they were coupled with each other okay Kalman gain okay is used is based on the covariance update okay can you again update the you have predicted covariance Kalman gain calculation updated covariance again you do Kalman gain calculation predicted covariance it goes in the cycle okay so likewise here I want to derive a recurrent relationship in time okay which are again called as Riccati equation except now it is done in the context of control other than the context of you know state estimation okay so now having optimized gn-1 you move to gn-2 you are moving backward in time okay so now I want to minimize this term here gn-2 with reference to un-2 because I have see this term here is influenced by un-2 okay and here this is influenced by un-1 okay and so this has already been optimized except there is one thing here there is one thing here which is influenced by un-2 what is it xn-1 see look at this look at this relationship here I want to point out that they are actually analogous see here this is this is xn-1 sn-1 xn so this term will appear here earlier we had see earlier you had xn-1 and jn okay you just shift in time okay and you have already chosen un-1 so now only you have to choose optimally un-2 okay so algebra is messy you can if you sit down and do it it is not difficult to do this algebra but just understand the concept if you are getting stuck anyway please stop me okay so again okay again see this expression looks very very similar except this matrix here has become little complex this sn-1 is now a very complex expression you see this okay apart from that see if you accept that somehow this sn-1 is coming from the preview from the one step ahead in time okay then this expression is philosophically same I mean it is the same expression okay so you have this term and then you have n-1 what is in the n-1 that you can influence using un-2 xn-1 see earlier what was what was the case even here when it was jn it was xn xn transpose some matrix xn transpose okay so u term was not there in the last term here also now after you implement this control law u term is not there you just have this optimal you just have this optimal jn-1 which is where this sn-1 is given by this big matrix here okay so now how do I minimize this again same thing complete the squares okay now using using this this term okay if you complete the squares you go back in time okay like this by completing the squares you get what are called as Riccati equations okay this Riccati equations are moving backward in time okay so where g is given by this and sn so your controller your controller is found like this you start from sn okay sn okay then for that you find out gk that is your control that is your controller gain at time instant capital N capital N is the last time in your optimization problem okay so you find this you find this gn okay for the time being keep aside the fact that how will you implement this just look at it as a you know optimization procedure okay we first fix u n-1 having fixed u n-1 we go and fix u n-2 having fixed u n-1 u n-2 we fix u n-3 so we go backward in time okay so actually we are what we are getting here is a series of time varying gain matrices we do not get a control law with a fixed gain you get see just like in Kalman filter we got gain matrices that are changing as a function of time okay so here to you get a sequence of gain matrices that are changing with time they are coupled with each other they are coupled with each other through these Riccati equation okay through this Riccati equation except that you start from capital N that is the last time and you go backward in time and you will get a series of gain matrices okay so a practical problem you might ask is that what do I do I mean suppose I am running some plant okay let us say I want to take a driver motor automatically on some road and every one second I have to take a decision and I have to go from here to the other and if it takes a you know one hour or two hours to go to one hour to go to the other so do I actually compute 3600 gain matrices do I save them and then at each time point I just retrieve one gain matrix multiply and find the controller that is impractical okay so we will start this problem on do not worry about this right yeah I finally want to reach xn is equal to 0 so now aim in this particular case is not xn is equal to 0 in this particular case is to make xn as small as possible very close to 0 asymptotically I want to go to xn is equal to 0 this is for any arbitrary action now what we will do now what we will do is we will say that under certain conditions you can find us see this is a coupled equation okay this is a different equation why it is a different equation k depends upon k plus 1 see normally you are used to different equations which move forward in time this difference equation moves backward in time okay this difference equation moves backward in time what is elegant about this solution is that you have a close form solution okay you get sequence of optimal gain matrices okay and you get a close form solution now this is not practical because many times you do not know how many suppose you are going from here to the other you want to you know automate your car and leave it on the road you cannot predict what time it will take then it will take 1 hour 20 minutes or 2 hours depends upon the road condition so you this is not unless in some cases if you are you know hitting some target and you know you are going to hit a particular target in so many seconds then probably you know you can be calculate all the gain matrices keep them and then it will do the job but this is not practical but this gives you a recurrent formula and how will I use this next will become clear next so let me just summarize this first what is the how do you how do I calculate this so my feedback gain is gk and then gk is calculated by this recurrent relationship these are appearing here in the this is this different situation back in time and then what is guaranteed is this sk is always a symmetric and positive definite matrix because you start with Sn is equal to Wn this is a symmetric for the different matrix and the way the whole thing is constructed you are guaranteed that this Sn matrix Sn will always be symmetric and positive definite but the problem here is that n should be no no priori how many n how many such gain matrices you find out okay so but now for the time being leave the other side you got a way of calculating the optimal solution okay we will find asymptotic solution of this particular problem by letting n go to infinity okay we will find that solution and we will use that solution to I just want to draw parallel to ekf you remember what we did in ekf you know we had this solution which was last time solution or infinite solution okay so in this case we just said that this goes to infinity and then time goes to infinity as k goes to infinity the solution converges to a steady state problem okay and then you can get here you can get l star infinity okay you can get this infinite time gain which is only one matrix okay so this way you do not have to compute lk matrix at every time instant so after some time when you program this you will realize that after some time if lk goes to a steady state you do not have to actually keep computing it every time all these matrix equations so they under certain conditions of course on observability and so on so I had talked about Kalman predictor you remember Kalman predictor in Kalman predictor we had two equations we had gain update covariance update right we had gain update and covariance update and then this LP star LP star was the optimal gain and then we computed this gain for moving forward in time and then we had of course conditions under which you know you have this coupled equation and you can solve this algebraic Riccati equation you can solve them you know the solution exists and I talked about this lemma you know which tells you under what conditions the solutions exist when this phi square root q is controllable and r is non-singular then a unique solution exist a solution exist such that I get values of this matrix are inside unit circle so we had solved this problem earlier and I just want to draw analogy here if there are two things here this covariance update or covariance equation and the gain equation and they are coupled okay same thing is happening here if you just look carefully at least this LQG nodes I will upload today okay see here there are two coupled equation this is something like covariance update this is something like covariance update this is I mean qualitatively this is similar to covariance update this is similar to gain Kalman gain update so there is a gain calculation covariance calculation similarly here this SN calculation and gain calculation okay so now what I am going to do is to let SK go to S infinity which what happens is that GK will go to G infinity this will happen only under certain conditions of controllability and you know like we had conditions for algebraic Riccati equation for the Kalman filter same things will be holding here so you can show that under certain conditions this coupled set of equations have a solution and this equation is actually G infinity okay let us call it G infinity this gain so now if you solve this coupled equations you have one gain matrix which is optimal when n goes to infinity okay and for practical implementation all that derivation which I did I am not going to use I am going to use this algebraic Riccati equation find out G infinity and use it for my control okay so that starting with last point going backward in time all that was in between steps all that was an in between steps okay now that I have derived it I am interested in this infinite solution okay and this infinite solution is what I am going to use for designing my so this is the state feedback controller which I have now okay how do you solve this coupled equation can be solved under just like you had conditions there you have conditions here okay if you consider if phi and gamma is controllable there we had a condition of observability phi and C were observable here you have phi and gamma is controllable okay and this matrix WU matrix okay if you take this square root of WU matrix and if this is observable yes so you take square root of that okay this is called square root this omega is called square you know about LU decomposition a positive definite matrix can be written as L transpose L transpose L so this is a diagonal matrix so it will be just square root WU will typically be a diagonal matrix so this should be just a diagonal matrix okay so if this pair is observable and if this pair is controllable okay then you can guarantee that there exists a unique solution to this particular problem so this problem has a unique solution and you will get a unique gain matrix G infinity which will stabilize the closed loop okay which will bring the system from nonzero initial condition to zero initial condition okay which will ensure that the poles of C if I substitute this control law if I substitute this control law what is the closed loop solution what is the closed loop behavior is this okay is everyone with me on this correct so now I will prove stability okay I will show that asymptotic this using this controller is equivalent to I will do asymptotic stability with this this controller okay if it is asymptotically stable it should go to zero yeah so well while doing the derivation whether I started by saying xn is equal to zero or not is a different story all that I said when I started a derivation that extent should be as small as possible I minimizing your objective function with respect to you okay so my objective there was to choose input such that xn is as small as possible okay I did not say it equal to zero okay I did not say it equal to zero I just said it should be very small so no but I do not have to choose now no see I have just now got rid of that xn business see now I am just going to say that n goes to infinity and then go to infinity okay then I actually have to solve this problem this coupled equation okay so that xn and all that business was an intermediate step to derive this algebraic Riccati equation okay solution of the algebraic Riccati equation is going to give me g infinity which I am going to use in as a my control law is everyone clear on this all that dynamic programming and you know all that I did just as an intermediate step to derive this particular control law okay but this is a state feedback control law the state feedback control law okay yeah so that is you know for existence of solution of that particular equation you have to have two conditions you can go and check this proof in the one of the standard text which I will be listing at the end of the so the proof is very elaborate I do not want to get into the proof so this is a I am just quoting that I have not written as a theorem but in any book this will be a theorem which talks about conditions under which the solution exists for the algebraic Riccati equation see this is a matrix coupled matrix equation it is not an easy equation to solve okay so because these are this g is a matrix s is a matrix and these are coupled equations so solving this matrix equation is a tough problem and then there has been lot of work on how to solve such Riccati equation so you can just I will refer you to the book so this is a decomposition any w you can write so this is in MATLAB we have a function called so this is that is standard see the nice thing about all these things you can just use linear algebra ultimately just algebra simple linear algebra that you applied you can view this entire control theory as you know in state space the domain has applied linear algebra just so is everyone clear with this now you know you is the infinity times this so my closed loop becomes this okay is this clear I do not see lot of convince people I can go over it again you tell me where you are stuck which part you did not understand see let us look at it like this I started doing this derivation let us go over it again so then what I do is once I do this I ignore the fact that gain is time varying I just look at gain at as n goes to infinity I am just finally interested in deriving the algebraic Riccati equation for that I am doing this is an intermediate set how to arrive at the algebraic Riccati equation so I am starting with unsteady state case okay where n capital N is some intermediate point you are not reached the origin yet okay capital N is some intermediate point okay I have not specified n okay I use this only to derive this recurrent relationship see what is the recurrent relationship recurrent relationship is between this I mean finally you know you got let us go to the yeah so you I just wanted to derive this recurrent relationship between the gain and this possibly definite matrix SK this is not covariance here this is not covariance but you know in the Kalman filter derivation this is somewhat similar to the covariance update so this matrix update and gain update matrix update and gain update that is how you got this equation and then finally what I am interested is not in this time varying solution I am interested in the solution which is as n goes to infinity okay so I am interested in the steady set solution and then finally I am never going to implement a control law like this where GK is time varying I am interested in this control law where G I am interested in this control law okay so all that dynamic programming I have just use as an intermediate step to arrive at this algebraic Riccati equation algebraic Riccati equations give me okay see now here you know going back in time everything has vanished because here when the solution is converged SK plus 1 and SK are same right when the solution has converged there is no difference between SK plus 1 and SK so that is why steady set solution there is nothing like moving back in time or moving forward in time with the solution so I am just interested in finding out the steady set gain I have used that final n as a vehicle to arrive at this steady set solution okay and then I am setting you that giving you the conditions under which the steady set solution exists okay yeah so maybe first few samples it is not but as if you take n to be very large then it is optimal okay so this G infinity is optimal G infinity okay so this is my this is my closed loop solution okay now I want to show that this is asymptotically stable okay I want to show that if I prove asymptotic stability okay then you know yeah then see we have seen one thing for the app on a function if if if Eigen values of this matrix are inside the unit circle okay then you can always construct a Lyapunov function okay whose you know derivative is negative and strictly negative and you will you know the system will be asymptotically stable and vice versa so force inside the unit circle and ability to construct a Lyapunov function for a linear system time invariant system are synonymous so if I construct a Lyapunov function for this it implies that the poles are inside the unit circle for linear systems these two things are equivalent asymptotic stability I am being able to construct a Lyapunov function okay for the particular system so instead of trying to find out the poles of this system and show that they are inside the unit circle I am going to construct a Lyapunov function and show that you know the Lyapunov functions you know rate of change of Lyapunov function is negative definite so this is asymptotically stable system which means the poles of phi minus gamma g omega have to be inside unit circle so this is stable asymptotically stable okay that is the argument I am going to use so I am constructing this I am constructing this this function v of x which is xk transpose x infinity x infinity is a positive definite matrix by the way okay x infinity is always a positive definite matrix so what do I have to do to find out through the Lyapunov stability find out the difference between two successive values of the Lyapunov function so this is vk plus 1 and this is vk okay. So is everyone with me on this I am just substituting for the closed loop okay so this becomes yeah this becomes negative definite because look at this term xk transpose phi s infinity xk will it disappear or I have missed out some algebra in between there is an algebra in between right because phi will come phi transpose I have to find out in between algebra you can work out this algebra the expressions are correct okay let us see how one step I have so this wx plus g transpose w is a positive definite matrix so you can show that define in this Lyapunov function okay you have this delta vx which is positive definite so I am just worried about one term we have to go back and check I will just take this and confirm okay. So the argument that I want to make here is that the closed loop system asymptotically stable for any choice of wx and w that is very very important I choose these waiting matrices will possibly definite but arbitrary okay now in reality you will not choose them arbitrarily but you are guaranteed to get a stable controller stable closed loop behavior for arbitrary choice of wx and w okay for any of any wx w you will always get this matrix to be positive definite and then I think you have to simplify this using g infinity you have to use that expression for g infinity expression is pretty complex g infinity expression is this right you have to substitute and then expand to get so from here to here from here to here it will do lot of algebra okay it is not obvious it is not obvious from the from this so from here to here I have missed out lot of in between steps you can sit and work out by substituting for g infinity that expression on the previous slide and then do a good expansion and then cancel all the terms that appear and so we will have to do lot of work to arrive from here to here okay but that is not important for us that algebra is not for important do not worry about the algebra this is a standard textbook material now this was worked out way back and this is you know since it is possibly definite you just for the time being what is important to know is that this expression can be reduced to this simple expression here and this is positive definite x trans this positive definite matrix so x transpose this into x this whole term is always positive and negative of that is always negative okay so the so you can construct a Lyapunov function for this particular system and if this is negative definite then for any choice you know you will get a stable closed loop behavior okay the nice thing about this particular controller is that it not only guarantees stability it also guarantees performance because how is the performance defined through the objective function you have given an objective function okay so using those Wx WU matrices make sure okay make sure that you have desired performance translated into an objective function okay and then that gets that gets you know accounted for when you do the gain calculation so this is like I said this is just placing the poles inside the unit circle is like saying that you know you have passed the course you are above four but here you know you are ensuring both performance and stability you are above four and then you are the topper okay so by choosing this Wx WU you have to have some experience as to how to choose Wx WU you can actually shape the speed of recovery from non-zero initial condition to zero initial condition you can you can choose priorities you can prioritize different inputs you can actually give more importance to one input and less importance to other input and all kinds of things you can do so you can wait a costly input more and not allow it to move too much if team is costly okay don't change it too much all that can be done by choosing this weighting matrices that is yeah so G infinity can see now actually when you solve it when you calculate you don't do all that optimization that is just a derivation when I am asking you to know the next problem is that implement quadratic optimal controller on the system that you are okay so what you have to do is not do all this optimization what you have to do is take this particular problem you just define WU to define Wx okay define WU define Wx and then give it to MATLAB there is a subroutine called ARE algebraic Riccati equation okay you just give phi gamma Wx WU okay see what all things you are able to specify phi you have to give phi matrix gamma matrix Wx and WU if these four things are given okay MATLAB subroutine ARE will solve or psi lab will be there will be some other subroutine equivalent to this it will solve this problem algebraic Riccati equation and it will give you these two things what is the what it will give you back G infinity and S infinity both it will give you okay of course you should make sure that these conditions are met phi and gamma should be controllable okay and choose WU such that this pair is observable what is observable what is controllable both you know now okay so now you make sure these conditions are met so ultimately the working recipe is that this one slide all that was the derivation yeah but typically this asymptotic solution if you start programming so if you take Kalman filter okay I am going to run this plant for you know let us say I am doing it every one second interval I am doing calculation I am going to run plant for 20 days okay so actually it is like saying K goes to infinity because you know within some 100 samples the gain will saturate and you will get a steady state solution similarly if you start here if you start solving the dynamic Riccati equation okay you will find that in some n equal to 100 you will hit the G infinity S infinity so you will get the steady state solution with no time you know so those first see suppose I am going to run a plant for you know 20,000 time steps and if the gain saturates to G infinity within first 100 okay I give first 100 I am not so much worried about 1 to 100 being non optimal I am if 100 to 20,000 is optimal I am willing to you know sacrifice on 1 to 100 not being optimal okay so long time solution which actually means within some you know 100 or 200 samples you will get this G infinity okay so very short time the solutions will convert if the conditions are met so how do you find out the closed loop holes so this can be done using of course you can solve for this determinant of this it can be shown that it is also a solution of this generalized eigenvalue problem and this particular solution this particular equation is called as Euler equation for the linear quadratic problem and you can you are guaranteed that for any choice of W U and W X you know you will get a stable solution okay so stability and performance both are guaranteed by this particular approach that is the key thing here okay now see these things you read the derivations and then of course once you start implementing it on the reason I have given you the toy problem is because I want you to go back and implement it on a toy problem immediately okay that is the way you will understand in this course to do a lot of algebra and you know you cannot understand things even by solving some tiny problems of 2 cross 2 matrices of 3 cross 2 matrices actually implement it on a real system at least in a simulator and see how it works okay you now have 5 gamma matrices to start over non-zero initial condition and see whether you go to the 0 okay so of course you can do what about we started this problem by saying X transpose WN x right this objective function of form is equal to X if you have you know a model which is developed from data then X may not have any physical meaning only Y has a physical meaning so you can actually solve this problem Y transpose W Y into U transpose W U get this all that I have changed is instead of X transpose X I have changed it to Y transpose Y but this problem can be easily transformed into X transpose X problem yeah this can be very easily transformed into X transpose X so that is not an issue so problem which is originally X transpose Y transpose Y can be converted into X transpose X problem okay what is this CR here is you know I have said that those those outputs which you want to control you may have modeled many outputs not all of them you want to control so CR will give you that part of the output from the state which you want to control okay so this is so you can convert a problem which is output regulation to see many times many times you know the model is developed not from physics is developed from data in that case X does not have physical meaning so it is difficult to choose weighting matrices and all that only Y has a physical meaning Y is a real measurement so you can define a problem with respect to Y and convert it into a problem of X because X and Y are related okay you can convert that problem and then you can solve this problem by you choose WX to be this WN to be this and then you can solve the design problem so that is well now let us relax one by one we said that there was no we said that there is no state noise there is no measurement noise okay so what do I do if there is a state noise and measurement noise what we do is of course we have a state observer and then we have observer controller pair okay so my process dimension is now this okay so I am not going to ignore WK and DK okay I am not going to ignore them I will implement the Kalman predictor for example I can do this with Kalman filter also I am just showing you this with Kalman predictor that is because Kalman predictor I have to deal only one with one equation it is easier to do algebra for me so that is why I put Kalman predictor here so this is my Kalman predictor I will get estimate of X hat K given K-1 okay my controller will be okay it should be here not LK it should be GK here sorry this LK is the observer gain this is the LK here is the I made a mistake it should be UK is equal to should be G infinity okay and even here this can be this is okay so how am I going to take care of measurement noise and you know state noise I have an optimal state observer combined with the optimal controller okay this is called as linear quadratic Gaussian control why this Gaussian business comes in because I am using Kalman filter to estimate to estimate the state to construct optimal estimate of the state okay I am using optimal I am using optimal way of computing the gain matrix using Riccati equations so this is computed using another set of Riccati equations this is computed using some other set of Riccati equations so both cases both cases we have used the same theory we have used the same theory okay optimal estimation theory optimal control theory they are just you know mirror images in some sense if you if you try to draw parallel you will see there are very close parallel except one deals with something in future other deals with something in the past okay so you have this so I can have this pair optimal observer controller pair and of course I have to worry about whether this is you know jointly stable or not so I will be talking about what is called as a separation principle in my next lecture where we will talk about stability conditions joint stability of Kalman filter and optimal controller pair okay so they can be shown to be jointly stable so if I design my observer to be optimal observer which is Kalman filter and then I design my controller to be optimal controller linear quadratic optimal controller and then I implement the control law through reconstruction of the state then I am guaranteed that under nominal conditions they are jointly stable okay I can separately design observer I can separately design the controller I can marry the two okay that together they form a stable closed loop pair okay effectively if you see here this is an output feedback controller why it is an output feedback controller see because how is x hat constructed using the model and measurement okay so the measurement is feedback to this observer which reconstructs the state that state is used to compute UK and that is injected here okay see the state is not measurable so the idea of stop sensing is implicit here I do not have I do not have direct measurement of x I have only measurement of y y is used through an observer to estimate x estimated x is used for closed loop controller I have a way of computing optimal gain for the observer given the noise model okay I have a way of computing optimal gain for the controller given an performance objective okay and I want to reduce it close to 0 as fast as possible by considering the input weighting and all that there is a problem here first of all we have to look at stability second thing we have to see you know we said that non-zero initial condition to zero initial condition very very restrictive I want to go I want to go from any output to any other output I want to specify a set point and move from one set point to other set point tracking okay the second problem is there are drifting disturbances there could be plan model mismatch okay and then this controller will not give you offset free behavior how do you get offset free behavior with this pair okay how to do some modifications to do that the Kalman predictor of course I have just summarized here okay so I need to I need to do couple of things I need to now so now I know how to compute gain matrices is that okay I need I know how to compute observer gain matrix I know how to compute optimal controller gain matrix now having done that I have to move on and talk about two things okay what is the model and the plant are not matching this is a mismatch within the plant and the model what if the disturbances see here when we derived this Kalman filter we said that is WK is a white noise okay in reality there could be a disturbance which is not a zero mean white noise it could be a drifting disturbance okay which is most likely the case in the material system there is never see one example of let us say this disturbance modeling is when you are driving a car so you know you have wind or air resistance okay so air resistance if there is no if there is no wind but there is general motion of the air it could be probably approximated as some white noise affecting the you know some local motions of air affecting your car but if there is a strong wind in certain direction then you cannot approximate as a white noise that particular disturbance might be something like a colored noise and then you have to account for it systematically in your controller calculation so how do you do that so that is going to be our next task so now I want to move yeah so it is the problem of regulation but for disturbances that are drifting okay so how do I now modify my controller in such a way that I can account for drifting disturbances okay I also account for the fact that the model and the plant may not be identical they might be a mismatch so how do I how do I so phi and gamma which I am assuming might be different from what is actually there in the plant I have identified them now today okay after a period of over a period of time the plant characteristics okay I am still using the same old model okay see now that you are doing perturbations in the plant and collecting data how many times you can keep doing this you will do it once and then you know you will do it after some time so in the meantime the plant can change okay so now what now will you be able to do the controls with this so I need to do some I need to do some tricks to account for the fact that the model plant mismatch can develop over time I also have to do something to make sure that I reject drifting unknown disturbances okay so so now what I am going to do is I am going to modify my control law like this okay I am going to modify my control law what was my originally control law my control law was uk equal to minus okay and I wanted to go to origin okay so what I am going to say is that the origin itself is not fixed okay I am modifying this control law by using a steady state target here this excess is called as a target state okay so instead of saying that take x to the origin I am saying that take x to a target state excess so difference between this should go to 0 okay what is this target state how do I compute this target set so I am going to I am going to compute this us and excess these two quantities this is target input this is the target state okay this target input and target state are computed to account for two things one is unmeasured disturbances second it is computed to account for set point changes see I may want to move the system from one set point to other set point okay using the same controller how do I do this okay so that is going to be done by managing so the control law looks very very similar if you see here earlier my control law was uk equal to minus g xk I am just putting some compensation terms here okay this compensation terms will allow me to take care of moving targets it will take allow me to take care of unknown disturbances okay so excess represents the final edit state target okay and it says it is changing at the function of time where I should reach because it depends upon the drifting disturbances and so this is a more realistic formulation of LCOC in which the disturbances are not wide okay the model is not perfect okay I am able to know I will now I will now solve a more realistic problem okay using now what I have sorted out till now is how to find out ginfinity okay all that I have sorted out till now is how to find out ginfinity okay having found out ginfinity now I am going to modify my controller to take care of drifting disturbances to take care of any set point okay earlier I was only worried about moving to origin 00 very restricted assumption now I want to change it to the how to do this okay that will see in the next lecture you have to do lot of so what I will do now is I will upload this hopefully within a day or so I will upload this well and the thing that I am expecting you to do is as a part of your project is implement these controllers okay implement state observer okay in open loop first and then implement linear quadratic optimal control and observer together and control your plan okay that is what is and actually when you do it that is the time when you will learn it for because this algebra here on the you know on the board becomes very very messy and then you do not get feel for what is happening unless you do it you put your hands into it actually control one particular simulator at least the system then you will get feel for what is really happening okay load this.