 discuss the Hamiltonian formulation for the solution of optimal control problem and then we will consider some numerical example. So, this is formulation is Hamiltonian formulation for solution of optimal control problem of course, using calculus of variation. So, if we recollect why we are going for Hamiltonian formulation, suppose the system dynamics or system plant is described in state based form means any system can be described with a n dimensional vector. So, n dimensional what is any system can be described with a n differential equation in first order differential equation, if you represent the system in state based form it is convenient to deal with Hamiltonian function rather than the Lagrangian function. So, let us recall our earlier problem consider the plant or system x dot is equal to f of x t u of t and t and this dimension is n cross 1, this is n cross 1 and this state number of states is n cross 1 and number of inputs to the plant is n cross 1. You can think of it we have a plant or system plant or system this one input is u of t whose dimension is n cross 1 and this state you can say output is the state you can say x of t whose dimension is n cross 1. So, this plant is described in general we have a dynamic system is there we can describe with a nth order differential equation which can be converted into a what is called n first order differential equation. So, once you convert into a state based form then our problem is here and the corresponding performing index j if you recollect this one we have written x of the terminal cost is a function of x t and t at t is equal to t f final time plus integration of t 0 to t f final time v x of t u of t and t d t. So, our problem is to find out the control law u such that this performing index is minimized not only that subject to the constraint that equation let us call this is the equation number 1 that is the equation number 2. So, our problem is to find out the control law u of t which in turn to find out the optimal trajectory x of t such that this performing index is minimized subject to this constraint equation number 1 that is what we discuss less plus and correspondingly we got what is called the necessary condition. If you say our necessary condition along with the boundary condition we got it that del l this x of t minus d of d t del l dot function of l is a Lagrangian function which is a function of x t u t lambda t and comma t. So, that differentiate with respect to x dot of t whole that you solve this one along the because you have to find out star means you have to solve this one whatever the solution you will get it that is the optimal trajectory or optimal control law from which in turn it will give you the optimal trajectory x star of t. So, that equation you have to solve. So, we have a in turn we have a n cross 1 differential equation is there. So, that differential equation in general it is a non-linear differential equation. So, that is let us call equation number 3 and not only this another x condition we got del u del t this one is equal to 0. So, this is equation number 4 corresponding to our this problem and we have a boundary condition that boundary condition if you see this one that we obtain boundary condition we obtain l Lagrangian function minus del l of with respect to x of t that since x is a vector. So, l is a square quantity that will be a vector column vector. So, we have to take transpose of that one and then multiplied by x dot of t. So, this you along the trajectory we x star then t is equal to t f this is the boundary condition we got it when t f plus then del l dot del x dot of t whole star t is equal to t f and delta x f is equal to 0. So, let us call this equation is 5. So, and we know what is the l the l expression we have written in terms of what is called Hamiltonian functions where l dot which is a function of x t comma u t comma lambda t comma t. So, this is a function of that Lagrangian function is split up into a what is called Hamiltonian function and then Hamiltonian function free form x dot. So, this is the Hamiltonian function and that function is called Hamiltonian function. So, Lagrangian function this Lagrangian function is nothing but a if you see what we have consider this that we will write it in this plus del s dot plus del x of t whole transpose x dot of t. So, this is a function of what is called v this is the that function if you see integral part of this integral 1 v plus Hamiltonian function plus lambda transpose of f that one. That means this two terms if you see this term and this term is free form x dot and that function we consider as a Hamiltonian function because why you have express that necessary condition what we got it here necessary condition equation number 3 4 and boundary condition 5. If you express this thing into Hamiltonian function then we will see if the system is described in a state based form. If you express this in place of Lagrangian function if you replace by a what is called Hamiltonian function a set of equation what will get it in a simpler form and it will be easier to solve what is called if the description is state based form. So, that we will see later of this one. This is our equation number let us call 6 now we replacing the you can say using 6 in 4 to using 6 in using 6 in equation 3. That is the 3 equation not 4 3 4 and 5 3 2 3 2 3 2 5 comma we get what let us see. So, our first form 3 we can write form 3 is del 3 you see del l del x. So, in place of del l I will just write it in terms of h that whole expression I will write it. So, del l del x in place of l I am writing h x 2 del x. x of t u of t lambda of t comma t this one plus delta s dot of that x of t whole transpose that dot of t plus del s this del t minus lambda transpose of t x dot of t. So, this whole thing if you just see the whole thing from here and including this one is nothing but our l Lagrangian function. So, I have just written the first term of our equation this is that one then second term if you see this one I am writing is minus d of d t. Then del l del l again this whole quantity differentiate this with respect to x dot. So, what is the del l I am just writing h x of t u of t lambda of t comma this plus del s dot del x of t whole transpose x dot of t plus del s dot del t this one minus lambda transpose of t into x dot of t. So, that is our we have written l this whole thing just same as this one same as this one we have written it here then this is differentiating with respect to x dot. Now, look at this expression that one chain rule we have used frequently that chain rule is like this way. Suppose, we have a function x is a function of x t t is the parameter and y t and z of t this is the function of this. Now, if you want to find out the differentiation of x with respect to d t this is nothing but a partial differentiation of f with respect to x then x dot then del f del y then another variable del y of t then y dot. So, what is this if you have a function x of function x of t y of t function of x y t which is a function of time each then I can differentiate on of x with respect to time t is nothing but a partial differentiation of x partial differentiation of f with respect to x multiplied by x dot of t. Similarly, partial differentiation of f with respect to y t is y dot of t and z dot of t. So, this things we will write it here. Now, see this one s is a function of what this and this I can write it s is a function of x t and t this is a function of if you see this is the function of x t and t this one. So, now I am writing it this one that this and this term and this term commonly I can write it. So, look this term by using the chain rule this term and this term commonly I can write it that del d s of del y of t. So, this is what we can write it this and this commonly I can write. So, rewrite this equation what we write it this one just say del l del x of t then h I am not writing h dot means is a function of x t u t lambda t and t this h plus d s dot d t agree. And now this what is left this and this d s of d t and lambda transpose minus lambda transpose of the x dot of t agree. This is the first part which we have simplified then second part of the Lagrange equation d of d t see this one that whole thing this quantity this quantity you are differentiating with respect to x dot h this h is not a function of x dot. So, if you partial differentiation if you do with respect to x dot this term will not be there agree the first term will vanish then this and this is a function of s is a function of x t and t. So, this is a function of x dot. So, we can take this is a constant because it is not a function of x dot t. So, if you do the differentiation of this one del s this del x del s partial differential s with respect to x t we will get it that that one agree and this is not a function of x t minus x dot t. So, this will not come into the picture. So, and now it is a function of x t. So, only this term and this term will be remain in the differentiation of the Lagrangian function with respect to x dot when Lagrangian function express in Hamiltonian form. So, this term ultimately it will come if you see is coming like this way if you just do it then it will come and if you recollect once again that if you have a function if you are differentiating that function with respect to x x is a vector let us call say and you are getting a transpose x a is a row vector a is a row vector of dimension n cross this is the row vector of dimension 1 cross n and this is n cross 1. So, it is a scalar quantity. So, that is differentiating with respect to a vector that results is a. So, now if you just use that one the differentiation of this is you can consider is a scalar this is you can consider is a scalar multiplied by a what is called differentiation this thing by a what is called a vector. So, that results will be a row vector and you are differentiating with respect to this. So, results will come the differentiation of this with respect to x dot that term. So, I will ultimately I will get differentiation of s with respect to x of t and this term similarly this is minus lambda transpose x dot differential x dot is one and this will be lambda of t minus lambda of t. So, this whole thing if you see this one you can put it that whole thing is in bracket of that you if you just do it here this is not l this is of that that one is star you can put it you like you can put it in star that one. Now, this equal to 0 because what from where you are getting that first equation of from equation 3 this equal to. So, right hand side of this one equal to 0 this one so from 3 I am writing from 3 this is 0 and I am getting that one. Now, see look at this expression differentiation of partial differentiation of that quantity with respect to x of t and this order of differentiation you can change it. So, if you change this one this is plus this is minus this this is cancelled only is left over is this one del h del because that is the function of you have to differentiate with respect to x t. So, that will be vanish that there is this is lambda t lambda transpose t x dot of t. So, you have to differentiate with respect to x ultimately it is a del h this del h. So, del x of t whole star this bracket this star is here this and this is also star from here to here. So, this equal to this plus this minus this minus plus equal to your this term is what differentiation with respect to t that. So, it will be coming minus minus plus if you take right hand side it will be lambda dot of t. So, let us call this equation we have used the equation number up to 6 let us call this is equation number 7. So, now see this one when you use the Lagrangian equation the del l del l the Lagrangian function you differentiate partial differentiation with respect to x minus d of d t del l del x dot is equal to 0. So, this expression is boils down to what is called a simple Hamiltonian function form this is and this function is free form x dot is a function of x you t lambda t and t. So, next is so this is the this equation and what is this equation another equation is that one from 4 what we can write from 4. So, your h is l is express if you see l is del l del u if you do del l del u and l you express that one. So, it is a function of your differentiating partial differential with respect to u there is no u is here. So, del h del u must be 0. So, from 4 we can write del h dot del u of we can write del h dot del u of t whole star is equal to 0. Let us call this is equation number 8. Now, you see this one another expression we call this equation is called what is called costate equation number 7 is called costate equation costate equation. So, this is the costate equation. So, another equation one can derive like this way del h dot del lambda del h del lambda is equal to x dot of t. Now, see the our objective function if you see if you refer to your last class note if you see we have written the what is the Lagrangian function we have written the objective function plus the lambda transpose constant that is the Lagrangian function. So, del l del one of the what is called KKT condition del l del lambda must be equal to 0. So, we can get this equation from Lagrange expression one can get this expression get this expression from Lagrange function or expression. So, this is also if you like you can write star this star this one. So, let us call this equation is equation number this is the equation number 9. So, this is called is optimal state equation and you see the costate equation dimension because lambda dimension is if you say n cross 1 same dimension of the state vector. So, that is why it is called costate equation this equation and this equation nature of this equation and this equation is there are costate one of costate of another. So, this is called costate of a state equation vector. So, now we have that this is the equation number 3 is now when you express Lagrangian function in terms of Hamiltonian matrix this expression simplified form you have got in del h that corresponding expression that one will get in simpler form del h del x equal to minus lambda dot of t. Another expression you have to in place of del l del u partial differential Lagrangian function with respect to u when l is expressed in terms of what is called that Hamiltonian matrix then we will get del h del u and the state equation that what we have equation is del h del lambda is equal to x dot of t. So, now see the boundary conditions if you see the boundary condition of that one then what we are getting it will suggest see the boundary condition now. So, note and the boundary condition what we will do we will replace the Lagrangian function in terms of Hamiltonian function that is all what is the final boundary condition we will get that we will see it. So, the boundary condition and our original problem a boundary condition is expression is equation 5. So, boundary condition 5 is now replaced by in the form of boundary. So, in the form of Hamiltonian function. So, boundary condition is now replace Lagrangian function in the form of now you replace Lagrangian function in the boundary conditions Lagrangian function in the form of Hamiltonian function now replace the Lagrangian function Lagrangian function in the form of Hamiltonian function. If you replace this one now you see first what will write it for this one l that equation number 4 if you see here. So, this is h you will get it that is l differentiate of this that is l differentiate of this with respect to x whole transpose x dot putting t is equal to t f that one and then del l del x dot t is equal to f del x t f. So, let us see what we can write for this one. So, we can write it for this one if I just put it here that total let us call rewrite 5 equation 5. If I rewrite this one l dot minus l dot of this x x dot of t whole transpose x dot of t. Then this trans star t is equal to t is equal to t f delta t f plus delta l dot del x dot del x dot whole transpose then your star t is equal to t f delta t f. Now, see this one our expression for that that one what we can write it l that one what we can write it l is a what you can write it this l l is the our expression like this function expression h plus. So, if I write it this one you just see what we are writing l l is nothing but a h dot del s del x of t whole transpose x dot delta dot of t plus del s of t del t this minus lambda transpose this is lambda transpose of t x dot of t. So, this I have written in place of l now this l is f see that one l is now we are differentiate with respect to x. So, if you do this one then what will get it this when we are differentiating with respect to x dot then it will be a del x this we are whole thing we are differentiate with x dot then it will be a if you do differentiate with respect to x dot l the here is x this. So, minus see this one I am differentiate this with respect to x dot. So, you have a minus x this. So, this will be del s del x whole minus there is a x dot here this is l I am differentiating this thing with respect to x dot. So, then what will be this one this will be a del s del x t minus lambda t minus lambda t of that one and this is the bracket that thing will be bracket because it is a minus sign is here. So, the whole thing is a bracket. So, I have written if you just this one l del l del x dot transpose x dot what is this value that I have written it that quantity then this star t is the equal to t f delta t f plus again plus again this term this equal to 0 because our boundary conditions from 5 this is equal to 0. So, next I am writing similarly that one will be delta s delta x of t there is differentiation with respect to x dot. If you are differentiating with respect to x dot that is x delta of x t this minus delta of t plus term and that you are making star t is equal to t f and delta x f is equal to 0. So, now see what is it this is l and this is the delta l you are doing that differentiation of that one with respect to x dot agree we got it that that one then again it is a differentiation of this with a x dot we will get it this one. Now, see this one what simplification we can do it here because now delta s and there I missed it here that your what is called x dot x dot you see this expression that this is the x dot is there that x dot is missed here. So, delta s this transpose also this delta x dot this x dot this this cancel now this is plus this is minus. So, this term this term cancelled this term this term cancelled what is left that is h plus delta s dot this is dot delta t this term whole star whole star t is equal to t f delta t f plus what is left here delta s dot and delta x of t minus lambda of t this is the transpose this is the transpose that lambda transpose this that is star t is equal to t f and l delta x f is equal to 0. So, that and this is our like the boundary equation is this. Now, see what we did it I just replace once again I just what I did it this is the necessary condition when you express in language and function and when language and function is expressed with a what is called emulation function and some other terms we replace it l by this and do simplification we got that expression condition del h del lambda del h del lambda is equal to x dot that is what we got it. Then this expression when you replace by a Hamiltonian function l expression then we will go we got it del h del u is equal to 0. Similarly, the boundary condition what we did it here when we just replace that expressed boundary with in terms of Hamiltonian we got it this function that is x dot this expression. So, our simplified form not this expression that is just now we have calculated this expression in terms of Hamiltonian. So, this is the important boundary condition in terms of Hamiltonian functions. So, now we will summarize this point. So, if you see this one this is the basic three equation we need to solve now this is the equation 7 8 del h del x is equal to minus lambda dot t when thus the system dynamics is expressed in terms of what is called state phase form. Then we can solve this one by using the Hamiltonian function where the Lagrangian function is expressed in Hamiltonian function form if we express in that form then del h del x star is equal to x dot what is called x dot lambda dot of t that one expression. Then del h del u is star is equal to 0 then another expression. So, this dimension is m cross 1 this dimension is n cross 1 the next equation what we got it here that equation 4 5 page number 5. So, this is another equation we got it the del h del lambda is equal to x dot of the state equation of this that one can get it from Lagrangian equation of this one then this is the boundary condition. So, this is the equation number 9 this is the equation number 10 let us call. So, if you solve the equation 7 to 10 then you will get the trajectory of control input as well as which in turn it will give you the trajectory of the state trajectory path in optimal path of this one which will minimize the our objective function or performing index. So, we will summarize the results like this way. So, if you see this our problem algorithmic steps now algorithmic steps. So, if you see our problem is like this way given the function our problem is x dot is equal to plant is given A x plant is given x of t u of t. And t and performing index is j the terminal cost is x of f t f t or if this or you can write it x t comma t t f is equal to t is equal to t f this plus t 0 to t f v x of t u of t of t this d t. So, what you form it you form first step is form the Hamiltonian matrix Hamiltonian matrix x of t u of t lambda of t and t what is the Hamiltonian matrix this integral plus lambda transpose of t into f of x of t u of t and t. And this dimension n cross 1 and the whole dimension is 1 cross n 1 cross n. So, this is the scalar. So, this is free form x dot this Hamiltonian function once you know the Hamiltonian function then our what is called step is there step 1 compute del h del u is assigned to 0. Let us call this is equation number once I know the Hamiltonian function from the performing index given n the constraint this is the constraint our problem is finally, we have to find out u of u t or u star of t such that this performing index is minimized subject to the constraint equality constraint. And that equality constraint may be linear may be non-linear dynamic equations. So, this is the first equation second equation the state equation that we have written it del del h dot del lambda of t this if you like you can put it in this equation. So, put it because you have to solve it and what is the solution you give that is the optimal star means optimal solution you get or you can omit this star it does not matter because you have to solve this equation. So, that equal to this equal to x dot of t. So, this is also n cross 1 let us call this is equation number 2. And this is the call this is state equation then third is lambda del h del lambda del h del x differential h with respect to that means what is called Hamiltonian function you differentiate with respect to x which is equal to minus lambda dot star. This is if you give star that one and this is the your costate vector see this one what we got it from this from the linear costate vector that expression is the third. So, this is the equation number 3 and then you have to solve this 3 equation by using what is called boundary condition that our boundary condition is del h sorry h Hamiltonian function del s which is function of x t and t del t whole star t is equal to t f delta t f plus del s dot del x dot of t sorry x of t this minus lambda of t whole star t is equal to t f into delta x f is equal to 0. So, this is a vector so transpose is there that. So, this transpose do not forget to give transpose that one. So, this is the boundary condition and this is the boundary condition in terms of Hamiltonian function boundary condition. So, if we just recall once again that given the plant dynamics this x dot is equal to f of x or performing index is that way this is the terminal cost and this is the integral part of what is called cost function. Then our first step is find out the Hamiltonian function that Hamiltonian function is nothing but the integral part of this integral this one plus the Lagrangian multiply into f of x. That is we have we have seen it how we have converted a constant optimization problem into unconstraining optimization problem using what is called Lagrange multiplier technique. So, this is the that Hamiltonian function once you find out the Hamiltonian function then differentiate that partial differential h with respect to u because h is a function of x t u t and lambda t t. So, next is once you find out then differentiate that h with respect to lambda t that will be x dot of t that we have derived this one. Next is you differentiate this with respect to x t partial differentiation of s with respect to x t that will be given lambda dot of t. So, this will be giving you the what is called boundary condition of this one that. So, and this is the four equation you have to solve simultaneously. So, next question is let us call we have taken we take a simple example and before that what is that that after solving this one what is the guarantee that objective function or the what is called your that performance index will give you the minimum value of the functional or maximum value of the functional. So, that is called sufficiency condition sufficient condition sufficient condition. So, that sufficient condition if you see this one is is provides to determine it provides or in order to determine in order to determine the nature of nature of optimization either functional value is minimum or maximum this is the nature we must confirm this one that can be done by using sufficient condition. We have already derived what is the sufficient condition is there in terms of Lagrangian function and if you replace that l in terms of h and some other terms we have shown it you replace l by Hamiltonian function plus some other terms then you will get it the second variation of the functional this equal to t 0 to t have delta x of t transpose then delta square h of this delta x square capital S. Let us call it right capital because we have started with the capital X vector this one this star and delta x of t plus twice delta delta x of t plus delta x of t plus delta x of t delta transpose then delta square h delta x of t delta x of t then delta u of t multiplied by delta that whole thing compute at along the trajectory optimal trajectory del delta x of t del u of t then plus del u of t whole transpose del square l dot del u square of t whole this del u of t that whole bracket and that you differentiate with respect to d t. So, we know this one the delta square of this must be positive if the functional value is function will be minimum if that value will be negative delta square will be negative if functional value is this functional value is this integration value is negative if the integration value is negative then function value is maximum. So, this you can write it one can write this thing into a matrix and vector form t 0 t f del x t and this is what this matrices this is a Hessian matrix symmetric matrix again this is a symmetric matrix you will get it. So, that we can write it now del u of t transpose this whole thing is a scalar quantity that we can write it in quadratic form. So, you are writing del h square dot del x square of t del second derivative of h with respect to del x del u the order can be changed del u del x this does not matter the results are same. So, del x del u of t then del square h del u square of t that whole thing along the optimal trajectory u t extra multiplied by delta x t and delta u of t this and then differentiating with respect to this. So, delta square the second variation of second variation of functional second variation of the functional if it is a negative that second variation is a scalar quantity if it is a negative indicates that this integral part that matrix whose dimension is this matrix dimension is n plus m into n plus m matrix dimension and that is a symmetric matrix and that matrix must be a if it is greater than 0 that matrix must be a greater than 0 means positive definite. If it is a positive definite matrix then we will call that our what is called functional value is minimum and at what point along the trajectory along the trajectory u star and x star and that u star solution and x star which you got once you find out u star x star also you can find out and that is the optimal trajectory that we obtain by solving equation just now we have mentioned it by solving equation that algorithmic steps if you see the algorithm by solving the equation that our that is just a minute that means just our equation is del h del u that means that 3 equation you have to solve it del h del u is equal to 0 that means if you write del h del u is equal to 0 then del h del lambda is equal to 0 that is equal to x dot of t that you have to solve it and del h del x is equal to again this one is equal to lambda minus lambda dot star this your lambda dot star the star you can write it this 3 equation along with the boundary condition that just now we have written the boundary condition and this boundary condition 4 equation number that means equation number 1 equation number 1 del h del u equation number 2 del h del lambda a x dot and equation number 3 del h del x is equal to minus lambda x dot and equation number 4 is necessary to solve the set of non-linear equation that boundary condition is required. So, that way we can solve it this one and once you solve the optimal trajectory in order to check whether the functional is a optimum means minimum or maximum to show this one nature of the optimization that this matrix that is this matrix I am now writing this matrix the Hessian matrix del h del dot del x square of t del h dot del x of t del u of t del square h dot del x of t del u of t del square h del x del u square of t. So, this matrix you can write this matrix if it is greater than 0 means positive definite matrix agree and this matrix is a symmetric matrix this matrix is symmetric matrix. If it is greater than 0 it implies that it implies that functional value is minimum that j star is minimum functional value is minimum if it is less than equal to 0 the negative definite this matrix is negative definite along the trajectory optimal trajectory what we got by solving just now mention equation number 1 2 3 and boundary condition that one if you and put it here in this matrix if you get a negative definite this implies this implies that j star is maximum functional value is maximum. So, this is the necessary and sufficient condition you have to necessary condition is required to find the trajectory of the what is called trajectory of control u of t and x of t once you get it optimal trajectory of u star of t control in afford u star of t and x star of t optimal trajectory of the state variables. Then next question is to know the nature of this optimality whether it is a objective whether the objective function or functional value is minimum or maximum then you have to test with this matrix and that matrix dimension you see n is the number of states m is the number of inputs. So, it is n plus m into n plus m that matrix you have to check it if it is positive definite j star is minimum if it is a negative definite j star is maximum. So, this is the basic theorem of this one. So, we will take an example and see that how to solve a problem a practical problem of this one and you can see this one we can just consider the three cases as a case a similarly as we discussed earlier fixed time fixed final time and fixed final state this is case one and fixed final state this is the case one. Then case two we can see the fixed final time and you see that case b free final time and fixed final state similarly case c we can say the case c is the fixed final time fixed final time and free final state and last one and last one case d both are free case d, but free final time final time and final state and free final state. So, this whatever the we have considered the boundary condition all this thing that is one of these cases will be a special case of that general boundary condition. So, we will stop it here next class we will just take an example and show how to solve this problem by using the Hamiltonian function method to get the optimal value of the functionals to solve the control problems agree. So, we will stop it here.