 an idea about the midterm, what's going to be on the midterm, and maybe sample problems of the midterm. Okay, so let's get started. Last time we introduced the idea of optimal control problems or what are the ingredients that are needed. See any questions on that? Let me start with an example of optimal control problem and more specifically time optimal control. So we're trying to minimize the time when something happens. Okay, so the physical situation is very simple. There's supposed to be a straight line and I have an object moving. Mass is one. So basically Newton's law says mass times acceleration equals force, right? So the force that are acting on this is simply a. So the control is actually exerted by as a force, okay? So the control variable is if you want the magnitude of this force. Okay, force is a vector, right? But magnitude is, well, not even magnitude. So let me say there's restricted to, so with constraints that use between negative one and one, okay? So you can exert a force of maximum strength, the magnitude can be one on either direction. So it could be right a pulling or a pushing force if you wanna think about it like that. And the state, the dynamics, so if you want to, is given by this second order equation, right? X is a function of time, is a position of the object at time t. I don't know, X is X, oops. So there's an origin at some point, right? X equals zero and you're exerting a force, a magnitude one at most in either direction. And what the goal is, so the objective is to bring the object at a stand still at X equals zero. And minimum time. So how should I, maybe the picture's not too suggestive here. If I plot it zero here, I really want the force to be, well, actually we don't know, right? The object might be moving to the right, to the left. And so we'd like to apply a breaking force, right? So that it kind of slows down, right? But maybe if the speed of the object is too high initially, what happens? You only have this much strength to apply, right? So even if you apply the maximum force, the object might actually decelerate, but go beyond zero, right? So what you want the goal is to bring the object at zero with speed zero in minimum time, right? So in other words is minimize t, right? Over all admissible controls, which are negative one less than or equal to U less than or equal to one, right? Such that XFT is zero and X prime of t is zero. So at final time, let me put dot here, is zero. So this is the final time, okay? The final state, okay? So the first thing is we want to represent this as a dynamical system, right? So first we write the dynamical system using the state variables. X1, which is the position, and X2, which is the velocity, okay? So why do I call this the state space? The state variables, and then the state space. Because the moment I've, at a fixed time t, the moment I've identified the position and the velocity, I know the state of the system, right? So this system has two degrees of freedom if you want. There are two variables, the values of which at each time t determine the system at time t, right? It's a very simple system. But I need two of them, I cannot just, right? If I only know the position, but I don't know the velocity, that this means I don't know the state of the system, okay? All right, so what is the dynamical system? What is dx1 dt, and what is dx2 dt? Well, dx1 dt is the derivative of the position, and that's the velocity, so it's x2. How obvious this is, or if I need to go through. But this is simply writing us an equation that second order differential equation is a system of first order equations, right? So simply I say because of the choice of those variables, the first equation is a restating of the relation between x1 and x2. The second equation is actually is a restating of the dynamics of x1 double prime is, which is x2 prime, right? So this is x1 double prime, and this is u, right? So maybe if I, so no, because this is x. I think the system is this, right? Again, you're thinking of the picture here. Well, I'm always considering the u as being in the direction of x. So if u is negative, then it's going to be a force applied in the opposite direction, okay? But if that's the meaning of u, the notation for u, then it's just that, okay? So this is, again, this is just x dot and this is x double dot, okay? So the system is kind of fairly, fairly simple, well, simple looking, okay? But remember what do we, what are we actually trying to achieve with writing it like a system like this? Well, we'd like to imagine of the trajectory in the phase plane. So in the, starting with an initial condition, so I should have said that we always have an initial condition that we start with, right? So initial condition, initially at time zero, we assume x1, excuse me, x. So the position is given and the velocity is given, zero and x dot is zero, right? So this means we have something at time zero, right? And we have some evolution depending on what our choice of u will be, right? So we can, what are the admissible controls? Anything between negative one and one, right? So we could have different paths, right? Depending on what u is. And we'll talk about what a few sample controls, a few admissible controls. But we could go from that initial condition in many different ways. The goal is to reach to the origin and not anyhow, but in minimum time possible, right? So we're trying, it's kind of a target, right? We're targeting the origin, but how? By following the vector field that's generated by this dynamical system, right? And keeping in mind that u, we need to decide on what u has to be, okay? So for example, take u to be always 1 for some time, for some time interval, okay? What does it mean? This means that dx1 dt is x2 and dx2 dt is 1. So we can actually solve this, I mean, we don't have to go to a p-plane necessarily to do this, right? Because x2 is then going to be what? T plus a constant, right? So again, I want to remind you, this really is saying, I'm solving the second order equation in x, x double, the acceleration equals always 1, right? So if I have an initial condition, x of 0 is some x0 and x dot is v0, then you can find out what the constant is, right? So x2 of 0 is v0 implies that the constant is v0, right? So you see, you actually solve for x2, x2 is t plus v0, okay? Yeah, well, I'm just giving an example of admissible control, okay? And next we'll take u to be negative 1, okay? You see, sometimes if your initial condition is that it's shooting to the left really fast, then you might need to actually pull it with a positive u. So you will see that at some time you might actually have to juggle between the two values, okay? But I just want to show you on this face portrait how it actually looks for this value of u, so you can do it by hand. This is a point, and once you have x2, you can go to x1 and solve this, x1 of t is going to be what? 1 half t squared plus v0 t plus another constant, let's call it d, right? And what is, how do you find the constant d from the initial condition for x1, so this is going to get d equals x0, right? Probably you've done this in your physics courses a long time ago. So I'm going to pull a p-plane a second so you can see how this various solutions might look like. But the computation here is fairly easy to make it also by my hand, okay? So this is x1 of t and x2 of t is, so these are the two boxed equations. I'll give you the x1 and x2 as a function of t, right? Now, these are the parametric equations of a solution curve in the face plane, right? So how would you plot these if you were given an initial condition like this? What would the curve look like? Remember, it's always hard when the curve is given parametrically. This means that each t, this is the x1 and that's the x2, right? But what's the x1 versus x2 that takes some additional work, right? Different points, right? So you could select a bunch of values for t and that's how the computer does it too, right? When you plot something parametrically. But in this case, it's fairly easy to see that you can, well, and this is very rare, right? But here you can eliminate t, so you can replace, you can find t in terms of x2 and then plug it back in here. So what you'll see is you'll see that x1 is a quadratic function of x2. So that's already giving, it's giving you a hand of how the solution will look like. They will look, they will be actually, x1 is kind of a parabola, but it's tilted, right? Because x1 is a quadratic function of x2, right? So this is basically how the solutions will look like. And of course the direction is also going to be given by the trajectory. It's going to be given by the direction field. So you can probably guess because dx2 dt is one, this means x2 is always increasing, right? So it means it's always going this way, okay? You can convince yourselves by plotting this direction field. So it's x1 is, we said it's what is x2 and x2 is u and u is 1, right? The origin is right here, right? Okay, and the picture should be, I don't know, maybe more symmetric, okay? So again, somebody gives you an initial condition, that's the plot of the solution, forward in time and backward in time, okay? So obviously if x1, let's see, no. If x2 is positive, if your initial velocity is positive, I mean as you're moving in the increasing direction of x, with applying a force equal to 1, you will never stop, right? But you see that if you are actually on the negative side of x2, so if you have a negative velocity, right? And it is likely that it can apply a positive u and stop. In fact, there's a very special solution here and I'm going to plot it backward in time that starts at the origin. There is the origin, this one, right? So there's actually a whole set of initial conditions that if you start along this curve, what's going to happen with your solution as time goes, it increases. It's going to go this way, eventually it's going to go to 0, 0, right? And of course if then you continue applying the force, it's going to keep moving away, right? So what strategy is, if your initial condition is, you see x1, that's the position is positive, you're to the, so here's the origin, you're to the right of the origin, right? But you have a negative velocity, and if it's just right, right? By applying that force of 1, you're going to actually after some time hit the origin with velocity 0, and then you stop pulling, right? Now that's, in essence that's really the only way you can reach the goal of being at the origin with velocity 0 if you apply u equals 1 all the time, okay? So you start here, say this is your initial condition, right? And you only have the option of applying u to 1, well, you're going to get 0 there, right? So it's not a question of minimum time, right? It's just only one way to do it. If you are forced to always apply the control to be 1, but as I said, you're not always forced to apply a control to be 1, you may actually apply control equal to negative 1 or negative 1 half or negative three quarters and so forth, right? And that's going to apply for points that are actually outside of this very special line here, okay? So, so back here I'm going to identify this as being a very special kind of set of like this piece of a parabola. By the way, all these parabolas should be, I think they're translates of each other. They don't really show on my picture, but they should show like here, right? And again, you can actually, maybe not, huh? Maybe, maybe not. So, so really the, to figure that out, you'd have to find, so express x1 in terms of x2. And I think you get something of this type. Well, I don't have it written down here. But what you have to do is you have to find t to be x2 minus v0. So you get x1 is one half x2 minus v0 squared plus v0 x2 minus v0 plus x0, so, okay? So, so yeah, so I don't think that's necessarily translate of each other, right? Because there's going to be an x2 times v0 here, hold on. No, actually I think the cancel, so, so let's, let's try this. This is one half x2 squared minus twice x2 v0, but plus one half of that plus v0 x2, that's canceled. So it's plus one half v0 squared minus v0 squared plus x0. So it is a translate of, so once you have one, one problem, then all the others are translates of that one, right? So my picture doesn't do a good job. But you can see it here, okay? Any questions on this? I want to switch to u equals negative one. And, and just to show you how easy it is to, you know, I mean, of course, when you have this, you just put negative one. And redo the same thing. And what you'll see is you'll see that, well, x2 is equal to, the derivative is negative one. So x2 is always decreasing, so it's going down, right? And then the problems are kind of pointing, pointing the other way around, okay? So for u equals negative one, the corresponding phase portrait looks this way, right? And again, these are translates of each other. So, of course, they don't intersect and they go down, right? And again, there is actually a special set of initial conditions that if you are there, all you need to do is apply x, u, the control to be negative one to get to the origin in a certain amount of time, right? Okay, so now what do you think needs to happen? Well, now basically you have to decide among the strategy for u, u can be anywhere in between negative one and one. So you can imagine if you allow u to be kind of varying between negative one and one, then of course you cannot plot a phase portrait anymore, because it would be a non-autonomous system, right? But the solution would have to kind of fit that time changing direction filled, which we cannot plot, right? So here's the idea, though, once you have those two, is as the combining the two phase portraits makes an artist or brings the artist from each of you. So I'm going to just kind of identify these two curves which are, we're going to later call them switching curves, okay? And so the phase portraits for u equals plus one and u equals negative one. But of course, keep in mind, you are allowed admissible control is you could have a half, right? For a while and changing, right? So what I'm doing here is very kind of limited still until we talk about this maximum principle. So I'm going to tell you that this maximum principle that we're going to introduce is going to say in this particular problem, your optimal control strategy is to use the extreme plus one and minus one only for some periods of time. And then with some switching in between, okay? So here's the illustration of that. So for instance, if your initial condition is summer here, okay, that's a time zero. How do you think you will achieve if you only had the possibility of using plus or minus one for your control, how are you going to get to zero? Well, first of all, you're going to have to kind of catch the ride on a parabola that points to the right which corresponds to u equals negative one, right? So it's like you look at the two phase portraits and you see how can I go, right? Well, why like this? Well, because you're going to actually hit this half of the parabola in some finite time. Remember, x2 is going with velocity negative one. So it's not like kind of lingering here, right? It's just going kind of steadily. So at some point it's going to hit this, right? Once it hits this, I'm just going to catch the ride using u equals plus one. And because we talked about a special curve here, you're going to actually hit zero zero and find out a lot of time, right? So that's a strategy, right? In fact, if you had only the choice of plus and minus one, this would be the only strategy. Because if you start here and you start with u equals plus one, well, actually, no, I take it back. It won't be the only strategy, right? In principle, you could start, so I'm going to use a different color here, maybe you could start with plus one. So you're going to follow some sort of a trajectory that's parabola pointing to the other way, right? You could start positive, but then you could go negative. And then you could go, right? It's just going to take a lot longer. So it's not going to be optimal in that sense, right? So if you only had these two possibilities, plus or minus one, you could actually imagine, I mean, you could have various ways in which you achieve that stopping task. But with lots of wandering around, okay? So you kind of see from this picture that this is going to be the optimal. I mean, to reach the origin, you have to land on this curve at some point. And the sooner you land, the smaller the time. So this is going to be actually the optimal strategy. Start negative, for this initial condition, start negative u and then switch to positive u once, okay? If x1 is positive, but above this curve. So x2 has to be somehow, right? Because if you are below here, so I'm going to use blue again. So this is going to be optimal, oops. But if you are starting here, you see you won't be able to. Because if you start negative, you keep going down, right? Whereas remember, it's best to kind of catch this other branch as fast as time possible, in quickest time. So you start with equals positive one and then negative one, right? Does it make sense? Yeah, exactly. So and it's not intuitive, I agree. So in other words, what I'm saying here is if you choose any other values of u, one series between negative one and one, you may arrive at that stopping point. But the time it will take is going to be longer than using this, okay? So this will be not optimal, okay? So this principle, this maximum principle is actually stating that and it has a proof and all that, but we won't be talking about the proof, okay? Now there's a better example which you probably experience a lot, quite a lot when you swing, well, somebody else, I guess. Or you can swing, you know, go on a swing, right? And you imagine like, okay, you have a child, right? And then you're swinging. And what you want is you want to bring that child to a stop in the shortest amount of time, right? What do you do? You're going to apply, well, let's imagine there's no resistance to air. It's just, basically it's a harmonic oscillator. So there would be x double prime plus x equals u, okay? What you do then is you actually, say you have a certain limited amount of force you can apply, right? What's going to happen is you're going to have to switch between pulling and pushing, depending on the position of the swing, possibly several times, to achieve the full stop, okay? But depending on the initial condition, that determines the number of switches. The picture would be a little bit more complicated, but anyway, let me just mention that. So stopping a pendulum is kind of the next complex problem. Oops, so x double dot plus x equals zero. And let's say I also have this force, right? Pendulum or spring mass, whatever, right? So right now you have some, so either as a pendulum or a spring mass, but it kind of moves both directions, right? So think about the dynamical system that you get. Well, you get this when you think about the extremes only. If you use one, what's going to be x double prime plus x equals one? And what's the phase border for this one? Well, x double prime plus x equals zero gives you this cosine and sine, right? Which are concentric circles around the origin, except here what's the steady state? The steady state is when x position is one and velocity is zero. So it's kind of concentric circles. So again, this is x1 versus x2. But center around this point, I don't know the, I think the counterclockwise probably, that's how they rotate, right? Colson and sine, okay? Whereas if u is negative one, the steady state is going to be at negative one and zero, and this is going to be the concentric circles, right? So when you do a combined phase portrait, it's going to be quite tricky. But imagine like I'm at this initial condition, right? So x1, x2. So position is positive and velocity is positive. So the pendulum is swinging away from the vertical, right? Or if you're going to think about a spring mass, it's, the spring is, right? It's kind of to the right and it's moving away from this. Obviously, you would like to apply a negative u, right? So you're going to be moving along. And I forgot to mention here, what's the direction of this? Clockwise, still, okay, so it's x1, dx1, d2 is x2, dx2, dt is minus x1, minus 1. Okay, the other one is plus 1. So let's see if x1 is, if x1 is positive, both x1 and x2 is positive, x1 is increasing when x2 is positive, right? So actually counterclockwise here and possibly, so excuse me, clockwise on both directions, I think. Okay, so you're going to be moving clockwise, but you're going to be kind of trying to circle around this one, right? Yep, and if you always keep negative one, you will never actually reach the origin unless in a very special case when you are on a circle that goes through the origin, right? So at some point you're going to have to hitch the ride on a circle that's concentric with respect to this one. And hopefully, let's say this is one. So there are these two special circles, right? Eventually, you have to land on a circle that's centered at 1, 0 or at negative 1, 0 that goes through the, that's tangent to the, that contains the origin. So you see if I'm going here, I'm going to go this way. And maybe I should exaggerate this a little bit so you can see. So let's see my initial condition is at this point. So the circle that I'm trying to, yeah. So the circle I'm first starting on intersects this, right? Then what will be my optimal trajectory? I'm going this way, right? And I'm going this way. See, it's the only one switch. But what if I'm actually at an initial condition that's kind of, my momentum and my position are kind of large enough that I won't be able to stop it in one switch? And I'm going to have to hitch to catch a ride on one of the circles, right? I'm going to have to have several switches. It's a little bit, it's actually the number of switches that one is required to make for optimal control is actually a very kind of delicate problem. Because it depends on the initial condition, okay? So that's why we typically don't start with these kind of problems because of the control spring mass, okay? But nevertheless, the outcome of this optimization problem is that you always need to go to the streams for this particular problem. No, I haven't shown you this. And actually, I'll show you in a second. But it will have to follow basically this strategy. I mean, this maximum principle, okay? Okay, so let me kind of start by describing. So what is the maximum principle called Pontragon maximum principle? Commonly called PMP, okay? So we start with the state variables and the dynamics. So the dynamics is dx1 dt is f1 of x1 xn and possibly control u and dxn dt is fn of x1 xn u. I'm going to try to stay with the notation that I have in this handout, okay? And again, the state variables are x1 through xn. Once you've identified the state variables at time t, you know this is the whole system, right? So, depending on the system, you might need two components, or you might need seven components, or you might need one component. Depending on the number of degrees of freedom. So given an initial condition, x at time zero is some given vector, some given, right? Each component has a given value. The goal is to find the optimal control. I think I stated this last time with two variables. But now we're going to be a little bit more careful with the notation. So we're going to find an optimal control strategy. So u as a function of time over some finite period of time. Now remember this finite amount of time may actually be part of the problem. So this t might actually be part of the problem to minimize. So sometimes we fix the terminal time t. Sometimes we don't fix the terminal time t, okay? If it's time optimal problem, we don't fix it. We want to minimize t, capital T, right? In other times, you fix the capital time t, okay? Such that the following functional is maximized. And again, this is just a convention. If you have to minimize something, you just put minus that and you maximize it. And the most general we're going to consider is the following. It's going to be a combination, or a combination of the final time, the final states, x1, the components of the final state. So x1 of t, xn of t. So it's a linear combination of those plus possibly an integral term, integral from 0 to t of some expression involving state variables and the control throughout the whole dynamics, okay? Now this is certainly not the most general type of objective function that one can encounter. Like this first part here, one could have some nonlinear function of the x of the states. Like I wrote last time, I call it like a g naught. But I like to focus on this just linear combination of them, just because we talk about linear programming. Remember in the linear programming we were always our objective was a linear function of the states, state values, right? So think about if for some problem you only have this first term. So in other words, if f naught is 0, right? The new objective function is just like in the linear programming. What's the difference between linear programming and this problem though? In linear programming, what were you doing? You're maximizing an objective function, a linear objective, right? Given some constraints, right? But it was like a steady state. There was no dynamics, there was no time, right? You just had some constraints and you had an objective function and you wanted to minimize, maximize that objective function, right? Whereas here, so you're trying to hit a vertex in a simplex, right? Simplex was defined by the constraints, right? Here it's kind of a similar idea except you're trying to kind of reach the vertex where this is maximized, right? You don't necessarily have a simplex. What you have is you have a dynamics. That you started some initial condition and you're trying to kind of hit a vertex where something is maximized, okay? And hitting that was kind of following a dynamical system rather than a, I mean, in linear programming you didn't have any dynamical system. So it's simply kind of a static or steady state, okay? The only constraint that is happening here is actually on the control, possibly, okay? So finally, among the following functions maximized, among all admissible controls. You know, u of t belong into some set, I don't know if I call it s. But we call s when we talk about linear programming and we said that's the feasible set, right? So there we had a feasible set. Here we have a feasible, a set of feasible controls. Possibly, it's the functions of time, okay? So this is kind of a grand picture. It's kind of, I mean, I'm not really expecting you to say, yeah, of course, so in a program I can do this, I can deal with this, right? Not only because it has this extra kind of complication, but you have this extra term here, right? But also because of what comes next. So this is the formulation of the problem, okay? Every problem that we're going to be dealing with in this chapter, just like before, we're going to try to fit it with this approach, okay? We're going to try to fit it in this mold of identifying the things that we need, okay? So here's the PMP, okay? You ready? And you'll see how nicely it kind of gives us the answer for that first problem and others. So it says construct first the so-called Hamiltonian. And this Hamiltonian has a lot of variables. It's a function of many variables. And I'm going to try to group them. The order doesn't quite matter, but let's group them by n. So there are n variables first that I'm going to call them psi's, and I'll tell you in a second what psi's are. The next n variables are the x, so these are the state variables. And the last one is the control variable, right? And by definition, this is what is going to be psi 1f1 plus psi 2f2 plus and so forth psi nfn plus f0. So you recognize all this. So what is f1? This is the right-hand side of the state variables, right? Of the state of dynamical system, right? The f1, f2, fn. What is f0? f0 is this guy that, as I said, may or may not appear in the objective function. If it doesn't appear, it means f0 is 0, right? But if it does appear, then that's the f0 that you need to add to those terms, right? So let me introduce this guy's here, so sorry about that. So here psi 1, psi n are called adjoint variables. X1, Xn are the state variables. And what are these adjoint variables, adjoint? Can I call them shadow variables? Adjoint is a fancy mathematical term. But we've talked about shadow prices. We've talked about adjoint multipliers. These are exactly those kind of things. They're things that kind of work and have some meaning. But we won't need them unless, I mean, we will work with them, but not need them unless explicitly asked by the problem, okay? So I wanted to think of it as an adjoint multiplier. In fact, it is an adjoint multiplier. In many places, you see this principle. They don't use psi, they use lambdas, okay? Just to be more illustrative. But I figured lambdas would make things a little bit more confusing. But so they are Lagrange multipliers. And again, not that I expected to say, yeah, of course, that's not understand what they are, okay? Because I didn't tell you what they actually satisfy. So the psi variables satisfy an adjoint system, dynamical system as follows. So it's d psi j dt is minus partial of h with respect to xj. J starts from 1 to n. This is getting actually now weird because, well, first of all, I wrote this system as in compact form. So what this really means is d psi 1 dt is minus partial of h with respect to x1, d psi n dt is minus partial of h with respect to xn. Agree with that? So it's a system. But the right-hand side of this system involves h. And h was something that was defined in terms of size, right? So when it's understood how this is actually done, well, think about what is the partial of h with respect to x1? In this guy here, that's h, right? So when you differentiate with respect to x1, where is x1 going to be hidden? Well, x1 is hidden in f1, in f2, fn in the f's, right? So when you take the partial with respect to each x, the psi will be left alone, and they're going to be linear. So the point is that this system, no matter how complicated and it's not explicit, really, it doesn't tell you what psi is explicitly. But it will be a linear system. So this is going to be a linear system of a linear dynamical system in psi's. And we'll see an example in a second. So this is kind of the most important thing. This is linear in psi1, psi2, psin. Now, why am I saying it's important? What do we know about a linear dynamical system that is linear? Can be solved. Now, it will not be autonomous, unfortunately, always, most of the time. Actually, that's not true. I mean, in principle, it can be solved. Because as a linear system, you can exponentiate the matrix of the coefficient and figure that out. But in practice, it may still be hard to solve this, right? But just the idea that you can solve this system for psi should give you a comfort of what its psi's are. I mean, these psi's come out of solving a linear system. So if the first time I wrote this psi, I probably got shocked by what are they? At least now, you should think about, OK, these psi's are some shadow variables that are obtained from solving a linear system. And again, we'll see this in each example. It may be a little bit different, but you will see what am I saying here. So it can be solved explicitly with one caveat. So when you have a system and you try to solve it, what do you need? You have a differential equation. How do you solve a differential equation? Well, you can find the general solution, right? But if you want a specific solution, you need to impose some initial conditions, right? At some time, you need to specify the psi's, right? Well, so here's the important thing, is it can be solved explicitly given initial conditions. But now I'm going to cross initially, and I'm going to call it terminal. So everything's kind of upside down. These terminal conditions are psi 1 at capital T, psi 2 at capital T, psi n at capital T. OK, so somehow, we need to know what these psi's are at the final time T. And if you look at this cheat sheet here, there are two cases that I'm listing. And these are not the only two cases. But case 1 says fixed terminal time and free terminal state. OK, so in this kind of scenario, we have a problem where we're given to optimize something over a fixed time period. But we don't know the terminal time, the terminal state of the system, right? So this is in contrast to the time control problem, time optimal control, where you don't know the time over which the dynamics takes place, but you know the terminal state. You want it to be a certain, like the origin, right? OK, so in the case 1, which is fixed, so fixed terminal state, excuse me, terminal time. It's not my friend here. So fixed terminal time T. This is kind of funny. The terminal conditions of this adjoint system are to be taken exactly those constants that appear in the objective functional. The constants that come in front of the objective functional. And the time optimal, in the case 2, time optimal problem, there are no terminal conditions explicitly imposed. You can think of like this. So the time optimal problem has some fixed terminal time for the state variable x. So for that dynamical system, you have initial conditions and you have terminal conditions for the x. So you don't need any for the actually psi. The only one that's needed is that h, the Hamiltonian at the terminal time T is equal to 0. So anyway, so far, unfortunately, we're not done with this. I've only introduced you this variable size. So only introduced the variable size. So here's the key step. So imagine now you're comfortable with building that function, Hamiltonian h, where size are some shadow variables. But now here's the key step. The key step says for each fixed T between the time period where you're doing this problem, choose the u star, the value of the optimal control that maximizes h as a function of u. Among all admissible values for u. OK? So this is pretty much nonsense until we do an example. So let's go back to our example. And you see, again, the only way to learn these kind of steps or to apply these steps to a new problem is to go through a few examples. So we'll start with a very simple one like this. The first step is to identify the state variables, right? And we've done that, x1, x2. And we write the state, the dynamical system, right? So now with a different color, I'm going to. So this is the f1. This is f2, right? Of x1, x2, and u. x1, x2, and u. At this point, I wanted to stop me with anything. OK? All right, so I have two variables. I have f1, f2, I've identified. So right off the bat, I'm going to write this Hamiltonian. h, psi1, psi2, x1, x2, and u. Let's see. Can somebody tell me what it is? Psi1, f1. So that's psi1, x2, correct? Psi2, f2, which is u. And I didn't tell you what objective function we had, right? We had time optimal problem, right? So that was what? j was, so time optimal problem was to minimize t, right? So what is in standard form? It was to maximize j, which was minus t, right? Which is the integral of minus 1 from 0 to t dt. So this is f0, right? Remember, f0? So this is minus 1. So we need that as well, right? OK. So this is a function of five variables, right? But really, x1 doesn't show. OK, so adjoint system. So I'll tell you, this can be done like, I mean, it's almost algorithmically. It can be done without any emotional involvement, really. The fsi1 dt is minus partial of h with respect to x1. But what is that? There's no x1 in h. So partial of h with respect to x1 is 0, right? And the psi2 dt is minus. I mean, this minus is important, by the way. Many of these things have reasons beyond, way beyond. I mean, we wouldn't be able to kind of talk about it in class because of the time. But at some point in natural task, why any of this, right? Why is this working? Well, think about it as kind of the same question. Why, when you look for maxima or minima, you take derivative of set it equal to 0, OK? There's the same thing except here there are seven steps rather than one step or two steps, right? And OK, so let me finish this. So this is minus. And what is partial of h with respect to x2? Psi1, right? So this system is incredibly easy, right? Because I mean, it's obviously linear, but you don't need to do any explanation or anything to solve it, right? Now, let's postpone that. We know we can solve it. We know psi1 and psi2 can be solved, right? We don't have terminal conditions or initial conditions. So there's going to be some things that are uncertain at this point, right? And let's go to the key step. What did I say? How should we choose u, the value of u, to maximize? So to maximize h as a function of u alone, I should have said that. So look at that h and you see what? Whereas u only appears as a in one term and as a linear term, right? So I forgot to, I mean, we talked about all the constraints. What was the admissible constraints? The admissible controls, the feasible set, negative 1 to 1, right? So what you're having here is having a linear function of u that has to be maximized over a finite interval. So find u such that u star, right? Such that h of u star maximizes this function of u, OK? Nothing easier than that since h is linear in u, it can either be, so I'm plotting h, what do I mean of u alone? It means I freeze everything else. Time is frozen, x1, x2, psi1, psi2 are frozen. So all you have is you have some sort of a linear function between negative 1 and 1, right? And a linear function can either be increasing or decreasing, right? So how do you maximize? This is a one-dimensional constraint optimization problem. How do you maximize a linear function over a finite interval? It's one of the endpoints. You don't take the derivative and set it equal to 0. Because if it's linear, the derivative, unless it's flat, right? But if it's not constant, at all constants, then you don't take the derivative set equal to set equal to 0, right? If this were not linear, then that's what you do, right? You take the derivative with respect to u, set it equal to 0. OK? So the first problem in the homework, if you read it carefully, there won't be any, there are not any constraints on u. So which means at this step, what you have is you have to maximize h over the whole line, right? All values of u. But guess what? H will not be linear. And how do we know it will not be linear? Well, just take a look at that problem. We're trying to minimize this objective function, right? So do you see that h is not going to be linear in u? Because inside that integral, the f0 is u squared. In fact, it's going to be minus u squared. Minus u squared, right? Because we're minimizing. So the first thing to do is to convert everything to maximizing, right? So then f0 is minus u squared, right? Then h is going to be a function of a linear term with u and minus u squared. So that's a problem offside down. That has a maximum, right? So this is going to be accomplishable, possible, right? But here, because h is linear, you have either u equals 1 or u star is either 1 or negative 1. Who decides whether it's plus or minus 1? So it's minus 1 if? Who decides if h is increasing or decreasing? Psi 2, right? So this is if psi 2 is how? Negative, right? Then it's decreasing so the maximum occurs at the left-hand point, negative 1. And it's 1 if psi 2 is positive. So now what do you need to do? Well, pretty much you've established that the optimal control has to have one of the two extremes, plus or minus 1. The only thing that's not established is when is plus 1 and 1 is minus 1? Well, for that, you have to solve psi 2, right? Psi 2, you solve that adjoint system. And I won't do it now, but let's see. I think in one of the handouts, the first handouts, you see that worked out in example 1 at least. So you just solve for psi. You can see psi 1 is constant, right? Psi 2 is going to be the integral of a constant. So it's going to be linear. Psi 2 is linear, right? So psi 2 changes sign only once. That's basically saying you always have to change this problem from negative 1 to positive 1 or from positive 1 to negative 1, only once. So that's why that combined face portrait is all in need. So again, try to look through those examples, identify at least up to this point the Hamiltonian adjoint system and how to pick U star. U star is going to be when h has to be maximized, OK? And we'll come back on one Monday. Thank you.