 Now, another way around this is what I will now talk about, it is what are called augmented Lagrangian methods. Now, the augmented Lagrangian method does something slightly different in the sense that it is not just looking for a minimization of a penalized problem, it is doing that, it is looking for the minimization of a penalized problem, but then it is not just doing this blindly, it is minimizing, it is looking at a certain type of penalized problem, it minimizes that penalized problem, but then the penalty associated with it is not going to be increased blindly, rather it is going to be tuned very carefully to the way we are getting solutions of solutions from this particular penalized problem. So, the effort is to eventually start mimicking these KKT conditions of the constrained optimization problem and at the same time making sure that you get feasibility at a finite value, even for finite values of the penalty parameter. So, let me give you, I will explain to you now what the augmented Lagrangian method is. So, recall what we know about, what we had used as what was called the Lagrangian, the Lagrangian was a function of X and the Lagrangian multipliers associated with it. So, I am going to look at a problem where the, so let me first try it out the optimization problem. So, let us first look at a problem where with simply an equality, with only an equality constraint, so you are minimizing a function F subject to say for simplicity, let us have these, only these equality constraints at J of X equal to 0 for J equals 1 to P. Now, the Lagrangian is defined in this way, so the Lagrangian was of X and here if these are Lagrang multipliers theta, so we define the Lagrangian as F plus summation theta J H J of X. The augmented Lagrangian is this particular quantity, the augmented Lagrangian is L of X, theta which is F of X plus this was summation from 1 to P, this again summation from 1 to P, so you have continue to have this particular thing theta J J of X plus you also include in this your quadratic penalty and outside you can put a penalty parameter C. So, you have your quadratic penalty added on top of the what is the usual Lagrangian, that is why this is called an augmented Lagrangian, so you are augmenting the Lagrangian with this additional term. Now, what we would do is that the algorithm would what it would do now is you can notice that you do not have just the primary variable X, but you also have the dual variable or the Lagrangian multiplier theta present here. So, it will continue to adjust X and theta at each step, so but and at the same time it will and it will keep increasing C k as well. So, actually to make the dependence on C explicit, let me write this as this given C. So, let us suppose we take the so what it the whole idea is to start to have an is to is to adjust your X and theta in such a way that you start with an initial estimate of X, you start with an initial estimate of theta and then as the iteration go along you start refining your estimates. And the goal is to get and is to refine them in such a direction that eventually at convergence you would end up the minimizing this augmented Lagrangian would amount to the same as minimizing the true Lagrangian and hence solving the KKT condition. So, let me explain this. So, suppose we have for sake of argument suppose we have fixed to motivate this suppose we have fixed C at C k. So, C equals suppose I fix C equal to some C k and I fix theta equal to some theta some theta k at iteration k. And now if we end up if at iteration k now with this suppose I minimize the augmented Lagrangian. So, then that would make that if I minimize the augmented Lagrangian to get a value X k the X k is obtained. So, you minimize L of X comma theta k given C k you minimize this you know and by minimizing this you get X k. Now, this would mean sorry augmented Lagrangian. So, now this would mean that the gradient of the augmented Lagrangian with respect to X must be close to 0 or exactly 0 if you come to the exact solution. So, we would need that at X k this theta k given C k should be approximately equal to 0. You calculate the gradient of the augmented Lagrangian that turns out to be gradient of f gradient of f plus summation j goes from 1 to p theta k j minus C sorry plus C k times H j of X k the whole thing times gradient of H j evaluated at X k. Now, if you look at if you if you compare this with your KKT conditions what you would want is that eventually this term here which is in the bracket this should start resembling we want that this should start resembling your optimal Lagrange multiplier. So, suppose this is so suppose theta j so suppose theta j star are the optimal Lagrange multipliers then in that case we would want this quantity the one that is that I have that I have put an under brace below that quantity should start resembling or start approaching theta j star. Now, I can rearrange this and write this in the following way. So, write this as theta j star to be approximately equal to theta k j plus C k times H j of X k or equivalently I say H j of X k is approximately equal to theta j star minus theta k j the whole divided by C k. Now, what this means is that if somehow I am able to get this the consequence of this little calculation is that it tells me that if I am somehow able to get this term here the numerator to be close to 0 that means if I have my Lagrange multiplier correctly figured out that means if theta k is close to theta star then even when C k is finite I should still be able to make this get close become feasible make my H j of H j of X k close to 0 that means that should approach feasibility even for finite values of C k. Recall now this is exactly what we were trying to achieve when we introduce this non-differentiable penalty function with the non-differentiable penalty function we were able to do this by keeping we were able to use a finite value of the penalty parameter and yet get to the true solution and get to a feasible solution here is something similar is happening with the finite value even with the finite value of the penalty parameter we are able to get feasibility. Now, this particular equation also gives us a hint on how exactly should one update how exactly one should one update the penalty how sorry the value of the theta k. So, what we did right now was we fixed a value of C k we fixed a value of theta k and we said we let us minimize the augmented Lagrangian to get to get X k and then what we are saying is well if we were at nearly at the true solution the way it should behave is that this should become equal to theta this should become equal to the to the next to the next to the optimal Lagrange multiplier. So, thinking from this what we the way we update now are Lagrange multiplier. So, the Lagrange multiplier update Lagrange multiplier update is simply this that I do theta k plus 1 j equals theta kj plus C k times nj of X k and does this for all j going from 1 to p. So, this is basically the idea. So, the overall augmented Lagrangian method is then to do the following you let you consider you consider a sequence of penalty parameters consider C k that increases an increasing sequence of penalty parameters. And so this can I have taken this to go to infinity but it is basically simply an increasing sequence alright. So, you choose an increasing sequence of penalty parameters for and start off with some value of theta theta 0 which is your initial guess about the Lagrange multiplier start off with some value of X 0 which is your initial guess about the primal solution and then you iterate by doing the following you minimize the at each iteration you minimize this overall X. And alongside after once you get the X you also update the update theta theta k plus 1 as theta k for the jth constraint plus C k at j of X k and you do this now in infinite loop. So, what are what one is doing effectively is one is simultaneous one is doing pipe doing these two things you are increasing the penalty for a parameter and you are all you are simultaneously adjusting the primal and the dual variables. So, this sort of a method is what is called a method which this sort of a method is what is an example of a primal dual method. Why is this a primal dual method because it is not just simply searching in the primal space it is searching in the primal space and also at the same time using the using what which is learned from the primal space to also inform or update the current guess about the dual variables. So, effectively it is you if you think about it it is operating in some sense simultaneously in the primal dual space though priority is in some sense given to the primal space and the dual space is treated almost as a parameter. So, one of the big lessons in optimization is that optimization is best seen you know through neither the primal space nor the dual space, but rather the jointly the primal dual space when we did when we did a study of optimization of duality in optimization. For example, the way of addressing that problem was through the cost constrained pairs that that could be attained by an optimization problem. So, that object was an object that lied jointly in the in the objective and the constrained space. So, analogous to that in the decision space is the that is in the value space. So, in the decision space the analogous thing to do is to get primal variables as well as simultaneously the dual variables. So, this we are the augmented Lagrangian method is so effective because of this because because one because at its heart it is trying to play with both both variable that one. So, it and is therefore attacking the problem in the in you know using all the levers available. Although it is not a full blown primal dual method a full blown primal dual method is an example of a full blown primal dual method would be an interior point method that is the method I will talk to you about next. But this is this this method come is sort of a stepping stone towards interior point methods. So, the main theorem that we can we can we have for this for an augmented Lagrangian method is that if essentially if so let X star be a local minimum of the constrained optimization problem constrained optimization minimize f subject to h j of X equal to 0 for all j going from 1 to p. Suppose, suppose LICQ holds at x star and second order sufficient conditions are satisfied theta j equal to some theta j star. Then there exists exists threshold or let us say a finite threshold finite threshold c bar such that for all c greater than equal to c bar X star is a strict local minimizer. So, it is a strict local minimizer of this augmented Lagrangian. So, from this theorem so this is now not a complete theorem of convergence of the augmented Lagrangian method that that takes a little bit more to state it is quite a mouthful to state that. But essentially from here what we are seeing is that if I fix the Lagrange multiplier at the optimal value then it is enough for me to keep a finite value of the finite value of the penalty parameter. And with that finite value of the penalty parameter I can I my original the solution of my original problem can also be obtained by minimizing the augmented Lagrangian function. Now what one can do is from build on this and then work with not the exact value of the Lagrange multiplier but rather a value of the Lagrange multiplier that is approaching the optimal. And then from there you get you one can conclude the convergence of the convergence of the augmented Lagrangian method to a KKT point of to a KKT point of the constrained optimization problem. So, this basically brings us to a close on this particular topic of augmented Lagrangian methods but there is an before I move to interior point methods there is another type of method which I wanted to highlight and wanted to teach you about which is what is called a cutting plane method. Now this sort of method actually relies on convexity and makes you know very clever use of the convex analytic geometry of the problem. So, but it is it is not a very general method but it is it is it is probably it is a very elegant method. So, I would say the let so I thought I will just talk to you about it.