 So, the trouble that we what we learnt was that basically the same geometric object can be represented in multiple ways using formulae or using algebra basically. And our way of calculating normals is by looking at derivatives of the algebraic function, but then that may not be the right, may not give you the true normals. Yes, could have arisen there, could have arisen there and I will come to that. So, I will put together, you will remember there that there was a condition I had used about the gradients of the constraints must be linearly independent and that is the condition that will save up. So, it turns out, so I will come to, so good that you brought this up. So, what is it that we need to fix this essentially. So, this is what this conditions that will basically ensure thing that, so if I, so what do we want, let us come back to this again. So, we would have liked that tangent cones can be described using formulae like this, using these sort of formulae, this is what we want. And if they are describable using that sort of formulae, then an additional thing that we want is that there is equality here. If you have equality here in these cases and then you have equality here, then we are done. Then you have a formula eventually for the tangent cone of the set as a whole. So, conditions that ensure that are what are called constraint qualifications, so a constraint qualification. So, we say that a constraint qualification holds that a constraint qualification holds at x star if the tangent cone ends up being equal to this. So, since I am looking at only equality constraints, let me look at, let us look at, I will just write it for, sorry inequality constraint, I will just write it for inequality constraint. So, this is equal to for all i in the active set. So, a constraint qualification holds if somehow you get that the tangent cone is equal to the intersection of the individual tangent cones and the individual tangent cones in addition are given by the formula that you expect them to be given by. Then we say that a constraint qualification holds. So, what kind of constraint qualifications? Well, there are many different conditions out there. For example, one of them is that the gradients of these constraints should all be linearly independent. So, for example, there is a constraint qualification called the linear independence constraint qualification which simply says that this if I look at all the great constraints, constraint normals and over collection of inequality constraint normals that are formed by the active constraints, then this set is linearly independent. If you have in addition inequality, if you also have equality constraints, then you need that this in addition to and all of these and the equality constraint gradients, these are all linearly independent. This is one example, but this is just one out of many if you have real constraint qualifications that are out there, eventually aim of all of them is to get equality here. They eventually ensure that you get that the tangent cone is given by this formula. There are many ways by which this can be attained. So, let us go back and look at the two examples we had seen. See one of the this linear, let us look at the linear independence constraint qualification. Linear independence automatically ensures that you will not get equality constraint gradients, the gradients of all the equality constraints. I will write this out more clearly. So, we had seen two examples. So, one example was just in the previous slide where I said that suppose you have it is possible that h of x, you have a constraint like this h of x is equal to 0, but then that is the same as h square of x equal to 0 h cube of x equal to 0 and so on. Now, if you look at the gradients of h square h cube, etc., these gradients will all end up being 0 necessarily by just h being on the constraint because you will get 2h of x gradient of 2h square, 3h square of x gradient of h, etc., etc. So, they being 0 will mean that you will have a problem. You cannot have a 0 vector in a linearly independent set. So, this prevents these kind of cases from happening. Another case which we had seen in the previous lecture was this case where you had 2 circles. Remember I showed you this case. We had these 2 circles that intersected in this sort of way and we looked at a point x star like this and you at a point at this sort of point x star, you said that the tangent cone would be you can look at you can think of the tangent cone in this way you have this and you have this and the common region here, this part would be the tangent cone at x star. So, here what was the logic? Well, here the logic was you said we will take the tangent cone with respect to one set, tangent cone with respect to the other set and look at the intersection of the 2. But then the intersection logic failed when we looked at this example. The same circles, I moved them apart a little bit in such a way that now they are just touching each other at x star. What would happen in this case? The tangent cone with respect to one set would be this tangent cone with respect to the other set. So, the tangent cone at x. So, we are talking of tangent cone at x star in the common region. So, tangent cone of x star with respect to s 1 intersection s 2, a tangent cone again of s star with respect to s 1 intersection s 2 that tangent cone should have been 0. But then if you look at the intersections of the individual tangent cone, then what you get is an entire line. And you see you can see here also what happened is that what has failed is linear independence. So, if you had if the gradients of the 2 sets. So, in this case the gradient in this with respect to gradient of the constrained s 1 here was normal here, gradient with respect to s 2 was normal here and their intersection, the intersection of the individual tangent cones was giving us the correct tangent cone. And the reason that was all working out nicely was because the gradients were actually because the gradients were linearly independent. Whereas, in this case, if you look at the gradients, well the gradients are actually because the sets that sets are perfectly tangent at that point, the gradients actually are collinear opposite direction but collinear. So, again linear. So, here again linear independence has failed. So, linear independence constraint qualification is one sweeping constraint qualification that actually takes care of both such both these issues. It takes care of issues of representation, it takes care of issues of that making sure that the tangent cone of the intersection is the intersection of the tangent. So, both of these gaps make which is the first one being ensuring that there is equality in these two places and the second one ensuring that there is an equality here. Both of these gaps are plugged once we have linear independence. And more generally any other constraint qualification. So, constraint qualification ensures that you have equality throughout here. Now, these two constraints, so here are my sets, s 1 and s 2. The gradients with respect, so I can look at the tangent cone of s with respect to s 1 as the tangent cone with respect to s 1 as d such that this is less than equal to 0 and the tangent cone with respect to s 2 as this. But then if I look at the common region of these two because g 1 and g gradient of g 1 and gradient of g 2 are collinear the common region of this is simply gradient of g such that sorry d such that gradient of g 1 gradient this transpose d is equal to 0. That would be the tangent cone with respect to s 1 and that is what sorry that would be the intersection of the two tangent cones. So, the intersection of the two tangent cones would turn out to be this line, this half hyper plane, but the true tangent cone is actually just 0. That concern is also solved that is what I am saying. So, that concern is also solved by the linear independence. So, then in that case you would not have the linear independence effectively. So, constraint qualifications take care of all of these. So, all that is what I said. So, there are two gaps here, two possible gaps. One gap is out here making sure that these hold with equality. The other possible gap is here making sure that this holds with equality. So, once we have both these plug we will get equality throughout. But power terms are one possible reason. The point is the unifying one unifying way of dealing with all of this is linear independence. Yes, but see for example, but how would you know that you are at the minimum power term or see power is one kind of reason why this happens. It was just a way for me to demonstrate that the same geometric region can be expressed with multiple different algebraic constraint. But it is not that easy to know whether this is you are at the minimal or not. So, that is what this constraint this condition is ensuring. It is automatically taking care of such matters. So, now go to a fresh page. So, what did we conclude? We said that well if you have a constraint qualification, then a constraint qualification basically ensures this. This is what I want you to remember that constraint qualification ensures that this holds. Formulate the, yes, so you can put it in the, there are many ways you can apply this. One way is that you should see if you can get rid of constraints or redundancies and so on. So, that you have left with only a linearly independent set. That is one possibility. Other possibilities to see if you can formulate the problem to begin with in such a way that linear independence holds. So, somehow you need to ensure that this some constraint qualification is holding. Linear independence is usually not that easy to guarantee because you have to check it point at each point. Then there are other constraint qualifications also that are you know that are slightly different kind. But then they will be more restrictive. They may not always, they may not apply for that many problems. No. So, the way we way this is usually applied is that one checks this whether this holds for all extra all points in the set. All everywhere is your gradient or your gradients linearly independent. That is what you would like to tell. So, now let us move forward. So, then now recall this, recall Parkash lemma that I had mentioned to you a couple of lectures ago. So, what did they say? Well, it said that the following statements are equivalent for all x such that A x is less than equal to 0. We have C transpose x less than equal to 0. And the second is the statement that there exists a lambda greater than equal to 0 such that A transpose lambda is equal to C. So, for all x such that A x is less than equal to 0, we must we have C transpose x less than equal to 0 and there exists a lambda greater than equal to 0 such that A transpose lambda is equal to C. So, here we call this is, so here A is just a matrix in R m cross n and C is a vector in R n. So, that is where that is the context of this. So, now what we will do is we will just apply Farkas lemma to the two things that we have where we have. We have what I proved in the previous lecture which is this statement which is that the gradient of F at x star transpose D is greater than equal to that if x star is a local minimum then of this optimization then gradient of F at x star transpose D is greater than equal to 0 for all D in the tangent cone. And moreover, if my optimization problem is given in this sort of form, if my optimization problem is given in this sort of form then the gradient then the tangent cone is given by this expression. So, let me summarize this. So, consider once again this optimization minimize function F subject to g i of x less than equal to 0 for all i going from 1 to m and let S be the feasible region then x star if x star is a local minimum then gradient of F at x star transpose D is greater than equal to 0 for all D in the tangent cone. Moreover, if a constraint qualification holds at x star then T of x star minus is the same as D such that gradient of g i transpose D is less than equal to 0 for all i in the active set. So, we have just summarized what was there on the previous two slide. So, we have one condition here which says that the gradient of the objective if you have a local minimum x star then the gradient of the objective must make an acute angle with all directions in the tangent cone. The other condition which gives a formula for the tangent cone it says that well D the tangent cone is comprised of D such that gradient of the constraints make acute angles with the directions in the tangent cone for all active constraint sorry obtuse angles with the direction of the tangent cone for all active constraint. So, now look at these two things and compare them with the Farkash lemma that is written right up there. Farkash lemma says that for all x such that this A x less than equal to 0 we must have C transpose x less than equal to 0 that is equivalent to saying that there exists a lambda greater than equal to 0 such that C can be written as A transpose lambda. So, let us apply let us do this. So, I will just state for you the final theorem. So, suppose this is let us call this optimization problem P let x star be a local minimum of this optimization problem P and suppose that a constraint qualification holds at x star then there exists lambda i greater than equal to 0 for i in A of x star such that gradient of F at x star plus some summation lambda i grad g i of x star i equals i in A of x star this must equal 0. What is this say? Well it says that if x star is a local minimum of this optimization problem and suppose that a constraint qualification holds at x star then you can find lambda i greater than equal to 0 for i in the active set of pertaining to x star such that the gradient of the objective is plus a linear combination of the gradients of these constraints is equal to 0. Now this you will this of course will remind you of something that you have already seen in the case of when we were talking of optimization over equality constraints what are these lambdas these are Lagrange multipliers lambda is actually the Lagrange multiplier. We will make this slightly more general in a moment but for the I will quickly show you how this is how this is derived. So, I know that this is greater than equal to 0 for all D in the tangent cone and tangent cone is the same as tangent cone is given by this particular expression which means that this is greater than equal to 0 for all D such that gradient of G i transpose D is less than equal to 0 for all i in A of x star. So, now what I have all I need to do is just apply the Farcash lemma. How do I apply Farcash lemma here? For Farcash lemma the condition here grad G i less of grad G i transpose D less than equal to 0 for all i in A of x star that is analogous to this condition A x less than equal to 0. So, and my grad F transpose D greater than equal to 0 that is analogous to this all I need to do is just change the sign. So, C becomes minus F and then I get back this condition. So, what is so effectively I am so my so apply Farcash lemma in the following way. So, your D becomes x the your minus grad F at x star becomes C and all these G i of x stars for i in this if I put these together they form my matrix A. So, this transpose put these together they will form my matrix A they are the rows of my matrix A. So, I put that together and I can apply Farcash lemma. Now, then for this Farcash lemma tell me this is this it tells me that this thing in the box holds that is equivalent to that there exists lambda greater than equal to 0 such that my C C is minus grad F minus grad F at x star is equal to A transpose lambda and A transpose lambda is what that is just some that is basically this summation that is this summation lambda i grad G i of x star i in A of x star. If I take my grad F to the other side I get the condition that I am looking for. So, this here is what I what I just wrote this is what is called this is called a KKT condition KKT stands for Karush, Kun, Tucker, Karush, Kun and Tucker conditions for the optimum. So, these so these give you necessary conditions for a for a local for x star to be a local minimum of an optimization problem assuming a constraint qualification holds. If a constraint qualification holds and what should be what must that x star satisfy that is what is given by these KKT conditions says that well x star is a local minimum and a constraint for qualification hold then you must be able to find Lagrange multiplies lambda i greater than equal to 0 such that this condition in this boxed condition is satisfied. So, this is now we can this can be general put made a little bit more general and I will explain that in the in the next in the next lecture. So, what I have not allowed for so far are equality constraints in this in this definition of this optimization problem p, but that is not very hard to do because any equality constraint can also be written as two inequality constraints opposing inequality constraints. So, those can be taken into account and that will give putting all that together will give you a comprehensive one comprehensive condition which can be applied for any type of optimization. So, we will do that in the in the in the in the following lecture. So, any questions? You can say well constraint qualification will be used is is used to get you to this point this equality here see without constraint qualification you would not have got concluded this right that is the that is the that is where the problem linear independence is one of them there are others I will I will give a homework based on those also. So, there are there are other type of constraint qualification linear independence is one simple one look at all the gradients and check if they are linearly independent.