 Let us move on to the, to a harder topic in optimization which is, which involves optimization with inequality constraints. Remember what we, the two observations we have made so far, one is that if you have an optimization problem where the constraint is an open set, if this is an open set then a necessary condition for optimization which means just to so long, which means a condition that any minimum minimizer must satisfy is simply that the gradient is, so if x star is a local minimum then the gradient of f at x star should be equal to 0. And if you have second order information the Hessian should be positive. Now if you have instead equality constraints, suppose you have minimizing this subject to, subject to equality constraint, let us say for simplicity a single equality constraint hx equal to 0, then in that case we found that the necessary condition takes a slightly different form that we, you need to introduce another variable into the calculation which is called the Lagrange multiplier corresponding to this constraint. So we, for every constraint that is a Lagrange multiplier, so the condition then becomes something like this gradient of f x star plus lambda times gradient of h evaluated at x star should be equal to 0. So the Lagrange multiplier there exists a lambda such that this is and of course that h of x should be equal to 0, h of x star equals 0. So these, so the point here is that when you are in an open set the problem has a, looks very different from where you are on the surface. Of course when you are on the surface also we were able to somehow reduce it to a problem of optimizing over an open set but that was a reduction but then rad reduction also brought in this new quantity which is called the Lagrange multiplier. Now the problem with inequality constraints is that it combines the features of both of these problems. So what happens when you are in it, if you have an inequality constraint like this, suppose you want to suppose minimize this subject to the norm of this less than equal to r. So what sort of region is this? So these are suppose my axis, this is my origin, this is a radius r and I am looking, I am optimizing the function f over a ball of radius r. So the ball contains all the points on the surface of so on the shell of the ball on the surface of the sphere as well as all these points that are inside. So if I consider a point x star here that is inside then and this is suppose a local minimum suppose I write this in red suppose I consider a point x star here that is in the interior of this ball and it is suppose a local minimum of this optimization problem then it is also what it means is it is a local minimum over this a small neighborhood around it which lies completely inside the ball. So I can take a small enough neighborhood around x star which lies completely inside the ball and x star is a local minimum over that is a minimum over that neighborhood. Then what that would do is that would then in that case the problem would sort of take the character of this problem, problem of optimizing a function over an open set. Whereas if my x star is here on the boundary or on the shell of this region then the problem would take the form of character of this sort of thing because you would be satisfying hx equal to 0 or in this case norm x equal to r. So what happens is once you have inequality constraints like this inequality constraints are basically capturing two conditions together they are either saying norm x is either equal to r or norm x is strictly less than r either of these is okay with that. Now strictly less than r is basically the interior of this sphere and that is like this problem that is like optimizing over an open set. The one that you have equal to r is the shell of this sphere that is like optimizing over an equality constraint. So all the complications in optimization appear because of this sort of character of this problem which is sort of changes there is a phase change that happens once you go form interior onto the boundary. The nature of the problem the way you think about it all of it changes okay and so what we will now build towards is ways by which you can again come up with conditions that a local minimum of such a problem must satisfy okay and then using those one can of course build algorithms for finding a local minimum okay. What this also entails is that we need to learn a little bit about what is called convex analysis okay convex analysis and so I will tell you what that I will introduce that subject today. So the convexity is or convex analysis is basically the study of overall shape of a region or of a body okay it is not just about the local shape but about the overall shape and so for that let us start introducing a few concepts. So suppose if I give you two points x comma y in Rn the this here we will just denote the line segment joining x and y. So what is that what is the line segment joining x and y how do I express that? So this is a say a point alpha x plus 1 minus alpha y where alpha can range over 0 to 1. So as I range alpha from 0 to 1 if here is my point x here is my point y as I range alpha from 0 to 1 at 0 I am at y at 1 I am at x okay I cover this entire segment. So this is this is the line segment joining x and y. A set S set of Rn is set to be affine the line passing any distinct points of S lies in S. So for so what this means is a set is set to be affine if the line passing through any two distinct points of S lies in S means that you should take you can take any two distinct points say point x and y these are points in S. You look at the line that passes through them now can you explain can you tell me what is the line that passes through them what should be the expression for this? The segment joining x and y I have defined above. So what is the line passing through x and y? So if I take so the line passing through x and y is all points like this alpha x plus 1 minus alpha y where alpha is any real number alpha can be positive negative greater than 1 less than 1 does not work. So it of course contains the segment but it also extends beyond beyond beyond x and beyond y. So that is this line. Now what is this what is the what is the definition of an affine set saying it is saying that set is set to be affine if the lines passing through any two distinct points of S lies in S. So take any two distinct points like this x and y the line passing through them which is which is this set it is the set of all alpha x plus 1 minus alpha y where alpha range is over R this this line should be in S right. So which means that this here so if x and y belong to S then alpha x plus 1 minus alpha y belong to S for all alpha y. This is what this is what this is what it means for S to be an affine set. Now can you give me examples of an affine set? So the the entire ambient space Rn is an affine space clearly any other a plane a plane in Rn. So I have to define for you what a plane is but you have not done that but if you if you are just willing to just let us look at R let us look at R3 for example and look at a plane in R3 R3 is the space around us plane and R3 would be say the wall here that the entire that wall you just extend from it to all the way till infinity up and down in all directions that will define for you a plane in R3. Now is that an affine set? Why is that an affine set? If I take any two points that lie on the wall look at the line passing through those points that entire line will still be on the wall that is that is clearly an affine set. Now suppose I ask you so the wall is an affine set what about the wall and all the air on this side of it? Why is that not an affine set precisely? So if I take two points like this that here take the line passing through this will go hit the wall puncture the wall and go on to the other side of the wall. And so it will not necessarily be it is not necessarily that it is such a line will be always in this set. So a plane that you just mentioned is an affine set I will explain what a plane is we have to define it but plane is an affine set more generally actually every subspace is an affine set right any subspace of Rn that you take is an affine set any subspace of Rn is an affine set. So the subspace is always closed under scalar multiplication and linear combination. So consequently subspace is an affine set. Suppose if you take K point suppose we take K points in an affine set okay suppose you take not just to take K points in an affine set how do you define okay so what one thinks let me ask this question in the following. Suppose I take K points in an affine set and you say X1, X2, X3 dot, dot, dot say it is K okay. Now look at this sort of point this point is summation theta i X i why I use earlier notation let us use alpha i summation alpha i X i where alphas are such that they sum to 1. So you look at this point so you have taken these K point which are in the in an affine set X1, X2, X3 all of them is and consider this point let us call this point Y. Y is defined as summation alpha i X i going from 1 to K where alpha i have this are chosen such a way that they sum to 1. Now what can you say about this point? Now let us take to let us suppose I make alpha 1 and alpha 2 positive and the alpha 1 and alpha 2 nonzero and the others as 0. So that will then capture as I vary my alphas that will then capture all the entire line passing through X1 and X2. If I take similarly alpha 2 and alpha 3 as nonzero and make everything else 0 that will capture for me the entire line passing through X2 and X3. What if I took alpha 1, alpha 2 and alpha 3 as nonzero if you think about it for a moment you will realize that this actually captures for you any point that lies in the same plane as alpha as X1, X2 and X3. The entire plane defined by X1, X2, X3 is being captured by this and as you keep doing this what you are doing is you are building you are creating a what you are creating is something what we what is called a hyper plane as you include more and more points like this eventually what you are going to create is something that is akin to a subspace. But what is going to be the property of that? It is not exactly a subspace I will be clear about that what would be the property of that? It would be such that you so what it would be such that if you take any two points that lie in that set the line passing through that those two points will be in the set. So the point I want to make is that if your set is affine and then you consider any k points in it and consider a point Y any k points like this X1 till Xk and consider a point Y that is defined like this by taking a linear combination which sums to 1. This is what is called an affine combination of the points X1 to Xk which are in S. So you take any affine combination of these points this point this affine combination also lies in S. So if S is affine then any affine combination points in S lies in S that is the definition of an affine combination. So it generalizes the previous definition in the previous case. So in the previous case here my alpha 1 was this was this alpha and alpha 2 was the 1 was 1 minus alpha 1 plus alpha 2 is still 1. So this condition that the alpha should sum to 1 is simply is just generalizing the is generalizing the previous condition. So what we are so here what we are the claim that is being made here is that if S is affine and not only that the line passing through any two points lies completely in S. You can take any affine combination of any point not just two any k point and k can be anything take any affine combination that entire affine combination lies in. In fact you can define you can actually say the following S is affine if and only if it contains all affine combinations of points. Now if you start thinking about it this way you start realizing that an affine set has many of the properties that you would think a subspace should have. Because if you take linear combinations of points in a subspace that also lies in the same subspace. The only thing that is missing is the property of scalar multiplication. If you scale the point in a subspace you have you are always in the subspace an affine set need not be like that. Take for example let us draw this in R2 say for example here is an affine set this is just a line take any two points on it. It is affine combination is the line itself it lies completely it is the line itself. So this is an affine set the problem is that if I however if I scale any point say take a point a vector x that lies here and scale it say suppose I double the length of it double the length of it that will end up here somewhere outside that is not on this line. So what is the issue the issue is that the line need not pass through the origin. So an affine set is actually a subspace that has been shifted. So this can actually be shown. So if S is affine and say x0 belongs to S and look at the set V is just the so V depend is this x minus x0 as x ranges through S. So what I am doing is I have taken this set S taken some point x0 in it and subtracted that from all the points in the set. Now V will have will pass through the origin V contains 0 why because x0 is in S. So V contains the origin so but it is essentially just a shift of S by all vectors have been you have subtracted x0 from every vector in S. So you have shifted it shifted the entire set S but now that you have shifted it will pass through the origin and it will now have the properties of scalar multiplication of being closed under scalar multiplication also. So V is actually a subspace. So if you take if you so an affine set is this sort of flat set which is not necessarily passing through the origin I shift it it will now be a it becomes actually a subspace after shift does not matter and it does not matter what I shifted by. I can choose any x0 in S and that every x0 works just as well. An affine set you should remember is just a shifted subspace. Now if it is a shifted subspace that also tells you something now a subspace is always can always be given by the null space of some matrix. So if you have any subspace so suppose if so if u is a subspace of Rn then you can always find then there exist some matrix like this A in some m cross n such that u is the same as the null space of this matrix. So u is the set of solutions of the equation A x equals 0. You can always find such a matrix the m and the m depends on the dimension of u. The point is that you can get you can always find a matrix A such that if you lake degrees null space of A that is actually the subspace u. Now which means any subspace is the solution set of linear system of equations with a zero right hand side. So these are linear equations with zero right on the RHS. So now what would happen if I shift this if I shift this instead of having a zero on the RHS what I would get is some other vector on the RHS right. So then an affine set if S is an affine set then there exists some A like this cross n and B in some Rn such that S is just the set of solutions of this system of equations. So an affine set is actually in is nothing but a solution set of a linear system of equations just expressed in a geometric way rather than in an algebraic way.