 So, now with this now we can begin the topic of non-linear optimization which is now our most general optimization problem which is where we have say minimizing some function f over a set say S. Now what we need for this sort of problem is to say find a very general condition so that which characterizes a point x star for a that is a local minimum. So, suppose x star is a local minimum, suppose x star is a local minimum then what then we need a condition that this sort of x star must satisfy unnecessary condition for this sort of x star alright. So, this condition is given through what is called the tangent cone of the set. So, I will define this for you what is a tangent cone. So, the tangent cone at a point x star for the set S is a collection of vectors D such that such that there exists a sequence x k lying in S x k converging to x star and a sequence say tau k that is positive. So, it decreases and decreases to 0 such that D can be expressed as this limit D can be expressed as the limit x k minus x star divided by tau k. So, the tangent cone is this is those vectors D for which there exists a sequence x k in S converging to x star and a sequence tau k of these are these are scalars positive scalars converging to 0 such that D can be written as in this form it is a limit as k tends to infinity of x k minus x star divided by tau k. Now what is so can you make a before I before we see what this set actually means can you make a few simple observations do you see that D must be the sorry the set T x star S this tangent what we call the tangent cone do you see that this must be a cone it is a it is very easy to see that because if I scale if I look at a scaled version of D that is that always exists in this set because all I need to do to construct such a D it would be to just simply scale the tau k appropriately right. So, if I look at say alpha times D then if I want to get alpha times D all I have to do is divide the tau k by alpha and that will that would give me alpha times D right. So, the tangent cone is always a cone always a cone and what does this set look like but it is a cone but what sort of set is it. So, to get some some intuition on that let us let us try to let us try to see what this this x this particular thing which is the limit of x k minus x star divided by tau k what is that actually saying. So, suppose I have a I have a region like this and suppose let us let us begin here let us suppose this is my point x star now what is this so this is my set S and this is my point x star. So, now for this sort of point what is the tangent cone. So, this is a point x star that lies in the interior of S right. So, x star lies x star belongs to the interior of S ok. Now in this case what is the tangent cone. Now the tangent cone if you look at the definition what it says is well it is those directions D such that you can construct a sequence x k converging to x star and a sequence tau k going down to 0 such that D can be written as x k minus x star divided by tau k. Now x k minus x star. So, if if here is my point x star and here is my point x k. So, x k is a sequence that is eventually converging to x star. So, it is a sequence that approaches x star. So, so here is the vector x k minus x star right. It is it is a vector I can think of its origin shifted to x star and pointing towards x k. So, here is my vector x k minus x star x k minus x star as x k goes to 0 sorry as x k goes to x star x k minus x star will go to 0 right. But what about x k minus x star divided by tau k what is what would happen to that where would that end up pointing. So, suppose let me amplify the picture here suppose here is my point x star and here is the sequence x k that converges to x star right. Because suppose x 100 then x 1000 etc. etc and eventually it comes eventually it is the various x k's and they eventually converge to x star right. So, so the distance x k minus x star or the vector x k minus x star that vector is eventually going to become 0. But if you look at x k minus x star divided by tau k what where would that end up pointing. So, the way to think of this is that you can think of tau k as keeping track of time in some sense. So, and x k is a trajectory it is a point it is a point that is moving and it is eventually converging to x star tau k is keeping track of time see since x k converges to x star the numerator is going to 0. But numerator divided by time would be what it would be the it would be the rate of change or equivalently the it would be the tangent to this particular trajectory right. So, it is the so x k minus x star divided by tau k as k tends to infinity would end up pointing along this sort of direction the direction the limiting direction by which you approached x star along this trajectory. So, if for instance you did so here was x star and you did something like this suppose you you did this this this this this and eventually came along the eventually approached x star along this particular direction the limiting value of that direction something like this is the is where that D would end up pointing. And of course, because this is a direction you can always scale it you can measure time in any units you can always scale scale the axis of time and that will give you a longer arrow and the longer vector D that is why this is a cone right. So, so this every every direction every D in in the tangent cone is a direction from which one can approach x star from within from within s while staying within the set s the directions by which you can approach x star is that the set of such direction is the tangent cone. Yes, yes, yes. So, so I was about to come to that question good. So, now come back to this question. So, suppose suppose x star is in the interior of s star s. So, x star lies in the interior of s, then in that case what would be the tangent cone then it would be R n because you can approach x star from any possible direction or any any any direction because there is the entire ball in R n can be fit inside the set s. So, you can approach direct you know, radially along any direction towards the towards x star from inside the ball right. So, so if x star belongs to the interior then is a subset of so then this is then the tangent cone is equal to R n the R n is the ambient space of s right. So, so what this means is that well for points in the interior this the tangent cone is given trivially it is always always R n. Now, what about what if the point is somewhere here. So, what if you have your if the point x star is now here on the boundary then what would be what would be the what would be the tangent cone. Now, every direction so if you are on the boundary and you have the set has an interior and you are on the boundary of it and it is possible to approach from the interior to the point x star then all such possible directions will be in all of these possible directions are included in the tangent cone. And you can keep including directions that just graze past the boundary of the set and you can keep doing that till the point where eventually you actually become just tangent to the set. All of these are possible directions by which you can you can approach you can approach the point x star right. So, if you are so if you so let me just draw this more neatly if you are if your point x star is here on the boundary right. So, then this all of this so in short this entire thing here would be the tangent cone all of these directions right. But then it is a little more complicated because of because of because of some reasons and I will I will explain to you what the reasons are see let us suppose you take suppose you take this sort of set suppose he has my set is now a point here in the interior we know what the tangent cone is. So, my set is this box in our R2 if I have my point in the interior then this point the tangent cone is just R2 because I can approach from any direction. Now if I am if I am here then what would be the tangent cone it would be the it would be this particular half space right it is this half space it is this half space because I can approach from all of these directions the half space formed by this this phase of my box. Now what if I am here at this corner if I am at this corner then actually the tangent cone would then be these 2 this particular so this this cone here would be my tangent right. So, now the reason this has changed is because on in both cases you are on the boundary okay in both cases you are on the boundary alright here on when you are on this boundary you have exactly one tangent to the to the set whereas when you are at this boundary what is the tangent to the set there is no the tangent is actually not defined what you have is tangents to this set this particular surface and tangents to this particular surface and from there we are trying to somehow get to the tangent of at this at this corner point. So, so you have to be a little this is where the complications start begin to arise that. So, if you have a simple you know if you have just of in simplest of cases it does happen that yeah you can think of you can just draw a tangent at that particular point and try to get the get the tangent cone by looking at the time at the tangent. But if you if you do not have differentiability and so on at that of the boundary at that point then you you know you end up having problems with the in defining the in in in visualizing the tangent one right. So, I will I will give you more examples of why this gets come why the tangent cone is a very deceptive object. So, if you have just a simple if you have only let me keep this on the side and start on the new page. So, suppose suppose let us let us consider let us consider something like this. So, suppose we have suppose I have a circle this is my origin a circle without the interior and I consider a point on the boundary. Now what is the tangent cone for this point? It is only the tangent plane right only this particular plane that would be the if I have a circle but with the interior considered and I and I take this point out here as my point x star then. So, circle with the interior then the tangent cone is this half plane half space right the half space on that lies on this particular side this the side on which the circle lies ok. Now, suppose I did the following suppose I have one circle here and suppose I took another circle like this and I included say the interior of both. So, I have this included this. Now all of these points here that I am marking shading with red they are now part of both both circles. So, let us call this set S1 let us call this set S2 the red region is part of both circles. So, if I take a point here suppose or if I take a point let us begin with a point like this. So, take a point here what is the what is the tangent cone at this point at this point tangent ok tangent cone with respect to what set. So, we are looking for the tangent cone at. So, with respect to the set the common region with respect to the set S1 intersection S2 ok. So, this point belongs to the common region what is the tangent cone at this point it is again going to be this half space ok what is the tangent cone at this point. So, if X star is this one which lies on the boundary of both circles. So, what you would you might you know visually you might be able to figure out that well I should be looking at this one I should be looking at this one and let me take the intersection of these two the common region between these two and the common region would be would be this. So, that would give me the tangent cone at this point at this point X star right. So, from by looking at examples like this you would you you are led to sort of thinking that this is actually the same as doing effectively just same as doing tangent cone of the intersection you would think is equal to equal to this the intersection of you just look at individual sets look at the tangent cones and then when you get to a point that lies in both look at the common part of the tangent cones of the two of the two tangent cones ok. But you will see soon see that this fails miserably ok. So, let us consider this here is one circle and here is another circle that touches this circle at exactly one point ok here is my it is. So, these two circles are tangent to each other here is my point X star. So, my and I am including the interior of the circle. So, this is my set S 1 this is my set S 2. So, now what is the S 1 intersection S 2 S 1 intersection S 2 they intersect at only one point then my S 1 intersection S 2 is just the point X star ok. Now, what is the tangent cone at X star of S 1 with respect to S 1 intersection S 2 it it is 0 only the vector 0 right it is always a cone. So, it has it the vector only the vector 0 because there is only one point in the set X k is equal to X star. So, if you are always going to get 0, but now what if you did your intersection formula what would you get from the intersection. So, you look at the intersection. So, he has the tangent cone with respect to S S 2 and this on the other side is the tangent cone with respect to S 1 right and you take the intersection what you would get is just the tangent plane. So, if you look at T of X star with respect to S 1 intersection T of X star with respect to S 2 that is that is equal to the tangent the entire tangent plane at X star and what is. So, what has happened here the intersection of individual tangent cones has turned out to be much larger than the true tangent cone now you can see this is this can very quickly get slippery. So, when you have multiple surfaces intersecting you it is not possible to get get to the tangent cone of the common region by just looking at each of them individually and then taking the intersection it is possible that you will get end up getting a larger set. So, this is this is one of the key reasons why study of tangent cones you need to do carefully there is another reason also I will come to that I will I will come to that in the next class, but the main so the main the main sort of slippery part of tangent cones is this. Now, why do tangent cones play such an important role and I will just I will state the result for you it is not that hard to prove the reason they play such an important role is in optimization is the following. So, suppose so here is the main theorem in which they appear. So, suppose x star is a local minimum of this optimization minimum minimizing f x in s x in s then you must have that grad f of x star transpose D is greater than equal to 0 for all D in the tangent cone. So, the gradient of f at x star transpose D is greater than equal to 0 for all D in the tangent cone at x star with respect to s the proof is very easy. So, the proof is so suppose suppose it is not the case suppose this is not true then in that case there exists a D in the tangent cone such that transpose D is less than 0. Now, D belongs to the tangent cone since D belongs to the tangent cone you can there exists your sequence x k converging to x star. So, x k in s and a tau k decreasing to 0 such that if I look at tau k D tau k D is all is equal to x k minus x star plus something that is small o of tau k. We have seen this notation before the small o notation small o just means that small o of tau k is a quantity which when divided by tau k also goes to 0. So, tau k D is therefore equal to x k minus x star plus small o of tau k. So, what we have done and moreover by Taylor's theorem f of x k is equal to f of x star plus gradient of f at x star transpose x k minus x star plus small o of norm of x k minus x star. So, now if I put if I substitute this in I would get that f of x k is equal to f of x star plus gradient of f at x star transpose D into tau k plus something with that is small o of tau k. So, this small o of x k minus x star I put that in I that will soon that will then become small o of tau k right by substituting. So, after substituting this here I get that this is equal to. So, f of x k is equal to f of x star plus gradient of x star transpose D in tau k plus something that small o of tau k which means that f of x k is equal to f of x star plus tau k times grad f of x star transpose D plus small o of tau k divided by tau k. And now small o of tau k divided by tau k is a quantity that this is a quantity that and this is a quantity that goes to 0 and since the first quantity is strictly negative by assumption. So, by assumption the first quantity is strictly negative the second quantity is going to 0 which means that eventually for large enough k. So, let me just complete it here for large k f of x k would be less than f of x star. But then x k converges to x star which means that if you are if you if x k is converging to x star and but for large enough k f of x k becomes less than f of x star then it cannot be that x star is a local minimum. So, this is a contradiction. So, this is a contradiction. So, which means that this must be the case. So, if suppose x star is a local minimum of this I forgot to write here f belongs to C1. So, that is differentiable then it then we must have that grad f transpose D is greater than equal to 0 for all D in the tangent group. So, you can see here this is a very general purpose condition any set any function so long as the function is differentiable you it must satisfy this. But the problem that we encounter is that if often our sets are defined using the intersection of multiple set and we do not know how to get to the tangent cone of the intersection. So, that is the next thing we need to sort of encounter overcome and then we will be able to write out optimality conditions for optimization problems. So, I will end here now.