 So, we will in the last lecture we ended with an important theorem which was the theorem of Weistras. Weistras was a great analyst. He had many theorems to name. So, this is this but the particular one I am referring to is the following that let S be a subset of Rn be closed and let f from Rn to R be continuous. The claim was that there exists a point that attains the minimum the infimum of f in the set. So, there exists an x star in S such that f of x star is equal to the infimum of f of x as x range is in S. In this case we said the word we use was that the infimum of f is attained. Now because a problem of minimizing a function f can always be converted to a problem of maximizing another function and when by just changing f to minus f and changing f to minus f preserves the continuity of f. So, f if f is continuous then minus f is also continuous. So, that does not change the continuity. So, consequently this theorem also gives us for free that there exists another x let us call this x hat suppose x hat in S such that f of x hat equals the supremum of f. So, both the infimum as well as the supremum is attained. Now let us just give some more intuition on what this theorem is actually doing because you will get an idea from this about what is happening when your algorithms in optimization do not converge and so on. So, I had told you two examples last time. So, let me draw those two examples again. So, one was this an example of this function. I am plotting here on the x axis the function is the function is a function of real numbers. So, this is what I am plotting here is just the x axis of real numbers. So, I am plotting R. So, the function is like this and there is some point A where the function value jumps to the bold point there this red point that is the value of the function at A and then after that the function then increases this way. So, the function decreases till this point till A, but at A its value is it jumps and then it increases in this sort of fashion. This is one case. Another case is this case which I drew last time which was the case of where the function is e to the minus x. So, here f x equals e to the minus x and we took s as the non-negative reals. So, what is the infimum of f here? The infimum of f in the second case the infimum of f in the second case is 0. So, as the least possible the infimum of f over s is 0 and we found that there was a problem here that there is there was no x there was no x star for which f of x star equal to 0. There is now a third case on top of that. So, suppose I consider the interval from just 0 to 1, but I exclude the end points. So, I have an open interval from 0 to 1. So, that is my s, s is my open interval from 0 to 1. Let us just simply take this sort of linear function that keeps decreasing this way. Now, in each of these cases can you tell me what the what the infimum of f is and whether it is attained. So, if you recall the infimum of f in this case was this height right this height here this here is the infimum of f. So, I am taking s and let us say r for some reason. In this case the infimum of f in this case is 0. What is the infimum of f here? What is the infimum of f in the third case? The infimum of f in the third case is again this height and now in each of these cases let us ask if the infimum is attained. So, what happens in this case? In this case in the first case, in the first case let us call this case 1, let us call this case 2, let us call this case 3. So, in case 1 what happens is the infimum the function is tending towards to becoming smaller and smaller as you approach A from the left you get lower and lower values, but at A the value jumps right. So, the problem in case 1 is it you can say is a discontinuity in the in F. What is the problem in case 2? The problem in case 2 is that the infimum is 0 but the infimum is attained quote unquote at infinity right and if infinity is not part of our domain we are looking at only real numbers right. So, the problem here is that the domain is unbounded domain or more specifically let us say S is unbounded. What is the problem in case 3? Is the infimum attained in case 3? Again what you see here is you see that the function value decreases as we go down here keeps decreasing as you keep going closer and closer to 1. The infimum is at the infimum is eventually the limiting value of this of this height it would have been F of 1 if F of 1 was defined in the same way and if 1 was part of the domain, but then 1 is not part of the domain right. So, what is the problem in this case? The problem is that the S is the boundary of S is not included in S. So, S is not closed. So, what if you go if you see these 3 issues that have propped up one is discontinuity the other is unboundedness the other is closedness. If you plug all 3 of them what Weister's theorem is telling you is that if you plug all 3 of these deficiencies then there is no problem of you are assured that the infimum will be attained right. So, if a function if they are if you are set is closed and bounded if your function is continuous then you are infimum and also the supremum of your function on S will be attained. This is what Weister's theorem is telling you okay. This also brings me to some couple of other points which we should mention because eventually the geometry of optimization depends on the kind of sets over which you are optimizing and the kind of function over which you are optimizing. So, let me mention a couple of other things. Remember what the definition of an open set was an open set S is set to be open if there exists a radius r greater than 0 such that B of x, r is completely included in S. S is said to be open if for all x in S there exists a radius r greater than 0 such that B of x, r lies completely in S. Now, if a set is not open then we can in some cases extract those points around which such balls exist that is what is called the interior of a set. So, I will define that. So, so let C subset of Rn be any set the interior of C denoted in this way it is denoted usually by C with a little circle on top. This is simply the union of all sets S that are such that S is open and S is contained completely in C. So, take all the open sets that are completely contained in C take their union that is what is called the interior of C. And there is an analogous notion of what is called the closure of C that is denoted by this C with a bar on top. This is called the closure of and this is the intersection of all sets S such that S is closed and S contains C. So, the interior of a set is the largest possible open set you can fit inside the set. Do you see that the interior itself is an open set because it is an union of arbitrary open set. So, the interior itself is an open set. So, interior is the largest therefore the largest open set in C and the interior and likewise the closure is also a closed set because it is an intersection of an arbitrary number of closed sets. So, it is the smallest closed set containing. Now, if you consider this set which is let us define this let us denote this by del C like this. This is what is called the boundary. This is simply defined as C closure minus C interior which means you look at the closure and remove from it all points that are there in the interior whatever remains is what is called the boundary. Now, if C is itself closed then C closure is equal to C. If C is itself open then C interior is equal to C. Is this clear? But if C is neither closed nor open then these may in general be different. Now, the reason I bring this up is because we have to as we start talking about optimization problems we have to pay careful attention to the kind of to the way we are defining this set. So, now suppose you have a if you encounter an issue such as for example the 3 cases that I just mentioned you have case 1, case 2, case 3 the weight and these are in all these cases you would have the issue you will have the difficulty that your infimum is not being attained. So, there is no solution to there is no optimal solution to an optimization problem either in this case or in this case or in this case. Now, the way to fix it now in case 1 the thing the culprit is the fact that your function has a discontinuity. So, you have your function itself needs to change what you are optimizing that itself needs to be changed in order and you need to consider something that is continuous how do you fix case 2 what will happen in case 2 is suppose you run an optimization algorithm it keeps searching for a better and better point that will give you a better and better value for it and in the process it will keep going further and further and further away towards infinity eventually you will reach a stage where you have run out of your precision of your machine. If you have used MATLAB you would have encountered this error called NAN not a number. So, this sort of error you this is the sort of error you would end up with. So, what this is suggesting is that your domain is too large you need to plug your domain because this is not the only way you might want to consider a different function and so on that is a separate matter but since here the domain is that to be blamed I am talking of the domain. So, your domain needs to be clipped and you need to clip it at some point you need to bound it consider a bounded domain. The third case tells you that simply bounding it without including the boundary point is not enough you have to make sure that eventually your domain has to be both closed as well as bounded. So, if your boundary points are not included you again have this problem that your algorithm keeps searching goes further and further closer to the boundary but then the boundary is not included. So, it keeps searching it will never actually get to a solution. So, you need so in the case if you are in case 3 you need what you need to do is make sure that if you are in case 3 one simple way of fixing this would be that you simply you make sure your set is closed and for that all you need to do is simply take the closure of your set. So, now let us again go back to an optimization problem and now from here onwards I am going to assume that an infimum is attained. So, if you remember my notation for when the infimum is attained you said we will write it like this we will call it the minimum. And so the way we will write optimization problem is that we will write here a function f that we want to minimize subject to this condition. So, when we when I write subject to something it means that this is the set over which I am minimizing and this is the function that I am looking to minimize. So, in this any point in S is called a feasible point it is called a feasible point S as a whole is called the feasible region points that are outside S that means that that is the belong to S complement they are called infeasible they are infeasible points.