 Welcome everyone to the next lecture on, so what I plan to discuss today is what is called transformations of optimization problems. Now, we should relate this to what we did when we were talking of optimization over equality constraints. So remember we were talking of this problem where we had this ellipse and we wanted to find the inscribed rectangle with maximum area and we said we can take a point like this x star, y star on the ellipse, suppose that gives you the maximum area and then what we did was we said you will use implicit function theorem to around x star, y star in a neighborhood of x star, y star solve for y in terms of x, so every y that was in this, every x comma y in this neighborhood was written in such a way that y could be obtained as a function of x. So, if I give you an x in this region here, you could give me a y that lied exactly on the ellipse. So, this helped you transform, this was the case of basically transforming a problem, a problem of which is in two variables x and y, problem in x, y to a problem only in x. Now, this is a very sophisticated transformation and it was affected by the use of implicit function theorem and moreover because of the nature of the implicit function theorem, the limitations of it, we can only say that it holds in this neighborhood, it was a locally valid transformation. Now, there are certain transformations you can make which hold on the entire region but do not change the solution of the problem, that is why I will be focusing on today. And there as a result of them being such general purpose transformation, they are also in some sense very simple, but sometimes they are not easy to notice and you may not, it is not, it may not be a priori obvious to you that this could be done, so this, so I want to spend some time discussing this. So, suppose we consider, let us consider an optimization problem like this, you are minimizing the function f of x, the decision variable is x over you have inequality as well as equality constraints. So, you have constraint like this, g i of x less than equal to 0 for all i from 1 to m and h j of x equal to 0 for all j ranging from 1 to p. So, this will be my canonical optimization problem and I will write this in a different equivalent form. So, first thing is to, let us take the first form. So, do you notice that this is actually equivalent to this optimization problem is equivalent to this problem. So, what I have done is I have multiplied the objective by alpha, multiplied the inequality constraints by beta, ith inequality constraint by beta i and the jth equality constraint by gamma j. Now, are these two equivalent alpha, betas, gammas are all scalars, these are all scalars. Are these two equivalent? For equivalence, you need that you need some constraints on the signs of these. So, alpha, beta, gamma, these are all scalars, I am going to take alpha as positive, all the betas also positive. And gammas, I will make sure that I need that gammas are not 0. So, if I do this, what happens as a result of this? So, the objective function scales, the objective function is being scaled by alpha when I do this. So, earlier whenever you are getting one unit of objective, now you get alpha unit of objective. Now, what happens to the feasible region? The objective function is scaling, what about the feasible region? There is a feasible region scale. So, the feasible region is this, the individual functions that define the feasible region are scaling. These functions, the constraints that define the feasible region, they are scaling. So, your gi of x is being scaled by beta i, scaled up or down depending on whether beta is greater than 1 or less than 1. Similarly, h j of x is being scaled by gamma j, they are all being scaled. But the x is that satisfy the constraint, they are still the same. So, what that means is, if there is any x, if there is any x that solves these constraints, that satisfies these constraints, let us call them, let us denote these by star. If there is any x that is feasible for star, then it is also feasible for the optimization problem here in 1. Likewise, if there is any x that is feasible for the optimization problem 1, it is also feasible for star. So, the search space, the space over which you are searching, the x is over which you are searching has not changed. So, the feasible regions of both optimization problems are the same. Let me note this, they have the same feasible regions. What about the objective? The objective of 1 is being scaled by, the objective of 1 is alpha times the objective of star. So, the objective value will be scaled. So, if you send these two problems into a subroutine that solves them, what are you going to get as same for both? Suppose they both have one unique solution, what would you get? What would be the same in both? The solution would be the same, the optimal x would be the same. What about the objective that comes out of it? So, the objective that comes out of 1 will be alpha times the objective that comes out of star. So, when I say equivalent, we have to be precise what we mean by equivalent. So, equivalent by broadly speaking, we mean that if you solve one, you should be able to solve the other or you should be able to recover the solution of the other. This is why what we mean by equivalent. Equivalence can mean something more specific that the solution of one is exactly the solution of the other. But sometimes it is not just that. It is not just that the solutions are the same. Through some transformation, you should be able to get from one solution to the other. So, in this case, it does happen that the solutions are actually the same. So, if you take the solution sets of both optimization problems are the same, just that the objective value is scaled. So, we are I am thinking of these as equivalent this transformation as these two optimization problems as equivalent because essentially what has happened is there is a simple way of going recovering the solution of one from the other. All right, let us look at another problem. Suppose consider this problem, you suppose you have consider a problem that looks like this. So, minimizing a function f of some phi of z where and gi subject to the constraint gi of phi of z less than equal to 0. Decision variable is z sorry and hj of phi of z equal to 0. Now, what is phi here? Phi is just let us say if my x is in here x is suppose in Rn and my phi is a function that from Rn to Rn and let us say it is 1 to 1. So, phi is some 1 to 1 mapping from Rn to Rn. So, now what do you see about what do you notice about problem 2? So, what do you notice? What can it is a function of a function? So, f has been composed with phi, all the gi's have also been composed with phi, all the hj's have been composed with phi. So, what do you notice about this? See what is going on here is if you think about this. Yes, you are searching over z's. You are searching over the search, the optimization problem is now written in the space of z. You are searching over all values of z, but the way z enters into the problem is always through phi of z. So, the way z enters into this is in terms of in the constraint is that what matters is not the value of z per se, but the value of phi of z. Similar in the in the that is there in the inequality constraints and likewise in the equality constraints, what matters is the value of phi of z. And similarly in the objective also what matters is the value of phi of z. So, what this means is effectively although the problem 2 does not have the same feasible region as problem 1. These are 2 very different space of x and space of z are different spaces. It does not have the same feasible region as 1, but you can still say or transform 2 back to 1, 2 back to star, I mean not 1 star. You can transform problem 2 back to problem star. And how do you do that? Yeah, what you what we need here is a change of variable is something like what you do in calculus integration and so on. You suppose I just denote x as phi of z, then what is happening when I change my z, I am as I as I vary my z, I am spanning over the entire all possible values of x. Why am I spanning over all possible values of x? It is because the function is 1 to 1. Every x has a z associated with it and every z has an x associated with it. Is this clear? So, because this is 1 to 1, as I range over the values of z here, I am also effectively searching also over the values of x. So, consequently I can just do this substitution, do this substitution. After this substitution, I get back to problem star. Is this clear? So, problem 2 is actually another way of writing problem 2 is simply that I am doing something like this, f tilde of z subject to g tilde of z less than equal to 0 and h tilde of z equal to 0, where f tilde is simply f composed with phi g tilde i is also g composed with phi and likewise h tilde is h composed with phi. Is this clear? So, if I had shown you a problem in this sort of form without it being made explicit that these are actually all composed on the right hand side on from the right by phi, it may not be immediately obvious to you that there is such a function, there is such a function lurking there. But this is one of the tricks to notice that is, you know, for example, changing x to minus x or changing, doing some sort of a transformation to change or your coordinates or the way the x is represented, all of these are one to one transformation. So, those can, they do not, they basically give you an equivalent problem. So, in what sense are these two equivalent? If I have a solution x here of star, if I have a solution, let us say x star is a solution of, if x star is a solution of star, then z star which is can be written, which is phi inverse of x star is a solution of 2 and likewise if z star is a solution of 2, then phi of z star is a solution of. Notice the, I meant, I want to emphasize once again that phi is one to one. Without that, this is not going to work. So, for example, if phi is not one to one, if it is many to one, so there are multiple z's that give you the same x, then what will happen? How would 2 and star be related? So, when you want to establish the relation between optimization problem, there are two things that you need to look at. One is the feasible region, second is objective. So, if well point at every value of, if I make this transformation x equal to phi of z into, what I am doing is I am just evaluating f of x at some other point. So, objective is not so much of a concern here. What happens to the feasible region? If phi is not one to one. So, which has the larger feasible region or are they not related at all? Why does someone saying here is that 2 has a larger feasible region? Why is that? There are many values of z that can map to x. Is that a correct answer? So, if phi is many to one, what is happening here? This is actually something that does not need much mathematics. You just need to notice what is going on. So, you are of course, what the way z enters into the problem is through its value under phi. What matters is phi of z. So, as z varies over this range, you get to a certain ranges of in the values of phi of z. Now, when you make this substitution and get rid of the connection between x and z, when you substitute phi of z by x and get rid of the connection that x is equal to phi of z, that connection is gone now. You have just substituted x equal to phi of z and what you are saying is you are now solving star. Now, you may be searching over x's that do not have a corresponding z. So, the search space, the feasible region of star includes all the values of phi of z that could have come from the feasible region of z, of 2. So, you take the feasible region of 2, feasible region of 2 as z's. This is in the space of z's, z that satisfy, this is the feasible region of 2. Now, look at what, but look at the, look at phi of the feasible region of 2. Look at the values of phi of z, such that z is feasible for 2. All the phi of z for which z is feasible, in which z lies in the feasible region of 2. And this is obviously contained in all the x's that x is feasible for star. So, when we search over, when we search in star, in the problem star, when we are optimizing over all the x's, we are including those x's that for which there is a corresponding z, but also those that for which there is no corresponding z. So, this will, this feasible region is larger. So, the optimal value that you will get from here will be lower than the optimal value you will get from problem 2, less than equal to the one from problem 2. So, if phi is not 1 to 1, phi is many to 1, optimal value of star is less than equal to the optimal value of. Now, this is a canonical mistake that many people make, where they make these changes of variables without checking if the function is 1 to 1. Yes, because you are now, so the optimization problem star is an optimization of all x's that lie in the feasible region of star. But the feasible region, so if you look at the phi of z's that are coming in the feasible region of 2, they are included in this. So, you are searching over a larger region in star, then you are searching over in 2. So, you will get a, and you are minimizing. So, you will get a lower value. You are looking to minimize the function over a larger region. So, you will get a, you will get in general a lower value. If you are maximizing, then the inequality will go in the opposite direction. You would get a higher value for star than you will get for 2. So, if phi is not bijective, yes, that is what I am saying. You cannot, these are not equivalent, there will be an inequality. And it is, so many things can happen here. One is that, one possibility is that by chance it can happen that the solution of star also happens to be, so your x star just happens to be such that there is a corresponding z for it. There is a corresponding z star for it, in which case you will get equality here, if that happens. It can happen that the solution of star is such that, so maybe I will list these out for you. So, x star can be such, is such that, so first simple case is x star is such that there does not exist as a z, so there is no corresponding z. In which case there is no way you can, there is nothing, there is no way you can get back to a solution of 2. All you can say is that, all you can say is that optimal value of star is less than equal to the optimal value of 2. This is the only thing you can say. Second case is suppose if x star is such that there exists a z such that phi of z is equal to x star. So, you get to a solution and it just happens that you can actually recover the right z for it. In that case, you will get inequality. These two have to be equal. Now, the optimal values are equal. Now, it can also still happen that x star is such that there is no, so it can also happen that it can also happen that the optimal values turn out to be the same. But the solution of star is such that there is no corresponding, does not correspond to the solution of 2. So, I will explain what this is before I write this in the form of an equation. So, suppose it can happen that, so let me write it this way. Third case is that you can have that this is equal to this. You solve, phi is not 1 to 1, but yet you solve star you get and you solve 2, you get the same optimal value. You solve star, you also get from it a solution say x star. You solve 2, you get a solution z star. But x star is not equal to phi of z star. Is this possible and why would this happen? Why would this happen? So, the objective values are the same, but the solutions are not in correspondence with each other under phi. So, the issue is that this is one solution x star likewise z star is one solution there may be other solutions that are in correspondence with you here. So, x star need not be the only solution there could be other solutions which you have not discovered. And in particular phi of z star this point phi of z star let us call this x hat this point is actually another solution necessarily. This is a solution. So, this sort of case can also occur. But be aware of this particular possibility because I have seen this happen multiple times where you make a substitution without checking if it is 1 to 1 and then you get an what you get is an inequality.