 Today I will be talking about what is probably the most beautiful and most profound part of optimization which is called duality. This is a subject that arises specifically in optimization due to the very way by which we go about deriving solutions of optimization problems. I will explain what this duality means. So, suppose we have a linear program like this. Today I am going to limit to linear programming duality. So, suppose we have a linear program which is written in standard form like this. The decision variable here is x. Now corresponding to this linear program, I will write another linear program which is this one. So, the first linear program is minimizing c transpose x subject to a x equal to b and x greater than equal to 0. This one is in standard form. The second linear program is maximizing b transpose y subject to a transpose y less than equal to c subject to a transpose y less than this is not in standard form, but it has been written in this I have written this particular problem for specific reason and I will explain what that reason is. So, you have a LP here on the left-hand side which is a minimization LP. The LP on the right-hand side is a maximization LP. Now what we the constants involved here that means the b that is there in this the objective of this is the same as the b that is in the right-hand side here. This we are talking of the same b. The c that is in the objective here is the same as the c that is in the right-hand side here. The a matrix here is the same as the a matrix here. Only thing it has been transposed and written in the constraint. So, these problems are involved the same set of constants, but arranged in a certain specific way. You will also notice that here there is a equality constraint because this was in a standard form whereas, this has an inequality constraint. Here the x is greater than equal to 0, but here the y is unconstrained. There is no constraint on the side. So, the LP on the right-hand side this is specifically crafted. It is very specifically crafted way with to have this kind of structure and we will soon see what the connection is with the one on the left. The one on the left we will call it the primal LP and one on the right would be called the dual. Now here is a quick observation you can make. Suppose I gave you any point that is feasible for the primal. So, let us write out some notation here. Let us write this Fp as the feasible region of the primal. So, it is x such that Ax equals b and x is greater than equal to 0 and let us write Fd as the feasible region of the dual. y such that A transpose y is less than equal to c. Now can someone tell me what are the spaces that these sets lie in? So, what is the dimension of x and what is the dimension of y? So, suppose my matrix A is an m cross n matrix and what is the dimension of x and what is the dimension of y? So, dimension of x is x is an n length vector and y is a m length vector. So, now, so these are not these two optimization problems the primal and the dual they are not all in the same space at all. I mean one is in Rn the other is in Rm one has n decision variables the other has m decision variables. Yet what is amazing is that they are very closely related. So, I will show you this. So, look at let us make the first observation. So, suppose I take a x suppose x belongs to the set Fp and y belongs to the set Fd. Now look at the value c transpose x or look at if I look at if I consider c transpose x or this is the inner product between c and x. x itself is greater than equal to 0 and x satisfies A x equals b that right x is greater than equal to 0 and x satisfies A x equals b. On the other hand, let us look at this value b transpose y b transpose y is the is the value attained by a by a feasible solution y of the LP on the right. So, y just satisfies A transpose y less than equal to c. Now, I know that if I know that look at if I form since y status y belongs to Fd. I know that c is greater than equal to A transpose y for this particular y. So, I know that c is actually greater than equal to A transpose y. Now, what I can do is this is a full vector right c is a vector and y A transpose y is also vector every component of that vector if I if I multiply by a number that is non negative the the inequality in that for that particular component will be resolved right and then I can add up all those inequalities and get a get a get a scalar inequality using all of those. So, what I am doing what I am going to do now is just take an inner product with x x I know is a vector that is non negative x lies here lies in Fp x is a lies in Fp it is a vector that is non negative. So, what I I can do is I can take an inner product with x. So, which means that would give me x transpose c is greater than equal to x transpose A transpose y right I am talking I am referring to this specific x and this specific y that I have told an x in Fp and and a y in Fp. So, I just multiplied x on both side and the inequality so to be in a product with x and the inequality is preserved because every component of x is greater than equal to 0 right. So, I so from from here I have from from this equation I was able to go to this equation no problem. But now what look at the right hand side right hand side is actually the same as A x the whole transpose y and let me write the left hand side also better let us write the left hand side as since it is just an inner product let me write it as c transpose x. So, I have basically c transpose x greater than equal to A x the whole transpose y now I make my other observation well again my x belongs to Fp x belongs to Fp which means A x is actually equal to b A x is actually equal to b since x belongs to Fp. So, what this means is c transpose x is greater than equal to b transpose y. Now notice what has happened here you have you started off with any point x in Fp and for that what I was able to show is I took this value c transpose x and I and I was able to show that c transpose x c transpose x is actually greater than equal to b transpose y. I started with any x in Fp and any y in Fd and I got this inequality that c transpose x is always greater than equal to b transpose y. It does not matter what my choice is every x that is feasible for the primal and every y that is feasible for the dual must satisfy that c transpose x is greater than equal to b transpose y. Now this let us take this one step further observe that the optimal the optimization problem p the primal optimization problem primal optimization problem is actually looking to minimize the looking for the minimum value of c transpose x. So, if the optimal value exists that means it is not minus infinity if the optimal value exists then the minimum value of the optimization problem is also going to be greater than equal to b transpose y for every y. So, the minimum value of the primal assuming this exists assuming it exists is greater than equal to b transpose y and this is true again for all y in Fd but then what is what let us compare that with the dual problem the dual problem is looking to maximize b transpose y. Now since this is true for every y in Fd it is also true for the y that gives the maximum possible value of the dual. So, therefore the minimum value in the primal is greater than equal to the maximum value of the dual. In short the optimal values of these two linear programs are related in this fundamental way that the optimal value of the primal cannot go below the optimal value of the dual the primal is looking to get the least possible value of a certain function of x. The dual is looking to get the maximum possible value of a certain function of y but they are they have this kind of tension between them. You cannot bring the primal below the optimal value of the dual and you cannot raise the dual any higher than the optimal value of the primal. Now this property is what is called write this in red it is what is called weak duality. Weak duality simply refers to this that for any that c transpose x is greater than equal to b transpose y for all x in Fp and for all y in Fp. Now the remember the thing that I want that I mentioned at the start this these are you might be first tempted to think that you know there is some this is actually somehow is some sort of simple operation in the sense that say for example you might be tempted to think like when you are doing minimum minimizing c transpose x it is like trying to come to the bottom of a function and maximizing c to b transpose y you are trying to get to the maximum of it is as if the same function has been flipped. It is not a flip of this it is not like you are taking a reflection of the objective of one to get to the other. These are two problems written on two separate spaces you are not flipping one to get the other there the there is a very there is a specific way in which these they have been crafted which gets you to weak duality. But at the same time it is really beautiful that you can actually say something like this because it tells you that if there is that you cannot bull down the value of an optimization problem beyond a certain level if it is a minimization problem likewise you cannot raise the value of the optimization problem if it is a maximization problem beyond a certain level. In fact it gives you this the following following results straight away let me just state this either primal or dual LP is as an unbounded optimal solution as an unbounded optimal value if either primal or dual has an unbounded optimal value means that the value since we are looking to minimize the primal that means primal value is minus primal optimal is value is minus infinity or since and we are looking to maximize the dual that means or dual optimal value is plus infinity. So, either of if either of these two is true if either the primal is unbounded or the dual is unbounded then the other must be infeasible. So, what this means what this says is that if the optimal value of the primal is minus infinity then you cannot have even it cannot be that the dual has a feasible solution. Why is that well what the reason for that is just by weak duality if the primal value is minus infinity that means the minimum here is minus infinity then you would get that minus infinity is greater than equal to the maximum value of the dual. Now in particular it would be greater than equal to the value of the dual for any y for any y in F D. So, but a finite value of y would give you a finite value of B transpose y there is no way that that can be less than equal to minus infinity. So, the contradiction is that there cannot be a single y that satisfies the constraints of the dual which means the dual has to be infeasible. So, if the primal is unbounded then the dual must be infeasible. Likewise, if the dual is unbounded so dual takes value plus infinity then there cannot be a finite value of x which satisfies the constraints of the primal because then you would get that you have plus infinity here on the right hand side and something here which is finite but greater than equal to the plus infinity. So, that is also impossible. So, in short if any of them is unbounded take so primal taking value minus infinity or dual taking value plus infinity then the other must be infeasible. So, this also gives you a way of testing if your problem is going to have minus infinity is going to have an unbounded optimal value because it amounts to checking if a bunch of other linear inequalities or equations can be satisfied. So, if just giving you the just as soon as I give you this LP here the primal LP I can I need to I can check if this is going to have a minus infinity as the solution. Well, one if this if this if a transpose y is less than a less than equal to c is satisfied by some y then it means that the dual is not not infeasible then there is no way that this one can have minus infinity as a solution likewise for the dual. So, this theorem is what is called the first property is what is called the property of weak duality and this here the theorem that I just wrote is just simply a consequence of weak duality. Now, but the theory of duality does not end here because there is an even more powerful result that is out there and that result actually says that these two these two problems if they admit solutions then their optimal values are actually equal. Weak duality simply says that the optimal value of the primal is greater than equal to the optimal value of the dual. It just only guarantees you a an inequality here but a stronger property is two which is which simply which says that if both are finite then they are both equal. So, so that is what is called that that property is what is called the property of strong duality strong duality and the strong duality theorem is this if if either primal or has a finite optimal value then so does the other and these values are equal. So, the if the primal has a finite optimal value then the dual must also have an finite optimal value and the two values should be should be the same. Likewise if the dual has a finite optimal value then so does the primal and its optimal value is the same as that of the dual. So, if one and from the earlier theorem we know that if it does not have a finite optimal value means it has if it is unbounded then the other one must be in position. So, now what we will do today is is prove the strong duality theorem. So, that is the agenda for today. The strong I will prove the strong duality theorem and I will also give you some an application of the strong duality theorem. Now, the one of the strong duality theorem actually rests on one key property of convex sets and the property is that way very is very easy to see but it is not that easy to prove. So, suppose I have a convex set here C suppose C is let us say C is closed closed and convex and suppose I take a point here X that lies outside C. A simple observation is that I can always draw I can always find a hyperplane like this a hyperplane that separates the point X from C what do I mean by separate separates means that the point X should lie on one side of the hyperplane and the set C in entirety should lie on the other side. So, my hyperplane suppose is so let me call this point something else let us call this X star my hyperplane is suppose X such that A transpose X equals B then if then this by separation of the set from of the set C from X star what we mean is well A transpose X star is less than equal to B say it lies on below the hyperplane and A transpose Z is greater than suppose greater than equal to B for all Z in C the entire hyper entire set C lies on the other side. Now, the way I have written this is this form of separation is where I have allowed both these inequalities to be weak which means both these both the point it is possible for the point X star to lie on the hyperplane it is possible for the set C to also touch the hyperplane touch but not cross the hyperplane right it can so you can have equality in both this is called weak separation this property is what is called weak separation but you can also talk of much stronger versions of separation and I will I will we will use a stronger version of separation in the stronger version both these inequalities will become strict you will so so it will be like the kind of picture I have drawn here your point X star lies on the other in one half space but not on the hyperplane and the set C also lies in the other half space and does not touch the hyperplane this is what is called these theorem that guarantee you something like this what are called separating hyperplane theorems separating they are called separating hyperplane theorems and are one of the key tools for proving proving bounds in optimization so or optimization and several other fields also where if you want to show that something is impossible try to show one of the key ways of useful tools in that is through separating hyperplane ok now what is the role of convexity in a separating hyperplane theorem you can see that by just looking for looking at another case suppose this set C was not not convex like this but say suppose shaped like this and my I had a point X star of this out here now is it possible for me to separate X star from the set C by a hyperplane no it is just not possible no matter how much you try you you try to you will never be able to find a hyperplane that separates it you can find some other curve that separates the two but not a hyperplane right this is the beauty of convexity convexity guarantees you this ok now so now there are other points that may be separable like for example a point X star that lies here this may it may be possible to separate from the set C ok but it is not possible for every point outside the set right so in a when you have when your set is set is convex you can set it can be separated from every point that lies close than convex it lie it can be separated from every point that lies in the exterior