 So, last class we have seen that how to solve the what is called optimization problem using the KKT condition necessary and sufficient conditions. And we also seen that there is a inequality constraints is there and equality constraints are there. If inequality constraints that X of K is less than 0 for J is equal to 1 to n that is 1 to small p m J. If this constraint this 0 is part of with a small positive numbers then and also the equality constraints is this is right hand side is part of with some positive number. I is equal to 1 to there are such p equality constraint then what is the effect of objective function value of this we have studied it. In other words we have studied the sensitivity analysis after what is called post analysis we have studied it. And also we have studied the effect of cost function scaling on the Lagrangian multiplier effect of cost function scaling on the Lagrangian multiplier. And also we have studied the what is the effect of scaling on constraints that constant this are the constraints. If you what is the effect of scaling if you multiply by this p i and h i is multiplied by m p i and g i g j of X inequality constraint multiplied by m i. Then what is this effect on the Lagrangian multipliers that we have studied it. So, today I will start and what is called convex set and convex function. So, what is convex set a set is a convex suppose this is the set this set will be convex. If any point on this set if you take any point on this set this is the set s the set s is a convex set. If we take any point on this set any point on this and form a line segment then this any point on the line segment belongs to that set. Then we will call this is the convex set. So, what we can write is set s is convex if the line segment the line segment between between two points any two points between any two points lying in any two points in s in that set lies the line segment between any two points in s lies in that set in s. Then we will call that set is a convex set. So, let us call we have a this set is there we form a line segment by joining the point p 1 and p 2. So, this two points that p 1 and p 2 we join any point on this line segment belongs to that if that set is there in that set. Then we will call the set is convex set. So, in convex set we can always write it let us call X is a any point here this p is a point whose vector is X 1. This is a vector of X 2 these two points then linear combination of theta or you can say that alpha into X 1 1 minus alpha into X 2 belongs to that set. So, if you consider it is a single variable case therefore, this X 1 and X 2 there are two points are there you join with these two points agree. Then any points on this line any points on the line if it belongs to that set that is what is called as set it can be in n dimensional variables X 1 X 2. So, only the condition is if you consider this is alpha 1 and this is the alpha 2. So, alpha 1 plus alpha 2 must be equal to 1 then when alpha 1 is let us call here when alpha is 0 when alpha is 0 is nothing but a X 2 point this point when alpha is 1 is nothing but a X 1 point. So, any point alpha value any point between 0 to the alpha value is less than equal to greater than equal to 0. So, any point alpha between 0 to 1 indicates the corresponding point on the line if all the points on the line belongs to that set then this will be called as a convex set. So, we can see this one by let us call we have a simple octagon regular octagon 1 2 3 4 1 2 3 4 5 this is the I am drawing octagon like this way this is the circle agree. So, this 3 4 5 6 7 8. So, these are the octagon 1. So, if this octagon belongs to a set S let us call this belongs to the set at any point on this line any point on this octagon if you join together let us call P 1 and this is a P 2 this point is P 2 any point on this line 2 points X 1 and X 2 or P 1 point P 2 we join together and any on this line segment any point belongs to that set then it is a convex set. So, the octagon of this nature is a convex this is a convex set convex set. So, let us call this is a another set is like this way this belongs to this belongs to this is a set S set, but in this case any point on this set does not belongs to that set you see any point if this is P 1 this is P 2 if you make a line segment like this way and the any points on this segment this point this point this points are not belongs to that set. So, this is not a convex set this is not a convex set what we will call non convex set. So, in this set this is the point this is the point if you join this two point thinking I have just considering as a chord we have joined it. So, any point on this line does not belongs to that set. So, this is not a convex set is a non convex set. So, geometrically if you have a set is there you just join two points in that set and you will form a chord. If all the points on the chord belongs to that set then it is a convex set in geometrically you can say chord between two points you joined a line or line segment. If any points on this line segment belongs to that set then it is a convex set next is convex function and convex set this condition must be satisfied any point x 1 and x 2 and it is a alpha 1 alpha 1 is 1 minus alpha the alpha 1 plus alpha 2 must be equal to 0 or equal to 1. So, if you have a this you can write it now a convex function a function is set to be a function in general of n dimensional a function of n dimensional variable n variables. So, let us call this function is f of x n cross 1 variable defined that function is defined on a convex set agree that function is defined on a convex set the function is defined variable defined on a convex set s is set to be convex function is set to be convex function is set to be is set to a convex function if and only if necessary and sufficient condition for any two point any two point two points x 1 whose dimension you can say n cross 1 and x 2 any two points on that set x and set is your s on that dimension of this is x 2 dimension n cross 1 this and there is a scalar quantity theta is greater than equal to this or let us call since I have used alpha use alpha is greater than this then one can write it this one f of alpha into x 1 plus 1 minus alpha into x 2 equal to less than equal to less than equal to f alpha into f of x 1 plus 1 minus alpha f of x 2 if this condition is satisfied then we will call the function is a convex function in the convex set this condition is satisfied one can easily see this one by a single variable case that can be extended for a n variable function case. So, what is the definition of convex function a convex function n a function of n variable is set to be convex function if it in a set as if and only if this condition is satisfied and where that our if you consider the alpha is let us call alpha 1 this you consider is a alpha 2 then alpha 1 plus alpha 2 is equal to 1. So, two points now see this one this two point when alpha is 0 it indicates the x 2 points in n dimensional vector x 2 point when alpha is 1 it indicates the x 1 point in n dimensional vector alpha in between 0 to 1 any value that indicates the any value on this vector that x 1 n x 2 within this within the with this two points that vector what will generate it indicates any point on that vectors for any values of alpha from 0 to alpha from 0 to greater than 0 to 1. So, if this condition is satisfied then we will call the function is a convex function in that convex set. So, let us see with an example of this one this that is if this is a this this condition is there then the function will be set to be a convex function and it is a strictly convex strictly convex strictly convex function when this condition is satisfied alpha into x 1 1 minus alpha into x 2 this is less than equal to not equal to less than equal to previously it is a less than convex form now I am telling it is a strictly convex if this function value is less than equal to alpha into f of x 1 1 minus alpha into f of x 2 values. So, what is this just see for a single variable case what is the meaning of that one. So, suppose this is the function f of x now you consider this is a single variable case x is a one variable. So, single variable this is f of x this is the function plot. So, we take a two points here let us call p is the point whose coordinates are let us call x x is x 1 this is p 2 is there two points on the line on the curve function is x 2. So, I told you that with this two points draw a straight line which is nothing but a chord you can say it is a chord. So, this indicates you see if it is a function is convex according to the definition that any point on this line between x 1 and x 2 any point I can express like this way any point on this line let us call x any point from x x 1 to x 2 any point x is equal to I can write alpha into x 1 plus 1 minus alpha into x 2. Alpha values is greater than less than equal to 1 greater than equal to 0. So, any value of alpha from between this range the x will lies anywhere between the x 1 and x 2 see clearly when alpha is 0 is nothing but a x 2 point when alpha is the extreme limit when alpha is 1 is nothing but a x 1 point. So, alpha value in between this range it lies between x 1 and x 2. So, the any value between x 1 x 1 and x 2 the function value is that one and what is this value x any value in between x 1 x 2 is that one that expression. So, what is the function value f alpha into x 1 plus 1 minus alpha into x 2 this value this value this coordinate this is the coordinate of that one that x if you consider this is x then this is the coordinate of you can say x and y direction is that one that is the functional alpha into x 1 1 minus alpha into x 2 this value. And this value always less than any point corresponding x what is the value on the chord value of this equation value on the chord this. So, this quantity always greater than this quantity. Now, how you find out this quantity that is simple geometrical configuration one can find out the equation of the straight line once you know the equation of the straight line that any point x what is the coordinate of this one you can find out which in turn it will be it can be express like this way this quantity is less than equal to you see when it will be equal to at the extreme points of the chord. The function value and the chord value at the extreme point of p 1 and p 2 both are same that is why equal to sign other than this point this two point this chord any point on the chord function value y function value that will be always greater than the function value of f of x you see any point here this function value whatever the function value is there at the chord value at this point is greater than this one equation of this equation of this one if you know because I know the equation how you know the equation you know the the ordinate of that one change in ordinate and change in x. So, slope you can find out once you know slope and it is the function of x in order to find out the y this function equation agree. So, we can write it this one always alpha into x of 1 plus 1 minus alpha x of 2 that is f of x of 2 how you get this one this point value this point value how you get I told you find out the equation that what is the equation of this straight line then you can you find the equation of this straight line then you find out the what value of x what is the ordinate of this one. So, that you can find out and after simplifying this one you will get this one. So, if this condition is satisfied if this condition is satisfied it indicates that function is convex function and belongs to in convex set that is this one can easily verify that one by the finding out the equation of this straight line or chord equation between point 1 and point 2 and find out the value of this function value of this chord chord value that this equation value at x is equal to x 1 is nothing but that one after simplifying. So, this is our convex what is the definition of concave function definition of concave function is similar way one can find out the similar way one can find out that the two points are there let us call this is the function f of x that let us call single variable case we are explaining then we can extend for n dimensional case. So, this point is p 1 and this point is p 2 p 2 this is p 1 draw a chord between these two points this is x 1 and this is x 2. Now, you can find out that linear combination of these two points any points on this line between x 1 and x 2 any points this one I can express like this way x is equal to alpha into x 1 plus 1 minus alpha into x 2 and this is the definition of our convex set if you recollect that what we have discussed the convex set is belongs to in the convex x is belongs to this is our convex set this one with this expression and this one alpha varies from 1 to 0 once again when alpha is 0 this is x 2 when alpha is 1 it is x 1. So, any point on this line any point on this between the two points I can take the alpha values in this range greater than 0 less than 1. So, let us call what is the function value you see in this case function value at any point between x 1 and x 2 let us call x is always greater than the what is the value of this on the chord what is the value of this equation of this straight line agree what is the y value at this point it is always less than the function value f of x function value. So, one can write it for convex function in a convex set I can write it alpha into x 1 same expression 1 minus alpha into x 2 what I will write this is the function value at this point it is greater than equal to this point function value this function value and this is the this quantity is that one and this quantity I am writing alpha into f of x 1 plus 1 minus alpha into f of x 2 this one. So, convex function if this condition is satisfied then I will call the function is concave function in the convex set this one this you see from graphically also this nature of the curve is concave curve provided this condition is satisfied this means that x belongs to any convex set agree and if this condition is satisfied that function is a convex function and how one can do it in the similar manner I can do suppose f is a function I will define if f is a function which is a let us call concave function I multiplied by let us call g is equal to minus f. So, this function g function will be convex function if it is a convex multiplied by f minus it is a convex function and I know how to if I can check the g is a convex function in the set s in other words I can say f is a concave function in the set s just multiplying f by s. So, our definition then convex function is that we can write it the definition for concave function definition of concave function. So, these are the called in the beginning also we told this is the called joining these two points agree. So, definition of concave function a function f of n variables defined on a convex set set s. If it is a defined on a convex set s then any point on the set I can express any point on the on this set x I can express alpha into x on two points if you know 1 minus alpha into x 1 x 2 and that x 2 if belongs to that set then it is a convex set in the set s is set to be a concave if the function function if the function that our f if the function f is multiplied by f minus 1 that g agree is convex if minus f is a convex then I will tell f is a our concave function on that belongs to that set convex set agree. So, this next is remark let us see remark a linear function is a convex as well as concave functions just see a linear function is convex as well as concave function. Any linear function any linear function I can treat as a convex or concave functions let us call this is again it is a single variable case if you see what this is our linear equation that belongs to in a convex set. Now, whether it is a convex function or not or concave function is not you see take two points on this set let us call p 1 and p 2 join to this join this two points and it is just the same line as the original straight lines. So, naturally it may be a convex as well as concave function because equal to sign is now it is belongs it is just valid equality sign only both cases. So, you are just remember a linear function it can be a convex function or concave function I can say both this one next is your the properties of that what is called convex functions some properties we will just discuss before we discuss the optimization problem using the what is called convex optimization problems. So, the properties convexity properties convexity convexity properties. So, one thing is if S is a convex set if S is a convex set S is a our convex set and alpha belongs to our real positive quantity. Alpha is a any positive real quantity then S is our set which is a convex set this is a convex set this S is a convex set then if you multiply by this convex set by alpha this if you multiply by this convex set by alpha this also a convex set what I am telling if your S is a convex set if you multiply by that set either you enlarge or you just attenuated this by alpha alpha may be greater than one less than one agree, but positive quantity if it is so this one then alpha into S is also convex set that means if you alpha is greater than one the set is enlarge if alpha is less than this one set is contracted agree. So, this is one property what is this property if S is a convex set convex set and alpha belongs to that r plus one then alpha into S is convex set this is one another property is there if there are two convex set is there agree intersection of two convex set is also a convex set we have a two sets are there S 1 and S 2 are the two convex set. Intersection of these two convex set is also a convex set, but union of two convex set is not a convex set it is a non convex set. So, we will write it this intersection of intersection of convex sets is convex set the two sets are there let us call S 1 and S 2 that two sets intersects which one is the intersects of this which one common set is there that belongs to a that that convex set that union of two or more convex sets convex set the union of of two or more convex set is usually non convex set. That means you have a S 1 is a convex set S 2 S 1 union of this one let us call this is a convex set and this is a convex set something then you see this is the union of this one it is not necessary a convex set because you would take a one point P 1 is here P 2 is here P 2 is there join these two points any points on the line of this one must lie in the convex set, but it is this points are not belongs to that convex set. So, this is not a union of two or more convex set is not usually not a convex set whereas the intersection of two convex set the intersection of two convex set this one belongs to a that our convex set. So, this third point is this one now if the if f is a convex function f is a convex function you if you multiply it by f by a scalar quantity which is greater than positive quantity which is positive quantity then resultant is a convex function in that convex set. So, another property is there if you have a two convex functions are there two convex function f 1 and f 2 two convex function there the sum of these two convex function is also is a convex function. So, last two properties is like this way if f 1 is a convex function if you multiply it by some real positive number this f 1 the resultant is a convex function in that convex set. So, our fourth point is if f is a convex function and alpha is a real positive number then alpha into f of x is a convex function in that set. Then fifth property is if just now you mention if f 1 and f 2 are convex function convex functions on the set f 1 s then f 1 plus f 2 is also a also a convex function on s. This proof you can do simple by the definition of our convex function that our definition of convex function from that one the four and five you can easily verify this one. So, if f is a function of convex function I multiplied by a scalar positive quantity. So, that function also belongs to that convex set now you write it because f 1 f f 1 and f 2 are two convex function on that set. So, you take any two points on that set right what is the condition for f 1 to be a convex set right what is the condition for f 2 to be a convex set then add to this two things and resultant you see whether you will be able to express in this form. If you can express this form then it is a convex function of that one that can be easily proved. So, these properties are useful in studying the what is called constraint problems concentrate optimization problems. So, these properties are very useful properties are very useful useful in the study of constraint optimization problems. So, this so let us see some of the facts of convex optimization convex functions. So, let us call f is a function and belongs to c 2 this indicates c 2 means it is c can continuously twice differentiable function is continuously two means twice differentiable if I write three is a three times continuously differentiable this function. So, this is a the function f of x is continuously twice differentiable see continuously power two means twice differentiable if this then f of x is convex function convex function over a convex set over a convex set s f f of s is a convex function over a convex set s containing an interior point. So, we have a f of function is there belongs to that set the f of function belongs to that set and there is a interior point one point in the in the set interior point. If only if and only if this function is a convex set convex function if and only convex function over the set s and continuing a interior point one interior point if and only if the Hessian matrix the Hessian matrix h of f f is a function we know in our earlier lecture how to find out the Hessian matrix. That means second partial differentiation of f with respect to x the Hessian matrix if and only if the Hessian matrix is positive semi definite is positive semi definite through out s set many convex set in the convex set any point on the convex set any point on the convex set if the Hessian matrix of f Hessian matrix of f means second partial differentiation derivative of the function f this is the Hessian matrix over the all points of the set if it is a positive semi definite matrix or positive definite matrix then I will call the function is a convex function this is the this. So, let us see this proof of that one that means once again I repeat that one suppose if you have a function f of x and which is belongs to c 2 means continuously twice differentiable then function is called convex function over the convex set s containing the interior point containing an interior point containing an interior point if and only if the function if you do the second derivative that twice if you differentiate this one the function with respect to x which is a Hessian matrix and this Hessian matrix is positive semi definite over the convex set then we will call the function is a convex functions or this Hessian matrix is positive definite matrix also it will be getting the convex function of this proof. So, let us call we have since we are considering that there is a interior point is there in the set of this one let us consider by Taylor series expansion now by Taylor's theorem we have suppose we have f of y f of y I am writing y is there is a interior point only around this interior point that is a y x is interior point around this one is y. So, I am writing y I am writing as if it is x this is a delta x now this equal to by Taylor series expansion I can write nearly equal to f of x plus gradient of this function transpose of f of x at this one multiplied by incremental change y of x in delta x plus the second terms in the third terms in the second order terms of this one is half that y minus delta this is a delta x transpose into that Hessian matrix of f of x multiplied by y minus x whether x is a vector or scalar this is true this one. So, it is nearly equal to this one. So, f f of y if you bring this part in that side left hand side f of y minus this part is equal to this quantity and this quantity will be positive right hand side will be positive or negative depending upon the value of the Hessian matrix over the that our set S if you can say that Hessian matrix value over the set S is a positive semi definite matrix it indicates this quantity is a positive semi definite this quantity that means that this quantity is positive semi definite means this value of this one either it will be 0 or it will be greater than 0 this one. So, now this I can write it now if you just see this one f of y minus f of x plus gradient of this f of x into y minus x this quantity is equal to nearly equal to I can write it half y minus x transpose and I am writing the gradient of that one and gradient of this one let us call x that is a not that is Hessian matrix of this one h into y of x. So, now this side value of this quantity value right hand side value will be positive or 0 when this matrix value over the set S if it is a positive or 0 when it will be this Hessian matrix should be positive semi definite what does it mean this indicates that this positive means this indicate f of y is greater than this quantity agree. So, let us see what is this meaning of this one f of y is greater than this quantity what is this means. So, I can write it is clearly I can write from the equation that or this x for any value of x in this because I told you this is our x and this is our y any point in this I can write it alpha into x plus 1 minus alpha into y any point all this one any point any two points in this line. So, I can write it this alpha into this if you can write it this the y plus y that it is this one I can write y that it is this one I can write alpha into x minus y just see this one in the x is x is this point y is this point I am multiplied by this one some positive number alpha means since this two points belongs to that set S any point joining this two point this called any point on this line belongs to that set means it implies that one. So, y is that one then alpha into x alpha into x minus alpha into y minus alpha into y that one this point so I can write in place of anything here point of this one x will be now replaced by that quantity you can replace by that quantity any point on this curve on this set I can replace. So, this positive indicates this one positive indicates that this Hessian matrix this matrix should be either positive or zero when this Hessian matrix will be positive semi definite matrix. So, this indicates that f of y f of y this indicates that f of y clearly this indicates that f on y will be greater than equal to f of x minus plus sorry plus gradient of gradient transpose of f of x into y minus x this indicates geometrically one can see because this is the condition if you see we are writing the f of x is twice differentiable the function is continuous the function is convex function over the set continue a point x if and only Hessian matrix of s is positive semi definite matrix. So, what is this one if you see for a scalar variable case x and we have a this type of functions is there. So, this is our f of x this is f of x and y is a some point is here y this is f of y this is y and any point in this we can express by a convex set definition is alpha into x plus 1 minus alpha into y is equal to any point on this you can write it let us call any point on this one is x bar I can write x bar this one. So, now see this one what is this meaning of that one f of y function value is that quantity is always greater than what is the function value is here plus this quantity and what is this is the gradient at this point for scalar case it is nothing but a slope at this point multiplied by change in this one that means this this slope multiplied by this what you will get it that quantity this is the your y minus x. So, y minus x and this is the slope of that one slope means this quantity divided by this quantity that slope is given multiplied by this multiplied by this if you know the slope multiplied by this you will get it this quantity. So, f x minus of this quantity means this quantity indicates this is f of x plus delta f of transpose y minus x this is always greater less than equal to this is always less than equal to that one. So, therefore, if clearly we can say clearly if the Hessian matrix H of the function of the function f of x is positive semi definite everywhere positive semi definite everywhere then function then f of x is a convex function everywhere then f of x is a convex function. So, this is that and there is a important theorem is the this one that one because this is a convex optimization problem this theorem is most important one that theorem is let f of x whose dimension is n cross 1 be convex function defined this function is defined over a convex set this function convex function defined over a convex set as belongs to that as in n dimensional case then the local minimum is the global minimum then the local minimum minimum is the global minimum provided the function is convex function mind it. If the function is convex function over the convex set S then local minimum of this function will give you the global minimum of the of this function local minimum of the function local minimum is global minimum of f x over the set over the set S. So, what is the proof is very simple if you say let us call x star x star which is n cross 1 b a b a local point b a b a point of local minimum this is the local minimum point local minimum agree that means you have a this that at this point is local minimum we got it x star agree. So, we can write it now f of x star is equal to less than equal to if you take any point around this let us call x. So, f of x and since it is a local minimum point f of x is less than equal to x star plus x minus x star this is you can say delta x x this is the delta x you can say. So, it is less than this is equal to our f of x this is x. So, you can write it there is a let us call there is an any now this two points are there and this two point belongs to a convex set any point on this line I can write it this one f of x bar this by definition of this one I can write it this equal to what is x bar let us call this point is x bar agree this point is x bar. So, I can write it this x bar is equal to that x alpha into x star alpha into x star plus 1 minus alpha into 1 minus alpha into x is x bar where alpha greater than 1 less than equal to 0 this you can write it now this we can write it by definition of convex function because it is a convex function definition we can write it that alpha into f star of x plus 1 minus alpha into that what is called f of x we can write it that that one again since f x is a convex function that that one. So, this we can write it since x bar this is the x star is a optimal point minimum point x bar other than this this. So, I can if I write it in place of x star if I write it x bar this quantity x bar is greater than x star function because x star is a optimal point plus 1 minus alpha f of x. So, if you take it this is that side this quantity that side this implies after taking this side this implied f of x bar is less than equal to f of x this agree this we have shown. So, this implies and we know f of x star is also less than equal to f of x. So, since if you see since f of x since x is a any arbitrary point x is any arbitrary point hence and hence f star of x is less than equal to f of x see this one x that is what x is any arbitrary point this may be here also this side because it belongs to that set now. So, this we can write it for all x belongs to that set. So, our conclusion is that x star is a point of the global minimum agree because these two points I have taken it this one these two points belongs to that convex set and x may be any arbitrary any point. Since f x bar is less than f x agree and we have shown it the f star is less than this one it indicates the f star of x is less than any point on the global means in the set itself always this f star is less than that one. So, it is a global minimum. So, our conclusion is if f x is a convex function over a set s agree and that minimum point that means a local minimum is nothing but a global minimum of the function if f of x is a convex function then only and over the convex set. So, next class we will just discuss other points agree we will stop it here now.