 Let us continue, we want to discuss applications of the Gram-Schmidt process. It follows immediately from the Gram-Schmidt procedure that the following result holds. So I will simply state this as a theorem and skip the proof. Every finite dimensional inner product space, every finite dimensional inner product space has an orthonormal basis. I will not write down the proof. The proof is it is a finite dimensional inner product space so it is a finite dimensional vector space so it has a basis consisting of finite many elements. Since it is a basis it is linearly independent apply Gram-Schmidt process to that to get an orthonormal set. The fact that these pans are the same proves that this is an orthonormal basis also, okay. So you can write down the proof on your own. I am more interested in applying this GS procedure to what is called as a best approximation problem, okay. Let us develop some machinery before stating this problem. So I have the following result which will be useful later. Let V be an inner product space. Let us call U as the set of all vectors U1 U2 etc Un. This is not necessarily a basis but an orthonormal set of V. Let V be inner product space and U be an orthonormal subset of V. Let us take an arbitrary element X and V, arbitrary but fixed element in V and define a vector U as a linear combination of these UJs in this particular manner. Define a vector U in this manner. For a fixed X then this vector U satisfies the following properties. Remember X is fixed. The first property is that X minus U this vector is perpendicular to span of U. The vector X minus U is perpendicular to span of U. What is the meaning? The meaning is that the inner product of X minus U with W equal to 0 for all W in span of U. It is orthogonal to every vector in the span of U. It is the first property. Property 2 is called Bessel's inequality which is the following inequality. The first one is an equation. What is norm U square? Norm U square is summation J equals 1 to n modulus of the inner product of X with UJ in the whole square and this does not exceed the norm of X. That inequality is Bessel's inequality. So there is a formula for norm of U square if you know the coefficients. If you know the numbers inner product X UJ and this number does not exceed the norm of X square. That is the second property. So norm U square less than or equal to norm X square is Bessel's inequality. Property 3. If U is a basis for V, U is already an orthonormal set. So it is an orthonormal basis. If you have that condition also then X equals summation J equals 1 to n X UJ UJ and norm X square in that case is summation J equal to 1 to n modulus X UJ the whole square. That is the view is a basis for V then X coincide with this little view that we started with. Once you realize that the other two are straightforward. This is called the Fourier expansion of X with respect to the basis that we started with. So this one is called Fourier expansion of X with respect to the basis that we started U1, U2, etc U n and this is sometimes called Perseval's identity. Let us look at a proof. First I must show that X minus U is orthogonal to span of U. U is this orthonormal set. So it is enough if I show that X minus U is orthogonal to each of these vectors. Span of U, anything in the span of U is a linear combination of these and so this is enough. So let us consider U with UK for a fixed K. We do not have the difference symbol then it is easy to understand. So U1, U2, U1 is orthonormal set. We call it U and one element we call it a small U. So if you do not have the difference symbol then it is easy to understand. I will change that everywhere. U is a single element so how can I say it is basically. So I will replace U by V. I am just renaming the set U by script B proof. So for each UI coming from that set script B, I look at the inner product of U with UK. I have the definition for U. Summation J equals 1 to n inner product X U J UJ. I take the inner product of that with UK but this is the first term goes out with goes out as it is the first scalar. Then I look at the inner product of UJ with UK. J is a running index K is fixed and K runs between 1 and n. So when J is equal to K that is the only term which will remain all the other terms are 0 because it is an orthonormal set. Script B is an orthonormal set. So this is delta JK. So this takes a value 1 when J is equal to K and in that case it is just X with UK and J equals K. This number is 1 all other terms are 0 and so what we have proved is that inner product X minus U with UK is 0 for all K and so it follows that X minus U is perpendicular to span of B okay that proves the first part okay second bezel is inequality. Just use this relationship between the norm square and the inner product in the X band. This is summation J equals 1 to n mod X UJ UJ. Second term I will use let us say R. R equals 1 to n X with UR UR. This is summation J equals 1 to n X with UJ summation J equals 1 to R equals 1 to n X with UR the whole thing goes with the conjugate I can apply the conjugate to each of the terms and then the rest is UJ UR. You can do it either way first look at this sum R equals 1 to n where J is fixed and then the summation is over J. So when J is fixed R takes a value J this will be 1 all other terms are 0. So when R is equal to J this will be X comma UJ with a conjugate summation J equals 1 to n X comma UJ into conjugate X comma UJ bar. See for the second sum R is running index and R takes a value J, J is fixed. Now this is simply complex number into its conjugate so modulus square okay so that is the formula for norm U square. So if you know the coefficients then you can compute its norm okay we have a second inequality that we need to prove norm U square is less than or equal to norm X square but remember that X minus U is perpendicular to U that is from the first part use Pythagoras theorem norm X square is equal to norm X minus U plus U square by Pythagoras theorem this will be norm X minus U square plus norm U square okay. If you look at norm X minus U square it is a non-negative number so this is I have added something non-negative so that must be greater than or equal to norm U square. So norm X square is greater than or equal to norm U square so norm X is greater than or equal to norm U that is special inequality use the fact that X minus U is perpendicular to U. U belongs to span of B by definition U belongs to span of B that is what that is why I could use use the first part X minus U perpendicular to U okay. Third is immediate once we prove that X is equal to U okay so let me I will prove that here itself I will prove that X is equal to U then it follows from what we proved in 2 that these two hold if I prove X equals U then we know the formula for U that I have written down this is a formula for U so if X is equal to U that first thing holds if X is equal to U from norm U square equal to summation modulus square this follows okay that is easy look at if let this be a basis for V then span of B is V span of B is V till now it is not a basis in 3 we are assuming it is a basis also the span of B is V now what does the first part say first part says X minus U is perpendicular to span of B so X minus U perpendicular to V which means X minus U inner product of that with V this is 0 for all V in B in particular for V equal to X minus U we get the following X minus U X minus U must be 0 but X minus X minus U is norm X minus U square okay you can even use the thing from the inner product so I will say this is 0 from the positive definiteness of the inner product it follows that X is equal to U and so the third part follows okay so if B is a basis then span B is V is what is being used this is just to develop machinery for a particular problem that I mentioned briefly the problem is the following often we consider the following situation given an inner product space and a fixed vector we seek a vector U in a subset W of V which approximates X which approximates X given a fixed X in an inner product space and a subset W among all the elements from that subset W I want to choose a particular vector U which satisfies the property that it will serve as an approximation for the X that I started with okay let me give one or two examples one example motivating example from analytical geometry suppose I am given a vector let us say 1 2 1 in R 3 I want to find I want to find a vector on this plane this is W, W is the set of all X in R 3 such that let us say X 1 minus X 2 plus X 3 is 0 for example we ask this question what is the length of the projection from a point on to a plane what is the distance of a point from a plane etc I have a point this is X this is my subset W in this case it is a subspace the point U can be thought of as an approximation to the point X 1 2 1 already belongs change the equation 1 2 1 is already there so let us say this is my equation X 1 plus X 2 plus X 3 is 0 the question is there a point U in W that will approximate X from analytical geometry of three dimensions we know that you need to just project X on to this plane that projection on to W will give the vector U that projection will give the vector U will give the vector U let us also observe in this case that the vector X minus U is perpendicular to the plane the vector X minus U is perpendicular to the plane okay remember what we proved in the first first part of the previous result so these are related X minus U is perpendicular to the plane because it is a projection okay this is one motivating example sometimes in the space of functions space of continuous function let us say C 0 1 we would be interested in approximating the exponential function in terms of just writing down as a linear combination of certain polynomials certain polynomials you look at the subspace spanned by those polynomials then I would like to write down the exponential function in that space so what is the approximation to that exponential function in that subspace this is one another question okay so let us pose this problem we need to understand this meaning approximation and then see how this problem can be solved okay see in the finite dimensional case this problem can be solved in the infinite dimensional case it is not there are examples where the problem does not have a solution okay but we will confine our attention to the finite dimensional case so let me rewrite this problem mathematically the problem is given given X and V we need to determine if possible U in W such that so it is an approximation, approximation is given in terms of norm, norm induced by an inner product given X and V to determine U such that norm of X minus U this particular U this norm should not exceed the norm of X minus W for all W this is the problem so it is not just an approximation it is a best approximation in that sense such a U is called a best approximation a best approximation to X it also depends on the subset W so it is a best approximation to X from W this will obviously change if I change W so I need to minimize this norm I need to minimize norm X minus W for all W and W to see whether there is a U in W that satisfies this condition this is called the error of the approximation norm X minus U is called error of the approximation. Now in many practical problems this norm is the two norm that is sum of squares or integrals of squares of functions so this is in many instances called the least squares approximation least squares approximation and this vector U sometimes is also referred to as a projection of X on to W there is a motivation coming from geometry the problem that we considered just now the vector U is a projection of X on to the subspace W sometimes U is called a projection of X okay that is another terminology in general as I told you in general this U may not exist it may not be unique okay but for in the finite dimensional situation the compute the existence is there the computation follows by using Gram-Schmidt process okay so let me prove that result next and go back to this problem and see how this problem can be solved by this method. So this proof is construct in the next theorem tells you how to construct the best approximation in case it is unique I have an inner product space and X is a fixed vector we need W to be a subspace let W be a subspace let U be a best approximation to the vector X from the subspace W see what this also means is that U belongs to W among all the vectors in W U is the one that minimizes norm X minus W U be a best approximation X from W then X minus U with I will use W this is equal to 0 X minus U must be perpendicular to W for all W in W the converse also holds that is if U is a vector in W that satisfies this condition then U must be a best approximation to X converse also holds okay let me just write down that is if U element of W satisfies okay that is a converse we also have the following I will call this one what we would like to have ideally if a best approximation exists then it must be unique that is always the case existence might be a problem but if it exists then it is unique in the case of an inner product in the case of a see this problem is posed as a problem involving norms the general norm linear space it may not exist but if it is a norm induced by the inner product in the general norm linear space it may not be unique but if it is a norm induced by an inner product if it exists then it is unique okay and second property second part tells you how to construct this U let this be U 1 U 2 etc U n be an orthonormal basis for W so this last part assumes that W is finite dimensional V can be infinite dimensional there is no condition on V we can be an possibly infinite dimensional inner product space but if W is finite dimensional subspace then the last part tells you the formula for the best approach then U is summation j equals 1 to n X U j U j is the unique best approximation again this should remind you of the formula for U that we defined earlier okay so if W is a finite dimensional subspace of V and if you have this orthonormal basis then the best approximation to X is given by this explicit formula okay you know X you can compute these numbers so this U can be determined maybe I will give an example we will take the same example and use this construction given in 2 and then solve the problem and then come to the proof of this result what is U that satisfies norm X minus U less than or equal to norm X minus W for all W in capital W probably we could look at the other methods of solving this problem for instance how would you solve this problem using techniques from analytical geometry of 3 dimensions do you remember yes any point alpha beta gamma then alpha then we find the equation of line X U is using direction percent 1 divided by 1 okay is a direction the issues okay that is not enough then U is perpendicular to the plane so use that and then solve this problem that is from analytical geometry can you use calculus because the minimization problem so how using the characterization X 1 plus X 2 plus X 3 0 by using Lagrange method you can find yeah you must use Lagrange multiplier method for function several variables to determine this minimum okay you try both those methods and also the method that comes from applied linear algebra okay what does this tell you this last part tells you that you must first construct an orthonormal basis for W okay let us consider a basis for W say this is the problem the problem is to determine U that minimizes this distance where X is this vector W is this subspace okay I want a basis orthonormal basis for W what is the dimension of W into 2 so I need to determine 2 vectors so let us call U 1 as 0 1 minus 1 and U 2 to be 1 0 minus 1 both these vectors belong to W and they are linearly independent one is not a multiple of the other okay I should not call it U 1 U 2 I will call it A 1 A 2 U 1 U 2 is a notation for an orthonormal basis A 1 A 2 is a basis so we need to apply Gram-Schmidt process for the first one for me U 1 is 1 by root 2 into 0 1 minus 1 again for convenience I am writing the vectors as row vectors this is U 1 and what is U 2 to determine U 2 I must use W 2 W 2 is U 2 minus sorry A 2 minus A 2 U 1 U 1 A 2 is 1 0 minus 1 minus A 2 with U 1 these two terms are 0 1 by root 2 there will be another 1 by root 2 that is 1 by 2 into U 1 0 1 minus 1 so that is 1 0 minus 1 by 2 minus 1 plus 1 by 2 minus 1 by 2 this is orthogonal to this and the norm of this so I can write this as I will take minus 1 by 2 outside then it is minus 2 1 1 so that I get 1 minus 1 by 2 minus 1 by 2 V 2 is what is the norm of this 1 by 4 2 by 3 root 2 by 3 into W 2 minus 1 by root 6 into minus 2 so this I should call it U 2 4 plus 1 plus 1 so norm is 1 this is orthogonal to this vector now determine U from the formula that is given here X is 1 2 1 right call U as summation so that is X X is 1 2 1 X comma U 1 X comma U 2 X comma U 1 this goes with minus 1 just 1 1 by root 2 into U 1 U 1 also has a 1 by root 2 U 2 goes with 1 by 6 in a product X with U 2 that is minus 2 1 1 minus 2 1 those 2 get cancelled I get just a 1 again into U 2 U 2 is minus 2 1 just check if my sign is correct minus so that is taken care of I should go with a minus 2 1 1 okay so that is minus 1 by 3 then 1 by 2 plus 1 by 6 2 by 3 minus 1 by 2 plus 1 by 6 minus 2 by 3 the last term is minus 1 by 2 plus 1 by 6 that makes it minus 1 by 2 plus 1 by 6 so this is 3 minus 1 by 3 so I can write this as 1 by 3 say minus 1 by 3 into 1 2 1 minus 2 1 this is my vector U so you please go back and check that this is the vector that you would get by the Lagrange multipliers method and analytic geometry okay. Lagrange multipliers method is conditional extremum conditional extremum minimize the distance between X and U subject to U being in a subspace that is conditional extremum okay in advance calculus you must have studied okay so let me stop here I will prove this next time do you have any questions? W is finite dimension then you must exist suppose if you take a dimension W is infinite dimension then is it possible to exist or not? No you can give examples it may not exist but if it exists it is unique in an inner product space that is in a normal linear space induced by an inner product in a general normal linear space it may not exist in a general normal space it may exist but it may not be unique.