 Okay so let me emphasize what I said yesterday, the notion of inner products, inner product spaces more generally normally in a spaces this has relevance to practical notions like approximation and convergence these will be made mathematically precise a little later okay so we are only trying to develop the background material for that okay. Let us also recall Cauchy-Schwarz inequality I want to give two examples Cauchy-Schwarz inequality in an inner product space okay model is inner product X, Y this does not exceed the product of the norm of X and the norm of Y this is true for all X, Y the right hand side norm comes from the inner product okay so let us just remember that again. What is this inequality say with regard to the three inner product spaces we have seen before in particular we have the following look at C n for instance I will call it one summation J equals 1 to n I am using X i, Y i bar and then I take the modulus this is the inner product of two vectors in C n this does not exceed norm of X into norm of Y norm of X is summation J equals 1 to n mod X i square to the 1 by 2 into norm Y a similar expression summation J equals 1 to n mod Y j square to the 1 by 2 okay this is one particular case look at what happens to the inner product space C n cross n with the trace inner product trace of A B star modulus of that this is less than or equal to norm A norm B norm A is trace of A A star to the half norm B similar expression trace of B B star to the half finally if you look at the infinite dimensional example infinite dimensional inner product space C 0 1 we have the following modulus integral 0 to 1 in a product X 1 f of t g t bar d t modulus of this does not exceed the product of norm f norm g what is norm f? Norm f is integral 0 to 1 mod f t square d t to the 1 by 2 the second factor is 0 to 1 mod g t square d t to the 1 by 2 okay inequalities are important when you discuss notions of approximation convergence etc so you will encounter these if not in this course some other course so these are specific instances of the Cauchy Schwarz inequality okay we have seen yesterday that the notion of a norm can be introduced for a vector in an inner product space more generally we have the following that is a norm need not be induced through an inner product one can have a general normal linear space a norm on a vector space V is a function it will be denoted by these two parallel lines I am sure you must have encountered this it is already there in inner product spaces it is a function from V to R unlike the inner product which can be a complex number so this is a function from V to R such that the following conditions are satisfied norm of X is greater than or equal to 0 for all X in V and this goes along with this norm X equal to 0 if and only if X is equal to 0 condition 2 is again a condition that we have seen in the context of an inner product space norm of lambda X is mod lambda norm X for all lambda in let us restrict let us look at the case of a complex vector space so I will take the scalars from C for all X in V second condition third condition is just a triangle inequality norm X plus Y must be less than or equal to norm X plus norm Y so a norm on a vector space is a function that satisfies these conditions a vector space together with this norm with a given norm is called a normed linear space normed vector space a normed linear space or a normed vector space vector space are also called linear spaces a long linear space is a pair it is a pair V, some norm where this is a given norm on V okay and so every inner product space is a normed linear space every inner product space is an example of a normed linear space it is a subclass it is a normed linear space with respect to the induced norm just to recall norm X is the positive square root of the inner product of X with itself what is also important is to observe that on a given vector space you can define several norms on a given vector space you can define several norms and what can be shown is that not all norms are induced by an inner product okay now let me give you at least two different norms on CN for instance these will also serve as examples of normed linear spaces consider V to be CN with the two norms defined as follows I will define two norms with respect to which CN becomes a normed linear space one is called as the one norm sometimes called the absolute value norm so it is it goes with the subscript one norm X goes with one this is summation J equals 1 to n mod X J where the usual convention is that X 1 X 2 etc X n are the coordinates of X this is called the one norm this is one norm and yeah all that I am saying is that this satisfies you can verify that this satisfies these conditions okay so CN with this norm is a normed linear space there is also called the so called supremum norm or sometimes a maximum norm this is equal to the supremum of actually it is maximum there are only finitely many numbers here maximum of mod X I 1 less than or equal to I less than or equal to n maximum of the moduli of the coordinates of the vector X this is called the infinite norm or supremum norm maximum norm CN is a normed linear space with respect to both these norms okay CN already has a norm with respect to the standard inner product induced by the standard inner product okay in numerical linear algebra one would like to ask questions like whether these norms are equivalent okay we will not deal with those but it is in that context you would like to know whether norms are equivalent if norms are equivalent then see a normed linear space can be shown to be a metric space D X Y equals norm X minus Y then with respect to this metric we can ask questions about convergence then the question is if it is if a sequence X n is convergent with respect to one norm should it be convergent with respect to another norm and this is related to the question as to whether two given norms are equivalent okay that is why it is of interest to know different norms on the same space and different norms have different different norms are suitable for different applications for example when we do calculus it is the it is a stand it is a norm induced by the standard inner product it is called it is called the two norm or the Euclidean norm okay let me write that also it is called the Euclidean norm or the two norm that is from the standard inner product so can you tell me what the two norm is J equals 1 to n so norm X square okay that is mod X J square this is called the Euclidean norm or the two norm in calculus it is a two norm which is important whereas in you know robot trajectory planning etc it is a infinite norm that is used okay so different applications ask for different norms the question however is we need to go back to this question the two norm is induced by the standard inner product what happens to these two the claim is that these two are not induced by any inner product okay these two are not induced by any inner product how do you prove it in order to prove it the following result is useful it is called the parallelogram law which holds in a vector space so let me state and prove that and then I will leave it for you to verify that these two norms are not induced by inner product by any inner product okay parallelogram law let me state that here and prove it there in an inner product space let V be an inner product space then we have the following this the rule that I am going to write is motivated by what we have seen in two dimensions even three dimensions norm of X plus Y the whole square plus norm X minus Y the whole square you can think of X and Y as two dimensional vectors on the plane vectors on the plane then if then X plus Y is the length of one of the diagonals X minus Y is the length of the other diagonal the sum of the squares of the diagonals must be two sides two times the sum of the square of the sides two times norm X square plus two times norm Y square for all X Y this law holds this is the parallelogram law in inner product space this holds where the norm is of course the norm induced by the inner product okay so if V is an inner product and this is the norm induced by the inner product parallelogram law holds if I have a norm linear space where the parallelogram law does not hold then it cannot be the norm cannot be induced by any inner product that is what you should use to prove that these two are not induced by any inner product you have to take sample vectors X and Y calculate these numbers and verify that this law does not hold for these two norms okay I am going to leave that as an exercise these two are not induced by any inner product that is an exercise but let me prove the parallelogram law rather straight forward you simply look at see this is induced by an inner product so you need to use that look at norm X plus Y the whole square plus norm X minus Y the whole square this is inner product of X plus Y with itself plus inner product X minus Y with itself just expand and simplify X with X is norm X square Y goes with Y for norm Y square and you have a Y X and an X Y the second term gives you norm X square plus norm Y square minus Y with X minus X with Y so you get the right hand side this is two times norm X square plus two times norm Y square okay so that proves the parallelogram law straight forward but it is still powerful in showing that certain norms are not induced by any inner product. So the exercise for you is so that the one norm and the infinite norm are not induced by any inner product the context is CN you also have similar results for the space of continuous functions over 0 1 so space of continuous functions over 0 1 there is a two norm induced by the inner product which I have given there but there are other norms that can be defined on C 0 1 so let me also mention on C 0 1 I will define two norms similar to the one norm and the infinite norm on C 0 1 norm F the one norm is any guesses about what this is F is a continuous function over 0 1 this is similar to the one norm integral mod F integral 0 to 1 mod Fp dt F is continuous mod F is continuous so the integral like this similarly the infinite norm what is the infinite norm supremum of modulus of F of t in 0 1 the supremum make this because F is continuous mod F is continuous composition of two continuous functions so there is a maximum and a minimum I want the maximum so in fact I can replace supremum by the maximum so with respect to these two norms 1 and infinity C 0 1 is a norm linear space it can again be shown using the parallelogram law that these are not induced by any inner product okay let us move on these are some of the basic notions. One of the motivations for an inner product space is that it should allow us to generalize the notions of the usual dot product the notion of angle between vectors in particular orthogonality okay let us look at these notions so in particular I want to look at the concept of an orthogonal basis and orthonormal basis I will simply say orthonormal sets okay notion of orthogonality so it is done through the inner product so this definition is natural let V be an inner product space take two vectors X and Y then X is said to be perpendicular to Y perpendicular to Y or X is said to be orthogonal to Y if the inner product of X with Y in this fashion is 0 okay if X if the inner product of X with Y taken in this manner is 0 then the inner product of Y with X Y first X next that is also 0 because of the conjugate symmetry okay. So then we can say that X and Y are orthogonal you can say that X and Y are orthogonal to each other okay for a subset A contained in V is called orthogonal a set is called an orthogonal set if distinct elements are orthogonal distinct vectors distinct elements in A are mutually orthogonal if distinct elements in A are mutually orthogonal 0 vector is the only vector that is orthogonal to itself that is if A, A is equal to 0 then A is 0 that comes from the positive definiteness of the inner product orthogonal we need something more A is called orthonormal if A is orthogonal and each vector in A has norm 1 each vector in A has norm 1. So such a set is called an orthonormal set that is for every A B in A we must have the inner product of A B is 0 if A is not equal to B it is 1 if A is equal to B. So we write like this distinct vectors are orthogonal and each vector has norm 1 so such a set is called an orthonormal set do you have examples of orthonormal sets look at the vectors that belong to the standard basis but before that I will give another example consider the following vectors u1 is 1 minus 1 u2 is 11 these two vectors form an orthogonal set not orthonormal these two form an orthogonal set not orthonormal because they do not have norm 1 norm of u1 or u2 is in fact 1 by root 2 sorry just root 2 u1, u2 both have norm root 2. On the other hand if you look at the standard basis vectors the standard basis vectors in CN are orthonormal in fact just to emphasize what the standard basis is look at E1, E2, etc EN where EI is 0 0 etc 1 0 0 0 where this occurs in the ith coordinate this is an orthonormal set this is this probably the simplest orthonormal set one would encounter I want to explain a procedure the question is the following given a linearly independent set can we get can we construct an orthonormal set out of it okay the answer is yes but before that we must understand that orthonormal vector orthogonal vectors are linearly independent okay but even before that I want to prove Pythagoras theorem then I will come to this Pythagoras theorem which holds we have seen in the plane holds in a general normally general inner product space so I want to prove this and then look at the process of constructing orthonormal vectors from an independent set linearly independent set Pythagoras theorem setting is an inner product space so if X Y belong to an inner product space such that X is perpendicular to Y then the inner right triangle there is a hypotenuse there are other two sides look at the square of the lens of the other two sides that sum is equal to the length of the hypotenuse norm X plus Y square is norm X square plus norm Y square okay just to recall if this is 90 this is X and this is Y these are the lens okay I think I should use alpha beta numbers then this is alpha square plus beta square alpha and beta are the lengths norm X norm Y are the lengths of the sides this holds in a general inner product space I will leave the proof okay you have to as before start with norm X plus Y whole square use the inner product and one line the proof okay so the high school notion of Pythagoras theorem you see holds in an abstract inner product space okay I told you that orthogonal vectors are linearly independent let me prove that let us recall the following let V be an inner product space and let me take this as a basis I am considering a finite dimensional inner product space B equals let us call the vectors U1 U2 etc Un let this be a basis of V okay so it is a finite dimensional inner product space given as before it is an ordered basis okay it is an ordered basis that is U1 is the first vector U2 is the second vector etc Un is the last vector so that when we write down the matrix of a linear transformation or the matrix of a vector we know what is the first component second component etc so this is an ordered basis it means that any X and V can be written as ordered basis see when you write down the matrix of a vector then it is always done with respect to a basis that is this X there is a representation this X can be written given this basis this X has the following unique representation alpha 1 U1 plus alpha 2 U2 etc plus alpha n Un where the numbers alpha 1 alpha etc alpha n are unique for this X okay and we always deal with the standard basis for the reason that we will have occasion to talk about the first coordinate of X second coordinate of X etc when we do matrix operations so it is natural to call alpha 1 as the first coordinate of X alpha 2 as the second coordinate of X etc do you remember this that we used to write the matrix of X relative to this basis then that is a column vector coming from the first term coefficient of the first term coefficient of the second term etc now what is to be understood is that this sum does not change if you alter the first and the second term for instance but when you write down the matrix of the vector corresponding to the basis it does make a difference okay so we will always have in mind that this is an ordered basis so there is a first coordinate second coordinate etc so this is an ordered basis I have this representation as I told you these numbers alpha 1 etc alpha n are unique for the particular X that we started with okay how do you compute these numbers given a vector X do you remember how we compute these numbers alpha 1 etc alpha n in a general vector space see u 1 u 2 etc they do not form an orthogonal basis orthonormal basis they form just a basis ordinary ordered basis so how do you find these numbers solving a system you can write this as okay see X is given I need to find these numbers so what we do is look at the matrix whose columns are u 1 u 2 etc u n and then I want to determine the numbers alpha 1 alpha 2 etc this left hand side is given I know what X is I want to determine the numbers alpha 1 the coefficients of X relative to this basis I know what this is this is also given the basis is of course given I need to determine this this is unknown so this is essentially solving a linear system of equation so you need to do elementary row operations and then determine the unknowns from the system of equations okay now that we know that is that takes a little effort in the case of a so to determine the coefficients of a vector X you need to solve a system of linear equations but if this is not just a basis but an orthonormal basis then this step is very easy that is advantage of an orthonormal set an orthonormal basis by the way what is an orthonormal basis a basis which has the property that the vectors are mutually orthogonal and have each have norm 1 is an orthonormal basis if this is an orthonormal basis so in addition if B is an orthonormal basis then this computation immediate there is no computation involved it is immediate then we have the following okay I will go back to this equation alpha 1 u 1 plus alpha 2 u 2 etc alpha n u n I take the inner product of X with u i I runs from 1 to n then this is alpha 1 u 1 u i plus etc alpha i u i u i etc plus alpha n u i okay how do I write the terms u 1 u 2 is the first I am taking on the right so u n u i since this is an orthonormal basis all terms cancel except this 1 all these are 0 this is 1 and so this is alpha i and so the coefficient alpha i is determined as inner product of X with u i so the coefficients can be computed by multiplication dot product by the dot product immediately but the price you have to pay is the computation of an orthonormal basis from a linearly independent set it is just a basis ordinary basis it is a linearly independent set there is some effort involved in going from a linearly independent set to an orthonormal basis there is a there is a there is a name process Gram-Schmidt procedure numerically it can be modified but we will simply look at the Gram-Schmidt procedure that tells us how to go from a linearly independent set to an orthonormal set so once you do that certain computations become easier okay I told you that orthogonal vectors are linearly independent can you see that happening here immediately in general orthogonal if X is 0 that is if I take a linear combination of the vectors u 1 etc u n equate that to 0 then is it clear that you come back see that the coefficients must be 0 so an orthogonal set is linearly independent not conversely any orthogonal set there is no orthonormality that we are using here any orthogonal set is linearly independent any orthogonal set is linearly independent but not conversely that the converse is not true has been exhibited already you look at those vectors 1 minus 1 1 1 I am sorry they are linearly they are orthogonal okay you give an example okay that is easy linearly independent vector 1 comma 1 1 comma 2 they are linearly independent but not orthogonal okay so this means we need to look at the procedure that takes a linearly independent set to an orthonormal basis this is called the Gram-Schmidt procedure let me discuss that next. So we have what is called as the Gram-Schmidt orthonormalization process okay what is this process I will state that as a result let u 1 u 2 etc be a linearly independent set in an inner product space v so I start with a linearly independent set in inner product space then I can construct then we can construct an orthonormal set we can construct an orthonormal set I will denote that by v 1 v 2 etc remember this can be an infinite set so you can apply this to an infinite dimensional space C 0 1 for instance we can construct an orthonormal set v 1 v 2 etc which satisfies the following such that see for one thing it is an orthonormal set they are mutually orthogonal and the norm of each vector is 1 there is another thing it satisfies such that the following holds look at the span of u 1 u 2 etc u j for any j you can show that this span is the same as the span of v 1 v 2 etc v j for each j step by step that is look at the span of u 1 that is the same as the span of v 1 span of u 1, u 2 is equal to span of v 1, v 2 etc for every j these two subspaces of v are the same okay the proof I will complete the proof today the proof is by induction the proof is by induction on j okay to apply the induction principle you need a basis step and then an inductive step okay basis step take the case of j equal to 1 j equals 1 I have the vector u 1 I must show how to construct v 1 such that span of u 1 equals span of v 1 okay but remember that we start with a linearly independent set so this u 1 cannot be 0 any vector that contains a 0 vector is linearly dependent so none of these vectors is 0 u 1 is not 0 so I can divide by norm u 1 so I will call v 1 as the vector u 1 by norm u 1 norm u 1 is not 0 because u 1 is not 0 then this v 1 satisfies the requirements for one thing norm v 1 is 1 and you do not have to take another vector to take the dot product etc this is the basis step there is only one vector this also clear is that span of u 1 is span of v 1 that is because v 1 is a multiple of u 1 anything that is in the span of u 1 is a multiple of u 1 that is obviously a multiple of v 1 so these two subspaces coincide so the basis step holds so we apply the inductive suppose that suppose that v 1 v 2 etc v n have been constructed such that such that span of okay such that for one thing this is orthonormal such that okay I want to write that again suppose this is orthonormal and this condition also must hold span of u 1 etc u n is equal to span of v 1 v 2 etc v n so you assume that you are able to construct n vectors then you must show that you can do it for n plus 1 vectors then by the induction principle it follows that this can be done forever indefinitely okay I need to give a formula for v n plus 1 then we are done okay given v 1 etc v n I must tell how to construct a v n plus 1 that is done as follows consider the vector it is a new vector that I will define I have n vectors v 1 etc v n I define a new vector w n plus 1 as let us take u n plus 1 the one that we started with and then subtract the following sum j equals 1 to n take the inner product of u n plus 1 with each of the vectors that we have constructed each of the vectors v 1 etc v n that we have constructed v j and then take the dot product of that with v j there is a geometric significance to this but this can be explained only a little later okay remember u 1 u 2 etc that infinite set is given to us so I know what u n plus 1 are I have computed v 1 up to v n only those I am using here so I delete this from the vector u n plus 1 the first observation is that this is not w n plus 1 is not the 0 vector can you see that if you can see that then I can skip that step w n plus 1 is not the 0 vector how do you prove as usual by contradiction if w n plus 1 is 0 then this vector is 0 so what is the contradiction if w n plus 1 is 0 then yes u n plus 1 can be written as this sum it is in the linear span of v 1 v 2 etc v n but v 1 v 2 etc v n the span of that is equal to this which means u n plus 1 is a linear combination of these contradiction to the fact that we started with this is a linearly independent set so no vector can be written as a linear combination of the previous vectors so this cannot be written as 0 so w n plus 1 is not 0 it makes sense to talk about norm w 1 and then divide that by divide a vector by that so I will do something that is similar to the first step call v n plus 1 as w n plus 1 by norm w n plus 1 this is well defined because denominator is not 0 the claim is that this v n plus 1 obviously has norm 1 but the claim is this is orthogonal to the vectors v 1 v 2 etc v n okay then we are through almost for one thing norm v n plus 1 is 1 how are they orthogonal how is v n plus 1 orthogonal to the previous n vectors that follows from this formula simply look at v j comma v n plus 1 okay look at v j comma w n plus 1 v j comma w n plus 1 so this is I am doing it for the first argument so using this formula it is v j comma u n plus 1 minus summation j equals 1 to n that is the first argument so this will go with a complex conjugate so can you see that this is what we have summation j equals 1 to n the complex conjugate u n plus 1 v j oh this is okay I need to change this the summation index is j I will call this l so this is l this is j this with respect to this so that is v l v j is that okay the summation index is j I do it for all l l is fixed l is fixed l runs between 1 and n l is fixed l runs between 1 and n I am looking at the inner product w n plus 1 with v l v l in the first argument so v l with u n plus 1 v l with this now this will go out with a complex conjugate v l comma v j is that clear j is a running index l is fixed this is 0 if j is different from l so all terms are gone except the term corresponding to j equals l when j is equal to l this is 1 this becomes u n plus 1 v l with a conjugate that is v l u n plus 1 that gets cancelled with this so this is 0 okay so v l is orthogonal to w n plus 1 for all l so how do I choose l this is true for all l such that 1 less than or equal to l less than or equal to n so w n plus 1 the new vector is orthogonal to v l alright but since v n plus 1 is just a multiple of w n plus 1 it is also orthogonal to the vectors v 1 etcetera v n. So v 1 v 2 etcetera v n together with v n plus 1 this is an orthonormal set the last point is to verify that the span of this is equal to the span of u 1 etcetera u n plus 1 okay just consider that I will prove one inclusion the other one is similar look at span of v 1 v 2 etcetera v n v n plus 1 this span is contained in I will keep the first n vectors v 1 v 2 etcetera v n and observe that v n is a multiple of w n plus 1 so instead of v n plus 1 I can use w n plus 1 okay but when I write w n plus 1 I observe just go back to the formula w n plus 1 is a linear combination of u n plus 1 and the other v 1 etcetera but that is a linear combination of u 1 etcetera which means w n plus 1 is a linear combination of u 1 u 2 etcetera u n u n plus 1 agreed yes v j u n plus 1 for j equals l because all terms are gone except j is equal to l minus no real number it is a complex number see you take the conjugate but then inner product x comma y bar is y x so this becomes v l u n plus 1 this is v l u n plus 1 so it gets cancelled okay is this step clear v n see this step is obvious I am sure because v n plus 1 is a multiple of w n plus 1 on the other hand w n plus 1 okay you tell me if this is clear what I am trying to explain is this is a linear combination of v 1 v 2 I am again writing v 1 v 2 etcetera v n u n plus 1 do you agree that is because w n plus 1 is with regard to you can write it in terms of v 1 v 2 etcetera v n and together with that you append u n plus 1 so this is fine but v 1 v 2 etcetera v n span of these vectors is equal to span of u 1 etcetera so this is again contained in span of u 1 u 2 etcetera u n u n plus 1 okay so this is one inclusion I want to show that the span of these two sets are the same this is one inclusion the other inclusion is similar you can simply retrace the steps you can simply retrace the steps okay that completes the proof. We will look at some examples next time and also applications of the Gram-Schmidt process in a certain optimization problem okay so let me stop here.