 So, today we are going to start looking at another grand generalization from school geometry or three dimensional geometry, this is inner product spaces or Hilbert spaces. As I said the nice property of three dimensional world or the geometry that we study, equivalent geometry that we study is orthogonality between two vectors. If you have perpendicular vectors, we can define models very conveniently, we can define vectors very conveniently. So, there are many, many advantages of Pythagoras theorem and we would like it to hold in a general space which consist of functions, which consist of polynomials and so on. So, we have to come up with new structure on a vector space which is equivalent to what you have available in three dimensions and then try to see to it that the properties that are important in three dimensions or in school geometry are also preserved in these newly defined vector spaces. So, now we impose one more structure, see we till now we started by defining norms, but just length or norm was not enough, it was helpful in defining generalizing the ideas of convergence, convergence to a limit and so on, but we need something more, we need angles. So, I remember two things, what I want to generalize, I want to generalize the so called dot product, if my x is nothing but r 3, then if I am given any two vectors x and y that belong to x, then what I do is I construct unit vector as x by two norm of x and y cap which is also unit vector y by two norm of y, so this is two norm, x two norm is x transpose x raise to half okay, we construct two vectors which are unit vectors in direction of x and y and then if I want to find out angle between x and y, I have to take a dot product between x and y, so a fundamental result that we have is cos theta, cos theta is equal to x cap transpose y cap, a fundamental result that I have is x cap transpose y cap is equal to cos theta angle between them, I want this particular idea to be generalized in a vector space, now what we know from trigonometry is that cos theta is cos theta is always bounded between plus and minus one okay, so another way of stating this in this equality is to say that mod cos theta is less than or equal to one or this also means that in three dimensions x divided by norm x, this is another way of writing the same inequality cos theta is always less than one, so x transpose, so I can see this is a scalar and this is also scalar right, so I can write this as mod of x transpose y is always less than or equal to right and then what was important property of that we said we want to have is orthogonality, so when x is perpendicular to y okay, when x is perpendicular to y, x transpose y is equal to 0, this was very very important for us, this was very very important for us, we extensively used orthogonality, we use orthogonal basis for example, the most well known orthogonal basis is i, j, k you know unit vectors perpendicular along the coordinate direction, so orthogonality is a very very important property, we want it okay, so I want to now come up with generalization of these results in a vector space, that is my aim, so I am going to define a new entity called inner product, I am given a vector space x together with a function called inner product, so I am given a vector space x and a function which is defined on x cross x okay, x cross x to the field f, so given x y that belong to x, I am going to define inner product x inner product y, this is the notation that we are going to use throughout the course, x inner product y is defined from x cross x to, so what are the axioms that govern this definition, there are four axioms, the certain properties which are generic to inner product in three dimensions, which I want to now generalize and come up with a generalized definition which is which will in a special case would be this dot product which you know in three dimensions, so my first axiom is well I may in when I am working with vector spaces in many situation I have to work with complex valued functions or complex valued vectors, so what this says is that if I change the order, if I take inner product of x with y okay, then and if I change the order what I get is the complex conjugate okay, so typically the field that we are going to work with is r or c set of complex numbers, so well if you are working with real valued vectors or real valued functions, then this is very obvious if I change the order, if I make x transpose or y or y transpose x, I am going to get the same value okay, with complex numbers you get a complex conjugate that is that is important, the second property I want the inner product to observe is that if I am given any three vectors any x, y and z that belong to x, so this inner product that we define should distribute over vector addition, so if I take x plus y and take inner product with z then that is same as adding these two inner products okay, x with z and y with z okay, that is the second important property of a function to qualify as a inner product okay, so what is the third important property, the third important property is that if I take a scalar lambda, I take a scalar lambda and multiply it with x, then this is same as lambda bar, this is same as but the way it happens with the second element and the first element is different in the inner product, well if you are working with real numbers these both results are same because lambda bar is equal to lambda okay, so if you are working with complex numbers you need to separate these two equalities, so if I take the first vector multiplied by lambda then it is same thing as multiplying inner product of x and y with lambda bar that is complex conjugate of lambda okay and if I take the second vector multiplied by, this is y if I take and multiply by lambda and it is same as okay lambda times, so this is another essential property or this is an axiom that defines a function to be inner product, I want to maintain this because we have to generalize a few things, so I want to draw parallels, so let this be there for some time, what is the last axiom, so the last axiom is, the fourth axiom is if I take inner product with okay, let us look at here, let us look at this property, if I take inner product of a vector x with itself what do I get, I get two norm in three dimensions, what is two norm, two norm is if I f x1, x2, x3 are three components x1 where plus x2 square plus x3 square whole to the power half okay, so this particular property is quite important in light of generalizing this, so this should always be greater than or equal to 0 and the inner product of x with itself should be 0 only if x is 0, this is also very very important here right in three dimensions only inner product of x will be equal to 0, x transpose x will be 0 only if it is origin, the same property is being generalized here okay, in fact this is what helps us in defining a norm which is tied up with a inner product okay, a norm which is tied up with a inner product, a norm which is tied up with a inner product plays very very important role in numerical analysis because this is a norm which comes up, which comes with a definition of angle okay, that is why two norm is something which is very very often used in applied mathematics okay, so now let us start looking at can we define there in three dimensions, we defined a norm using inner product, can I do it here in a general vector space okay, so I said any function that obeys the four axioms qualifies to be an inner product, so it is not necessary that we have only one particular way of defining inner product, we have a generic way of coming up with a definition of inner product that is suitable to our application okay, what I mean by this will become clear as we go along okay. So let me define some examples of inner products which are even on R3 I will show you that there are different ways of defining inner products on three dimensions, but before that let me just state what is a Hilbert space okay, sometime back in the last lecture we you heard about Banach spaces, what are Banach spaces, complete norm linear spaces, so what happens in complete spaces, every Cauchy sequence is convergent in the space right, so Hilbert space a complete inner product space, a complete inner product space is called as a Hilbert space, it is this is this name is given after a great mathematician Hilbert who led foundations of functional analysis okay, so now some examples of inner products, my first example is to show that there is no unique way of defining inner product in three dimensions also okay, let me take my x as R3 okay, now well when you say R3 the field is R I am not going to write it every time just to keep the things simple, I am going to define a inner product on this now which is different from what we have done earlier you know just x transpose x okay, I am given so let W be a positive definite matrix, so now we define a inner product using this positive definite matrix W, so my inner product for any two vectors x y that belong to R3, so I have this x and y belong to R3 and x inner product y I am going to give a little subscript here W okay is going to be defined as x transpose W y, a simplest example of W would be a diagonal matrix okay, see for example I can simplest example of W would be you know matrix which is 100.1 and 1000 okay, now you might ask me why do you want to define some funny matrix W and then call it as a inner product where is it useful, let us say I am working with a reactor okay and my x, x is a vector that consist of say temperature, pressure and concentration fractional okay, so this is in 10s and 20s let us say degree centigrade temperature pressure let us say is you know defined in Pascal's mega Pascal's, so it is in 10 to the power 5 something here you know x is in fractions okay, if I use my old good old way of defining inner product or length I have trouble because you know fraction this mole fraction is always going to be a small number square of it is going to be smaller number okay, so many times I need to work with scaled variables okay, I need to work with scaled variables at such times it is useful to have inner product definition which normalizes the unit differences between different variables okay, so I am not just defining this W matrix just like that there is a purpose behind this and then there are many situations where you will hit into this kind of normalization business where you have to use a matrix W, now let us see whether this particular inner product satisfies properties that are specified what is the first property, just go back and look at your, so first thing is first property is my first property is that xy should be yx bar but we are not working with complex numbers, so since we are not working with complex numbers in this particular case it is if I interchange it should not matter right, so this I do not have to prove to you that x transpose W y is same as y transpose W x, why this is true, is this always true I need one more property I have missed out something here, no I just said it is a positive definite matrix I need something more, no yes, so I need this matrix W to be positive definite and also this W has to be symmetric, W is equal to W transpose this should be symmetric otherwise this does not hold okay, otherwise this does not hold, so this matrix being positive definite is not sufficient it should be a symmetric positive definite matrix okay, then what will happen if I take x transpose W y transpose that is y transpose W transpose x which is y transpose W x because W transpose is equal to W okay, so symmetry is very very important okay, so I need a positive definite matrix and a symmetric matrix okay, what next lambda times x what will happen, what is the next property I think x plus y the second property distribution is very obvious you do not have to prove this x plus y z is equal to x plus y transpose W z which is nothing but x z plus y z I think this is just it just follows very simple what is the third thing if I multiply one of the vectors by a scalar inner product should get multiplied by now we do not have to do mod here there is no bar here we just have to take the scalar out because we are working with real numbers, so the third property is very very obvious that is lambda times x y is lambda x transpose W y which is lambda x transpose W y okay, I do not take complex conjugate because we are working with real numbers, what about the fourth property does it hold x transpose W x if I take inner product of a vector with itself what is the meaning of positive definiteness all the eigenvalues are greater than 0 there is no 0 eigenvalue all the eigenvalues are greater than 0 okay, the definition of positive definiteness itself means this is the definition of positive definiteness okay, a matrix is positive definiteness the fundamental definition of positive definiteness is that if x transpose W x is always greater than 0 if x is not equal to 0 if x is equal to 0 it will be equal to 0, so only vector that will give you x transpose W x equal to 0 is 0 vector that is what we wanted right all four axioms are satisfied, so this is another way of defining inner product on three dimensions these kind of inner product we very very routinely use in numerical methods because we need to do scaling of variables, x will consist of pressures, temperatures, concentrations all kinds of variables which have different units and then if you want to find out length of such a vector you cannot just say x1 square plus x2 square plus x3 square you need to multiplied by a suitable weighting matrix that is why you need this, so yeah that is why I said that W has to be positive definiteness and symmetric, symmetric is important, x transpose W y is symmetric no, so just positive definiteness is not enough we need symmetry also so symmetric positive definiteness matrix is important and then yeah lambda bar is complex conjugate but we are working right now with real numbers, so complex conjugate will be the real number itself there is no okay, so this I can very easily change to my second example where you talk with Rn I could have talked with Rn and the same thing would hold okay I have symmetric positive definiteness matrix okay and then I can define a norm which is I can define an inner product which is using any symmetric positive definiteness matrix which is n cross n which will give me all the properties that I need for defining inner product okay we still have not established the connection between the last axiom and the norm I have been just taking I am just saying that well it is related to the inner product gives you a norm which is here but actually we need to see that connection okay we need to see that connection, so I will give one or two examples and move to proving that actually inner product in a general space defines a norm just like in three dimensions x transpose x gives you a norm okay you will also get a norm defined through inner product okay before doing that let me give you one or two more examples of inner product spaces so my second example would be Rn or I can easily move to Cn a complex valued n tuple and so on where the matrix there should be hermitian not symmetric positive should be hermitian okay but moving on from finite dimensional spaces let me give you third example so set of square integrable functions over an interval ab set of square integrable functions over an interval ab you have come across this kind of a set when you worked with Fourier series expansions okay now you will soon realize what are the connections so if I am given any two functions say ft and gt that belong to x then I can define a inner product between ft and gt as integral a to b set of all square integrable functions okay so integral over a to b typically when you study Fourier series in your undergraduate we look at ab that corresponds to 0 to 2 pi or we look at ab that correspond to minus pi to pi remember something like this when you do Fourier series expansion you take sin theta or sin t into ft dt integrals that is actually in a product okay and you can just check whether all four axioms are satisfied let us look at first axiom what is the first axiom if I interchange f and g will the integral be different so first axiom is satisfied if I take if I multiply ft by some lambda what will happen to the integral it would be lambda times right second property is satisfied what about distribution if I take f plus g inner product with some ht very obvious right the third property if I take ft plus gt inner product with ht this will be integral a to b ft plus gt which is same as integral a to b everyone with me on this everyone with me on this so the third third axiom is satisfied what about the fourth axiom if I take inner product of a function f with itself what will happen will always be a positive number why so my fourth my fourth axiom is integral of ft with ft this is nothing but integral a to b ft square dt okay which is always greater than 0 okay if ft is not a 0 function if ft has even one nonzero value in interval a to b ft square will be positive ft square dt will be positive so ft as long as this will be 0 this will be 0 when f is 0 everywhere on a b if f has nonzero values this integral the summation this integral will always be nonzero okay so all the four properties that you need for an inner product space or inner product to be defined are satisfied okay I could further modify this inner product see just like from x transpose x from x transpose x I said x transpose wx where w is a symmetric positive definite matrix is also inner product I could expand this definition by putting a positive weighting function here so I can have another definition my fourth example would be my fourth example would be I will take a weighting function wt ft gt dt okay wt is strictly greater than 0 on wt is strictly greater than 0 is a positive function wt is a positive function it has only positive values in the interval a b this is my interval a b on which inner product is defined on which the space is defined okay just like we could use a positive definite symmetric matrix there if I modify my definition of inner product by multiplying by a positive weighting function that also satisfies inner product and these kind of weighting functions we are going to hit upon soon in when we come up with different ways of defining inner product on set of continuous functions which are square integrable we will also come up with these kind of inner products you will need them when you solve partial differential equations so boundary value problems when you solve in the mathematical methods course okay so these there are different ways of defining inner products yet we have to establish two major connections one is with the angle and other is with the norm okay so let me start preparing for this I need to prove an inequality I need to prove an inequality which is essence which exactly captures this part in order to show that a inner product defines a norm I need to prove an inequality called as Cauchy-Schwarz inequality and this inequality will help us to come up with connection between inner product and the so called two norm okay this is two norm and we want a connection to be established in a general okay so what all things that you need for a function to be a norm when you call a function to be a norm what are the three axioms okay so one is norm x is greater than 0 if x is not equal to 0 and this is equal to 0 if x is equal to 0 okay that is the first axiom okay what is my candidate norm definition my candidate norm definition is I want to use in a inner product space I want to define a norm actually we will call it two norm but right now okay let us keep calling it two norm so two norm here is I want to say x x raise to half that is what I want to do x transpose x okay this is my candidate function now does this follow the first axiom for norm does it follow from the definition of the very definition it will follow nothing to worry okay what is the second thing about norm scalar multiplication so if I take alpha times x then that is equal to mod alpha norm x what about this does it follow okay let me see alpha x alpha x what is this equal to alpha alpha bar alpha x x right first element alpha bar second element alpha okay so which is mod alpha square x x right so alpha x alpha x raise to half is equal to mod alpha x x raise to half you have proved whatever we wanted so far so good okay now comes the third problem what is the third thing triangle inequality okay now triangle inequality is where we need this to be generalized okay you cannot go to triangle inequality unless you generalize this result in inner product spaces and here we need a little bit of work okay I am going to prove this on the board why this defines a inner product why this inner product defines a norm and how you can generalize this result this result in inner product space is called as Cauchy-Schwarz inequality so what is it that I want to do I want to generalize this particular result from three dimensions okay except here it is written x transpose y I want to prove I want to arrive at mod of x y is less than or equal to norm x if I take inner product of any two vectors x y its absolute value is less than okay this is what I want to show in any inner product space okay that is generalization of the result cos theta is always less than 1 okay and then with that I will move to triangle inequality because I have to establish triangle inequality to come up with okay so how do I do that now let us first look at a situation where y is equal to 0 vector if y is equal to 0 vector does this hold always yeah because inner product with 0 will give you 0 0 is less than or equal to 0 no problem okay so if y is equal to 0 so we do not want to look at a trivial case 0 vector case okay now to prove this inequality now I am going to play a trick okay so let lambda be a scalar nonzero scalar such that I am going to take a vector x-lambda y and take inner product of x-lambda y with itself everyone with me on this lambda is any arbitrary scalar okay so does this hold for any nonzero lambda this inequality holds for y inner product of a vector with itself is always greater than or equal to 0 okay so this always holds holds for any lambda not equal to 0 or any lambda for that matter y not equal to any lambda it is holds fine it holds for any lambda so what is this quantity on the left hand side can you expand this so this will be x inner product x I am taking first with first then x inner product lambda x y-lambda bar y x look carefully lambda times lambda and x distribution I am using the distribution property okay plus everyone with me on this I will just expand it the right hand side okay this also of course has to be greater than 0 this is x inner product x always greater than 0 mod lambda square y inner product y always greater than 0 now I have two quantities in between okay x and y and y inner product x okay so let us preserve this part here because this is what we are generalizing okay so this holds for any lambda am I correct that inequality which we proved there holds for any lambda so I am going to pick one specific lambda now okay I am going to pick one specific lambda inner product is a scalar ratio of two scalars okay this is why is not 0 so why is not 0 so since why is not 0 this is a positive number okay and this lambda is a valid lambda so this should hold for this lambda also okay for this particular lambda what is lambda bar is that right I just use the first property okay now I am going to substitute this lambda and this lambda bar in the inequality that we developed earlier okay so using this using this lambda and lambda bar I have 0 greater than x inner product x okay before that let us let us do a little bit of work so this implies that minus lambda x inner product y minus lambda bar y inner product x this is equal to if I just substitute this if I just substitute this lambda and lambda bar okay then what I will get is that this is nothing but two times x y yx just check what is lambda yx I am substituting in the first thing there xy right and what is lambda bar xy but there it comes yx right just algebraic jugglery okay this is equal to two times well minus is here of course minus sign minus sign will persist so this is equal to minus two times x inner product y x inner product y bar right y inner product y okay I will move on to here now is everyone with me on that okay so this is equal to minus two mod okay minus two mod now so this quantity here this quantity here this quantity here can be now replaced by our new value okay so I get 0 greater than x minus 2 our lambda is if I substitute for lambda square where lambda square would be if I substitute for lambda square it would be this and then finally the inequality that I get is I finally get an equality which is 0 greater than x x minus okay so this minus this is always positive and I am just doing algebraic jugglery now there is nothing specific if you are not followed right now just go through meticulously through the the notes you will see the steps just substitutions okay I have just eliminated by see this is a scalar so this square this will cancel with this square and then you can do the jugglery it is not so difficult okay so what does this imply this implies that the above thing implies that mod of x y is less than or equal to x x raise to half y y raise to half this is I take this on the left hand side and take the square root see this is this is square of this in a product of x and y right I take this I take this quantity on left hand side because see this is this is greater than this right otherwise this cannot be greater than 0 right and then I am just doing multiplication yy I have bought it on this side I have just omitted one in between step everyone clear about this no problems okay so now so what is this what is this this inequality is same as this results in three dimensions which we know no difference you know x dot product y mod of that is always less than which is nothing but cos theta less than 1 okay so so I have proved an equality which is called as Cauchy Schwarz inequality I have proved inequality called Cauchy Schwarz inequality and this helps us to prove that triangle inequality how will you prove triangle inequality now what is triangle inequality so triangle inequality should be norm so we want to prove we want to prove x plus y 2 or x plus y is less than or equal to norm x plus norm y we want to prove this inequality finally and I want to use this I want to use this result this is this is Cauchy Schwarz inequality this is generalization of this result okay well once I once I declare x transpose x to be norm of x I can actually even move to this inequality because this is a scalar I can divide take it inside and so on we will move to that little later the next class will will start from this inequality Cauchy Schwarz inequality and move on to proving triangle inequality once we prove triangle inequality we are done okay once we prove triangle inequality we have shown that inner product defines a norm three axioms of norm two of them we already proved the third one was triangle inequality to prove triangle inequality we need Cauchy Schwarz inequality but Cauchy Schwarz inequality not only helps you to prove triangle inequality it also gives you a way of generalizing definition of angle okay it will also give you a way of generalizing so we will be able to define orthogonal vectors in any inner product space these vectors could be two functions like sin and cos okay or this vectors could be two polynomials we will talk about orthogonal polynomials why do we talk about orthogonal polynomials why do we talk about orthogonal functions they are very very useful when you do mathematics applied mathematics but why were they called orthogonal why were they called orthonormal or whatever okay so that those questions will get answered if you understand this basis that is why I am doing all this this proofs okay so let us in the next class we will move on to triangle inequality and then more properties of inner product spaces we will see that the famous Pythagoras theorem which we study in your eighth grade also holds in any of these inner product spaces what a relief you can work with orthogonal vectors okay.