 So, good morning we were looking at projections and projections general projection theorem in Hilbert spaces. So, in the last class I talked about distance of a vector or a point from a subspace in a general Hilbert space. So, we wanted to project just like we projected in three dimensions we projected a vector onto a two-dimensional surface in the same way we could project a vector in general Hilbert space which could be infinite dimensional onto a subspace which is finite dimension ok. So, this we derived a formula or we derived equations for computing the coefficients of the least square approximation and so we are looking at what is called as the normal equation. I will just go through over it very quickly. What we wanted was we have a Hilbert space and we have this subspace S. S is a subspace of X is a subspace of S and then we are given a vector u a vector u that belongs to X a vector u is given to you that belongs to X. Now X may or may not belong to the subspace S ok. I want to find out the point in the subspace which is at a shortest distance in the least square sense from u. So, what I essentially want to do is so let us say S is equal to span of some basis vectors so these are linearly independent basis vectors there are m basis vectors and so this is a finite dimensional subspace S is a finite dimensional subspace. I want to find out so any vector that belong to this subspace will be of the form alpha 1 a1 any vector that belongs to S will be a linear combination of these basis vectors ok. What I want to do is to find out alpha 1 to alpha m such that 2 norm of u minus or let us say 2 norm square I want to minimize sum of the square of errors when it is finite dimensional I want to minimize the integral I want to minimize the integral if it is infinite dimensional space I want to minimize this integral over the domain this is 2 norm square that means inner product of this vector with itself this is the error vector u is the original vector in x this is the projection this p here is the projection p here is the projection and then we want to find out a vector p that belongs to the subspace S which is at a minimum distance from so what I wanted to emphasize again and again it is not different from say this is the point ok and this is my plane here ok this is 2 dimensional subspace and this is the point the shortest distance of this point is obtained by dropping perpendicular ok. So what classical projection theorem in Hilbert spaces says is that the way to go about the way to find this vector p which is at the shortest distance is to use the fact that this error vector e that is u minus p this is perpendicular to ai i going from 1 to m ok this error between the original vector and the projection the original vector and the projection this error vector is orthogonal remember that orthogonality can be used only in the inner product space or Hilbert space and that is why we work with 2 norm or least square approximations that is why least square approximations are so popular because you can use orthogonality ideas ok so this is the basic idea with this we derived what is called as a norm equation this norm equation gave us way to compute the least square estimate of alpha 1 to alpha m and that we constructed as follows. So we got this normal equation this was a 1 a 1 inner product of a 2 a 1 a 2 a m likewise ok so using the fact that the error is orthogonal to each of the basis vectors a 1 to a m we arrived at this equation this equation is called as the normal equation in Hilbert spaces this when I solve this these inner products these are the inner products here once you compute this a 1 to a m these basis vectors are known to you you can compute the inner products ok so this will be a matrix what is the dimension of this matrix this will be a m cross m matrix there are m there are m vectors this will be m cross m matrix this m cross m matrix is invertible in fact it is symmetric positive definite invertible matrix you can check that and then you can solve for alpha 1 to alpha m you can get a least square solution what you will get here is the least square solution of alpha 1 to alpha m ok you will get the projection vector once you use this alpha 1 to alpha m you get the projection vector onto the subspace s which is at the least distance from u ok so this is the best approximation of now I began in the working on this using finite dimensional spaces I had derived a formula which said that a transpose a inverse theta so this in the finite dimensional spaces this just go back and have a look at what we have done this will reduce to the same thing will reduce to a transpose a this will reduce to theta this is this will reduce to a transpose u in the finite dimensions this equation will reduce to ok the matrix equation which we have derived earlier so I am just extending the idea from finite dimensions to a general Hilbert space here I could work with any of the see for example I began my lecture last time by saying I have this vector u t which is a plus b t and t belongs to 0 to 2 pi ok t belongs to 0 to 2 pi I want to find out an approximation p t which is alpha sin t plus beta cos t I want to find out approximation alpha sin t plus beta cos t ok let us say we can so for the time being let us take only 2 vectors what are the 2 vectors what is a1 a2 here this is my a1 vector this is my a2 vector this is my a1 vector this is my a2 vector ok how do you find out alpha and beta so what I have to do here to find out alpha beta how do I set up the problem to find out least square estimates so I have to find out sin t sin t sin t cos t what is sin t cos t or 0 to 2 pi can you do this integrals what is the inner product defined here how is the inner product defined inner product between some f and g is integral 0 to 2 pi f t g t inner product is defined here like this ok inner product is defined here like this so now we know that over 0 to 2 pi sin and cos are orthogonal so this is 0 this is 0 ok if you notice what we are actually deriving are the first two coefficients of the Fourier series I am doing Fourier series expansion of ut in terms of sin and cos ok the nice property of sin and cos over 0 to 2 pi with this as my inner product is that they are orthogonal so this is 0 this is 0 I get a diagonal matrix solving for this diagonal matrix is very very easy it is very very easy it is not so difficult to solve for so what is integral sin t sin t it is pi inner product of sin t ut and inner product of cos t ut so this is equal to pi this is equal to pi this is 0 this is 0 ok the least square estimate the best estimate of in the least square sense in terms of sin and cos of this function ut ut in this particular case I have taken a plus bt it could be it did not be you know the this kind of function I can take for example ut to be any other complex function I could take this as you know 1 plus 5t square minus 7t cube sin t let us say this is my function does not matter this is a function ok in the space of in the set of continuous functions over 0 to 2 pi it is a continuous function over 0 to 2 pi ok I want to find out best approximation to this function using sin and cos ok I can go on if here I have included only 2 basis vectors well somebody might say just using 2 is not sufficient you want to use 4 ok in which case I would use sin t cos t sin 2t cos 2t and so on ok so the approximation could become more and more complex if you want so I can have alpha 1 beta 1 plus alpha 2 sin 2t plus beta 2 cos 2t plus alpha 3 sin 3t plus beta 3 cos 3t let us say I want to develop approximation like this to this problem how will you do it we just methodically apply this formula ok now now there are 6 vectors the subspace what is the subspace span subspace span by 6 vectors sin t sin 2t sin 3t cos t cos 2t cos 3t we want to find out best approximation of the given function ok best in the least square sense ok in the subspace span by these 6 vectors what I do is I find out this inner products ok I evaluate this inner products between 6 vectors this gives me this matrix on the right hand side I have to compute inner product of the given function with each of the vectors ok and when these vectors are orthogonal actually what we recover is the Fourier series expansion the Fourier coefficients ok when these are orthogonal you know only the diagonal elements of this matrix will be nonzero ok only diagonal elements will be nonzero all the off diagonal elements will be 0 because orthogonal vectors inner products are 0 if I take an orthogonal set why do we actually want to approximate something using an orthogonal set that is because the projection problem approximation problem gets very very simple ok also there are some other advantages like actually what you are doing here is you are when you do this orthogonal projections you are expressing a vector in terms of its orthogonal components why do we work with x i y j plus z k in three dimensions because x is a component along you know unit vector in x direction y is the component along ok they are they are orthogonal they are not related to each other ok they are not related to each other the same way what we are doing here is that we are expressing a vector in terms of a basis ok we are projecting onto orthogonal basis now the complete vector you can write if you take all when you see if I take a vector x which is given by three coordinates x y z ok when will I get when will I get the entire vector if I take all the three components together in the infinite dimensional space when I when will I get entire vector if I take all the basis if I take all the basis vectors and take you know projections along each one of them ok if I take projections along each one of them then I will get the Fourier series I will just write down this whole thing it will become more clear but there is one more thing I just want to tell you see this vectors right now I have chosen them to be orthogonal I have not chosen them to be orthonormal if I choose them orthonormal what will happen suppose these vectors are orthonormal what is the property of orthonormality in a product is equal to one right and in a product between a i a j zero and with itself is equal to one what will happen to this matrix if I take orthonormal vectors this will be identity matrix if I take orthonormal vectors to find out the projection all that I have to do ok is to ok so so the concept is not different from I will just leave this and come back there so the concept is not different from see if I have this three-dimensional vector if I if I have three-dimensional vector V belongs to R3 ok how do you find out the component of V along x-axis I take inner product of V with i i is the direction so I am talking here about this is my ijk orthonormal basis for three-dimensional space ok if I take inner product of V with i what will I get I will get x component right I get x component inner product of V y component is inner product of V with j and z component is inner product of V with k right how do we write how do we write this vector how do we express this vector we express this vector as V is equal to x i plus yj plus zk where ijk let us put this cap here let us say they are unit vectors i cap j cap and k cap are unit vectors ok but this this writing is same as inner product V i i plus inner product V j I could as well write this like this right this and this is same you agree with me ok what we can do is that in a general inner product space if you are given a basis now this x is my inner product space or Hilbert space ok is a Hilbert space and I give you as a basis so this Hilbert space could be any of the Hilbert spaces that we have looked at ok so need not be need not be just finite dimension it could be any of the infinite dimensional Hilbert spaces that we have looked at and then I give you a set of orthonormal basis in this see for example I can create if it is 0 to 1 you know I can create shifted degenerate polynomials with which are orthonormal ok I can create a basis which is shifted degenerate polynomials which is orthonormal and I can use that to you know define a subspace so in general you are given a basis set you are given a basis I will call this as e 1 e 2 e n this could be a finite basis this could be an infinite basis depending upon what kind of inner product space you are looking at ok if you are looking at set of continuous functions twice integrable continuous functions then this will be infinite basis ok so so I may have finite number I may have infinite number of elements in my basis set these are orthonormal vectors what is the meaning of orthonormal vectors orthonormal vectors which means e i e j is equal to 0 if i is not equal to is not equal to j and this is equal to 1 if i equal to j if i is equal to j then this is equal to 1 these are orthonormal vectors their magnitude is 1 and they are orthogonal to each other ok so this is the orthogonal basis this orthogonal basis is same as i j k that we considered in three dimensions why do we like i j k in three dimensions it is orthogonal basis it is very nice ok I can express any vector in terms of components along each direction ok the same thing is true about any inner product space now how do I express an arbitrary vector an arbitrary vector u that belongs to the inner product space how do I express this vector in terms of these basis how do I get those components I can use the I want to project what I want to do I want to project this vector onto the space spanned by space spanned by all possible linear combinations of this ok let us take there are there is an infinite set ok so the if you start writing the normal equation what will be the matrix matrix will be i because because you know because of orthonormality this is one if it is diagonal elements will be 1 of diagonal elements will be 0 ok so by virtue of this ok you will get the coefficients if you start writing the normal equation you will get coefficients you will get coefficients i into this vector say alpha 1 alpha 2 and so on this is equal to inner product of u with even inner product of u with e2 and so on because this is i on the left hand side you have orthonormal vectors ok the inner products are between i not equal to j are 0 inner product of with itself is equal to 1 so the left hand side is all i and i which is of infinite dimension let us say if there are infinite vectors i of infinite dimension ok so these are the coefficients how did we write the vector here we wrote the vector v as v component of v along i times i times in the sense this is this is the direction i plus component of v along j times j and so on ok same thing i can do here in the inner product space i can write the vector u as inner product of u even even plus inner product of u e2 e2 plus is everyone with me on this is this clear i am just writing this vector u now i am not approximating when i take the basis when i take this even e2 e3 e4 en or e infinity whatever it is when i take this as a basis ok i am not approximating it is equality ok if you have infinite basis you will get infinite sum here so this is equal to sigma i equal to 1 to infinity u ei ok actually what i have written here is generalized Fourier series expansion of any vector u in terms of orthonormal basis in fact ok so actually actually when you write this just remember this when you are writing this you are actually writing Fourier expansion of v you are writing Fourier expansion of v ok in terms of orthonormal basis that is what you are doing i am just using the same idea in any other space probably when you studied Fourier series for the first time in your second year of engineering yeah you start wondering why i mean somebody comes and says take this function and write it as sin and cos what do i what do i gain out of it ok actually what you are doing is same thing as writing xi plus yj plus zk what you do in three dimensions it is nothing different same idea extended to any other Hilbert space any other general space what is nice about orthogonal basis you know you can look at individual components separately ok you can look at individual components separately it is very very easy to work with orthogonal vectors we know that in three dimensions that is what we want in any other three-dimensional vector space that what we want to do in any other vector space which is a Hilbert space it is possible only when you have in a product you have definition of you know angle generalized otherwise it is not possible to do this that is why Hilbert spaces that while these square approximations are so important you know in engineering or most of the applications why we look at least squares why we use two norm all the time why not one norm why not infinite norm okay because two norm comes tied up with angle orthogonality you know everything that is nice in three dimensions okay so this is generalized Fourier expansion if i if i give you if i give you a general function say a specific example of this would be the classical Fourier series which you study in second year of engineering is for the space x is for this the set of continuous functions over either minus pi to pi or we study over 0 to 2 pi right square integrable functions over minus pi to pi or 0 to 2 pi this is where you look at these expansions and then we are given this basis vectors unfortunately sin cos are not or to normal they are orthogonal okay and if you remember you get when you when you are when you are taught this Fourier series you get this you know ai is equal to 1 by pi integral remember this formula ai 1 by pi then 0 to 2 pi ft sin i t dt something like this right where does this pi come from this pi comes why does this pi come i mean i used to have this problem when i studied this why suddenly put pi there first of all remember that this is this is nothing but this is nothing but inner product of ft sin i t divided by inner product of sin i t sin i t okay this is the normalizing factor 1 by pi which comes because sin i t is not a orthonormal vector i want to get an orthonormal direction so this is this is what it is okay in this Fourier expansion just look at how will you find the coefficients see this is my Fourier expansion okay if i take if i take inner product if i take inner product of u if i take u e j what will i get u e 1 inner product of u e 1 is 0 u e 2 is 0 u e 3 is 0 right what will i get you will get u e j you will get u e j back because inner product of e j with e j is 1 right inner product of e j with e j is 1 and inner product of e j with e i i not equal to j is 0 okay so all the terms in this series will cancel only one term will remain that is projection along e j okay so this very very nicely ties up with your xi plus yj plus zk which you know from three dimensions just remember that Fourier series is nothing but nothing but extending this idea into a okay so so far nice so we have little bit diverted because we are not going to use Fourier series in this course but Fourier series will be useful in the mathematical methods course will be we will be looking at all kinds of Fourier series you know in terms of Bessel's function and in terms of some other familiar names will appear but now they should fall in a place you know you should be able to see them in light of this general development of what is projections on two orthogonal spaces yeah not subspace in r3 three dimensional subspace in r4 r3 would be a yeah perpendicular will lie on r4 perpendicular will lie outside r3 perpendicular outside r3 okay so actually you are splitting you are splitting a vector into two components one along r3 one orthogonal to r3 which will of course lie in the general space so this Fourier series is just a side note it is it is important here of course but it is more important when you develop analytical solutions okay so before we move on I know I still not coming to applications to boundary value problems or PD but before I move on I want to explain something which is very very important well I have been telling you that you know using polynomial approximations okay or polynomial interpolation high dimensional polynomial interpolation is the problem same thing is true about actually polynomial approximations if you start if you start developing a polynomial approximation which is even if you are large data set if you have a polynomial approximation of this form I have a function ft in which belongs to set of continuous functions over 0 to 1 okay and then I want to write I want to develop an approximation pt which is you know alpha 0 plus alpha 1t plus alpha 2t square alpha m t to power m I want to develop this approximation okay so how do I go about doing this what are the basis vectors now 1t t square up to t to power m okay I should set up the normal equation how should be the normal say my inner product here my inner product is defined as 0 to 1 ft my inner product is defined like this okay well there are two ways of looking at this problem I am sorry let us say I know the continuous function over the entire domain then I can approximate the other problem that we have looked at earlier is you know if you know this function at the finite number of points okay if you know this function at finite number of points let us say n points then it is the we will took this not ft we took this ut if I know u at finite number of points then I found that formula right a transpose a inverse that least square approximation formula which is which can be used to okay this is this is this is knowing the function at every point if I know ft let us say ft is some function like tq plus phi sin t or whatever okay so this is a function which I know over the entire domain everywhere and I want to do an approximation well so just to give you an insight why polynomial approximations are ill conditioned I kept on telling you that you know polynomial approximations are ill conditioned high order polynomial approximations that is why you have to do in in orthogonal collocations we do piecewise polynomial approximations we have this spline functions so spline is fitting a low order polynomial by dividing the entire region into smaller segments okay why we do all this business so that is why now I want to explain you through this okay so the question is now here a classic problem I want to develop a mth order polynomial approximation now what I know from Weierstrass theorem is that I can develop an approximation arbitrarily close to the function see this only tells you there exists a polynomial which is it doesn't tell you how to reach that polynomial how to find is a different story okay Weierstrass theorem only gives you existence now when it comes to actual computing there can be trouble unless you do some smart tricks unless you do some smart tricks okay so what are the smart tricks will come to that okay so let's first look at this problem mth order m is let's say large I want to have some 10th order polynomial fitted here okay and my ut is let's take some ut which is ut is 5t square minus minus 7t cube sin t this is the ut I want to develop a 10th order polynomial approximation of this function in the least square sense what is the meaning of these square sense that means the two norm of difference between ut and pt should be smallest okay we know that this can be found by using normal equation so we go about you know finding out this inner products so my inner product is 1 with 1 then 1 with t 1 with t2 power m right I am constructing the normal equation okay I want to estimate these square estimates of alpha 1 to alpha m okay then sorry t with 1 t with t and t with t2 power m and so on so this this times this times alpha 1 alpha 2 alpha m and then how do I get the right hand side inner product of 1 with my ut inner product of t with ut and so on inner product of t2 power m with ut and you know classical projection theorem tells us that if you solve this you will get alpha 1 to alpha m okay now let me tell you what is this integral well these integrals may not be that difficult to evaluate right because you know polynomial integral 0 to 1 okay you can actually write a general formula for this so if this matrix this matrix in my notes I have called this as h matrix this is m plus 1 cross m plus 1 matrix there are m plus 1 oh sorry there are alpha 0 alpha 1 right we start with alpha 0 alpha 0 alpha 1 not alpha 1 alpha 2 alpha 0 alpha 1 so this is a matrix which is this matrix is m plus 1 cross m plus 1 matrix is m plus 1 cross m plus 1 okay what is the element of this so ijth element of this is given by 0 to 1 t to power i plus j minus 2 dt this is equal to 1 upon i plus j minus 1 well why I have called this h will become clear soon because this what you get here is a famous matrix called Hilbert matrix okay you can show that you can show that elements of this matrix see these are inner products of t square t cube t 4 t to the power 5 so I have just given a formula for general ijth element of this you can very easily verify this you will get this okay now if I actually compute this if I actually compute this and then fill in the fill in the matrix okay the matrix that I get here is like this it's a very very nice looking matrix 1 by 2 1 by 3 1 by m 1 by 2 1 by 3 1 by 4 1 by m plus 1 1 by m 1 upon 2 m minus 1 this matrix is very very difficult to invert it's a highly ill conditioned matrix now yet we have to define what is ill conditioned matrix we have to wait a little bit for defining ill conditioning formally okay but what you get here is a matrix okay just to pre-empt what it means is that this is a basically first of all remember that this is a symmetric matrix okay this is symmetric matrix not only that it's a positive definite matrix okay this follows from the fact that this is actually you know what we have obtained in the case of projection this is a projection matrix this is obtained by you know projection matrix is obtained from projection matrix related to the projection matrix so this is actually a symmetric positive definite matrix but it is highly ill conditioned because the eigenvalues of this matrix are very very strange the ratio of the highest eigenvalue to the smallest eigenvalue okay which is what for a symmetric positive definite matrix which is what we will define ill conditioning we will see this later systematically is so large that computations become impossible very very difficult okay so Hilbert matrix more than 4 or 5 m that is third or fourth order polynomial becomes difficult to invert okay if you ask matlab to invert Hilbert matrix it will give you some junk and say that don't believe the results okay we will tell you that this matrix is ill conditioned it will say it for it will say it in a nice way not not it will not tell you don't believe the results it will say that the solution may not be reliable okay this matrix is highly ill conditioned it will give you a number r conditioning is equal to something when you don't know what what when you are not done all these theory you just tend to ignore matlab is giving something very very important message that your results could be completely unreliable now to get to get alpha 1 to alpha m I need to invert this matrix okay but if this matrix is highly ill conditioned you know the inverse is unreliable if inverse is unreliable alpha 1 to alpha alpha 0 to alpha m calculated are unreliable okay and then you are fitting you know a wrong polynomial because not because you know your formulation is wrong but you just cannot compute properly you know there is no way of computing you are stuck because this is a ill conditioned problem okay this is a ill conditioned problem and but well if you are smart enough and done this course and still remember things that I have taught you you will say well that's not a way to go I don't want an ill conditioned matrix here okay so if I want to do polynomial approximations what I will do instead is instead of using this raw polynomial like this I would choose to use orthogonal or orthonormal polynomial okay on 0 to 1 what is the orthonormal polynomial that we have constructed earlier shifted the general polynomials okay so instead of developing this approximation instead of developing this approximation see this if I want if you want an equivalent of three dimensions what you are doing here is you are trying to express a vector in terms of vectors which are not orthonormal or orthogonal see a given vector if I if I give you one vector say this one okay I can choose to express this in terms of three orthogonal components or basis need not be always orthogonal basis can be any three linearly independent vectors okay so see the basis need not be like this even basis is like this three linearly independent vectors okay but if the three linearly independent vectors are something like this which are very close to each other you may have trouble expressing this any vector in terms of these three vectors just because they are linearly independent does not mean they are convenient okay orthogonal vectors are convenient you know because you can express them in a very nice manner so what you would do instead instead of using this polynomial you will say I will express this p t as alpha 1 say l 1 t plus alpha 2 l 2 t alpha m l m t what does what are these l m these are shifted these are shifted general polynomials okay I do not have to worry here now about the order I take why I do not have to worry because when I use shifted general polynomials I am not going to get the Hilbert matrix by the way in matlab if you want to play with Hilbert matrix just use command h i l b and give into the you know into bracket the dimension you want h i l b phi will give you Hilbert matrix of phi cross phi it will generate this matrix same matrix okay and just just use inverse of that hill okay it will clip that this matrix is ill conditioned try to multiply inverse into that matrix matlab is fairly good till 12th h 12 it does a very nice job but beyond h 12 that is 12 cross 12 it starts breaking okay so amazing that even for so highly ill conditioned matrices it is able to do it now okay if I use the general polynomials what will happen to this matrix this matrix will not be a full matrix because okay let us call them instead of alpha 1 alpha 2 let us call them some other numbers you will get confused otherwise say beta 1 beta 2 some other the coefficients are different because the basis has changed the coefficients are different so now if I want to find out beta 1 beta 2 to beta m I have to take here inner products inner products will change inner products will change this will be l 1 u this will be l 2 u and this will be l m u now what about this matrix what we know is that for shifted general polynomials okay inner product of li l j is equal to 0 if i is not equal to if i is not equal to j what will happen to off diagonal elements 0 what will be the diagonal elements if i take orthonormal 1 okay this will be just 1 1 1 1 this will be a 0 here should be a 0 here I do not have problem of so if i want to develop a high order polynomial approximation okay it is easier to go through the root of orthonormal polynomials than to use the raw you know 1 t t square t cube t to the power 5 and so on is it clear why why we are obsessed with orthogonality why we are why we want orthogonality so much in every application that is because of this nice property if I take shifted general polynomials which are orthonormal this matrix which in earlier case was the Hilbert matrix of highly ill conditioned I could not invert now see this is the best matrix is no better matrix to invert than the identity matrix right so you get identity matrix okay you get the projections and now there is no problem with what order you go okay so it is not that you are not developing a higher order polynomial approximation except you just turn around change the basis you get much better solution okay you get much better solutions that is why that is why we always want to work with orthogonal polynomials now this orthogonal business will form we looked at orthogonality we looked at roots of the orthogonal polynomials in orthogonal collocations right so this in this case here if I use orthonormal set approximations are you know just taking inner products with the vector in fact what you are done here is find out the Fourier series coefficients that is it this is conceptually same same as writing a vector v as xi plus yj plus zk that is all you have done that is all you are generalized to any other dimensional space if you remember this that Fourier series nothing but xi plus yj plus zk extended to any other space that is enough okay so now you know how to make approximations not only that you know how to make good approximations you can make good approximations if you use orthogonal polynomials that is why we are concerned about generating orthonormal series orthonormal functions orthonormal basis and so on okay so the next class we will now start with the two things one is how is v square used for developing different engineering models I will briefly go over that in the beginning and then move on to you know using these methods for discretizing od boundary value problem or pd and and so on so this will lead to so called you know galerkin method or finite element methods fem you may have I am not going to do full of fem because that will consume rest of the semester I am just going to touch the tip of the iceberg and say that this is what it is rest is for you to discover okay so we continue in the next class about more applications of orthogonality