 So in my last lecture I discussed about Hilbert matrix and approximating a function using a polynomial function mth order polynomial function and what we saw that we get this Hilbert matrix which for large order polynomial is highly ill conditioned, difficult to invert, well the problem is not different if I just passingly mentioned, yesterday we saw that we had this function ft which belong to set of continuous functions and then maybe we took some specific thing here 0 to 2 pi and then ut which is function ut which is this and then we had this approximation and then we wrote the normal equation and said that particularly when you pick this to be a polynomial that is when you pick this to be alpha 0 plus alpha 1 te plus alpha 2 e square for this particular case I showed that you end up into this Hilbert matrix and this Hilbert matrix is trouble inverting so in principle that you get an equation you cannot you cannot solve it properly because of ill conditioning. Now ill conditioning of a matrix will formally define little later, right now all that I would tell you is that the Eigen values of this matrix or singular values of this matrix to be very precise, singular values or Eigen values of A transpose A, Eigen values or singular values of this matrix the smallest one and the largest one are far apart and that causes problems and computations. Now this is when you are trying to approximate the entire function suppose you know this function instead of the knowing the function knowing the entire function what it is and trying to approximate it using a polynomial suppose you know this function values you do not know the function but that is a more practical problem in engineering you just know function values at different points so t1 this is t2 and so on so tk and in general you know value of this function at several points so these values will be u1, u2, uk, un okay u2, uk, un and then you know at each point you will write this equation that is approximation or ut, uti is equal to alpha 0 plus alpha 1 ti plus up to alpha m ti to power m you get these equations uti is nothing but ui this is nothing but ui so uti, ui okay this plus error i okay and i going from 1 to up to n you have these equations we have done this earlier okay and we get this matrix A matrix so we will get this huge A matrix so this will be 1, 1, 1, t1, t2, tn up to t1 raise to m, tn raise to m and alpha 1 alpha 0 alpha 1 alpha m is equal to u1, u2, un in this case you have this equation you are trying to fit a lower order polynomial say tenth order polynomial and this n here number of data points is let us say 1000 so you have to put a tenth order polynomial in thousand points okay you have lot you know this function value at large number of points simple example cp versus temperature okay i know cp values at many, many temperatures in some range i am trying to fit i am trying to correlate i will get something like this there t will be temperature okay in that case and then u1, u2 will be cp values and of course you have this error coming up here so this error 1, error 2, error n this vector will add here so this is my A matrix this is my A matrix okay so A matrix and we have solved this problem we know that the solution the least square solution the least square solution is given by the least square solution that is alpha least square is A transpose A or i will write it in a little different way so this is A transpose A alpha least square i think we named this as theta right in the earlier this thing we have called as theta so let us keep calling it theta so theta least square theta least square is equal to A transpose u, u is all the values of dependent variable which we know okay we know this now this as i told you is nothing but the Hilbert matrix this is nothing but the Hilbert matrix so this actually what you can show that this matrix this matrix in this normal equation will tend to Hilbert matrix this is not Hilbert matrix in the earlier case it we directly got the Hilbert matrix if you have large number of data points okay what you can show is that this matrix this matrix the proof is just given here i am not going to write it down on the board this matrix will tend to Hilbert matrix so for large values of see A transpose A this tends to this tends to 1, 1 half, 1 third, 1 by m, 1 by 2, 1 by 3, 1 by 4 okay for large number of data points here okay this particular matrix A transpose A okay you can look for the arguments here this matrix will tend to this matrix which means you know basically if you want a good approximation what will you do if you want a good approximation you will take more data points which is logical okay if you take instead of taking you know 100 data points if you take 500 data points you will get a better approximation okay you will get a better approximation but as you take more data points and try to fit the polynomial this matrix A transpose A will tend to the Hilbert matrix and Hilbert matrix is difficult to invert because this is a hill conditioned matrix because this is a hill conditioned matrix we have a problem that we cannot get reliable estimates of theta okay we will not get reliable estimates of theta and because we cannot get reliable estimates of theta approximation that you develop set 10th order or 12th order is not reliable you know is not reliable polynomial approximation a better way here would be to use orthogonal polynomials instead of using orthogonal polynomials we will make this matrix there are slightly different ways of constructing orthogonal polynomials for the discrete data case which I discussed in the notes so I am leaving it for you to read but the idea is that if you use orthogonal polynomials then this difficulty can be avoided there is something that we already have discussed is how do you how do you use the square approximation for fitting certain kind of correlations in chemical engineering we discussed about Sherwood number correlation or we discussed about Prandtl number and you know Nusselt number equal to Reynolds number raise to something and if you if you take a transformation then you can linearize that problem and then solve the problem of estimating you know constants in this correlations using linear v square so that we have already seen so we have already seen one application of linear v square in approximation okay so right now right now when I am doing Cp as a function of temperature I am trying to approximate a continuous function using a polynomial function this is one common application you have some algebraic static map and then you are trying to approximate using some other some other known functions and the problem is transformed to estimating this theta okay and we also saw in any general Hilbert space how you could construct the normal equation I am uploading extra problems today for v square and so you can start looking at the problems where you will see a lot of engineering applications where you can you will be using this v square so one of the things that I promised you is that we will now move on at some point to solving boundary value problems or partial differential equations using this method well that is what I am going to do next that is see how I can apply this method for solving a partial differential equation so now I am going to use the idea of v square for constructing a solution of a boundary value problem okay so right now I am going to restrict here to showing you two methods one is method of v square for a partial for a boundary value problem I will not show explicitly an example of partial differential equations I will also talk about a variant of it called Galerkin's method now these two methods though will be looking at them very briefly right now we are not going to look at the details of this beyond the point because these methods become very complex and what actually you know this is actually we are getting into the method of finite element okay if you have to actually implement this you have to get into method of finite element and finite element methods are fairly complex when it comes to computer implementation so it will go beyond the scope of this particular course you know I have to stop at some point problem discretization and move on into you know solution techniques so I am just going to give you very very brief introduction not really complete introduction but at least the philosophy of this problem discretization that is say boundary value problem how do you discretize this using least squares approach philosophy will be clear we will not go into the details okay so details will be I mean if I have to get into details it will take many lectures and we have to move on and do other things other than just problem discretization okay but this once we do this you know it sort of completes the picture we have three different ways of problem discretization one is problem in this particularly I am referring to boundary value problems or partial differential equations we discretize using Taylor series approximation we discretize using second is interpolation we got collocation methods okay third is this least squares okay we get this finite element method first is finite difference do not confuse finite difference method then we had orthogonal collocations then we also had orthogonal collocations on finite elements okay and this method but that basically is interpolation based method and the third one is this least squares method interpolation we want the polynomial to pass through every point okay that is the difference approximations we are trying to fit a polynomial in some least square sense we do not want the polynomial to pass through every point or the function to pass through every point okay so let us look at this method of least squares for solving so this method is called as minimum residual method and well why can you solve a problem using optimization and just to give you a little bit of a background before I move into least squares okay we will revisit what I am doing right now on the board again but I am just preempting something see we know how to solve a x equal to b right now from your undergraduate how do you solve a x equal to b Gaussian elimination the best method you know is Gaussian elimination you do you know triangularization and then solve in the reverse direction so that is one of the most efficient methods that we know can I use the ideas of optimization to solve this problem okay now let us see whether we can use the ideas of optimization to solve this problem what we wanted in optimization we want to minimize something okay and the minimum the minimum should be the solution okay now let me take this let me take this before I get into this let me let me say that I want to solve a x equal to b I want to solve a x equal to b okay I want to solve this exactly okay I could I could say that well this problem is like saying that you know let me define an error vector let me define an error vector which is a x minus b a x minus b everyone with me on this okay and then I will define an scalar objective function okay I will define a scalar objective function so my scalar objective function is going to be now well what I am going to do now is very very similar to what is happening here so this is my error vector this is a theta and this is u except in this particular case in this particular case we had you know this matrix a was tau matrix it was not a square matrix right now what I am writing there is a square matrix okay square matrix so I want to solve square matrix problem by optimization is it possible so now let me formulate an objective function phi which is e transpose e transpose e I formulate an objective function e transpose e and then what I do is which is what is it e transpose e a x minus b transpose a x minus b okay and now I say that well minimize phi with respect to x how do you get the minimum right so doh phi by doh x is equal to 0 can you solve this what will you get solve this if I use the rules of differentiation of a scalar function phi is phi a scalar function phi is a scalar function right I am differentiating a scalar with respect to a vector we have looked at the rules of differentiation of a scalar with respect to a vector if I use those rules what I will get is this equation a transpose a x is equal to a transpose b I will get a transpose a x minus a transpose b is equal to 0 this is the equation that I will get if I differentiate and set derivative of phi equal to 0 I will get this equation okay is this fine just see this you will get if you expand this you will get a transpose you will get x transpose a transpose a x then you will get a transpose x transpose into b you will get four terms you will get one term which is b transpose b b is not function of x b doh b transpose b by doh x is 0 okay so if you differentiate you will very easily arrive at this particular equation okay I just want to show that this is same as this is same as a transpose a x minus b is equal to 0 okay now if a if matrix a is non-singular if matrix a is non-singular okay when will this be equal to 0 a x minus b equal to 0 okay so actually by solving this optimization problem by solving this optimization problem you have solved you have reached the solution of a x equal to b and later at some point we are going to see a method for solving a x equal to b using optimization and iterative procedure which is faster in arriving at a solution many times then is this the optimum why is this the optimum why is this minimum if you take second derivative what will you get a transpose a transpose is always a symmetric matrix whatever is a a transpose is symmetric positive definite matrix very nice matrix okay so we have reached the minimum okay so this is so I can actually formulate a problem of solving a x equal to b as an optimization problem yeah this will be exact solution okay earlier we saw approximate solution so in the situation where in the situation where you know a is a tall matrix okay which means a is a non-square matrix it is it has more number of rows than the columns okay then you will get the least square solution when you when a is a you know square matrix invertible square matrix this will give you exact solution so look at look here this will give you exact solution if a is invertible when can you solve that problem a x equal to b when a is invertible if a is not invertible you cannot solve that problem right so now we are starting with the assumption that a is invertible okay and we want to solve the problem so this obviously gives me the solution if I if I if I do this okay so optimization could be a root to arrive at a solution of a particular problem okay what I am going to do for what I am going to do for the case of boundary value problem is more similar is similar to the case where a is tall okay so a is non-square what I am going to do for boundary value problem is qualitatively similar to a being non-square that is a has more number of rows than the columns okay it is possible to do at least formulate the problem to get an exact solution of a boundary value problem using optimization and that discussion is included here in the notes under the it is very very nice you should read this section though I am not going to do it on the board it takes lot of time and this actually forms the foundation of finite element method and finite element methods are very very useful in solving partial differential equations so I would say there are these two competitors finite difference is very easy to understand but finite difference is not so efficient because you need large number of grid points okay two methods that are competing good methods you know which balance between computational efficiency and good solutions really a good balance of these two things are orthogonal collocation collocation interpolation based methods or this finite element methods so now let us move on to boundary value problem okay now I want to consider a specific problem here boundary value problem so there is some operator L which is operating on UZ okay I suppose now you are familiar with these kind of things because you are most of you are attending the other course on mathematics mathematical methods and you have looked at these kind of problems right L operating on Z U of Z is equal to this is equal to some FZ and boundary condition one this corresponds to U0 is equal to 0 and BC2 is equal to U1 is equal to 0 so this is a classical boundary value problem that you get when you are solving a partial differential equation this will typically yield if you solve it exactly if you know the solution exactly it will yield the eigen function, eigen function expansions are you doing eigen function expansions yeah so this will yield the eigen function expansions right now my mandate is not to solve it exactly my mandate is to solve it approximately okay I am going to solve it now here I am assuming that L is a differential operator which is operating on U conceptually we are not doing anything different than ax equal to b and we have talked about it right this is a this is equivalent to operator ax equal to b we are trying to find out inverse problem classic inverse problem so right now I am not interested in the exact solution I am interested in some approximate solution to this problem and I am going to hypothesize a solution approximate solution U cap Z is equal to alpha 1 okay I am going to hypothesize a solution now what are these U1 U2 U3 these are some known functions these could be simple polynomial if you want it that way but you know polynomials is not a good choice okay this could be this could be Lichander polynomials shifter Lichander polynomials it could be sin and cos it depends upon how you choose these functions it is up to you and you have to choose these functions such that the boundary conditions are met of course so I have these functions which I have chosen here let us assume that so let us assume that we have chosen these functions for the time being okay that such that at the two ends they satisfy the boundary conditions okay we have chosen the functions such that at two boundary points we satisfy the boundary conditions okay so you have to choose the choose these functions in this particular case carefully now what I am going to do is I am going to define this residual I am going to define this residual okay the terms used are little different but the meaning is same in the square parameter estimation we call this error vector okay here we call residual vector okay residual is between left hand side and the right hand side okay so residual is between L uz minus fz this is my error vector in the case of finite dimensional least squares what did we do how did we find the minimum sum of the square of errors was minimum what was actually sum of the square of errors 2 norm square what is 2 norm square in a product of the vector with itself right in a product of vector with itself so for the finite dimensional case for the finite dimensional case we had this just keep this in the background in the finite dimensional case we wanted to obtain theta so theta least square was minimum with respect to theta you know in a product of error and error right sum of the square of errors least square estimation error vector was where error was defined as a theta u minus a theta right let's keep this in the background okay I just want to do the exactly same thing here for this particular problem now I have this I have this functions I have this functions u1 u2 u3 so what I am going to do is I am going to pose this problem okay I am going to pose this problem as minimize I am skipping the algebra which can be a little bit tedious I will write the final expressions but whatever we have done till now from that you can very easily derive what I am doing if I have to do the same thing here I will have to write this as summation of these functions and then start working with those summations and the expression will become complex but that is just an algebra the concept is exactly identical so you should not so what are the unknown here what is the theta here what is theta here alpha values alpha 1 alpha 2 alpha m right alpha 1 alpha 2 alpha m okay now what is r u zi what is r u zi well you will have to substitute one by one each that is not ready to go about okay so we want here r zi other what is r zi what is r zi r zi is l operating on summation i going from 1 to m alpha i ui ui of zi ui of zi let me let me do a small modification here let me let us call this j let us call this j and let us call this j this is this is residual at this is the residual at z equal to this is the residual at z equal to this is the residual at z equal to zi what do I need the points here I am not going to go same way as the grid points I am going to go in the I am going to go in the parameter space directly so what I am going to do here is instead of forcing the residual to be 0 at each grid point well what I am going to do here is to pose my problem as minimize with respect to theta phi theta which is equal to inner product of r r z well this is this is something different from okay you should realize this is something different from I am not taking any grid points right now I am not taking any grid points okay so what is what is this what is the inner product what is the inner product defined as inner product integral say 0 to 1 now how do I get the equations from this how do I get equations I have to differentiate this I have to differentiate this with respect to also with respect to theta right so so now what I have to do is to get the solution to get the solution I should see how many equation how many unknowns are there there are m unknowns I need m equations okay so if my phi theta is this then what is how do you get the optimum doh phi doh theta is equal to 0 okay so which actually means doh by doh alpha i of r is equal to 0 for i going from 1 to up to is this okay doh phi by doh theta equal to 0 which is same as this is my objective function error right this is my error square some of the square of errors but it is not some it is integral of error it is integral of error not at one point not at few points but over the entire domain okay there is a difference you can see here between what I am getting the way I am constructing the solution and finite difference method or orthogonal collocation method here we are setting this with respect to 0 okay if I if I solve for this if I solve for this okay what do I expect to get see what is rz here so you have to you have to actually solve this problem of doh by doh alpha i okay into what is this this is fz minus summation i going from 1 to m alpha i u cap i right what is rz rz is fz minus this right this equal to 0 okay and this equation you have to solve for i going from 1 to up to m oh yeah l of this right right operator l operating on this operator l operating on it yeah so thanks for this correction so l operating on this and l operating on this and I have to take I have to take derivative and set it equal to 0 but if l is a linear operator if we make an assumption l is a linear operator okay then this alpha i will come out and then you can differentiate and solve the problem very very easily so if l is a linear operator then what I can do is if I actually write those equations and collect the terms together okay I will get this I will get this equation which looks very very similar to the normal equation you can work out the algebra I am just skipping it it is just matter of doing it meticulously you will get this equation u1 okay you get this equation if you rearrange if you actually differentiate do a little bit of take a trouble of rearranging all the equations and then put them together you will get this equation which looks very very similar to the normal equation okay except that operator l is operating on each one of those basis functions which you have chosen okay and then if I solve this if I solve this then I will get alpha 1 alpha 2 alpha 3 alpha m I am getting an approximate solution in the least square sense okay there is one difference there is one difference here between the previous methods of discretization and this method of discretization what is that what is the difference here here these integrals are over the entire domain not at few points okay I am not evaluating this integral at few points I am evaluating this integral over the entire domain okay so the finite dimensional approximation comes because of the finite number of basis vectors we have taken the finite dimension business here comes because of not because of the grid points okay not because of the grid points here you have constructed a solution which is not see earlier case when you get the solution you must have done it now for the boundary value problem which we are looking at tram problem it is some points right finite after all you get only few points actually you know that the solution is a continuous function by this approach we are getting the solution which is a continuous function I will just take a simple example so that the ideas will become clear okay is this algebra clear is there any doubt about this see here l this residual is this residual is fz-l operating on this function okay fz-l operating on this function we are taking l to be a linear operator okay linear differential equation and then we are solving this problem of you know estimating alpha 1 alpha 2 alpha m. Now let us look at a specific problem well this particular method will work by this way because when the operator is linear if the operator is not linear what do I do we will come to that a little later but right now right now we are just looking at the basis of the method which is least squares method and the operator is linear so we get this nice so let us look at a specific problem okay that will give you better insights into well I want to solve this problem of my example is l uz this is same as dou 2u or d2u by dz square minus u equal to 1 so this is my fz I want to solve this problem I want to solve this problem this is equal to 1 okay over the entire domain this should be equal to 1 and my boundary condition 1 is u0 is equal to 0 my boundary condition 2 is u1 equal to 0 well u is not equal to 1 please note that u is not equal to 1 we are saying that this operator operating on u should be equal to 1 okay do not confuse okay I am going to take a very very simple solution to this problem this particular problem you know you must have seen in the other course that this particular problem can be solved analytically you will get analytical solution right now the motivation is to convey some concept okay so that is more important right now so let us take this approximate solution u cap z okay so my approximate solution is alpha 1 sin pi z plus alpha 2 sin 2 pi z does this satisfy the boundary conditions satisfies the boundary conditions at 0 and at 1 boundary condition is satisfied now I want to see if I get difference between finite difference method okay and this method is that if I get alpha 1 alpha 2 I can find out value or approximate value of the solution at every point z not at finite number of points you get a difference if I can get least square estimates of alpha 1 alpha 2 I get okay so now what is what is the what is the normal equation the normal equation normal equation so what is my u1 cap what is my u1 cap z is sin pi z and what is my u2 cap z sin 2 pi z right sin pi z and sin 2 pi z these are my u1 cap and u2 cap right okay what is the normal equation what is the normal what is the equation that you get well it normal like equation let me not call it normal equation because not exactly the normal equation so what is the equation that you will get here how do you get alpha 1 alpha 2 so this is inner product of l operating on u1 l operating on u1 cap inner product of l operating on u1 l operating on u2 cap l operating on u2 cap l operating on u1 cap u2 cap l operating on u2 cap this matrix okay this matrix times into alpha 1 alpha 2 this matrix time alpha 1 alpha 2 will be equal to to get alpha 1 alpha 2 what should this be equal to inner product of u1 cap 1 inner product of u2 cap 1 so what is inner product of l operating on u1 what is l so let's go back here so what I do what is the first inner product l u1 l u1 this is inner product 0 to 1 what is l u1 d by dc square of what is u1 sin pi z minus d2 by dc square d2 by dc square of sin pi z minus sin pi z this so this integral I have this integral I can do I can just it is not so difficult to do this integral you have to be little bit patient and then do these integrals okay if you work out these okay well I will write down what you get here actually just if you take this what you can show is that l u1 this is equal to minus pi square plus 1 sin pi z you can show this very very easily you can also show what is l u2 okay and then you just have to find those integrals if you find those integrals what you what do you see here is you will get pi square plus 1 whole square by 2 0 0 and 4 pi square plus 1 whole square by 2 right hand side will turn out to be minus 2 pi square plus 1 by pi and 0 if I do patiently all those four integrals okay then I get this I get this equation I have chosen orthogonal basis functions here sin pi z and sin 2 pi z are orthogonal so that helps me in this particular case that helps me in getting these to be 0 this comes out to be diagonal matrix and what happens here is that I get the final solution so what is the final solution if I if I compute alpha 1 and alpha 2 I get the final solution to be my solution comes out to be minus 4 by pi pi square plus 1 sin pi z so which means alpha 2 comes out to be 0 alpha 1 comes out to be this and this solution is pretty close to the in this particular case the true solution can be found out the true solution is the true solution is uz equal to this is the true solution this is the approximate solution to solution can be found by some different means not by this this approach yeah yeah and then force the residual to 0 so now I choose some of these functions from a standard basis no I can choose them from a standard basis like sin cos I could choose them from the general polynomials shifted general polynomials I could I could have worked out this problem using shifted general polynomials if I have those shifted general polynomials which satisfy the two boundary conditions in this in this particular case sin and sin functions are convenient because they satisfy the two boundary conditions they are not be orthogonal but in this particular case orthogonality always helps you always work with orthogonal functions okay we try to work with orthogonal functions as far as possible orthogonality is but this solution there is one difference between this other solution other solution is exact at known points exact means it is not exact it exactly satisfies the differential equation at known points here this solution is obeying the differential equation in the least square sense okay at every point it is not equal to 0 but if you take the least square sense it is 0 okay integral of the square of error is 0 error is not equal to 0 okay integral of the residual square is 0 okay even in the other case we are forcing residual to 0 only at finite point what happens to the residual at the point which are in between they will not be 0 right so we are only forcing at a finite number of points even here you know you do not know which points exactly the residual will be 0 but the solution will be crossing you know 0 at many points so this is both approximate solution this is one way of getting approximate solution and you have other way of getting approximate solution which is what we have seen earlier okay now the problem is that in every everywhere where you are solving this your L will not be in general you know it will not be linear there could be a nonlinear differential equation we are solving this Tram problem right it is not a linear operator okay so what do I do what will I get what will I what how will I solve that problem so in that case you know I just use the concept of orthogonality between the subspace and error the concept that we had earlier and we will get the method of you know Galerkin method so I will discuss it briefly in my next class but the next class after doing that half an hour of Galerkin method something about a Galerkin method I want to move on to solving problems because we are only two classes away from the end same and then complete same and we will solve some problems of v square so that you get better insight into what is happening but the key thing here is that right now I have not used polynomial approximation have you general functional approximation the solution is obtained by posing the problem as a v square problem okay and then you are getting a solution approximate solution of the boundary value problem using v square support okay it is possible to extend this to partial differential equations we will have to compute double integrals in x and y and so on okay it is possible to do that in algebra will become complex the concept is not different