 In the previous talk we talked about QR decomposition for the over-determined case. QR decomposition can also be applied for the under-determined case. I am going to quickly point out some of the key steps. Let again z is equal to h of x, h is m by n, m is less than n, m is the number of observation, n is the number of unknowns. I have more unknowns than the number of observations. In this case please remember I have done the QR decomposition of h m by n when m is greater than n. We have already done with that. In this case I have h m by n, here m is less than n. If I took the transpose of this h that is n by n, n is greater than m. So these 2 cases are the same if you interchange m and n. Therefore QR decomposition of h for the over-determined system is the same as the QR decomposition of h transpose for the under-determined system. That is a quick mathematical enterprise that you can utilize. If you utilize this you can see the QR decomposition for the lower triangles for the under-determined case and the over-determined case are not too different from each other. So I can compute h transpose is equal to QR as above. Since in this case n is greater than m by interchanging the role of m and n I can apply everything that we saw thus far. Q again is Q1 and Q2, R is again R1 and R1, R2. R1 is an upper triangular matrix, R2 is a 0 matrix, Q1 transpose Q1 is im and h is equal to R transpose Q transpose. So essentially the whole thing flies without any much trouble at all. So in this case again my ffx is equal to square of the normal residual. So let us go back to the previous case. The problem reduces to h is equal to R transpose Q transpose. In view of that hz-hfx I am sorry in view of that z-hfx I am sorry once again let me correct myself z-hfx h is equal to R1 transpose Q transpose therefore z that z-hfx now becomes R transpose Q transpose x and this is what is used in here. So ffx is the square of the sum of the errors, errors represented this way. So this is an alternate expression for the residual which can be when multiplied can be given by this. You can compute the gradient of this, this is independent the first term is independent of x this is linear in x this is quadratic in x. From the module on multivariate calculus we know how to compute the gradient of a linear function. We know how to compute the gradient of a quadratic function by applying those rules it can be verified that the gradient is given by this expression at the minimum gradient must be 0. I can also simultaneously compute the Hessian. So the solution for the least square problem is obtained by solving the gradient to be equal to 0 and that gives rise to this equation. So R R transpose Q transpose x that must be I am sorry x must be equal to Rz therefore if I change if I can multiply both sides by R inverse if I multiply R inverse I get the overall solution. So to that extent I am now going to express various operations in here y is equal to Q transpose x which is given by this partition form R plus R transpose is essentially given by so there is no plus here sorry there is no plus here. This is given by this matrix there is R1 R1 transpose 00 therefore if I assemble all these things I get this equation which is the equation I need to solve R1 R1 is a non-singular matrix if I multiply both sides by R1 inverse I simply obtain the least square solution by solving R1 transpose y1 is equal to z. So this is again R1 is an upper triangle system R1 transpose is a lower triangle system this can be solved in O of n square. So the whole theme of the exercises whether it is over determined or under determined by invoking the QR decomposition of the rectangular matrix H in either case one can reduce the solution of the least square problem to one of solving a lower triangle system or upper triangle system which is much easier to solve. This is the motivation for using the QR decomposition because the pathway leads to a very simple problem in the end and we already know solution of upper triangle system lower triangle systems are very simple. This is another look at the least square solution the least square solution is now given by x is equal to Q of y in the case of under determined system y is y1 y2 Q is Q1 Q2 therefore x is equal to Q1 y1 plus Q2 y2. Now please remember y2 is arbitrary so there are infinitely many solutions and that is consistent with the under determined system in the case of under determined system there are infinitely many systems infinitely many equally infinitely many solutions for the equations whether because there are more unknowns than the knowns. From the previous analysis we get the linear solution of the linear system arising from the least square problem is given by this therefore the the norm of xls so this must be the norm of xls the same xls in here that is equal to the norm of Q1 y1 square plus Q2 y2 Q1 transpose Q1 is im Q1 transpose Q1 is i n minus m im is the mth order unit matrix i n minus m is the n minus nth order unit matrix Q1 is an orthogonal transformation or the length is invariant in the orthogonal transformation therefore this term reduces to this term because of orthogonality again this term reduces to this by orthogonality. So y2 is arbitrary therefore the least square solution is larger than or equal to y1 the norm of the square of y1 so we are going to we are going to sorry we are going to we are going to any solution x is greater than the least square solution therefore the least square solution is given by the formula that we have already given which is here so this corresponds to this corresponds to this corresponds to this therefore any arbitrary there are infinitely many solution any one of the infinite many solution has a norm that norm is greater than the least square solution therefore the least square solution is the unique solution of minimum norm so that is the the end game the end result and this is the result of applying the QR decomposition to the under determined system as well. So summary of the QR algorithm over determined system H is m is greater than n compute q1 compute r1 such that H is equal to q1 r1 using a method called Gram-Schmidt orthogonalization so that is where we are now leading to that is the reason for the summary. I only said that such a thing can happen but we do not know how to make it happen so if I can express H as q times r q is orthogonal r is a triangular we saw all the beauties in the analysis that essentially assumed I can do now it is time for us to be able to tell how to make the decomposition happen that decomposition H is equal to q1 r1 that is a reduced to qr decomposition is often done by a very famous procedure called Gram-Schmidt orthogonalization method we will talk about it shortly. So once you have q1 r1 I can compute q1 transpose z once you have q1 transpose z that defines the right hand side I need to solve the lower triangle system r1 x is equal to q1 transpose z that gives raise to the linear least square solutions r1 minus t minus t means what this is r1 inverse transpose it is denoted by is also equal to r1 transpose inverse so these two operations commute because they commute for simplicity in mathematics we simply call it minus t. So if I say minus t is that is a combination of two operations the order in which you apply these operations is a material these two operators commute. So the least square solution is given by a solution of a lower triangular system which is which is which is given elegantly by this. So the whole method is beautiful and very nice so we need to concentrate on being able to express H as q1 times r1 the reduced decomposition and that is done by Gram-Schmidt orthogonalization so that is our next term given H how do we convert H into product of q and r. So we in our method what is that we assume let us pretend we can do the decomposition H is a q and r and how if I can do that the solution the solution the structure of the solution becomes after having achieved that it behooves us to ask a question how do I make it happen how do I make H express as the product of q and r. So this is the next part of the exercise I also want to touch upon the under driven case when H is when m is less than n it is an under driven case again it is a complete summary of all the analysis we have done H transpose. So whatever we did for H now I am going to do for H transpose operation on H and operation of H transpose are not too different from each other they are mathematically equivalent again H transpose can be expressed the product of q1 and r1 q1 r1 are given by these respective sizes solve the lower triangular system and least square solution is given by q1 times r1 inverse transpose times z. So with this we have completed a discussion of the qr transpose algorithm qr decomposition algorithm modulo the method for decomposition qr decomposition and that is given by the method called Kramp-Schmidt orthogonalization. So now I am going to conjure up a case and illustrate how given a matrix H and I can deliver the q factor and the r factor to that end let H be a matrix is the m by n matrix each of the h i's are a vector in rm. So there are n vectors each of size n each of size m i runs from 1 to n m is greater than n. So there are n columns each column have m rows let the so I am going to assume the columns of h are linearly independent. So what is implied h is a full rank so rank of h is equal to minimum of m n in this case m is greater than n so minimum of m and n is n if the rank of I am sorry rank of h that is what I should have said rank of h is equal to minimum of m and n is equal to n. So it is a full rank matrix and that is guaranteed by the columns of h being linearly independent now you can see the basic concepts from vector space theory comes into play right through. So what is our aim our aim is to be able to find a q matrix with n columns which are q 1 q 2 q n I would like the columns of q q 1 q 2 to be orthogonal system in fact it is been orthonormal system that is given by the following if I took any two vectors with distinct entities i i and j not equal q i transpose q j is 0 if i is equal to j q i transpose q i is 1 so this is orthonormal. So what is the problem given the columns of h I need to find the columns of q such that the columns of h are linearly independent but the columns of q are orthonormal this is simply a procedure for converting a set of linearly independent columns to a set of orthonormal columns it is a very simple procedure and this procedure is very fundamental is used repeatedly in several different applications in linear algebra. We have already talked about the applications in solving linearly square problems now we are simply going to concentrate on how to do the decomposition itself. So let me let me go back and say this so given h 1 h 2 h n find q 1 q 2 q n given that h are linearly independent I need to find a set of vectors q that are orthonormal so what is my first vector q 1 first vector q 1 is arbitrary I can start with any place so what I am going to do I am going to start with q 1 is equal to my h 1 divided by the norm of h divided by the norm of h. Norm of h refers to the length of the vector h 1 so r 1 1 is the first element of r matrix it refers to the norm of the vector h 1 it is the two norm of that so if I divide the vector by its norm I get a unit vector so u 1 is a unit vector so first vector u 1 is very simple it is simply normalized version of h 1 now I need to compute q 2 I am going to compute q 2 as a linear combination of h 1 and h 2 q 1 has already information about h 1 so I am going to now express q 2 as 1 over r 2 2 times h 2 minus r 1 2 times q 1 please remember q 1 is known h 2 is known r 1 r 1 2 is not known r 2 2 is not known so there are two unknowns r 1 2 and r 2 2 how do I find these two unknowns these two unknowns are found by imposing two condition what is the first condition q 1 q 2 must be orthogonal to q 1 that is give rise to value of one of the unknowns then the second one is obtained by forcing q 2 to be of unit length so orthogonality condition and the normality condition two conditions are enforced by fixing two parameters so if I work to require q 1 to be orthogonal to q 2 it requires 0 is equal to q 1 transpose q 2 the assumed form of q 2 is above I substitute this in here therefore why this is r 1 2 I would I would I would have q 1 transpose q 1 is 1 please remember that I have already utilised in here q 1 transpose q 1 is equal to 1 therefore when I equate this to 0 r 1 2 is simply q 1 transpose h 2 so I got 1 1 1 constant what is r 2 2 now I know r 1 2 I know q 1 I know h 2 that is the vector I can compute the norm of the vector r 2 2 essentially divides this vector by its length so r 2 2 must be the length of that vector which is h 2 minus r 1 2 q 1 so this gives rise to a formula for computing one of the course unknowns this gives you a formula for computing the other unknown so from the known I am computing the unknown so here look at this now I have recovered q 1 I have recovered q 2 I have recovered r 1 1 I have recovered r 1 2 I have recovered r 2 2 so I am recovering both the system simultaneously q and r so let us go back to a general go to a general case when there are j unknowns in other words I can now express q j so let us pretend I have found out q 1 q 2 q 3 all they have to q j minus 1 I am going to compute q j q j is going to be expressed as a linear combination of all the previous vectors h j q i has information about h i therefore I can express q j by the linear combination that is given here divided by a constant r j from the previous calculations you can really see r j j is going to be the normalizing constant r i j are the constants using which I am going to force the orthogonality so q j has to be orthogonal to q j 1 j 2 q 1 q 2 q 3 q j minus 1 so there are j minus 1 constant that are required to force the orthogonality one constant is required to force the normality and that is the clear story so by multiplying q j by multiplying q i transpose q j for i less than j and equating to 0 I compute all the values of r i j to be q i transpose h j so I have computed j minus 1 such constants then r j j so this must be r j j is computed by the norm of the entire vector q j j therefore r j j is equal to h j minus the norm of that vector so I can now embed this in a pseudo code given h that are linearly independent that are linearly independent I want to find these are orthonormal repeat the following steps 2 to 5 for j running from 1 to n step 2 v j is h j for i is equal to 1 no this must be v 1 must be equal to h 1 that is correct sorry sorry one second yeah that is correct v j is right because j is running from 1 to n so the overall do loop is on j that is right that is correct the overall do loop is on j so step 2 v j is equal to j that means j is equal to 1 so initially j is equal to 1 v 1 is equal to h 1 step 3 for i is equal to 1 to j minus 1 I am going to make v j perpendicular to the previous vectors so that is how I compute this I now compute the norm of v j and then q j is given by this so this procedure is repeated this procedure delivers q it also delivers the matrix r as r 1 1 r j i r j j compute the num this must be I am sorry r j j this is r j j therefore in every case for every j I am trying to find all the i running from 1 to j minus 1 in here r j j here so together I compute all the required quantities for every j and that completes the procedure that completes the procedure for computing the q r d composition so with this with this with this algorithm with this sort of code with this sort of code sorry with this sort of code with this sort of code we are able to express any q h to be q and r r is a triangular q is orthonormal and this is the reduced q r d composition the reduced q r d composition so here we saw how to multiply how to decompose h into a multiplicative decomposition q r I have already utilized if h is equal to q r how to utilize it in my in my in my in my solution process for the linearly square problems these two together help you to solve the linearly square problem by the q r d composition so that is method 2 now I am going to go for the last of the three matrix decomposition methods which is called the singular value decomposition singular value decomposition is a chronymous s v d this is the third alternative method for solving linear systems I am going to assume my matrix h is again a full rank matrix given a rectangular matrix there are two Grammians h transpose h h h transpose both the Grammians are of rank n or rank m there are full rank both of them are symmetric both of them are positive definite in this case it is positive semi definite if it is positive definite means all the Eigen values are positive if it is positive semi definite means the Eigen values are 0 or positive that is the only difference between the two and when m is greater than n when m is greater than n this is the smaller Grammian because n is smaller this is the larger Grammian this is the larger Grammian when the rank of h transpose h is n it has n non zero Eigen values when the rank of h transpose h the rank of h itself is n therefore the rank h h transpose is deficient if this rank deficient some of the Eigen values in this case are allowed to be 0 that is why it is symmetric and positive it is a semi definite that essentially comes from the rank conditions for product of matrices and the given rank of h and the given rank of h because we are considering the case m is greater than n that is fixed but these two are two different Grammians one is smaller another is larger because it is cheaper to work with smaller matrices now I am going to consider first the analysis Eigen analysis of h transpose h. So, consider h transpose h which is an n by n matrix h transpose h by this definition is spd therefore it has n Eigen values which are the largest of them is lambda 1 the smallest of them is lambda n even the smallest of them is strictly positive. If lambda i is an Eigen value let V i be the corresponding Eigen vector corresponding to Eigen value lambda i for each i Eigen value pair. So, if these are the n different Eigen value Eigen vector pair for h transpose if that means h transpose h V i is equal to lambda i V i that is the fundamental relation that comes from the definition of Eigen value there are n such relations I am going to by invoking to matrix relation succinctly denote the den relation is our own single relation to that n I am going to concoct a matrix V the matrix V is simply n by n matrix whose n columns are the n Eigen vectors of h transpose h lambda is a diagonal matrix whose diagonal elements are lambda 1 to lambda n. So, V is a n by n matrix diagonal matrix lambda is also n by n matrix we have already alluded to the fact that the Eigen vectors of a real symmetric positive different matrix are orthogonal to each other. Therefore, V is again an orthogonal matrix now you can see orthogonal matrices are occurring again and again one is you are decomposition now in Eigen decomposition of symmetric positive different matrices. So, there are very many different uses of orthogonal matrices in different applications if V is orthogonal V transpose V and V V transpose is equal to identity. Therefore, this relation can be expressed succinctly as h transpose h V is equal to V lambda if I am if I post multiply both sides by V transpose both this relation comes to be that means h transpose h is equal to V lambda V transpose this is called the Eigen decomposition of h transpose h this is called the Eigen decomposition. So, so far so good. So, what does it tell you given any matrix h of size m by n h is a full rank I can I can consider I can compute two Gramian smaller larger compute the Eigen structure of the smaller Gramian I can express the Gramian as the product of V lambda V transpose V is the matrix of Eigen vectors of the Gramian lambda is the diagonal matrix of Eigen values of the Gramian. So, this is the very basic fundamental results that comes from the symmetric positive theory of symmetric positive different matrices. Now I know V I know lambda I know h. So, I am going to now consider a linear transformation define a vector ui is equal to 1 over square root of lambda i lambda i is a positive I can take the square root h is a m by n matrix V I is a n by 1 vector. So, h V I is a m vector this is the constant. So, ui are m vectors I would like to remind you. So, this converts a vector from this transformation converts the vector from Rn to Rm. So, this is Rn this is Rm if there is V I this is ui this transformation h helps to convert an n vector into a m vector that is a linear transformation ui. Now I am going to multiply h h transpose please remember h transpose h is a smaller of the Gramian this is the larger of the Gramian. I am going to take the larger of the Gramian that is m by m matrix I am going to multiply that by a m vector I already know the value of u I substitute this value which is here I now simplify if I simplified it and I get h h transpose ui is equal to lambda i ui I am going to let you follow through the simplification procedure it is very simple sequence of arguments in matrix algebra. So, what does this tell you this essentially tells you the following if V I is such that h transpose h is equal to lambda i V I that immediately tells you h h transpose ui is equal to lambda i ui that is the fundamental result. So, that goes to tell you if V I's are the Eigen vectors of h transpose h ui are the Eigen vectors of h h h transpose but they share the same Eigen value. So, they share the same non-zero Eigen values this matrix must have m Eigen values this matrix must have n Eigen values m is larger than n. So, in this case they share the same set of non-zero Eigen values the m minus n Eigen values are 0 or 0 Eigen values here 0 Eigen values that is where the semi-definiteness comes into play. Therefore lambda i ui are the Eigen vectors of Eigen vector value Eigen vectors of h h transpose the rest of the m n Eigen values of h transpose are 0. So, we talked about the semi-definiteness of this positive to semi-definiteness of this matrix h h transpose. Now these are the general results I would like to concentrate on the definition of this and that is what is going to help us to be able to look at the linear transformation leading to SVD. So, let u 1 to u n be a matrix u ui is equal to please remember this is equal to this is equal to h v i 1 over square root of lambda i h v i h v i. So, u u transpose you can readily see is i therefore, u u transpose has also the orthogonal property that one would require I would like you to follow through this simple relation. So, that essentially tells you the columns of u are also orthonormal. So, we have proved the properties of v we have proved the properties of u u and v are related to the matrix h transpose h and h h transpose. So, we have seen all the Eigen structure Eigen vectors of the two Grammians h transpose h and h h transpose. Now we are going to concentrate on what is the singular value decomposition of h that comes from basically the definition of u i and is a relation to v i please remember this that is the way we define u i. So, this by cross multiplying both sides by square root of lambda i this can be rewritten as h v i is equal to u i lambda i this is for each i. So, this relation if I concoct for all i can be written in a matrix form as h v is equal to u lambda to the power half what is lambda to the power half lambda to the power half is simply the diagonal matrix which is lambda 1 to the power half lambda 2 to the power half and lambda n to the power half the square root of the diagonal matrix lambda v is orthogonal. So, I can multiply both sides by v transpose if I multiply both sides by v transpose because v v transpose is i I get this relation look at this now h I have now expresses the product of u lambda to the power half and v transpose and this decomposition is a new decomposition it is not l u it is not cheloski it is not q r it is a new animal is a new form of decomposition it is a multiplicative decomposition this decomposition is called singular value decomposition of h. Why this is called singular value lambdas are the Eigen values of h transpose h. So, square root of lambda are called the singular values of h we have already alluded to these things in our module on matrices because square root of lambda as or the singular values of h this is called the singular value decomposition therefore h can be expressed as the product of the matrix u the product of lambda to the power half and the product of v transpose like this. If I do the multiplication you can readily see h can be expressed as a product of square root of lambda i. So, that is a scaling factor u i is a column vector v i transpose is a row vector. So, this is an outer product matrix each matrix is multiplied by square root of lambda i. So, this is the weighted sum of outer product matrices where lambda i are the Eigen values of h transpose h are the Eigen values of h transpose h I am sorry Eigen values of let me correct that once again Eigen values of this must be this must be h transpose h lambda i are Eigen values of h transpose h and lambda i to the power half are the singular values of h by definition. Therefore, this expression this expression this expression they are all called singular value decomposition of h it is a very powerful device in fact it is one of the most fundamental algorithms in numerical linear algebra one of the important application of this in numerical linear algebra is the following suppose somebody gives you a rectangular matrix h and asks you to find what is the rank of h a computational process for determining the rank of h is to compute the Gramian h transpose h and do an Eigen value decomposition and organize the Eigen value in the decreasing order the number of non-zero or the number of positive Eigen values is equal to the rank of the matrix. Therefore, this algorithm for computing the rank of the matrix using SVD is a very stable procedure for computing the rank not only it is used in computing the rank but also it can be used in compute in in in solving the least square problems. So, the Gram-Schmidt procedure the SVD the things that we have developed even though we have developed within the context for specific aspect of inverse problems these algorithms are so fundamental to numerical linear algebra in computation each one of these find multitudes of application that is why these are considered to be some of the nuts and bolts of computational linear algebra. Now what I have to come back and close the loop I have expressed h in the form of SVD if I am able to express h in the form of SVD how does it help me to solve the linear least square problem that is what I would like to come back to because the whole aim in these lectures is to solve inverse problem. So, h z is equal to h of x I want to solve h is a full rank I can express h as SVD u lambda to the power half v transpose here v v transpose v transpose v is i n u transpose u is i n the orthogonality of u and v are already established. So, f of x is equal to z minus h of x transpose z minus h of x, but I can express h as u lambda to the power half v transpose u lambda to the power v transpose I multiply both sides I utilize all these properties it can be shown f of x is essentially this term this is the constant term this is the linear term that is the quadratic term. So, if I compute the gradient and equate it to 0 the gradient of this expression is given by the linear term is given by the first term the quadratic term is given by this this must be a plus instead of a minus therefore, if I solve this system I simply have v lambda v transpose x is equal to v lambda to the power half u transpose z I can multiply both sides and the left by v transpose I can multiply. So, if I multiply both sides by v transpose v transpose v lambda v transpose x must be equal to v transpose v lambda to the power half u transpose z, but this is i this is i that reduces to v transpose I am sorry once again that leads to that leads to this is i that is also equal to i we are left with lambda v transpose x is equal to lambda to the power half u transpose z now I can multiply both sides by lambda to the power inverse lambda inverse. So, if I multiply lambda inverse lambda v transpose x is equal to lambda minus 1 lambda to the power half u transpose z this is equal to i this is equal to lambda to the power minus half you can readily see that is equal to lambda to the power minus half I hope you can see that maybe I will write that clearly this. So, this is equal to this is equal to lambda to the power minus half u transpose u transpose z therefore, v transpose x is equal to lambda to the power minus half u transpose z I again multiply both sides by v v transpose x is equal to v lambda to the power minus half lambda to the power minus half u transpose z. So, this gives it x is equal to v lambda to the power minus half u transpose z beautiful say that is the least square solution. Therefore, if I am able to express my h to be equal to u lambda to the power half v transpose v transpose I have found u I know lambda to the power half I know v transpose I use v I use lambda to the power half here and u. So, all the factors are used here. So, I can readily use the factors that I find in the SVD and express my least square solution to be this. Now, this is a very beautiful interpretation. So, z is the observation vector g is given to me u transpose is an orthogonal transformation. Orthogonal transformation essentially rotate they do not elongate or shrink. So, u transpose z simply a rotated vector z then I multiplied by lambda to the power minus half diagonal elements. So, that essentially scaling. So, this is rotation. So, I am going to give a geometric feeling this is rotation multiplying this is scaling and then v is again orthogonal that is again rotation. Therefore, the least square solution is obtained by first rotating the given observation and scaling the given observation by the diagonal matrix which is the inverse square root of lambda and then again rotation by v. So, you get the least square solution that is a beautiful way to be able to express the least square solution using the method of using the method of a singular valid decomposition. So, it has a beautiful geometric view as well in addition to be able to analytically express the solution in simple form. Now, look at this now in here I do not have to solve any system in the previous case I have to solve either a Chaloski decomposition I have to solve an upper triangular system lower triangular system. In the case of q r decomposition I have to solve an upper triangular system here I do not have to solve any system why the inverse occurs only for lambda to the power minus half what is lambda to the power minus half lambda to the power minus half is equal to lambda to the power half inverse. Inverse of a diagonal matrix is essentially inverse of the diagonal elements inverse of the diagonal elements there are n elements I can compute the reciprocals of each stick them along the this. So, that is the only inverse I need to compute. So, there is no cost associated with computing inverse except for n reciprocal computation. So, once u lambda to the power half and v are available it is simply you do not have to do any operation except rotation simple scaling and another rotation. But I want to quickly add to compute u lambda and v we have to solve an Eigen value problem Eigen value problems are intrinsically more expensive. Therefore, this method could overall be very expensive but this is a very very good stable method and very often recommended as one of the alternate methods. So, among the three methods we have seen the Chaloski the q r and the SVD q r and SVD relate to orthogonal transformations. Any algorithm in matrix theory that involves orthogonal transformations are generally more stable numerically. Therefore, these methods these two lateral methods are much preferred the method of normal equation based on Chaloski method is very good method but is subjected to lot more numerical round off errors. So, the three methods have different strengths and weaknesses but before I go to compare I would like to summarize the SVD quickly here H is u lambda to the power half v transpose that is the SVD composition compute u transpose c that is rotation I am sorry u transpose c that is simply a rotation I have already talked about that in the previous slide this is essentially scaling and this is essentially another rotation. Therefore, we can very easily see the x star is the least square solution. So, with this we have covered three methods Chaloski sorry with this we have covered three method Chaloski q r and SVD these two involve orthogonal methods orthogonal matrices this method essentially is a variant of value decomposition Chaloski method is computationally not as stable in the sense of numerical round off as q r and SVD. So, when you are trying to learn these methods it is better to take one particular problem and solve it by three methods when I say solve it the three methods I have given you algorithm for each of these. So, you can develop your own subroutine to do Chaloski your own subroutine for q r your own subroutine for SVD. So, this way you can develop your own mathematical packages ground from ground up. So, that you do not have to depend on any built in library such as MATLAB library or C library or Fortran library this way you can develop mathematical software to solve data simulation problems from ground up rather independently from by using most of the basic algorithms. There are lots of exercises we have we have given I would like to encourage the reader to work out through all these examples. In fact working through these examples is a fundamental aspect of thorough understanding I also would like you to develop your own MATLAB program that is the important thing you can develop your own MATLAB program you may say hey MATLAB already has this program why do I do this. But that is an exercise suppose you work you want to go to work for a national agency such as a meteorological forecasting agency they may try to develop all these systems ground up independent of any existing libraries because they would like to have a better control over everything. So, in order to be employed in order to be able to develop such systems you must do an exercise in developing these programs. So, LUD composition solving lower upper triangular systems Chaloski decomposition Gram-Schmidt method SVD I would like you to be able to program these methods and compare them and that is a very good part of the exercise. These are other examples these modules follow closely developments in chapter 9 of our book we are not considering iterative methods iterative methods for solving linear systems are covered in several excellent textbook Golobin-Vellon as well as Hagman and Young. So, in view of time we will not be able to cover the iterative methods dark methods we covered iterative methods are pursued in here with this I believe I have provided you a very good summary of matrix methods especially direct methods to solve linearly square problems. Thank you.