 We are considering Gauss elimination method for solving system of linear equations. Today what we are going to show is that this Gauss elimination method is equivalent to writing l u decomposition of matrix A. So, the A is the coefficient matrix, l is going to be unit lower triangular matrix that means the diagonal entries they are going to be equal to 1 and u is upper triangular matrix. As I said these two methods like Gauss elimination method and l u decomposition they are going to be equivalent. So, in terms of number of computations we do not save anything, but when we consider backward error analysis of Gauss elimination method there this l u decomposition becomes convenient. And for a positive definite matrix we want to show that matrix A has a Cholesky decomposition that means it can be written as G into G transpose where G is going to be lower triangular matrix and N G transpose will be upper triangular matrix. So, for that we are going to need l u decomposition. So, what I am going to do is I will first illustrate it for 3 by 3 matrices and then consider general case n by n matrix. So, let us recall what we have done last time. So, we have a system of equations n equations in n unknowns which we write as n by n in the compact form as A x is equal to b, A is the coefficient matrix, B is right hand side vector which is given to us and x is unknown vector x 1, x 2, x n. We had assumed that matrix A is such that if you look at principle leading sub matrix which is formed by first k rows and first k columns. So, determinant of this A k is not equal to 0 for k is equal to 1 to up to n this condition is stronger than invertibility. The invertibility it means that determinant of A is equal to not equal to 0. Now, we want to write A is equal to not only determinant A to be non 0, but also determinant of A k not equal to 0. Now, this Gauss elimination method we will write, we will show that it is equivalent to writing A as l into u. So, the Gauss elimination method will be equivalent to saying A is equal to l into u, l is unit lower triangular, u is upper triangular matrix. This is what we want to show and then that will be the LU decomposition. So, we will always assume that LU decomposition means l is unit lower triangular and u is upper triangular matrix. What we are going to show is yesterday we saw that using elementary row transformations we had reduced A to an upper triangular matrix u. So, that is going to be our u upper triangular matrix and l unit lower triangular matrix will consist of the multipliers. In order to introduce zeros or to reduce matrix A to an upper triangular form our first step was multiply the first row by A 2 1 by A 1 1 subtract it from second row. So, this M 2 1 is equal to A 2 1 by A 1 1 is going to be multiplier. So, lower triangular matrix we are going to show that it will be such that it will have one on the diagonal and the remaining entries will be multipliers in the Gauss elimination method. So, l is lower triangular along the diagonal 1 and here you have got M 2 1 M 3 1 M n 1 these were needed to introduce zeros in the first column below the diagonal and so on. So, we are going to first consider 3 by 3 matrix. Now, here are some notations E j will denote canonical vector which has got only one at j th place all the other entries they are 0. When we talk of vector x it is always going to be a column vector. So, this will be equal to x 1 E 1 plus x 2 E 2 plus x n E n i j th entry of n by n matrix we denote by A i j if you look at A multiplied by E j E j is this canonical vector then what we get is j th column of A you can verify this. If you look at E i transpose A then you will get i th row of A and if you consider E i transpose A E j A is n by n matrix E j is n by 1 vector A is n by n matrix E i transpose will be 1 by n vector. So, the result is going to be scalar and that is our A i j. So, now, let us look at 3 by 3 matrix. So, here is a 3 by 3 matrix and in the Gauss elimination what we are doing is the first step is going to be look at A 2 1 by A 1 1 that is our M 2 1 and then R 2 minus M 2 1 R 1. Then you look at A 3 1 by A 3 1 1 that is M 3 1 and then R 3 minus M 3 1 R 1 this is the first step of Gauss elimination method. The way we have chosen M 2 1 and M 3 1 the entries here they are going to be 0. So, you are converting matrix A to this matrix. Now, this entry A 2 2 that will be modified to A 2 2 superscript 1 which will be A 2 2 minus M 2 1 and then the corresponding entry A 1 2. A 2 3 will be modified to A 2 3 minus M 2 1 and then M 3 that is the result of R 2 minus M 2 1 R 1. Then A 3 2 becomes A 3 2 minus M 2 1 and then A 3 3 1 A 1 2 and A 3 3 1 becomes A 3 3 minus M 3 1 A 1 3. Now, these operations we could have performed by multiplying our matrix A by this matrix. So, the matrix has 1 along the diagonal here you have entries to be 0 and here it is minus M 2 1 minus M 3 1. So, when I do pre multiply matrix A by this matrix first row into first column will give us A 1 1 then first row into second column A 1 2 and first row into third column A 1 3. So, no change in the first row then minus M 2 1 times A 1 1 plus A 2 1. Now, what was M 2 1? It is A 2 1 by A 1 1. So, A 2 1 by A 1 1 multiplied by A 1 1 multiplied by A 1 1 that will give you A 2 1 because of this minus sign it will be A 2 1 and then you are subtracting. So, here this entry will become 0 then minus M 2 1 A 1 2 plus A 2 2 which is nothing but the right hand side here and so on. So, thus the first step of Gauss elimination method is pre multiplying our matrix A by this matrix. So, now consider matrix E 1 if I define M 1 to be this vector 0 M 2 1 M 3 1. When I look at M 1 E 1 transpose E 1 is our canonical vector E 1 transpose is 1 0 0 M 1 is this vector 0 M 2 1 minus M 1 M 3 1. When you multiply you are going to get matrix of this form this is our E 1 this is M 1 E 1 transpose. So, E 1 will be identity matrix minus M 1 E 1 transpose. If you look at E 1 transpose M 1 that means you are interchanging the order. So, it is going to be 1 0 0 0 M 2 1 M 3 1. So, that is going to be 0 using this you can check that I minus M 1 E 1 transpose into I plus M 1 E 1 transpose is equal to identity which means inverse of this matrix E 1 is going to be identity plus M 1 E 1 transpose. So, this is the first step of Gauss elimination method. Now, we are going to look at the second step. So, in the second step we are looking at only 3 by 3 matrix. So, we will have to make only 1 entry to be 0. We are going to look at the third row minus M 3 2 times R 2. This is after the first step of Gauss elimination method we have got this. Now, you define M 3 2 to be A 3 2 1 divided by A 2 2 1 and subtract from the third row the second row multiplied by M 3 2. You have zeros here. So, these will not be affected this entry will become 0 and this entry will get modified. So, A 3 3 2 is equal to A 3 3 1 minus M 3 2 A 2 3 1 and you can verify that this can be achieved by pre multiplying our matrix A 1 by this matrix. You have got 1 0 0. So, the first row will remain unchanged. Second row here is 0 1 0. So, the second row also remains unchanged. When you look at the third row the third row into first column will give you 0. Third row into second column because of the choice of M 3 2 this becomes 0 and A 3 3 1 gets modified to A 3 3 2. As before let us look at matrix E 2 and define M 2 to be vector 0 0 M 3 2. M 2 E 2 transpose that is going to be a matrix which has 0 everywhere except M 3 2 here and hence E 2 will be identity matrix minus M 2 E 2 transpose. Consider I minus M 2 E 2 transpose into I plus M 2 E 2 transpose this is going to be equal to identity because E 2 transpose M 2 is going to be 0 you will have identity then minus M 2 E 2 transpose and plus M 2 E 2 transpose. So, that gets cancelled and then minus M 2 E 2 transpose M 2 E 2 transpose E 2 transpose M 2 will be 0. So, you are left with identity. So, E 2 inverse is identity plus M 2 E 2 transpose. So, thus our Gauss elimination is equivalent to doing E 2 E 1 A and then you get matrix A 2 which is equal to our upper triangular matrix U. We have seen that E 1 and E 2 these are invertible matrices. So, we have got E 2 E 1 into A is equal to matrix U which will mean that A is going to be equal to E 1 inverse E 2 inverse U and this E 1 inverse E 2 inverse we will show that that matrix is going to be a lower triangular matrix with one along the diagonal. So, that is going to give us a L U decomposition of our matrix A. It is a bit technical, but the idea is simple. It is just that whatever operations we are doing in the case of Gauss elimination method they can be performed by pre multiplying our matrix A by an appropriate matrix which is say E 1 then E 2 and so on. So, let us look at E 1 inverse E 2 inverse that is identity plus M 1 E 1 transpose and identity plus M 2 E 2 transpose multiply. So, that will be identity plus M 1 E 1 transpose plus M 2 E 2 transpose and E 1 transpose is row vector 1 0 0 M 2 was 0 0 M 3 2. So, that is 0 and hence this term is not there. Now, identity matrix that means you have got these ones and remaining entries 0 M 1 E 1 transpose we had seen that M 1 E 1 transpose is the matrix 0 everywhere except M 1 E 1 and M 2 1 and M 3 1 and hence you have M 1 E 1 transpose will be contributing M 2 1 M 3 1. M 2 E 2 transpose was matrix with 0 everywhere except entry M 3 2 and thus our E 1 inverse E 2 inverse is going to be a lower triangular matrix with diagonal entries to be equal to 1. So, this was for 3 by 3 matrix. Now, I am going to quickly do for n by n matrix, but not going into all the details the idea is similar. So, look at n by n matrix and look at the first step. So, in the first step you are multiplying the first row by m i 1 and subtracting it from r i, where m i 1 is a i 1 divided by a 1 1. This is our matrix a we want to introduce 0's here. So, you consider a 2 1 by a 1 1 multiply the first row and subtract from the second row. So, as we had seen before this first step method can be performed by pre multiplying our matrix a by this matrix call this matrix to be E 1. So, we have E 1 a is equal to A 1 that is the first step of Gauss elimination method this matrix E 1 as before define vector M 1 to be 0 M 2 1 M n 1. So, we had done it for the 3 by 3 matrix. So, in that case our matrix our vector M 1 was 0 M 2 1 M 3 1. So, now the only difference is instead of a 3 by 1 vector you have got n by 1 vector. So, 0 M 2 1 M 3 1 M n 1 then our matrix E 1 is nothing, but identity of the matrix E 1 is identity minus M 1 E 1 transpose exactly same as before only difference is instead of 3 by 1 vector you have got n by 1 vector. So, this matrix E 1 which is identity minus M 1 E 1 transpose it is going to be an invertible matrix and its inverse will be given by E 1 minus M 1 E 1 transpose same proof as before. So, E 1 is identity minus M 1 E 1 transpose and E 1 inverse is going to be equal to identity plus M 1 E 1 transpose. Next in the second step of Gauss elimination method you have a i j 2 to be equal to a 1 minus M i 2 a 2 j 1 i j varying from 3 up to n. So, here now you define your vector M 2 which has the multipliers. So, the multipliers are M 3 2 M 4 2 M n 2 and then E 2 is going to be your matrix which has 1 along the diagonal minus M 3 2 minus M n 2. If you look at E 2 multiplied by A 1 that is going to give you A 2. So, you have started with A you pre multiplied by E 1 and you got a modified matrix A superscript 1. Now, you multiply by E 2 and then you get A 2. This E 2 is also invertible matrix E 2 is identity minus M 2 E 2 transpose its inverse is identity plus M 2 E 2 transpose and in general your E k is going to be identity minus M k E k transpose E k inverse will be identity plus M k E k transpose and then when you look at E n minus 1 into E n minus 1 into E n minus 2 up to E 1 multiplied by A then you are going to get your upper triangular matrix U exactly the same matrix which we had obtained in the Gauss elimination method. In the Gauss elimination method our system A x is equal to b we had converted into an upper triangular system U x is equal to y and if it is a upper triangular system then we can do back substitution. So, we first determine x n then x n minus 1 and so on. So, this U 1 can obtain by pre multiplying A by invertible matrices E n minus 1 E n minus 2 up to E 1 then that gives you A to be equal to E 1 inverse E 2 inverse up to E n minus 1 inverse. Now, if you remember in yesterday's lecture we had said that our aim is to reduce the system A x is equal to b to a system U x is equal to y and that should be a equivalent system that means the original system and the new system they should have the same solution. So, now here is a proof of this that when you do Gauss elimination method then the new system is equivalent to the earlier system. So, we have got matrix A U pre multiply by matrix E. So, E is going to be matrix obtained by multiplying E n minus 1 E n minus 2 up to E 1 each of E j is invertible matrix. So, E is going to be an invertible matrix. So, we have A x is equal to b which is same as E A x is equal to E b where E is invertible and then E into A will be our U. So, you get U x is equal to y. So, if x is a solution of A x is equal to b it is going to be solution of U x is equal to y and the converse is also true. If x is a solution of U x is equal to y then it will be a solution of A x is equal to b. Now, look at E 1 E inverse it will be E 1 inverse E 2 inverse E n minus 1 inverse E 1 inverse is identity plus M 1 E 1 transpose E 2 inverse is identity plus M 2 E 2 transpose and E n minus 1 inverse is this. You multiply and when you simplify what you are going to get is E inverse to be this lower triangular matrix with one along the diagonal and the entries here to be the multiplier. We have E into A is equal to U. So, A is equal to E inverse U and now E inverse is equal to L. So, you get A is equal to L into U. Now, you have to note that the Gauss elimination method it can be performed provided at no stage our pivot is becoming 0 and that means we have proved that at A can be equal to L into U if at no stage the pivot becomes 0. So, let me look at the system A x is equal to b. This A we have written as L into U. So, you have got L into U x is equal to b. Now, this I will split into two systems U x is equal to y and L y is equal to b. So, this is given as B is given vector. So, look at L y is equal to b. L is matrix lower triangular 1 1 1 then you had here M 2 1 M n 1 and so on. These are all entries to be 0 y 1 y 2 up to y n is equal to b 1 b 2 up to b n. So, when we look at the first equation it is y 1 is equal to b 1 vector b is given to us. So, you determine y 1. So, you determine y 2. In the next equation you will have y 1 and y 2 y 1 is already determined. So, you determine y 2. So, this is going to be forward substitution. So, you determine y 1 y 2 and y n. Once you have done that then you look at U x is equal to y. Now, in U x is equal to y the right hand side we have determined and there you are going to do back substitution. So, we have either you consider Gauss elimination method or you look at A is equal to L into U and solve two systems of equations. Once forward substitution, once backward substitution and both of these they will need the number of computations to be of the order of n square. Whereas, finding L and U that will be of the order of n cube by 3. So, what we have done is we have proved that the Gauss elimination method is equivalent to or both of them are the same. So, you know one can show that if you have performed Gauss elimination method. So, you have obtained U, you have got multipliers construct matrix L and then you have got A is equal to L into U. So, now maybe what one can do is try to write this A is equal to L into U directly. So, before we do that trying to write directly, let us show that such a decomposition is unique. If you say that A should be equal to L into U where L is lower triangular U is upper triangular such a decomposition is not unique. But if you say that L should be unit lower triangular that means all the diagonal entries should be equal to 1 that is what makes the decomposition to be unique. Now, the proof of uniqueness is straight forward. There what we are going to use is if you have got two upper triangular matrices and if you take their product that is going to be again an upper triangular matrix. Similarly, if you have got two lower triangular matrices if you take their product then it is going to be again a lower triangular and if these two matrices are unit lower triangular their product is also unit lower triangular. This we will do as a tutorial problem the verification that product of upper triangular matrices is upper triangular. We also have a result that if your matrix is lower triangular and it is invertible then its inverse is also lower triangular. So, using these two results we are going to show that a U decomposition of a matrix A is unique where L is unit lower triangular and U is going to be upper triangular. Our assumption is determinant of A k not equal to 0 where A k is principal leading sub matrix of order k which is formed by first k rows and first k columns of our matrix A. So, under these assumptions our matrix A is going to be invertible matrix. Now, determinant of A will be determinant of L into determinant of U. We have proved existence of L U decomposition. Start with a matrix A with the property that determinant of A k is not equal to 0 perform gauss elimination method. The final matrix upper triangular matrix that is U construct L using multipliers and that gives you L. So, we definitely know that a matrix A can be written as L into U. Now, we want to show that such a decomposition is unique. Determinant of upper triangular matrix or lower triangular matrix is product of the diagonal entry. Now, L has all the diagonal entries to be 1. So, determinant of L is going to be 1. Determinant of U that is same as determinant of A because what we have done is we have used elementary row transformations and elementary row transformation of only one type which is multiplying the row by a non zero constant and subtracting from other row. So, such an operation does not change value of the determinant. So, this determinant of A is going to be determinant of U. So, determinant of U will be not equal to 0 because determinant of A is not equal to 0. So, our U is going to be a invertible matrix and also L will be invertible matrix. So, let us start with 2 decompositions. A is equal to L 1 U 1 and also L 2 U 2 and then show that L 1 has to be equal to L 2 U 1 has to be equal to U 2. So, A is L 1 U 1 plus is equal to L 2 U 2 that will mean that L 2 inverse L 1 is equal to U 1 inverse U 2. Determinant of U 1 is determinant A which is not equal to 0. So, U 1 is invertible determinant of L 2 is equal to 1. So, L 2 is invertible. So, it is L 2 inverse L 1 is equal to U 1 inverse U 2. U 1 is upper triangular and hence its inverse will be upper triangular. Product of two upper triangular matrices is upper triangular. L 1 is lower triangular, L 2 inverse is lower triangular. So, their product is also going to be lower triangular. So, you have on one hand a lower triangular matrix on another hand an upper triangular matrix. So, these two are equal provided both of them those are diagonal matrices. So, our L 2 inverse L 1 and U 1 inverse U 2 they are going to be both of them diagonal matrices. In addition L 2 inverse L 1 is going to be unique lower triangular. So, that means all the diagonal entries they are equal to 1. So, your L 2 inverse L 1 then U 1 inverse U 2 both of them they will be diagonal matrices with diagonal entries to be equal to 1 that means identity matrix. So, you get L 1 L 2 inverse L 1 is equal to identity that gives you L 1 is equal to L 2 U 1 inverse U 2 is equal to identity that gives you U 1 is equal to U 2. So, that is uniqueness of L U decomposition and this uniqueness we are going to need when we want to show that a positive definite matrix can be written as G G transpose that is the Cholesky decomposition. So, now let us see like we know that a can be written as L into U and such a decomposition is unique. So, let me see whether I can try to determine the elements of L and U directly that means not going through the Gauss elimination method and then multipliers and then constructing L and all. What I can try to do is write a as L into U where L is unit lower triangular matrix U is upper triangular matrix and try to determine the entries of matrix L and matrix U. So, matrix A is given to me the entries of L and U these are not known. So, you will multiply the corresponding entries and then try to determine. So, again I am going to quickly do this. So, we are writing A as a unit lower triangular matrix. So, you have got one along the diagonal and then L 2 1 L 3 1 L n 1 that will be the first column and so on. All the entries above the diagonal they are going to be 0. U will be upper triangular matrix. So, you have U 1 1 U 1 2 U 1 3 U 1 n first row then in the second row you will have 0 here U 2 2 U 2 3 U 2 n and so on. Some of the entries above the diagonal in U also can be 0, but what we know is definitely below the diagonal they are all 0. Now, you look at first row into first column. So, that is going to be U 1 1. So, that U 1 1 will be equal to A 1 1 like that then first row into second column, first row into third column and so on that will give you A 1 j to be equal to U 1 j that means you have determined the first row of U. Our L the first row is only has only one and remaining entries 0. So, when I consider first row of L multiplied by various columns of U what comes into picture are only the entries of the first row of U and then you get A 1 j is equal to U 1 j. So, you have determined first row of U. Now, you consider the L 2 1 U 1 1 that is going to be equal to A 2 1 then third row first column, fourth row first column and so on. When you do that you are going to have L I 1 U 1 1 is equal to A I 1 and L I 1 is equal to A I 1 divided by U 1 1. So, that means you have determined first column of L. So, we write A as L into U L unit lower triangular U upper triangular and then the first row of U is determined first column of L is determined. Now, we will determine the second row of U second column of L third row of U third column of L and so on. In this order we can determine all the entries of L into U. Now, what you have to notice is that when you do this thing all the diagonal entries of U they have to be not equal to 0 because in the first one we notice that we are dividing by U 1 1 the matrix A is given to us and we are trying to determine the entries of L and U. Now, in the second step the first column of L is known first row of U is known. So, you consider second row of L and jth column of U that is going to give us A 2 j. So, this A 2 j is going to be equal to L 2 1 U 1 j plus U 2 j j going from 2 up to n. A 2 j's are given to us L 2 1 we have already determined U 1 j elements of the first row are determined. So, that will determine the second row of U then Ith row of L into second column of U that will determine the second column of L. In the second row of L these being all zeroes we have already determined second column and second row of L. So, you have so far determined first row of U first column of U second row of U second column of U and similarly for the matrix L. So, one continues this and one determines L and U. You can do the second row of U then number computation of number of operations. We have already computed the operations in the Gauss elimination method and we saw that they were of the order of n cube by 3. Now, we are doing it differently, but you will see that here also the total number of operations they are going to be n cube by 3. So, we do not gain as such in the number of operations whether you do Gauss elimination method or whether you determine L and U directly you are going to be the number of operations they are going to be the same, but as I said this writing A as L into U that is useful in doing the backward error analysis for Gauss elimination method and for considering the Cholesky decomposition of the matrix A. We have got L U decomposition. Now, we have got another decomposition which is known as L D V decomposition. So, what are this? L is unit lower triangular, V is unit upper triangular and D is going to be diagonal matrix. A is equal to L into U with all U I I's to be not 0 then you define consider D to be diagonal U 11 U 22. And then V is equal to D 1 inverse U. So, this is my definition when you try to look at the entries of V. So, V I I will be given by D inverse it is I k th entry into U k I k going from 1 to n the matrix multiplication D being a diagonal matrix the only term which remains in this summation will be D inverse I I and U I comma I will be nothing but U I I. So, you get V I I to be equal to 1. So, that means if you define V to be equal to D inverse U where D is diagonal entries diagonal matrix consisting of D inverse I I. So, we have diagonal entries of U then our matrix V it becomes an upper triangular matrix with diagonal entries to be equal to 1. So, we have A is equal to L into U V is equal to D inverse U that means U is going to be equal to D into V. So, we have A is equal to L D V. So, starting with L U decomposition we have proved existence of L D V decomposition where L is unit lower triangular V is unit upper triangular and D is diagonal matrix. Now, soon it will become clear to you why we are going to this that we had L U decomposition that why not be satisfied with it, but go now to this new decomposition L D V. So, let us show that this L D V decomposition is also going to be unique. We have proved the existence and uniqueness of L U decomposition from the L U decomposition we deduced L D V decomposition, but there can be some other way of finding L D V. So, we want to show the existence. So, we have A is equal to L D 1 V 1 is equal to L 2 D 2 V 2 where L 1 L 2 these are unit lower triangular D 1 D 2 are diagonal matrices and V 1 V 2 are upper triangular and unit. The diagonal entries to be equal to 1. So, now, let me look at this way. So, by uniqueness of L U decomposition what we get is L 1 is equal to L 2 and D 1 V 1 is equal to D 2 V 2. This will imply that D 2 inverse D 1 is equal to V 1 inverse V 2. Now, is V 1 invertible yes, because V 1 is going to be unit upper triangular. So, determinant of V 1 is going to be equal to 1. So, we have got D 2 inverse D 1 that means a diagonal matrix V 1 inverse V 2 that means it is going to be a unit upper triangular matrix. So, both of them they have to be diagonal matrices and since the diagonal entries of V 1 inverse V 2 they are going to be all equal to 1. All both of these they have to be equal to identity matrix. So, that gives you D 1 is equal to D 2 and V 1 is equal to V 2. Today's lecture we have shown that the Gauss elimination method can be written can be expressed as a L U decomposition of the matrix. Then we proved uniqueness of L U decomposition and then we also saw how to determine L and U by starting with the formula A is equal to L into U and then taking the matrix multiplication and identifying various entries. Then we have talked about L D V decomposition we proved is uniqueness. So, in tomorrow's lecture we will use this for showing that a positive definite matrix has got a Cholesky decomposition. So, thank you.