 Hi, we are learning direct methods for solving non-singular linear systems. Recall that direct methods give exact solution to a linear system when there is no rounding error involved in it. We have learnt Gaussian elimination method, do little factorization method and crude factorization method in our last class. In this class, we will learn another LU factorization method called Cholzky's factorization. A matrix A is said to have a Cholzky's factorization if there exist a lower triangular matrix L such that A can be written as L into L transpose. Observe that if L is a lower triangular matrix, then L transpose is an upper triangular matrix. Therefore, this form of factorization is indeed a LU factorization. Also, you can observe that the right hand side matrix is a symmetric matrix because we have L into L transpose. Therefore, if you take the transpose of that, you again get L into L transpose. Therefore, Cholzky's factorization is possible only for symmetric matrices. Also, we will see that Cholzky's factorization exists if the matrix A is a positive definite matrix. Let us recall from our linear algebra course what is mean by a positive definite matrix. A symmetric matrix A is said to be positive definite if X transpose AX is positive for all non-zero vectors X. You can see from this definition that it is not that easy for us to check whether a given symmetric matrix is positive definite or not. For that, you have to take all the non-zero vectors and compute the expression X transpose AX and check if it is a positive number. That seems to be very difficult. Therefore, we have some equivalent properties which we would have studied in our linear algebra course. Here, we will just recall those properties in the form of a lemma. Suppose we have a symmetric n cross n matrix A, then we say that A is positive definite if and only if all its principal minus are positive. That is one equivalent condition for positive definiteness. You can see that principal minus can be computed relatively in a easy way. Therefore, this condition is more handy for us to check whether a symmetric matrix is positive definite or not. Another equivalent condition is that a symmetric matrix A is positive definite if and only if all the eigenvalues of A are positive. Again, this can also be checked while finding eigenvalues is little difficult computationally than finding principal minus. However, this can also be checked. We will not get into the proof of this lemma because it is a part of linear algebra course, but we will use these properties in our theorem on existence and uniqueness of Cholsky's factorization. The theorem says that suppose you have an n cross n matrix with real entries, note that we will always work with matrices of real entries. Therefore, even if I do not tell this, you should keep in mind that we always work with matrices with real entries. So, we have a matrix n cross n such that it is a symmetric matrix and also it is positive definite. Then we can always find a lower triangular matrix L such that you can write A as L into L transpose. Moreover, if all the diagonal elements of this lower triangular matrix L are positive, then we can say that that is the only lower triangular matrix such that A is equal to L into L transpose. That is, we get a unique Cholsky's factorization. So, when you say Cholsky's factorization, it means finding the lower triangular matrix L such that A is equal to L into L transpose. We are imposing this condition that is positive diagonal elements is just to make sure that we have a unique way of obtaining the matrix L. In our computation, you will see that all the diagonal elements of the matrix L will appear as L i i square is equal to sum number. Therefore, when you take square root, you get L i i is equal to square root of that number. Therefore, you may have plus or minus also. So, you have two choices here to be very specific. We will always make our mind that we will choose the positive sign for those diagonal elements. This is just to make the algorithm more precise without any ambiguity of what sign we have to choose. In that way, the theorem says that you will have a unique such matrix L. That is what the statement is. Let us try to prove this theorem. The proof is by induction. It means you first prove the theorem for the matrix A which is a 1 cross 1 matrix. Then, you assume that the factorization is possible for a k cross k matrix and then prove it for k plus 1 cross k plus 1 matrix. If you do so, already you have proved it for 1 cross 1 matrix. Therefore, it is true for 2 cross 2 matrix. Once it is true for 2 cross 2 matrix, then it is true for 3 cross 3 matrix and so on. So, that is the idea of induction. Therefore, the first step is to prove that a unique factorization is possible when A is a 1 cross 1 matrix. Let us take A as A 1 1. Then, obviously, you can choose L is equal to root A 1 1. Then, you can see that A can be written as L into L transpose. Of course, you can also choose minus root A 1 1, but just because we made our mind to pick only the positive sign, we will have plus root A 1 1 here. Otherwise, you can as well take minus L. There is nothing wrong in that. Therefore, the Cholsky's factorization is true when the matrix A is a 1 cross 1 positive definite matrix. It means A 1 1 should be positive. That is how you are getting a real matrix L. Remember that L should be a lower triangular matrix with all its entries as real numbers. Then, only we will declare that Cholsky's factorization exists. Now, we will fix our induction hypothesis. As per that, we will assume that the Cholsky's factorization holds for any k cross k matrix for some natural number k. So, we are taking a k and then we are assuming that if A happens to be a k cross k matrix, then you can always find a L k such that A can be written as L k into L k transpose. Unique such L k will exist with all its diagonal elements as positive. That is the assumption we are making. As per the induction method, what we have to prove if we have a matrix which is k plus 1 cross k plus 1 matrix, then we have to prove that the Cholsky's factorization exists. So, let us assume that A is a k plus 1 cross k plus 1 symmetric positive definite matrix. Now, we have to find a L such that A is equal to L into L transpose. Remember, we have assumed that the Cholsky's factorization exists for any k cross k matrix. Therefore, what we will do is we will write our matrix A as a k which is the principal sub matrix of order k for A and then we will write the remaining column vector as A and its transpose as A transpose and then the lost diagonal element like this. Just to visualize, let us take a 3 cross 3 matrix as A 1 1, A 1 2, A 1 3, A 2 1, A 2 2, A 2 3, A 3 1, A 3 2 and A 3 3. So, here k plus 1 is equal to 3 we have taken in this example. Therefore, our k is 2. So, A k should be the principal sub matrix of A of order 2. It means it should be this matrix. This is your A 2 here and then your column vector small a is taken as this. This is your A. In all our discussions we will always take a vector as a column vector. Therefore, its transpose that is the row vector will always be written as A transpose. So, this is what is A transpose. Remember, it is a symmetric matrix. Therefore, this and this will be the same and this and this will be the same. That is A 3 1 will be equal to A 1 3 and A 3 2 is equal to A 2 3 and then you have this element sitting here. So, this is how we are just splitting A into block wise where A is the k cross k principal sub matrix of A and this vector A is the column vector at the last column removing the diagonal element that is A k plus 1 A k plus 1. You can observe that A is a symmetric matrix. Therefore, A k is also a symmetric matrix. Also you can see that A k is a positive definite matrix because A is positive definite by our lemma all its principal minors are positive. In particular, the principal minors of A k are also principal minors of A with lower orders. Therefore, they all are also positive. In turn A k is also a positive definite matrix. This is an observation. Therefore, by our induction hypothesis, you can find the Cholsky's factorization for A k. That is, you can find a unique lower triangular matrix L k such that A k equal to L k into L k transpose where all the diagonal elements of L k are positive. This is the assumption as per our induction hypothesis. Now, let us see how to construct the Cholsky's factorization for A with the help of the Cholsky's factorization of A k. Let us propose that the Cholsky's factorization of A looks like this matrix where L k is coming from our Cholsky's factorization of A k and then all these elements are 0 because it is a lower triangular matrix and you have a vector L which is written in the row form. Therefore, it is L transpose and then this is a number L k plus 1 k plus 1. Now, here we know this. This is known to us as per our induction hypothesis. Therefore, we do not need to compute that, but we need to compute the vector L and the real number L k plus 1 k plus 1. How we have to find that? We should find these quantities in such a way that A is equal to L into L transpose. So, that is our Cholsky's factorization. So, we have to find this vector and this real number. How are we going to find this? Let us see. Well, what we can do is you take this the first block of the matrix L. This is the row block and multiply it with the lost column of L transpose. That will give you L k into L is equal to we have to compare that with the corresponding entries of the left hand side matrix which happens to be the vector A. So, the vector A is known to us and therefore, we got a lower triangular system with solution as the unknown vector L. Now, how will you find it? Well, you can just use the forward substitution to get the vector L because L k is a lower triangular matrix. You do not need to go for any elimination process. You can simply use the forward substitution to get the vector L provided the matrix L k is invertible. How do we know that the matrix L k is invertible? Well, you can see that from the way we have constructed L k. We have constructed L k such that A k equal to L k into L k transpose. Now, you take the determinant on both sides. You have A k determinant is equal to determinant of L k square. But this is positive because A is a positive definite matrix. Therefore, this is surely non-zero. That is what we can see from the way L k was computed. Therefore, L k is an invertible matrix. So, you will get a unique vector L such that L k L is equal to A which is a known quantity. So, we obtained this vector L. Now, we need to only find this real number. That is the only part left out for us. Let us see how to find that. Again, what we will do is you take the lost row of L and multiply it with the lost column of L transpose. That gives us the vector L transpose into L plus L k plus 1 k plus 1 square. That needs to be compared with the corresponding element of the matrix A and that is going to be A k plus 1 k plus 1. Now, that gives us the vector L k plus 1 k plus 1 is equal to A k plus 1 k plus 1 minus L transpose L. Now, from here you may take L k plus 1 k plus 1 is equal to root A k plus 1 k plus 1 minus L transpose into L. Of course, you have to have plus or minus, but we have already made our mind that we will take only the positive sign. Therefore, I will take here only positive sign, but then the question is, is this real? In other words, we have to first justify that is this greater than 0 that is the question. Because remember, in order to say that the Cholesky's factorization exists, we have to find a lower triangular matrix with all its entries as real numbers. Therefore, this number should also be a real number, but it need not be because if this number happens to be negative, then you will have L k plus 1 k plus 1 as a imaginary number. Therefore, you have to justify this. How will you justify it? Well, again go back to the form that we are writing. You take determinant of A and that is equal to determinant of A L into determinant of L transpose which is nothing but L square. What is determinant of L? Determinant of L is determinant of L k into L k plus 1 k plus 1 and that square means this is square and square. Now, determinant of A is nothing but the product of all its eigenvalues. Therefore, you can say that the product of all the eigenvalues of A is equal to this is a positive number because we already know that L k exists. It means all its entries are real that is already assumed. Therefore, this is a positive number and this is what we do not know whether it is positive or not. Now, you can see that all the eigenvalues of A are positive. Why? Because A is a positive definite matrix. So, we have stated one equivalent property of positive definite matrix in the last lemma that all its eigenvalues are positive. Therefore, the left hand side is the product of positive numbers. Therefore, it is positive. That shows that L k plus 1 k plus 1 square is a positive number. Remember just because you are squaring this does not mean it is positive. Say for instance if L k plus 1 k plus 1 is equal to i then its square is minus 1. Therefore, you just cannot directly say that this is a positive number. You have to justify this because we have not yet proved the existence of Cholesky's factorization. In fact, that is what we are trying to justify that this is positive and that comes from this representation. Therefore, we have proved that this is positive and that implies that L k plus 1 k plus 1 is a real number and that proves the Cholesky's factorization exists and also you can see the way we have constructed that the Cholesky's factorization is unique. Why? Because from the induction hypothesis L k is unique and then this system has a unique solution L and then finally, this also is a unique representation. Therefore, with all this we can see that our Cholesky's factorization is unique provided that all the diagonal elements sin or fixed. So, as a unique sin either it should be positive or negative. You can see that at every stage of induction you are taking the square root for getting the diagonal element. There you have two choices you may go for a plus sin or minus sin. So, we made our mind to take only the plus sin in all the steps. In that way we have a unique factorization that you have to keep in mind. There is nothing wrong in taking minus in all the steps also that gives a different Cholesky's factorization. Therefore, as such Cholesky's factorization is not unique, but if you fix the diagonal elements sin then it is unique. Let us take an example. Let us take the matrix A as given like this. You can observe directly that it is a symmetric matrix. Also you can see that it is positive definite. How will you do that? One easy way out is to check all its principal minus or positive. The principal minor of order 1 is 9 for this matrix that is positive. The principal minor of order 2 is 9 into 2 minus 9 that is again 9. That is also positive and you can also see the determinant of A is positive. Therefore, this is a symmetric and positive definite matrix. Therefore, we can find a unique Cholesky's factorization with all the diagonal elements being positive. Let us see how to compute that. There are many ways to compute, but we will just follow the construction procedure we adopted in the previous theorems proof and try to construct the Cholesky's factorization for this matrix. For that we have to first compute L 1. What is A 1? A 1 is nothing but 9. Therefore, L 1 can be taken as 3 or minus 3. Again I am emphasizing we will always take the positive sign. Therefore, we are taking L 1 as 3. So, with that we will go to find what is L 2. For that we will write L 1 L 2 as L 1 0 and then L 2 1 L 2 2. L 1 is 3. Therefore, it is 3 0 L 2 1 L 2 2. We have to find what is L 2 1 and L 2 2 such that A 2 which is nothing but 9 3 3 2 and then equal to 3 0 L 2 1 L 2 2 into 3 L 2 1 0 L 2 2. That is going to be equal to 3 times L 2 1 is equal to 3. That implies L 2 1 is equal to 1. Similarly, L 2 1 square plus L 2 2 square is equal to 2. That will again imply that this is going to be L 2 2 square equal to 1 or L 2 2 is equal to 1. Again remember whenever there is a diagonal element we will always get it in terms of the square of that element and then when you take the square root we will always take the positive sign and that gives us L 2 as 3 0 1 1. Let us now compute L 3 which is also the required Cholzky's factorization. For that we will take L as L 2 and then L transpose L 3 3 and then this is a 0 vector. That gives us L 2 1 A is equal to L 2 0 L transpose L 3 3 and then it is transpose. That is L 2 transpose L 0 transpose and L 3 3. When you take the first block row with the last column here we get L 2 L is equal to this vector. So, this vector is equal to 0 vector minus 2 3. That gives us L 3 1. So, remember this is L 3 1 L 3 2 and L 3 3. L 3 1 is equal to minus 2 by 3 and L 3 2 equal to 11 by 3. So, that is what we have here. This is our L 2 and this is the 0 vector and we have L transpose here. Now, we have to find the diagonal element L 3 3. For that we will multiply this with the last column of L transpose that gives us L transpose L plus L 3 3 square is equal to 23. That gives us L 3 3 square equal to 23 minus 4 by 9 plus 121 by 9 and that 9 into 23 minus this is going to be 82 by 9. Therefore, if you take the square root on both sides and choose the positive sign we will have L 3 3 equal to root 82 by 9 and that gives us the Cholski's factorization for the matrix A given like this. This is not the only way to compute Cholski's factorization. Recall in the do little case we computed the do little factorization by direct comparison right. What we did? We wrote A is equal to L into L transpose of course, in do little all the diagonal elements of L are 1. So, you have to write such an L and then find the other entries of L as well as U. So, in fact, this was a U there in the do little factorization but here in Cholski's factorization we have to take it as L transpose itself. Now, just like how we did with do little factorization what we did the right hand side product of two matrices we just multiplied them and then compared the elements of the right hand side matrix with the corresponding elements of the left hand side matrix and we got all the elements of L and U there. The same idea can be followed in constructing Cholski's factorization also you need not go step by step as we did in the last computation. This is to make the proof of the theorem more rigorous we used it in the form of an induction otherwise you can also go for a direct comparison calculation which will also lead to a very efficient algorithm. Let us see how these expressions look like well L 1 1 can be directly obtained as root A 1 1 remember again we are fixing our sign as plus that is why we got it otherwise you can also take minus root A 1 1. Now, L 2 2 is obtained by multiplying this row of L with this column of L transpose right and that gives L 2 2 as this similarly to get L 3 3 you can multiply this with this and that gives you L 3 3 right all the diagonal elements are therefore, obtained in general if your matrix is a n cross n matrix you can just look at these expressions and try to generalize how this expression will look like for a n cross n matrix that will be given as L i i is equal to root A i i minus you have the sum starting from k is equal to 1 and goes up to i minus 1 right. So, here it is 3 therefore, it goes up to 2 similarly if you are computing the diagonal element at the ith row it goes up to i minus 1 because i is already there that is what you are computing i plus 1 onwards the entries are 0 because it is a lower triangular matrix right therefore, this will go only up to i minus 1. So, therefore, the diagonal elements are given by this expression how the non-diagonal elements are obtained again let us see to get this element you can just multiply this row with the first column of L transpose and that gives you L 2 1 L 3 1 we have to find for that you take this and again you do multiplication with the first column of L transpose thus gives you L 3 1 and how will you get L 3 2 well you multiply this with the second column and that gives you L 3 2 again you can just observe the expression and try to generalize it for any n cross n symmetric positive definite matrix A and that gives you this expression remember this has to go for each column other than the diagonal element all the elements after diagonal elements are also not computed therefore, j should go from 1 2 up to i minus 1 and this has to be done for all the rows right therefore, the row index i should go from 1 to n. So, this way also you can compute Cholsky's factorization. So, since we are working with symmetric matrix Cholsky's factorization is more efficient than Gaussian elimination and the do little or crude factorization because we are making use of the property that A is a symmetric matrix therefore, you are only computing L you are not computing U explicitly in that way you gain lot of computational time. Now, how to compare two methods in terms of their computational time well that can be done by counting the number of arithmetic operations involved in this methods. In the next class we will compare Cholsky's factorization with Gaussian elimination method and see which method is more efficient in terms of the computational time well that will be in terms of how many arithmetic operations are involved in them this we will do in the next class. Thank you for your attention.