 In the last lecture, we have started our discussion on matrix norms. We have also introduced an important concept called matrix norms subordinate to vector norms. In this lecture, we will continue our discussion on matrix norms, where we will define what is meant by condition number of a given matrix and we will also learn how the condition number can be used to study the sensitivity of the matrix that we will be using in solving the linear systems on a computer. Let us start our discussion with a theorem, which will tell us why we have to define condition number of a matrix in a particular form. Let us consider an invertible matrix A, which is an n cross n matrix. We will also consider a linear system Ax is equal to b. Let the solution of this system be x. Now, instead of taking the right hand side vector b, we make a small error in it and consider the right hand side vector with error as b tilde. So, we want to solve Ax equal to b, but we land up solving the system Ax equal to b tilde. Now, the solution that we obtain from this perturbed system is going to be different from the solution x. So, let us denote this solution from the perturbed system Ax equal to b tilde as x tilde and now the question is what is the error involved in x tilde when compared to the exact solution x that is the question. The theorem says that the relative error in x tilde when compared to x which is defined as norm of x minus x tilde divided by norm x. Remember this quantity is obtained for a given vector norm that is you have to take a vector norm and with respect to that vector norm, we are computing this quantity which can be taken as the relative error in x tilde when compared to x. Now, the theorem says that this relative error in x tilde will be less than or equal to this quantity times the relative error in the right hand side vector b. In this we are choosing a vector norm and this quantity is computed using the matrix norm subordinate to the vector norm that we have considered here ok. So, that is a very important point that we have to keep in mind when we are working with not only this theorem, but also whenever you are working with the error analysis of any linear system, the matrix norm that we take should be the subordinate matrix norm with respect to the vector norm that is given to us. In this theorem, the vector norm and therefore, the subordinate matrix norm can be anything that you choose the result will hold, but in the computations we will generally fix one of the three vector norms that we have introduced in the last class that is L 1 norm or L 2 norm or L infinity norm. We will choose one of these norm as per our convenience or requirement and then we have to choose the corresponding matrix norm subordinate to our chosen vector norm. So, this is how we will work in all the problems that we do from now onwards, but this theorem as I told will hold for any vector norm and the corresponding subordinate norm that we choose ok. Let us try to prove this inequality. Remember, we have to find an estimate for this relative error right. So, we will first start with x minus x tilde. Remember, from the linear systems we can write Ax minus Ax tilde is equal to b minus b tilde right. From there we can write A into x minus x tilde is equal to b minus b tilde right. Since A is invertible therefore, we can write x minus x tilde is equal to A inverse into b minus b tilde. Now, let us take norm on both sides. Remember, this is the vector norm and this is also the vector norm right. Now, you recall from our last lecture we have proved an important property for the subordinate matrix norm which says that Ax which is a vector and you take the vector norm on that. Then you can write this as less than or equal to norm A which is the subordinate matrix norm times norm x right. We have proved this inequality in the last class. Now, you see you can just imagine this as A and this as x and you can apply this inequality to get norm x minus x tilde is less than or equal to norm A inverse because you have inverse here into norm b minus b tilde that is the vector here right. Now, we have this in one hand right. See remember we have to prove something for the relative error. So, we have norm x minus x tilde. Now, we have to divide both sides by norm x that is what we have to do here. Now, you remember this is less than or equal to norm A into norm A inverse into b minus b tilde divided by b right, but here we have x, but we want b tilde here right. So, we have to somehow dominate this x by b here. How will you do that? Well, let us go back to our original system A x equal to b and take norm on both sides ok. Norm on both sides you will get norm b is equal to norm A x. Now, whenever you see this immediately our property should come in your mind and you can write it as norm A x is less than or equal to the matrix norm A into vector norm x right. That will immediately tell you that 1 by norm x what I am doing I am just taking this to the left hand side and that is less than or equal to norm A and this I am bringing to the right hand side and that will come as norm b. So, this is precisely what we were intent to do this is norm x divided by norm x was what we had in the previous inequality. Now, this part is less than or equal to norm A divided by norm b right. Therefore, you can write this is less than or equal to norm A inverse into norm b minus b tilde these are there already. Now, instead of 1 by norm x we will put this that is why I have this less than or equal to sign here I have norm A divided by norm b right. That gives me precisely what we want I will repeat once again what we are doing is we are taking this to the left hand side and b to the right hand side and writing this inequality from there we are getting what we want by putting this inequality into this. So, remember first we have divided by norm x on both sides and then we put this inequality here to get this and this is precisely what we want to show. So, this is a easy proof, but the result is very important which tells us that the relative error in the approximate solution when compared to the exact solution is amplified by this factor with respect to the relative error in b tilde when compared to b that is the error you have committed in your input data. So, remember this theorem is only considering the error in the right hand side vector whereas, the coefficient matrix is taken exactly, but in practical situations we will have error even in the coefficient matrix. So, let us see how the estimate for the relative error in x tilde will look like when compared to the relative error in the coefficient matrix and that is given by this inequality. Here also you can see that this expression is coming as the factor here in the previous one also the same expression was there. So, that shows that this quantity that is norm A into norm A inverse is very important in understanding the relative error in the approximate solution of your linear system when compared to the exact solution ok. So, now how to prove this theorem well it is not very difficult we can prove it with a little mathematical manipulations. Let me quickly tell you you can fill up the gaps you start with this A x minus A tilde x tilde this is how we have started our proof in the last theorem also, but now you see you do not have any error in your right hand side vector right. You are now considering in this theorem that the right hand side vector is exact, but you have error only in the coefficient matrix ok. In the previous case it was the other way round in fact, in practical situations we have error in both these inputs, but for the sake of simplicity we are considering these two cases separately and getting the inequalities. You can also combine and get any inequality, but that you can do it once you understand these two steps therefore, I am only doing these two steps separately ok. Let us start with this since the right hand side vector is same you will have A x minus A tilde x tilde equal to 0 right. Now what you do is you add and subtract A x tilde that is A x which is already there minus A x tilde which you newly insert into this equation then you add the same quantity and then you take A tilde x tilde which is already there equal to 0 right. That can be written as A into x minus x tilde equal to A minus A tilde there is a minus sign here because you are taking to the other side and x tilde right. That implies since A is invertible x minus x tilde is equal to minus A inverse into A minus A tilde x tilde. Now you take norm on both sides. The remaining part of the proof will go more or less in a similar way as we did in the previous theorem. I will leave it to you as an exercise to complete this. Now you see the important point as I mentioned this quantity seems to be very important in quantifying at least as an upper bound for the relative error. Therefore, we will take this quantity and give the name as condition number of the matrix A. So, we will define norm A into norm A inverse as the condition number of a given n cross n invertible matrix. We generally use the notation kappa of A for this. So, let me summarize when we have the error only on the right hand side vector that is B is taken as B tilde with some error. Then the relative error in the corresponding solution of the linear system will have such an estimate and similarly if you have the error in the coefficient matrix that is instead of A if you take an approximation to A which is denoted by A tilde then the corresponding relative error in x tilde when compared to x is given by this estimate. In both the case the amplification factor is nothing but the condition number kappa of A. Now you can see that if kappa of A is very small for a given matrix A then if you assume that this relative error, equivalently this relative error is very small then the upper bound is going to be very small. Generally it is fair to assume that this relative error in the input data is very small because that comes as most probably the rounding error on your computer or it may be some small error that we may make somehow. Therefore, these relative errors may be assumed to be very small. Therefore, if the condition number of your coefficient matrix is very small then the relative error in the approximate solution will also be very small. On the other hand if the condition number is very large, so large that this product becomes a big number then what happens it gives us a possibility that your relative error in the approximate solution may also be very large. It will not tell that the relative error in the approximate solution will be surely large, but it gives us a possibility that this may be large. So that is a kind of caution that this condition number will give us. If your condition number is very large then the computation of the solution of a linear system may be very sensitive to even small error in your input data that is what the message we are getting through this analysis. Here we should note very importantly that the condition number of a matrix depends on the subordinate norm that we use that is we will first decide what vector norm we will be using and from there we will have a subordinate norm corresponding to that vector norm and the condition number is then calculated with respect to that matrix norm. Generally we will use infinite norm because that is the easiest norm although it is not physically realistic, but it is easy to handle infinite norm rather than the L 2 norm. So in all our analysis we will fix L infinity norm and compute the condition number of a given matrix. Let us take an example consider this system where the right hand side vector is 0.7 and 1. The corresponding solution is 0.1. Now what we will do is we will make a slight perturbation in the right hand side vector thereby we will consider b tilde as our right hand side which is given by 0.69, 1.01. So this is slightly different from the right hand side vectors given here right. So we have made a slight error in our right hand side vector therefore we are considering this system, but we are actually intended to solve this system and get this solution, but we are actually solving this system and therefore we are getting its solution which is given by minus 0.17 and 0.22. Let us compute the relative error in x tilde when compared to x and see how it looks like. Now coming to precise computation like this we have to choose one particular norm to do this computation. As I told you it is very easy to handle L infinity norm therefore I will take the relative error in x tilde with respect to the infinite norm. Remember infinite norm of a vector is defined as maximum of the components of the vector its absolute value. So this is how it comes. Now if you take norm x minus x tilde infinity that is going to be 0 minus of minus 1.75 therefore its first component is 0.17 sorry it is 1.7 not 7.5 and the second one is 0.1 minus 0.22 that will give us 0.12 and therefore this is 0.17. Similarly infinite norm of x you can see that it is 0.1. So the relative error in x tilde when compared to x is given by 1.7. Now let us see what is the relative error in the right hand side vector. The right hand side vector relative error is 0.01. Now you see in percentage you have made only 1 percent error in the input data that gave us 170 percent error in the solution. So that seems to be very bad therefore the coefficient matrix in this system is very sensitive to the input data right. Let us see what is the condition number of this coefficient matrix. The condition number of the coefficient matrix is 289. How I computed this number? Well I have taken the coefficient matrix as 57710 and now I have to find the subordinate number matrix norm of A. What is that subordinate matrix norm? That is subordinate to the infinite norm in the last class I have given the formula for this. So you go back to our previous lecture and get the formula for this and compute this and also you find the inverse of this matrix and similarly you can find the subordinate matrix norm of the inverse of this matrix and find this number also and then multiply both of them because the condition number is nothing but norm A. Here we are taking infinite norm therefore you have to use that formula into A inverse again infinite norm of A inverse that will happen to be 289. So you can see that the condition number of this matrix is pretty large it is 289 and therefore when you are working with a matrix whose condition number is very large then you have to be very very careful because even if you make small error in your input here we made a error in the right hand side vector. Similarly you may make a error in the coefficient matrix also accordingly it will also magnify the relative error in the approximate solution right. A matrix with a large condition number is said to be ill conditioned whereas a matrix with small condition number is said to be well conditioned. Such concepts were also introduced in the case of evaluating a function a c 1 function at some point in one of our previous classes right. So here the words large and small is in general very difficult for us to quantify how large it is and how small it is because it depends on many factors more importantly it depends on the computational power a more powerful computer can handle a very bad matrices relatively better than a small computer also it depends on what kind of applications that we are working in. Therefore, in general we cannot quantify what is large and what is small. Let us recall a very famous matrix called Hilbert matrix which comes quite often in many mathematical models just with n equal to 4 one can see that the condition number of the Hilbert matrix H 4 is something like 28000. Now you can imagine if a matrix of condition number is amplifying the error in the solution so much what will happen to such matrices. This really tells us how serious it is for us to understand and then work with the matrices just going blindly with matrices and their computation is very dangerous on a computer right. Before ending this lecture let us prove an important theorem which tells us how one can identify a bad matrix that is a matrix with large condition number. Generally it may not be possible for us to use this theorem in practical situations however it is very interesting from the mathematical point of view. The theorem says that let us take a non singular matrix A then for any singular matrix B you have 1 by condition number of A is less than or equal to the relative error in B when compared to A. Therefore if A is very close to B you can see that the relative error is going to be very small right and that will make the reciprocal of the condition number to be very small it means the condition number is very big. So, this theorem says that if you are working with a matrix which may be non singular but if it is very close to a singular matrix in this sense then the condition number of your original non singular matrix will be very large that is what the theorem says. Let us quickly go through the proof of this theorem you take the left hand side 1 by kappa of A I am just putting the definition of the condition number. Now I will fix this term and apply the subordinate matrix norm definition only for the second one that is norm A inverse if you recall from our last lecture this is a definition of the subordinate matrix norm. And now I will simply take this instead of this maximum I will take a arbitrary vector y then what happens I will just forget this maximum therefore, I will have less than or equal to remember norm A inverse y divided by norm y is less than or equal to maximum of this y not equal to 0 of the same quantity. Therefore, 1 by this will be 1 by this but the inequality is reversed that is what I have written here. So, we got this inequality for our condition number let us see how to go ahead with it I will just put y is equal to A z just to make this term to look little better here thereby I have 1 by norm A into this goes to the numerator I have A z divided by A inverse y is now written as z right. So, this holds for any z that is what we have seen. Now, you take a non-zero z such that B z equal to 0 how is that possible well you go back to the theorem and see B is a singular matrix therefore, this is possible that is you can find a non-zero vector z such that B z equal to 0. Once you have such a z remember this inequality holds for any z I will in particular take my z such that B z equal to 0 therefore, I can write this term as norm of A z minus B z right there is no harm in writing it because this is 0. So, I will write 1 by cop of A is less than equal to A minus B z divided by this this is there. Now, I will use wherever you see norm A x kind of expression you should immediately remember our property of the subordinate matrix norm you put that here this is less than or equal to norm of A minus B into norm of z right now z z gets cancelled and you got what you want and this is what the theorem says let us see quickly an example for this. Let us take the matrix B which is equal to 1 1 1 1 which is not an invertible matrix. Now, let us put up the matrix B little bit by adding epsilon to some of its terms and call this as A as long as epsilon is greater than 0 this is a non singular matrix therefore, you can find A inverse for this and now what is norm A with respect to the subordinate matrix norm to L infinity norm that is given by this quantity you just go back to our previous lecture recall what is the formula for this come back and apply it to this matrix you will get this answer and similarly norm A inverse is given by this therefore, the condition number of the matrix A is given like this which is in our case is given by this expression and that can be greater than 4 by epsilon square. Now, you see if you take epsilon very small you can see that the condition number of the matrix A is going to be very large right. So, that is what is given in the previous theorem also we are just cross checking the theorem that we have proved just now in this example just to be more precise let us take epsilon to be something like 0.01 and that leads to your condition number of the matrix A has something like 40,000 you can also see that your matrix A is pretty close to the singular matrix B whenever epsilon is very small that is what is written here and consequently if you just take epsilon going to 0 you can see that the condition number of A will in fact, tend to infinity right. So, this tells us that if you are working with a matrix which is very close to singular matrix in the sense that is defined in the previous theorem then you are likely to have very sensitive computation for your input data. So, in this lecture we have understood what is mean by condition number of a matrix and how to learn the sensitivity of the matrix in computing solution of a linear system in terms of condition number of a matrix. Thanks for your attention.