 In our last lecture, we defined vector norm and induced matrix norm. So, here in today's lecture, whenever I talk of vector norm, unless it is specified otherwise, it is going to be either one norm, two norm or infinity norm and matrix norm is going to be always the corresponding induced matrix norm. We are considering solution of system of linear equations. We have seen methods to solve these. So, these methods, they are exact methods. The error is going to be because we do the computations using computer. So, you have got finite precision and that is what introduces the error. So, today we want to look at the error in the computation. So, we will start with a system A x is equal to b because of finite precision. The matrix A, the coefficient matrix A in fact is going to be A plus delta A. Each entry of the matrix can have some error. Similarly, on the right hand side b, instead of the vector b, we will be looking at vector b plus delta b. So, you have exact solution x. Your computed solution is going to be x cap. So, we will like to look at the error norm of x minus x cap. Now, that is going to be absolute error. So, it is more appropriate to look at the relative error. So, our assumption is A x is equal to b. A is in binary matrix which is invertible matrix and the right hand side is a non-zero vector. So, as a consequence our exact solution x is also going to be non-zero. In practice, you are going to solve A plus delta A x cap is equal to b plus delta b and our aim is to bound x minus x cap its norm divided by normic. And as I mentioned the norm is going to be either one norm or two norm or is going to be infinity norm. Now, what we are going to do is in order to simplify our analysis, first we will assume that there is only error in the right hand side. Next, we will consider the case when there is error also in the coefficient matrix delta A. So, this is for simplification. Finally, we are going to look at the general equation that is A plus delta A is the coefficient matrix right hand side is B plus delta B. But to start with exact equation is A x is equal to b and we look at only perturbation or error in the right hand side. So, we have the error delta B it is supposed to be a small error this is your computed error x cap. So, a computed solution x cap. So, let me write x cap is equal to x plus delta x where delta x is equal to x cap minus x. Now, this is our claim that norm delta x by norm x is less than or equal to norm A into norm A inverse into norm delta B by norm B. Norm delta B by norm B is the relative error in the right hand side norm delta x by norm x is the relative error in the computed solution. And we are going to show that norm delta x by norm x is less than or equal to norm delta B by norm B and it gets multiplied by this number norm A into norm A inverse. So, this number is going to play an important role that is going to be our condition number of the matrix. So, let us derive this formula that norm delta x by norm x is less than or equal to norm A into norm A inverse into norm delta B by norm B. So, we have our A x is equal to B then A x plus delta x is equal to B plus delta B. So, using the fact that A x is equal to B you get A delta x is equal to delta B. Now, look at norm B norm B is equal to norm A x norm A x is less than or equal to norm A into norm x. So, it is here that I use the fact that we are using the induced matrix norm that is why we have got this fundamental inequality. Now, from here you get 1 upon norm x to be less than or equal to norm A divided by norm B. Our assumption is norm B is not equal to 0 that implies x not equal to 0. So, we make sure that we are not dividing by 0. This equation A delta x is equal to delta B. We write as delta x is equal to A inverse delta B then take the norm of both sides and again use the fundamental inequality. So, you will get norm A inverse into norm delta B. Now, combine the equation 1 and 2 to obtain norm delta x by norm x is less than or equal to norm A into norm A inverse into norm delta B by norm B. Norm A into norm A inverse that is called a condition number and if this condition number if it is small. So, we will specify what we mean by small then norm delta B by norm B small will also imply that norm delta x by norm x is not too big and which will mean that the system is not overly sensitive to perturbations in B. First thing we are going to show is that condition number of a matrix A is always going to be bigger than or equal to 1. Condition number is defined for invertible matrices. It is defined as norm A into norm A inverse. We look at induced matrix norm. So, the induced matrix norm has got consistency condition that means norm A B is less than or equal to norm A into norm B. So, we will make use of this relation and also the fact that if you are looking at induced matrix norm then norm of the identity matrix is going to be equal to 1. So, using these two results we will show that condition number is always bigger than or equal to 1 and this is something consistent that you are starting with some error norm delta B by norm B. You are doing some computations. So, this result or the error we start with if at all it will get increased it cannot reduce. So, norm delta x by norm x is less than or equal to some number which is bigger than or equal to 1 into norm delta B by norm B. So, the best result will be when your matrix has condition number to be equal to 1. Then norm delta x by norm x will be less than or equal to norm delta B by norm B. So, let us first show that condition number is always bigger than or equal to 1. So, we are looking at the induced matrix norm. So, norm A is maximum of norm A x divided by norm x x not equal to 0. Here norm A x will be a vector norm you can keep in mind that it is either 1 norm or 2 norm or infinity norm. So, if I look at identity matrix then you have got norm i is equal to maximum norm x by norm x x not equal to 0. So, it is going to be equal to 1. So, this is the first result and then you have got 1 is equal to norm identity which is equal to norm A inverse which is less than or equal to norm A into norm A inverse. So, that is the consistency condition and this is our condition number of A. So, the best possible condition number is going to be kappa A is equal to 1. Norm of identity matrix is equal to 1 that is true for induced matrix norm. If you consider some other norm then this result may not be true. Like remember we have defined Frobenius norm. So, the Frobenius norm was defined as norm of A f is equal to summation i goes from 1 to n summation j goes from 1 to n A i j square whole thing raise to half. So, this Frobenius norm or it is like for the vector that is the Euclidean norm. And here you treat A as a vector of length n square then you have got this Frobenius norm. When you look at the identity matrix it is 1 along the diagonal and 0 elsewhere. So, norm i Frobenius norm is going to be square root of n whereas, if you consider norm i 2 which is equal to norm i 1 which is equal to norm i infinity these are going to be equal to 1. So, if you consider the condition number with respect to the Frobenius norm that means norm A f norm A inverse f this is going to be bigger than or equal to root n. Let us look at some examples of matrices for which the condition number is going to be equal to 1 that is the best possible condition number. So, I am going to look at two examples. So, one is orthogonal matrix. So, the orthogonal matrix has the property that if I call it matrix Q then it is Q transpose Q is equal to identity. So, we are going to show that condition number with respect to two norm of orthogonal matrix is going to be equal to 1. And then we will look at permutation matrices. So, the permutation matrix is obtained from the identity matrix by finite number of row interchanges. So, let us first look at orthogonal matrix. So, let me recall that on R n we have got we have defined inner product. So, inner product of x with y vectors in R n it is defined as y transpose x. So, it is summation j goes from 1 to n x j y j then norm x 2 is the norm induced by this inner product. So, it is inner product of x with x and it is positive square root. So, it is x transpose x raise to half this is going to be equal to summation j goes from 1 to n x j square whole thing raise to half. Look at matrix Q which is of size n by n and which satisfies the property that Q transpose Q is equal to identity. Then our claim is that condition number of this orthogonal matrix with respect to two norm it is going to be equal to 1. So, for a orthogonal matrix if you consider condition number with respect to some other norm like our one norm infinity norm then that need not be equal to 1. But if you are looking at the induced matrix norm induced by Euclidean vector norm then the condition number is going to be equal to 1. Why are these matrices with condition numbers equal to 1 important because they do not augment the error you start with. You start with error norm delta b by norm b in the computed solution the error is 0. So, it is less than or equal to the error you start with. So, that is why if you can work with orthogonal matrices or if you can work with matrices with condition number is equal to 1 that will be best computationally. Now, in order to show that condition number of orthogonal matrix is equal to 1 we will first show that two norm of orthogonal matrix is equal to 1 and from that it will follow that the condition number is equal to 1 and then we will consider an example of a orthogonal matrix. So, look at norm of q x two norm square that is equal to inner product of q x with itself by definition of inner product this is q x transpose into q x. Now, q is equal to x transpose into q into x it is transpose is equal to x transpose q transpose the order gets reversed. So, you have x transpose q transpose q x q transpose q is equal to identity. So, that means this quantity is equal to x transpose x, but x transpose x is nothing but Euclidean norm of x square. So, thus for a non zero vector x norm q x by norm x is going to be equal to 1 and what is the matrix norm it is maximum of such quotients over all possible x not equal to 0 vector, but the quotient is always equal to 1. So, that will give us norm q two is equal to 1 if q is orthogonal matrix q transpose also is orthogonal because look at q transpose it is transpose into q transpose that will be nothing but q q transpose and orthogonal matrix is such that q transpose q is identity q q transpose is identity and hence if q is orthogonal q transpose is also orthogonal then by this result you get norm q transpose it is two norm is equal to 1, but orthogonal matrix means inverse of q is nothing but q transpose. So, thus norm of q inverse it is two norm is equal to 1 and then the condition number with respect to two norm of q which is norm q multiplied by norm q inverse. So, that is going to be equal to 1. So, now let us look at an example of a orthogonal matrix. So, I am going to look at two by two matrix and for that matrix we will also calculate its one norm and infinity norm. In our last lecture we had seen that for one norm and for infinity norm you have got a formula when you want to look at one norm then that is nothing but the column sum norm. So, if you want to look at each column take the moduli of the entry sum it up. So, like that you will get n numbers look at the maximum and that is a one norm. If you want to look at infinity norm then it is the row sum norm that means what we did for columns you have to do it for rows. So, in each row you consider the moduli of the entries sum it up there are n rows. So, you are going to get n numbers the maximum among that is going to be your infinity norm. On the other hand for two norm we do not have a such a formula we have to be satisfied only with an upper bound and that upper bound is the Frobenius norm. For orthogonal matrix we have seen that two norm of the matrix Q is going to be equal to 1. So, here is an example of two by two orthogonal matrix it is 1 by root 2 1 by root 2 first column 1 by root 2 minus 1 by root 2 is equal to second column it is a symmetric matrix. So, Q transpose is equal to Q and you can verify that Q square is equal to identity matrix because it is a orthogonal matrix norm Q 2 is equal to 1. On the other hand when you want to consider norm Q 1 look at the first column sum of the elements is going to be root 2 the second one also the sum of the moduli. So, modulus is important that is again root 2. So, norm Q 1 is going to be equal to root 2 and norm Q infinity where you have to look at the rows that also gives you root 2 the matrix is symmetric. So, this is for orthogonal matrix now we will look at permutation matrix. So, you start with identity matrix and to start with you interchange only two rows any two rows you interchange the matrix which you get it is going to have property that p square will be equal to identity because you have identity matrix you interchange two rows. Now, if you interchange these two rows again then you are going to get back to your identity matrix. So, your p square is going to be equal to identity matrix and your matrix p is going to be a symmetric matrix. So, p transpose is equal to p. So, this matrix being orthogonal matrix its two norm is going to be equal to 1 because that we have proved just now that for any orthogonal matrix the two norm is going to be equal to 1. Then you look at one norm. So, it is a special matrix we had identity matrix and then we are interchanging the rows look at the matrix. So, it is this is my starting matrix now what I am doing is I am interchanging two rows. So, let us interchange first row and third row for example. So, the third row is 0 0 1 and then remaining entries 0. The second row will remain as it is 0 1 0 0 and third row we are interchanging first and third row. So, third row now becomes 1 0 0 and then remaining entries 0 and remaining part of the matrix is remaining unchanged. So, here you have all 0. So, this is my matrix p. Now, if I want to look at its one norm I have to look at its the columns of norm. Now, in each column there is only one entry which is non-zero and that is equal to 1 1. So, norm p 1 is equal to 1 norm p infinity also will be equal to 1 because in each row you have got only 1 non-zero entry and that is equal to 1. You have p square is equal to identity because if you interchange again the same rows you are going to get back the identity. So, you have norm p 2 is also equal to 1. Now, p inverse is equal to p because p square is identity and that gives us condition number of p with respect to one norm or two norm or infinity norm is equal to 1. So, this permutation norm is equal to 1. So, this permutation matrices they came into picture when we wanted to look at Gauss elimination with partial pivoting. So, there at each stage we may need to interchange the rows. These interchange in rows they were accounted for by pre multiplying your matrix by a permutation matrix. So, by a matrix obtained from the identity matrix by interchange of one row. Then what we wanted to do was take all permutation matrices together and take all lower triangular matrices together. So, that we did it using the fact that our p square is equal to identity and then we got the result that Gauss elimination with partial pivoting it amounts to a l u decomposition of matrix not a, but p into a where p is going to be product of these matrices p 1 p 2 p n. So, let me just look at two such matrices. So, I have got p 1. So, this is obtained from the identity matrix by one row interchange p 2 is again a matrix obtained by from identity by one row interchange. These two rows they will be now different. So, we have got p 1 square is equal to identity p 2 square is equal to identity. Let us look at matrix p 1 p 2 and then we also have p 1 transpose is equal to p p 1 p 2 transpose is equal to p 2. So, I look at matrix p 1 p 2 I take its transpose and then I look at p 1 p 2 this is going to be equal to p 2 transpose p 1 transpose p 1 p 2, but p 1 transpose p 1 is going to be identity because p 1 square is identity and p 1 transpose is equal to p 1. So, you have got p 2 transpose p 2 and p 2 transpose p 2 is nothing but p 2 square. So, it is identity and which will mean that p 1 p 2 is orthogonal matrix. So, the condition number of p 1 p 2 will also be equal to 1 and in this case even the condition number with respect to one norm or infinity norm is going to be equal to 1. Let us find a lower bound it is not always possible, but in this case because see what the result which we are getting is norm delta x by norm x is less than or equal to condition number of a into norm delta b by norm b. So, this less than or equal to it means that may be condition number of a into norm delta b by norm b that product is big, but still norm delta x by norm x can be small. So, what is ideal is to have an upper bound and also an lower bound. Now, in numerical analysis such a thing is not always possible. In fact, most of the time you have to be satisfied with only a upper bound and then one tries to give the upper bounds which are tight or which are the best possible and so on. So, now we are going to look at the lower bound in the case when a x is equal to b and a x plus delta x is equal to b plus delta b. We have already found an upper bound and now we are going to find lower bound the method is similar. So, you have a x is equal to b that means x is equal to a inverse b. So, that gives you norm x to be less than or equal to norm a inverse into norm b. From this we will get 1 upon norm a inverse into norm b to be less than or equal to 1 upon norm x. Then a delta x is equal to delta b. So, norm delta b will be less than or equal to norm a into norm delta x and then that gives you norm delta b by norm a to be less than or equal to norm delta x. When we obtained upper bound we had done other way. We had written norm b to be less than or equal to norm a into norm x and delta x to be equal to a inverse delta b. Now, we apply the fundamental inequality to x is equal to a inverse b and a delta x is equal to delta b and then combine these two results. So, you will get norm delta x by norm x the relative error in the computed solution to be bigger than or equal to norm delta b by norm b from here and then 1 upon norm a into norm a inverse. This upper bound we had already found. So, here condition number of a is in the numerator, here condition number of a is in the denominator. If the condition number of a is equal to 1, then you are going to have norm delta x by norm x is equal to norm delta b by norm b. So, this is the best situation possible that you start with some error and you do computations. So, we are going to perform various operations, but then in the end the relative error in the computed solution is the same as the relative error in the relative error which we started with. So, this is for this a x is equal to b and only the perturbation in the right hand side. Now, let us look at a example. So, this condition number of a norm a into norm a inverse if it is big. So, afterwards we will see what we mean by big and small, but I want to give you an example where the condition number is of the order of 10 raise to 6. So, it is really a big number and then you perturbed the right hand side slightly. So, your norm delta b by norm b is going to be small, but the error in the computed solution it becomes very big because of your big condition number. So, look at this example a is a 2 by 2 matrix, this matrix its inverse is given by minus 900 raise to 6. So, it is a 999, then 999 and then minus 1000. So, when you look at norm a 1. So, look at the first column, look at the second column, add up the entries, look at the maximum. So, norm a 1 is going to be 1999. Then when you consider infinity norm you have to do it with the rows. So, again norm a infinity is 1999. Norm a inverse its one norm and infinity norm they are the same. So, when you consider the condition number you have to look at norm a into norm a inverse. So, that means it is going to be 1999 square. So, that is approximately equal to 4 into 10 raise to 6 or to be more precise 3.996 into 10 raise to 6. Now, you look at this system we choose right hand side. So, that the exact solution is 1 1, if x 1 is equal to 1, x 2 is equal to 1 you are going to get the first entry to be 1999, second entry to be 1997. Per tub system has right hand side to be 1998.99 and the second one is 1997.01. So, it is a very small perturbation. So, your delta b is going to be equal to minus 0.01 and 0.01. So, very small number when you solve the perturb system. So, your exact solution is 1 1 and you are changing the right hand slightly and computed solution is 1 1. So, your exact solution which you are getting it is 20.97 and minus 18.99. So, your exact solution is 1 1 and you are getting the solution which has nothing to do with your earlier solution. You were starting error was 5.0 into 10 raise to minus 6 and now your error is not 0.1. So, norm delta x infinity by norm x infinity is 19.99. So, this situation this is something one has to be careful about that you want to solve some system. Now, if it perturbs slightly and that is bound to happen you are using computer. So, there is going to be always some error. Now, whether this error gets magnified by such an amount that you get a completely different solution. So, when we looked at our error norm delta x by norm x it was about 20. That means the error is of the same order as your solution. So, that is why the big condition number that is something one has to worry about. Here remember there is no approximation the 2 by 2 matrix you are doing exact calculations, but there is a error which is not acceptable. Now, what we could have done in this case the matrix A is given to us. So, if the matrix is given to us its condition number is fixed. So, what can I do I have I have been given a system I solve it and then it happens. So, if such is the case then you cannot do much, but what you have to be careful is your starting matrix is well conditioned, but in the process you are making it to be ill conditioned. So, that is what happens when you divide by a small number. So, we are considering Gauss elimination method in that Gauss elimination method when you determine the multipliers you consider A 2 1 by A 1 1 that was our multiplier M 2 1. So, if you divide by a small number your starting system is well conditioned, but you are making it ill conditioned. So, this is something you should avoid another is there are some ways to make the matrix to be well conditioned it may not work for all matrices, but for some matrices it will work. So, that part we will see how to do it. So, here it was I wanted to give you an example where the condition number is big and that affects your solution in such a manner that you get a completely unacceptable solution. So, now we had looked at perturbation only in the right hand side. Now, let us look perturbation in the coefficient matrix actually the most general condition is you will you are going to have perturbation in the coefficient matrix you are going to have perturbation in the right hand side. What I am going to do is now consider only perturbation in the coefficient matrix with right hand side to be exact obtain the result and then state the result in the general case. The methodology is the same it is just question of writing it down. So, now A x is equal to B is our original system the perturb system is going to be A plus delta A x is equal to B our assumption is that A is invertible. So, first we have to make sure that our perturbed system the coefficient matrix A plus delta A is going to be an invertible matrix. So, we are going to find a condition on delta A which will imply that A plus delta A is going to be invertible. These perturbations whether it is in the right hand side B or whether it is in the coefficient matrix they are supposed to be small perturbations. If you change your right hand side drastically then of course, the solution which you get is going to be something totally different then we do not expect it to have any relation with our original solution. So, what we are doing is when you are perturbing it slightly how does the computed solution behave. So, we are first going to find a condition on delta A which will guarantee that A invertible implies A plus delta A invertible. So, for that we are going to prove a result that if a matrix C has norm less than 1 now which norm any induced matrix norm. So, if norm C is less than 1 then I plus C is invertible. So, this is the first result we will prove and then we will prove then we will obtain a sufficient condition for invertibility of A plus delta A. So, C is an n by n matrix such that norm of C is less than 1 I want to show that I plus C is invertible. So, I will show that it is 1 to 1 in order to show that it is 1 to 1 consider I plus C x is equal to 0 vector that will mean that x is equal to minus C x take norm of both the sides. So, you will have norm x is equal to norm C x and by fundamental inequality it is less than or equal to norm C into norm x that is why I said that consider any induced matrix norm. Now, here norm of C is less than 1. So, this relation implies that norm x has to be equal to 0 vector because if norm x is not equal to 0 then I can say that 1 is less than or equal to norm C cancel norm x, but then norm C is less than 1 and I am getting 1 less than or equal to norm C. So, contradiction. So, norm x has to be equal to 0 not a 0 vector it should be norm x is equal to 0 it is a scalar and this implies that x is equal to 0 vector by property of norm. So, I plus C is 1 to 1 and hence it is invertible. Now, we use this fact to obtain a sufficient condition on delta a which will guarantee invertibility of a plus delta a. So, let a be invertible and delta a be such that norm delta a by norm a is less than 1 upon condition number that means 1 upon norm a into norm a inverse. So, this condition I can write or it will imply that norm delta a into norm a inverse is less than 1. So, we have norm delta a to be less than norm a inverse we have norm delta a into norm a inverse to be less than 1. So, actually norm delta a is less than 1 upon norm a inverse. Now, I look at a plus delta a this will be identity plus delta a into a inverse a I take a common I know that this is invertible this is my matrix C. So, what is norm of C? Norm of C is norm of delta a into a inverse which is less than or equal to norm delta a into norm a inverse by consistency condition and then from this condition you get this to be less than 1. So, you will get i plus c to be invertible and a is already invertible. So, a plus delta a inverse will be equal to a inverse into i plus c inverse using the fact that if a and b are invertible matrices a b inverse is equal to b inverse a inverse. We look at the perturbation in the coefficient matrix a and we assume that norm delta a is less than 1 upon norm a inverse or equivalently norm delta a by norm a is less than 1 upon condition number of a. So, norm delta a by norm a it is the relative error in the coefficient matrix. So, norm delta a by norm a it is the relative error in the coefficient matrix. And we assume that it is less than 1 upon condition number of a and that will guarantee that a plus delta a is invertible. So, a x is equal to b a plus delta a x plus delta x is equal to b then a x is equal to b will give us a times delta x is equal to. So, a x is equal to b. So, that will get cancelled and then a delta x is equal to minus delta a into x plus delta x taking this on the other side. Now, a is invertible. So, take a on the other side and then you will get norm delta x to be less than or equal to norm a inverse norm delta a using the triangle inequality here norm x plus norm delta x. Now, I am multiply and divide by norm a and then write norm a into norm a inverse to be condition number. So, I have condition number of a delta a by norm a norm delta a by norm a norm x plus norm delta x. Whatever is related to delta x I take it on the other side. So, I will get 1 minus condition number of a norm delta a by norm a into norm delta x to be less than or equal to condition number of a norm delta a by norm a into norm x that will give us norm delta x by norm x to be less than or equal to this condition number of a this bracket and then norm delta a by norm a. Remember that this quantity we are assuming it to be less than 1. So, even in this case what is going to matter is the condition number. You have got the relative error is less than or equal to condition number of a divided by something into norm delta a by norm a. So, tomorrow we will consider some conditions or we are not going to calculate the condition numbers. So, by looking at a matrix we should be able to sort of guess or we should have some way of knowing whether it is ill conditioned or well conditioned. So, some geometric picture these are the things we will do in our next lecture. So, thank you.