 We are considering perturbed systems and today we are going to study ill conditioned matrices in more detail. Yesterday we had considered first the perturbed system A of x plus delta x is equal to B plus delta B that means only there was perturbation in the right hand side then we considered the system when there is perturbation only in the coefficient matrix. Today I am going to state the result for the perturbed system when there will be perturbation in the coefficient matrix and also in the right hand side. I am just going to state the result because the proof is something similar to what we have done before and then after that we will take up the study of ill conditioned matrices. So, our exact system is A x is equal to B if there is perturbation only in the right hand side then we proved that the relative error norm delta x by norm x is less than or equal to condition number of A into norm delta A by norm A where the condition number is norm A into norm A inverse and our assumption is A is invertible operator right hand side B is not equal to 0. So, that will guarantee that x is also not equal to 0. So, you can divide by norm delta x here this norm delta A by norm A actually it should be norm delta B by norm B then when you consider perturbation in the coefficient matrix then you get norm delta x by norm x less than or equal to condition number of A and then 1 minus condition number of A into norm delta A by norm A and this is the perturbation norm delta A by norm A. For this result you assume that norm delta A by norm A is less than 1 upon condition number of A that guarantees that A plus delta A is going to be invertible and in the denominator 1 minus condition number of A into norm delta A by norm A will be a number bigger than 0. Now, look at the case when there is perturbation in the coefficient matrix perturbation in the right hand side the assumption will be norm delta A by norm A is less than 1 upon condition number of A. In this case the relative error norm delta x by norm x is less than or equal to condition number of A multiplied by relative error in the coefficient matrix plus relative error in the right hand side divided by 1 minus condition number of A norm delta A by norm A. So, if delta A is equal to 0 that means only perturbation in the right hand side then you will have condition number of A multiplied by norm delta A by norm A into norm delta B by norm B and denominator will be 1. So, we get back our earlier result if norm delta B is equal to 0 that means only perturbation in the coefficient matrix then you have condition number of A multiplied by norm delta A by norm A divided by 1 minus condition number of A into norm delta A by norm A and if the coefficient matrix and right hand side then this is the result. So, here what you have to notice is that the perturbation is small it is supposed to be small. So, norm delta A by norm A is going to be a small number. So, if the condition number is something reasonable then the denominator is going to be approximately equal to 1. In the numerator you have got condition number multiplied by the relative error in the coefficient matrix plus relative error in the right hand side. So, this condition number is going to play a crucial role your starting error will get multiplied by this condition number. So, if the condition number is small then we say that the system is well conditioned if it is big then we say that it is ill conditioned. Last time we saw that the condition number is going to be bigger than or equal to 1. So, the least condition number is equal to 1. Now, when we say that condition number small condition number big there is no precise boundary it depends on your computer it depends on your starting error and it depends on how much accuracy you need. So, you have suppose your norm delta B by norm B is approximately equal to 10 raise to minus 4 and if your condition number is about 100 then norm delta x by norm x is less than or equal to 10 raise to minus 2. So, you are starting with the error to be 10 raise to minus 4 of the order of 10 raise to minus 4 after doing your computations then your error in the computed solution is going to be 10 raise to minus 2. So, this can be acceptable like if you are starting error is 10 raise to minus 6 then your error in the computed solution will be 10 raise to minus 4. So, this can be something acceptable condition number is of the order of 10 raise to 4 and your starting error is 10 raise to minus 4 then your relative error norm delta x by norm x it is going to be about 1 that means norm delta x is about the same as norm x. So, the error is of the same order as of your solution and then this generally it will not be acceptable. So, if the condition number is between 1 to 100 generally it is considered to be a well conditioned system otherwise it is ill conditioned. So, now let us see whether we can do something about the condition number our condition number is norm a into norm a inverse norm have property that if I look at norm of alpha times a it is going to be equal to mod alpha times norm a and if I multiply throughout by alpha that means a x is equal to b is our exact system and instead of that if I consider alpha times a x is equal to alpha times b where alpha is a non zero number then I do not alter my system and then I can make norm alpha a to be as small as I want. Unfortunately, this multiplication of the coefficient matrix a by a non zero number is not going to change the condition number because you have norm a into norm a inverse. So, when you look at norm alpha a you have to look at norm alpha a and you have to look at norm of alpha a inverse and then you are going to have exactly the same condition number. So, kappa a is norm a into norm a inverse alpha not equal to zero. So, kappa of alpha a is equal to norm alpha a into norm alpha a inverse this will be mod alpha times norm a alpha a inverse will be a inverse divided by alpha. So, that will be 1 upon mod alpha norm a inverse. So, mod alpha and 1 upon mod alpha will get cancelled and then you get exactly the same condition number. So, this trick of multiplying the system by a non zero number that is not going to work whether the condition number has something to do with the determinant like we know that matrix a is invertible provided determinant of a is not equal to zero. So, now, whether small determinant means ill conditioned system that means whether that will mean that the condition number is going to be big. So, that is not the case like the condition number of matrix has nothing to do with small determinant. So, we can look at a simple example of diagonal matrix. So, look at diagonal matrix with diagonal entry into a is to be epsilon and epsilon is greater than zero. Now, if you look at any induced matrix norm then norm a epsilon is equal to epsilon inverse of a epsilon is going to be diagonal matrix with diagonal entries as 1 by epsilon. So, norm a epsilon inverse will be 1 upon epsilon and that gives you condition number of a is equal to 1 the best condition number we can have. So, now, determinant of a epsilon is epsilon square I can make it as small as I wish by choosing my epsilon appropriately. So, so long as epsilon is greater than zero the condition number is going to be equal to 1 determinant of a epsilon will be epsilon square. So, I can make it as small as I wish. So, there is no connection between small determinant and ill conditioning. Now, the ill conditioning it can be because of poor scaling. So, what I mean is you look at a system where the coefficient matrix is 1 0 0 epsilon x 1 x 2 is equal to 1 epsilon. So, where system is x 1 is equal to 1 and epsilon x 2 is equal to epsilon. If I have epsilon to be very small compared to 1 then our a inverse is going to be 1 0 0 1 by epsilon. When I consider norm a that is going to be equal to 1 because epsilon is much smaller than 1 when you look at compare 1 and mod epsilon then 1 is going to be and here take epsilon to be bigger than 0. So, norm a will be 1 whereas, norm a inverse will be 1 by epsilon. So, that will give you condition number of a with respect to either 1 norm or infinity norm to be 1 by epsilon and which is going to be it will be big because you can choose epsilon to be small and then 1 by epsilon will become big. Now, in this equation if I multiply the second equation by 1 upon epsilon then what I get is the system 1 0 0 1 x 1 x 2 to be equal to 1 1. This is the well conditioned system in fact the condition number is going to be equal to 1 the best possible condition number. So, see there is a difference between multiplying the coefficient matrix by alpha and also right hand side by alpha and multiplying only one of the row by alpha you have system of linear equations. So, you have n equations in n unknowns if I multiply one of the equation by a non 0 constant. So, I multiply the left hand side as well as right hand side in that case my solution is not going to change. So, I will have the same solution as before, but as we saw just now it can change the condition number of your matrix. So, if you multiply one row or more than one row several rows by non 0 constants and you multiply the corresponding right hand side also by the same constant then your system remains the same the solution remains the same. But the condition number of your matrix can change, what if I multiply the columns like I look at one of the column and then I multiply by a constant then you are changing the system. But in tutorial problem we will see that this multiplying say j th column by a non 0 constant c its effect on the solution will be the corresponding component will get multiplied by 1 by c. So, let me repeat you have got coefficient matrix A you are looking at system A x is equal to B I look at j th column. I multiply j th column by a non 0 constant c. So, I am going to get a different system, but the solution of this system the new system will get affected only in the j th component all other components of the solution they will be same as the earlier solution and j th component will get multiplied by 1 by c. It is easy to show this and this we will do as a tutorial problem, but when you multiply a column by a non 0 constant it can change the condition number of your matrix. So, you have got a system A x is equal to B we are going to see that if your say columns or rows they are out of scale. That means, one of the column you take some norm we have decided that we are going to take either one norm, two norm or infinity norm. So, suppose one of the column has a big norm and you have got another column which has got small norm then your matrix is going to be ill condition. So, this is what like we are trying to understand what well condition matrix and what ill condition matrix means. So, it has nothing to do with small determinant, but it has something to do with the norms of the columns or norms of the rows. And then what one can do is multiply rows and multiply by columns by numbers. So, as to have the norms of all the columns and norms of all the rows to be about the same it may not be possible to do this always, but that is one of the way of making your system to be well conditioned. So, you have scaling A x is equal to B multiply an equation by c not equal to 0 you get the same solution this is known as row scaling C multiply j th column by c not equal to 0 solution is altered only in the j th component and the change is the j th component is multiplied by 1 upon c. So, this is our exercise and this is known as column scaling. Now, I define maximum magnification of A that is maximum of norm A x by norm x x not equal to 0. So, this is nothing, but our norm A induced matrix norm you define minimum magnification of A to be minimum of norm A x upon norm x x not equal to 0. So, minimum magnification of by A we are going to show in tutorial that condition number of A is equal to maximum magnification of A divided by minimum magnification of A. Now, what happens if minimum magnification of A is equal to 0 because I am looking at when I consider maximum magnification of A that means it is norm of A. We know that if your matrix A is a non-zero matrix then the norm of A has to be non-zero because if you get the maximum of norm A x by norm x x not equal to 0 vector is equal to 0 then it will mean that norm A x is 0 for all non-zero vectors. So, you look at the canonical vectors and then you conclude that A is a 0 matrix, but now it is something different. We are looking at minimum of norm A x by norm x x not equal to 0 vector. So, what if this minimum is equal to 0? Now, this minimum cannot be equal to 0 because our matrix A is invertible. In fact, we will show that minimum magnification of A is nothing but 1 upon norm A inverse and that will give us condition number of A to be equal to maximum magnification of A divided by minimum magnification of A. Now, using these ideas about maximum magnification and minimum magnification, we are going to show that if the columns of the matrix are out of scale that means one of the column has big norm as compared to some other column then such a matrix is going to be necessarily ill conditioned. So, let us show this now. So, you choose the vector norm to be either 1 norm, 2 norm infinity norm, condition number of A is going to be norm A into norm A inverse and we are going to show that norm A inverse is 1 upon minimum magnification. So, condition number of A is equal to norm A divided by minimum magnification of A. Since norm A is maximum of all these quotients, if I look at x is equal to E j, canonical vector with 1 at j th place and 0 elsewhere. So, I will have norm A to be bigger than or equal to norm A E j, but A E j is nothing but the j th column. So, you get norm C j when you consider minimum magnification of A that is minimum of norm A x by norm x x not equal to 0. So, this minimum magnification of A will be less than or equal to norm A E i which is equal to norm C i. So, thus condition number of A is bigger than or equal to norm C j by norm C i. So, j and i they can be any numbers like that they denote the j th column and i th column. We have proved that condition number of A is bigger than or equal to norm C j by norm C i. If j is equal to i, this result tells us that condition number of A is going to be bigger than or equal to 1. That result we have already proved, but suppose your norm C 1 is 10 cube. And your norm C 3 is 10 raise to minus 3, then your condition number will be bigger than or equal to 10 raise to 3 divided by 10 raise to minus 3. So, it is going to be bigger than or equal to 10 raise to 6. It will also be bigger than or equal to 10 raise to minus 3 divided by 10 raise to 3, but we know that we have better result. So, condition number of A is bigger than or equal to norm C j by norm C i, where C j denotes the j th column and j and i can take any values. So, if you have got one column with a big norm compared to norm of another column, your condition number is going to be big and then your system will be ill conditioned. Now, I am going to go back to our example of ill conditioned matrix which we had considered last time and for that matrix we will calculate or we will try to find out the direction of maximum magnification and the direction of minimum magnification. So, we have A to be a 2 by 2 matrix, 1,999, 999 and 998. Inverse of this matrix is given by diagonal entries to be minus 998 and minus 1000 of diagonal entries are 9999 and 999. So, these are symmetric matrices norm A infinity is equal to norm A inverse infinity is equal to 1999 and that gave us condition number to be of the order of 10 raise to 6. If you look at A of 1, 1 that is 1999 and 1997. So, norm of A x infinity divided by norm x infinity x is 1, 1. So, it is infinity norm is going to be 1, infinity norm of this is 1,999 and that is norm of A infinity. So, this 1, 1 it is going to be direction of maximum magnification by A look at A inverse of minus 1, 1 this you get it to be 1,997 and 1,999. So, norm A inverse x infinity will be maximum of these 2 entries. So, that is 1,999 and norm x infinity will be 1. So, it is norm A inverse infinity. So, this minus 1, 1 is going to be direction of maximum magnification by A inverse and this is going to be direction of minimum magnification by A. This equation tells you that A of this is going to be equal to minus 1, 1. So, 1, 1 is direction of maximum magnification by A and this is going to be direction of this 1,997 and 1,999. This is going to be direction of minimum magnification by A inverse. So, the example which we considered last time where the exact solution was 1, 1 you perturb it slightly and then you get a solution which is completely different than the original solution. So, this example was constructed to illustrate the effect of ill conditioned matrix. So, what one had done was this 1, 1 it was the direction of magnification maximum magnification by A. So, that is what we chose and in the right hand side the perturbation which we chose. So, the perturbation was 0.01 and minus 0.01. So, that is that was chosen in the direction of maximum magnification by A inverse. That is why we got such spectacular result. So, that was for the sake of illustration. So, now we are going to look at what does ill conditioning means geometrically. So, what we will do is we will look at a 2 by 2 system like consider two equations into unknowns and then we will try to interpret the ill conditioning of the system for this particular 2 by 2 system. So, we are not going to take an example, we will just take two equations into unknowns and then try to see what does ill conditioning means. If A is a ill conditioned matrix whose rows and columns are not severely out of scale, that means we take out this or we rule out the possibility that ill conditioning is coming because of the rows and columns which are out of scale. So, suppose this is not the case, they are approximately the same. Now, you normalize A so that norm A is equal to 1. We have seen that the condition number does not change if you multiply a matrix by a non-zero constant. So, without loss of generality I can assume that norm of A is going to be equal to 1. Now, suppose the condition number is much big. So, it is we write this as 1 is 2 times less than norm A. So, now condition number of A is norm A into norm A inverse. We have normalized A so that norm A is equal to 1. So, our condition number of A is equal to norm A inverse. So, that means our norm A inverse is going to be a big number, but what was norm A inverse is going to be 1 upon minimum magnification of A. So, we have got condition number of A which is equal to norm A inverse which is equal to 1 upon minimum magnification of A. So, this 1 upon minimum magnification of A is going to be much bigger. So, that means if you take its reciprocal minimum magnification of A will be much small. So, we have minimum magnification means you look at minimum of norm A x by norm x x naught equal to 0 vector. Now, if minimum magnification is small that means there will exist a vector such that x such that norm A x by norm x is going to be small. It will not be 0 because our matrix is invertible, but norm A x by norm x is going to be small. So, once again I normalize now we have vector x to be to 1. So, norm x is equal to 1. So, this minimum magnification of A is much small will mean that there exist a vector x of norm 1 for which norm A x is small or it means the vector A x is going to be about a 0 vector. So, minimum magnification of A less than or equal to 1. So, there exist x belonging to R n norm x is equal to 1 such that norm A x is much less than 1. E j's are our canonical vectors. So, we write our x belonging to R n it can be written as x is equal to x 1 e 1 plus x n e n where E j is canonical vector with 0 everywhere except 1 at j th place. Then A x is going to be equal to x 1 A e 1 plus x n A e n. Now, what is A e 1 it is going to be c 1 plus x n A e n that is n th column which is c n. So, this is about 0 0 vector and that means c 1 c 2 c n these are almost linearly d n dependent and norm x is equal to 1. So, linearly dependent will mean that you were there exist on a scalars x 1 x 2 x n such that x 1 c 1 plus x 2 c 2 plus x n c n is equal to 0 vector. Our matrix A is invertible. So, that means the columns are linearly independent, but then the condition number big means that the columns are about or almost linearly dependent. So, the ill conditioning has nothing to do with small determinant. So, we have to write it down, but it has to do something with linear independence and linear dependence of the columns. So, now we look at 2 by 2 system. So, it is A 1 1 x 1 plus A 1 2 x 2 is equal to b 1 and A 2 1 x 1 plus A 2 2 x 2 is equal to b 2. If you look at only one equation A 1 1 x 1 plus A 1 2 x 2 is equal to b 1, this is going to be an equation of a straight line. Then second equation also depend it is also an equation of a straight line. The straight line defined by the first equation this straight line will be perpendicular to the vector A 1 1 A 1 2. So, think about it the second line which is given by A 2 1 x 1 plus A 2 2 x 2 is equal to b 2. So, this second line is going to be perpendicular to A 2 1 A 2 2. So, we are looking at 2 equations in 2 unknowns each of the equation represents a straight line. Now, the x 1 x 2 which satisfy both the equations that means, your x 1 x 2 should lie on the first straight line and your x 1 x 2 also should lie on the second straight line. So, that means, the solution is given by intersection of these two straight lines. So, you can have two equations in two unknowns its solution is nothing but intersection of two straight lines. The first straight line is going to be perpendicular to vector A 1 1 A 1 2 and second straight line is going to be perpendicular to vector A 2 1 A 2 2. So, now, let us look at the straight line which are given by 1000 x 1 plus 999 x 2 is equal to b 1 and 999 x 1 plus 998 x 2 is equal to b 2. So, this is the same example which we had considered yesterday and today for the maximum magnification. So, now, the slopes of these two straight lines they are given by slope of the first straight line will be minus 1000 divided by 999 slope of the second straight line is given by minus 999 divided by 999 98. So, the first slope is about minus 1.001 and then 001 the second slope is minus 1.001 002. That means, the two straight lines they are nearly parallel. So, you have say the two straight lines they are shown by solid state line. So, these are our straight lines then when you perturb. So, if instead of b 1 I consider some b 1 plus delta b 1 then what I am doing is I am looking at a parallel straight line I am not changing the slope the slope is determined by 1000 and 999. But, if I change the right hand side I am looking at a parallel straight line. So, the first equation or the first system of equation its solution was given by intersection of these two straight lines. So, this is going to be your original solution then now you are looking at you are taking instead of this straight line you are looking at this parallel straight line which is obtained by perturbing b 1 slightly. So, now, I have to look at intersection of this new dotted straight line and then the other straight line. So, this and I look at the intersection so, this is the original intersection and this is the new intersection. So, there is lot of change and that is the geometric interpretation of ill conditioning that we had two straight lines almost parallel. So, when I take slight change in the right hand side it changes my solution completely. So, when we look at this matrix the example which we have considered which has 199999 and 998 when you look at their columns then the columns their one norms or their infinity norms they are they do not differ much same thing for the rows it is a symmetric matrix. So, the ill conditioning is not coming from the scaling problem, but it came from the fact that it is two rows and it is two rows and it is two rows and two columns or two columns they are almost linearly dependent. So, let me summarize what we have learnt about the ill conditioning today that the condition number will not change if I just multiply the coefficient matrix by a number alpha it will change if I multiply only some of the rows by non zero constants and similarly for the columns. Then if the rows or columns if they are out of scale then they are going to make your system to be ill conditioned and ill conditioning has nothing to do with small determinant, but it has it has a relation with linear independence or linear dependence of columns. So, now you start with a system A x is equal to b what you can try to do is try to make the columns and the rows to have about the same norm you do as much as it is possible. So, that is the column scaling and the column scaling is not possible. So, that is the column scaling and then row scaling after that the system is given to you. So, you do not have much control over it, but at least you should know that you are solving some system of equations you are getting approximate solution. So, then how reliable that solution is that means how near it is to the exact solution, because we are interested in the exact solution this is our limitation that we have to use computer. So, then you are going to solve a perturbed system, but then at least one should know that what is the limitation of my method this much one can do. Other thing which is in our hands is not to make a problem our system which was well conditioned to become ill conditioned in the process that we have got look at Gauss elimination method. So, we look at our coefficient matrix we want to introduce zeros in the first column below the diagonal. So, we subtract appropriate multiples of the first row from the second row third row and so on. Then in the next stage we work on only n minus 1 by n minus 1 sub matrix which consists of the second third and nth row and second third and nth column. So, your matrix is well conditioned, but this new system sub matrix you have to be careful that you are not going to make it ill conditioned. So, I am going to illustrate this by an example. So, here is the system you have got three equations in three unknowns for 3 by 3 matrix you can calculate its inverse and you can verify that it is going to be a well conditioned matrix. Now, here the whatever analysis we are going to do we are going to assume that you are doing computations using 4 digits like no matter how powerful your computer is your number of digits number of significant digits that is going to be limited like you consider representation of any number. So, that number one will represent as 0 point and then we want the first digit to be a non-zero number. So, we have got 0 point then say alpha 1 up to alpha 2 up to alpha 3 up to alpha n into 10 raise to and then here you will have some exponent. So, this is going to be representation of your number I am writing to the base of 10 it can be a binary base or it can be 16 and then you consider alpha 1 to be not equal to 0. So, any number you are going to represent in this form. So, this n this is going to be something fixed and no matter how powerful your computer is this n is going to be something finite. So, there is going to be always some error some numbers you can represent exactly, but most of the numbers there will be floating point representation and then there is going to be some error. So, here given a number say suppose you are looking at 1 by root 2 then you look at its approximation 0 point alpha 1 alpha 2 alpha n 10 raise to something this is what you do. Now, in the process you are going to do you are going to multiply you are going to add you are going to subtract. So, at every stage you will have to either chop or you will have to do round off. So, that means always you are going to have say your number it should be always of the form 0 point d 1 d 2 d n into 10 raise to some exponential. So, for this example in order to illustrate we will assume that we have got 4 digits available and then we will see that this number point 0 0 2 it is a small pivot if you it is non 0. So, I can very well do the Gauss elimination method, but this small pivot is going to create problem and your well conditioned matrix in the second step it becomes ill conditioned. So, this is what we will like to avoid. So, this thing we will consider next time and we will consider some related issues like whether I can make this small pivot to be big arbitrarily by multiplying the row by the constant. So, we will see these things in the next lecture and we will consider also backward error analysis. So, thank you.