 We have considered some methods for solving a system of linear equations. So, the methods which we have considered are Cholesky decomposition, then Gauss elimination method with and without partial pivoting. Now, these methods they are meant for solving big systems of linear equations using a computer. If you are doing hand computations, then you will be solving small system say n is equal to 3, n is equal to 4 and then it does not matter which method you use, but the methods which we have considered and their relative merits, it is meant for a big system of linear equations. So, you are necessarily going to use computer. Now, in computer no matter how powerful your computer is, you have finite precision. That means, after the decimal point you are going to have a fixed word length. So, that is going to introduce a error. Also, the linear system which you are trying to solve, it may be coming from some experimental data. In that case, there is going to be experimental error also. We want to solve a exact equation A x is equal to b, but because of the finite precision of the computer, you are going to solve a perturbed system. That means, there is going to be error in the elements of your matrices, matrix, coefficient matrix. Similarly, there will be error in the right hand side. So, you get a computed solution. Now, one wants to know how near your computed solution is to the exact solution. So, for that we need to have a major to decide how near your computed solution is to the exact solution. So, that is why we are going to consider norms. Now, the norm will give us a distance way to calculate distance between two vectors. You are familiar with Euclidean norm. So, in R 3, the Euclidean length is going to be x 1 square plus x 2 square plus x 3 square whole thing raised to half. Then, we are going to define some other vector norm, and then we will be considering the matrix norm. So, today's topic is going to be vector and matrix norms, and as has been the case so far, our numbers, they are going to be all real numbers. So, matrices involved are real, then vectors involved, they are real. So, let us define a norm. So, x is going to be a vector in R n, and we will define norm. So, norm is a function from R n to R plus. So, R plus is all non-negative real numbers, which should satisfy the property that norm x is bigger than or equal to 0, actually that is included. When I say that norm is from R n to R plus, that means norm x is bigger than or equal to 0. It should be equal to 0, should imply x is equal to 0 vector, and the converse also should be true, that x is equal to 0 vector, that should imply that norm x is equal to 0. The second condition is alpha is a real number, alpha times x, that means you multiply each component of x by alpha. So, norm of this new vector should be equal to modulus of alpha times norm x. This should be valid for all alpha belonging to R, and for all vectors x belonging to R n. And the third condition is known as triangle inequality that norm of x plus y is less than or equal to norm x plus norm y. Two vectors in R n, you add component y's. So, if you have a vector x, x 1, x 2, x n, y is vector y 1, y 2, y n. When you want to add x plus y, you add the corresponding components. So, it will be x 1 plus y 1, x 2 plus y 2, x n plus y n, and the scalar multiplication is alpha times x will be vector with the components alpha x 1, alpha x 2, alpha x n. Now, let us look at some of the examples of norms. You can define norm in various manners. We are going to consider three norms. So, one is the Euclidean norm, then second is known as a one norm, and the third norm is the maximum norm or infinity norm. And the reason we consider one norm and infinity norm will become clear when we look at corresponding matrix norm. So, here Euclidean norm is summation j goes from 1 to n x j square whole thing raised to half. Then norm x 1 is equal to summation j goes from 1 to n modulus of x j, and norm x infinity is maximum mod x j 1 less than or equal to j less than or equal to n. Now, it is easy to verify that all these three definitions, they will satisfy the three properties of the norm. So, that norm x should be bigger than or equal to 0. So, look at Euclidean norm. You are considering summation x j square, and then you are taking positive root, positive square root. So, it is going to be bigger than or equal to 0. If it is equal to 0, then your sum of the squares is 0. So, it has to be, the each component has to be 0, that means x is the 0 vector. And if x is the 0 vector, norm x will be equal to 0. Similarly, when you consider one norm and infinity norm, it is going to satisfy that it is bigger than or equal to 0, and is equal to 0 if and only if x is equal to 0 vector. It is simple to verify. Next is norm alpha x should be equal to mod alpha times norming. So, this also for all the three definitions, you substitute, look at the formula, and then you can verify. The third property is the triangle inequality. So, this triangle inequality, it is easy to verify for one norm and infinity norm. It is straight forward, like look at the one norm. So, norm of x plus y, it will be summation j goes from 1 to n, modulus of x j plus y j, x j and y j, these are real numbers. So, that will be less than or equal to mod x j plus mod y j, and then separate the summation. Similarly, for the infinity norm, it is straight forward. The triangle inequality for the two norm, one needs to use what is known as Cauchy Schwarz inequality. So, I am not going to prove Cauchy Schwarz inequality, but I will state Cauchy Schwarz inequality and show you how using Cauchy Schwarz inequality, we get the triangle inequality for the two norm. Now, this inner product on R n. So, it is inner product of x comma y, x and y are victor's in R n. It is defined as summation j goes from 1 to n x j y j. We will come across inner product little later, and then that time we will study the properties of the inner product. In fact, we did use inner product when we had talked about the Gauss points. The Gauss points, they were zeros of the Legendre polynomials that was they were needed in Gaussian quadrature, and that is where we had talked about the inner product. So, you are familiar with inner product. Now, look at the inner product on R n, and then you consider positive square root of x comma x. So, that is nothing but our two norm of x. So, x comma x raise to half is equal to norm x 2. So, the Cauchy Schwarz inequality is modulus of inner product of x with y is less than or equal to norm x 2 into norm y 2. So, that is modulus of summation j goes from 1 to n x j by j, because that is our definition of inner product of x with y is less than or equal to summation j goes from 1 to n x j square raise to half, which is norm x 2, and this quantity is norm y 2. So, this is the Cauchy Schwarz inequality. Now, look at norm of x plus y, it is two norm and then square of it. So, that is summation j goes from 1 to n x j plus y j square, x j plus y j is going to be j th component of x plus y. So, I expand a square and I will get summation over j x j square plus summation over j y j square plus 2 times summation x j y j. This is nothing but two norm of x square summation over j y j square is norm y, it is two norm square plus. Now, I use the Cauchy Schwarz inequality to dominate this by 2 times norm x 2 norm y 2. So, this is nothing but norm x plus norm y square. So, now, you take positive square root and then that will prove the triangle inequality for two norm. So, this is about the vector norms. Now, I want to consider matrix norm. So, I have got a to be a n by n matrix. So, I want to define the norm. So, it should satisfy our three properties, the positive definiteness, triangle inequality and the third one was that alpha times a should be equal to mod alpha times norm a. So, why not treat our a as a vector of length n square? Because what is matrix? It is an arrangement of n square number. We write them as rows and then columns. So, as such the matrix a, it has got n square element, it has got n square element. So, I can treat it as a n square, a vector of length n square. I know how to define vector norm. So, then I will consider the corresponding norm. Norm, which corresponds to the Euclidean norm will be a i j square. Now, instead of single summation, you will have summation i goes from 1 to n, summation j goes from 1 to n and then whole thing raised to half. So, this is known as Frobenius norm and then norm a max. Now, I am not writing infinity because norm a infinity, I want to reserve that symbol for something else. So, norm a max will be maximum of modulus of a i j, 1 less than or equal to i, 1 less than or equal to 1 less than or equal to j, less than or equal to n and then one can verify that these definitions, they will satisfy our three properties of norms. Now, what we want to do is, we want to relate our vector norm and the matrix norm because look at our system of linear equations. It is a x is equal to b. So, we are considering matrix into vector. So, I will like to have some relation between a, the matrix norm and the vector norm and another thing is, I can multiply two matrices a and b. If they are, I am considering square matrices. You cannot multiply two vectors, but you can multiply two matrices. So, then again norm of a, b and norm a, norm b, I will like to have some relation between them. So, in order to do that, what we are going to do is, we are going to consider induced matrix norm. So, we will have, we are going to define a general way of defining a matrix norm. So, we are going to start with a vector norm and from that we will give a general definition of induced matrix norm and then since we are interested in our one norm, two norm, infinity norm, we will see what are going to be the corresponding induced matrix norm. So, now first definition of induced matrix norm. So, a is in binary matrix. You fix a vector norm, any norm which you want. That means, it should satisfy the three properties of norm and now I am going to look at maximum of norm a x by norm x, x not equal to 0 vector. That is my definition of norm of a. So, now the first thing we want to show is that if I define norm a in this manner, then it satisfies those three properties of norm. So, the first property is positive definiteness that norm a is bigger than or equal to 0 and it is equal to 0 if and only if a is a now 0 matrix. The second property will be norm alpha a should be equal to mod alpha times norm a and the third property is the triangle inequality norm of a plus b is less than or equal to norm a plus norm b. So, these three properties we are going to deduce using the fact that our vector norm satisfies these properties. You will see that the proofs are straight forward. So, norm a is bigger than or equal to 0 that is clear. Norm a is equal to 0 it will imply that norm a x is equal to 0. Remember our norm a is maximum I have written here. So, norm of a will be maximum of norm a x divided by norm x x naught equal to 0 vector. So, if norm a is 0 then norm a x is going to be equal to 0 for any non-zero vector in R n. So, now look at our canonical vectors e j is going to be vector with 1 at j th plus and 0 elsewhere. So, I will have norm e j a e j to be equal to 0, but what is our a e j a e j is nothing, but the j th column. Now, using the property of vector norm you will get c j to be a 0 vector and you get a to be a 0 matrix. So, we are using the property of vector norm. The second property is look at norm alpha a. This will be by definition maximum x naught equal to 0 vector norm of alpha a x by normic but then our alpha times a into x will be nothing, but alpha times a x. What is the difference? Here you are multiplying matrix a by alpha. Here you are first calculating a into x and then multiplying by scalar alpha. You have norm alpha a is maximum x naught equal to 0 vector alpha times a x divided by normic and using the property of vector norm you get mod alpha times norm a because mod alpha comes out. It does not depend on maximum. So, it will come out of the maximum and maximum of norm a x by norm x x naught equal to 0 vector is norm a and similarly triangle inequality. Norm of a plus b is maximum x naught equal to 0 norm of a plus b x divided by norm x. Use the fact that norm of a plus b x is less than or equal to norm a x plus norm b x. So, maximum of norm of a plus b x x naught equal to 0 vector divided by norm x will be less than or equal to maximum of norm a x by norm x x naught equal to 0 vector plus maximum of norm b x x naught equal to 0 vector divided by norm x. So, this proves that norm of a plus b is less than or equal to norm a plus norm b. We defined induced matrix norm and then we showed that it satisfies all the properties of our vector norm. Now, we want to we are going to prove one of the basic inequality which we will be using frequently later on. But that inequality is going to relate the matrix norm to vector norm and we are going to show that norm of a b is less than or equal to norm a into norm b. Now, these two properties they will follow from our definition of induced matrix norm. So, our norm a is maximum norm a x by norm x x naught equal to 0 vector. That means for all non 0 vector norm a x by norm x is less than or equal to norm a which will give us norm a x to be less than or equal to norm a into norm x for all non 0 vector. But if x is a 0 vector norm x will be 0 a x being 0 vector again this will be 0. So, this fundamental inequality will be true for all x belonging to R a. Now, I want you to note one thing that here norm a x that means vector norm. Norm a means induced matrix norm and norm x that means again vector norm. So, it is clear from the context about which norm we are talking about. A being a matrix this will be a matrix norm, x being a vector this will be a vector norm. So, this is the first inequality and now let us show the consistency condition that means if you have got a and b to be two matrices take their product. So, a b is going to be again a in by n matrix. So, norm a b is less than or equal to norm a into norm b. So, this fact will be proved using our basic inequality that norm b is less than or equal to norm a x is less than or equal to norm a into norm x. So, you have got norm a b x to be less than or equal to by fundamental inequality it will be norm a into norm b x. Treat b x as one vector and a as our matrix. So, you have got norm a into norm b x. This is less than or equal to now use fundamental inequality for b x. So, norm b x will be less than or equal to norm b into norm x. Now, if x is not equal to 0 vector norm x will not be 0. So, I can divide and you will get norm a b x upon norm x to be less than or equal to norm a into norm b for all non zero vectors in R n. So, now, take maximum of this over x not equal to 0 the right hand side is independent of x. So, you get norm a b to be less than or equal to norm a into norm b. So, this is consistency condition. So, when one talks about vector norm it should satisfy those three properties. When one talks about matrix norm it should satisfy the three properties of vector norm and in addition norm a b to be less than or equal to norm a into norm b. This is a very elegant definition start with any vector norm. You define the induced matrix norm showing that it satisfies the three properties of vector norm and then the consistency condition it was straight forward. So, very nice situation, but then given a matrix a I will like to calculate its norm like given a vector x. If I want to calculate its Euclidean norm I know that take the squares of their components add them up take the positive square rule. If I want one norm then I will take the modulus of each component add it up. If I want infinity norm look at the moduli of its components and look at the maximum. So, now suppose I take one of these three norms and then I want to know what is the induced matrix norm. Now, in the definition of induced matrix norm you have got maximum over all non-zero vector of certain quotient that quotient is norm a x by norm x. So, how am I going to calculate this maximum I have got infinitely many vectors. So, then I cannot calculate this induced matrix norm. So, now we are going to derive a formula for calculating norm a 1. That means when you fix vector norm to be 1 the corresponding induced matrix norm. So, we will have a formula in terms of the components of the matrix. So, it is going to be something one can calculate. Similarly, for infinity norm of the matrix we will have a formula whereas, the nice case of Euclidean norm well one has to be satisfied only with an upper bound. So, the Euclidean norm of matrix is something which you cannot calculate. So, now let us derive a formula for norm of a 1. We are fixing our vector norm to be 1 norm and then I am going to look at norm a x 1 divided by norm x 1 and look at their maximum. So, I have got norm a 1 is maximum of norm of a x 1 divided by norm x 1 x not equal to 0 vector. I want to find a formula for this. Now, the components of a x let me write as a x i. This will be summation a i j x j j going from 1 to n that is the matrix into vector multiplication. Norm of a x 1 this will be summation i goes from 1 to n modulus of a x i that is our definition of 1 norm. Now, let me substitute for a x i. So, it is going to be summation i goes from 1 to n modulus summation j goes from 1 to n a i j x j. Now, this will be less than or equal to summation i goes from 1 to n summation j goes from 1 to n modulus of a i j modulus of x j. So, we are using triangle inequality. Now, this is finite summation. So, I can interchange the order of the summation. So, I am going to have. So, this is our norm a x 1 is less than or equal to this. So, this is same as summation j goes from 1 to n summation i goes from 1 to n modulus of a i j and then mod x j. Now, summation is over i here you have x j. So, it will come out of the summation sign. So, it will be summation j goes from 1 to n mod x j summation i goes from 1 to n modulus of a i j. So, here what we are doing is you fix j. That means, you are fixing the column. Look at the entries in that column, take their modulus and then add it up. So, I can say that this is less than or equal to maximum over j of the quantity a i j. So, here what we are doing is you fix j. Summation i goes from 1 to n modulus of a i j into summation j goes from 1 to n mod x j and this is nothing but norm x 1. Let me call this quantity to be alpha. So, what we have is norm a x 1 is less than or equal to alpha times norm x 1. Hence, you will have norm a 1 to be less than or equal to this number. So, here we have norm a x 1 is less than or equal to alpha times norm x 1, where alpha is this number. Summation i goes from 1 to n modulus of a i j 1 less than or equal to j less than or equal to n. So, I can very well calculate alpha. I will look at the columns of my matrix a. So, look at the first column. So, take the modulus and add it up. So, that means I am considering one norm of the first column, then one norm of the second column and one norm of the third column. So, I will get n number. The maximum among them that is going to be my alpha and my norm a 1 is going to be less than or equal to this alpha. But we will like to show that it is not only less than or equal to, but it is equal to. So, we have proved that norm a 1 is less than or equal to this number. Now, let us show that in fact it is equal to maximum. So, you have norm a x 1 is less than or equal to alpha times norm x 1, where alpha is this quantity. Hence, you have got norm a 1 to be less than or equal to alpha. Now, look at this maximum. This will be equal to summation i goes from 1 to n modulus of a i j 0 for some j 0. Such a j 0 need not be unique. It depends on your matrix. May be all the columns they are going to have the same one norm, but you are looking at maximum of n numbers. So, look at at least one number which is equal to maximum and let that column be equal to j 0 th column. Now, if you consider a times e j 0, e j 0 is a canonical vector with 1 at j 0 th place and 0 elsewhere. So, a e j 0 that gives you j 0 th column. So, using this fact we will show that alpha is equal to norm a 1. So, consider a e j 0 that is nothing but the j 0 th column. So, norm a e j j 0 1 is equal to alpha and if you consider norm e j 0, it is one norm that is going to be equal to 1. So, you have got alpha which is equal to norm a e j 0 1 divided by norm e j 0 1 divided by norm e j 0 1. This is going to be less than or equal to norm a 1. So, thus we have proved that norm a 1 is equal to alpha and that is known as column sum norm, because what you do is in each column you are adding the moduli of the entries and looking at their maxima. So, here is a formula for norm a 1. Now, we are going to look at norm a infinity. Now, norm a infinity we will see that. So, here what was the crucial result? It was that you had a finite summation. So, you interchange the order of the summation. Now, in case of the infinity norm you do not have to do interchange, because there is going to be only one summation and one maximum. Here in order to show that the as such norm a infinity it is going to be row sum norm. That means what you do is look at each row, take the moduli of the entry add it up. You will get n numbers. Among these n numbers whichever is going to be the bigger one that is going to be norm a infinity. So, this fact now we are going to prove. Now, in order to show that norm a infinity is equal to this, showing less than or equal to will be simple. Showing that it is equal to that is going to be a bit involved. In case of one norm it was easy you just looked at the corresponding canonical vector. Here we will have to construct a vector where the infinity norm is going to be attained. But then the proof is going to be something similar. So, you fix infinity norm and then our aim is to calculate norm a infinity which is maximum of norm a x infinity divided by norm x infinity x not equal to 0 vector. So, let us start with norm a x infinity. So, norm a x infinity will be maximum of modulus of a x i, one less than or equal to i less than or equal to n by definition of infinity norm. Now, a x i as before will be given by summation j goes from 1 to n a i j x j. Then for this modulus use triangle inequality. So, you will get less than or equal to maximum one less than or equal to i less than or equal to n summation j goes from 1 to n modulus of a i j mod x j. So, there are no two summations here. Now, this mod x j I will dominate by norm x infinity. So, norm x infinity is going to be maximum of mod x j j going from 1 to n. So, if I dominate that it will come out of this summation and in maximum. So, you have norm x infinity maximum summation j goes from 1 to n mod a i j one less than or equal to i less than or equal to n. So, here you are fixing first i is equal to 1. Then you are looking at you are varying j that means you are looking at the elements in the first row taking their modulus summing it up. Then put i is equal to 2. So, you do it for the second row and put i is equal to n. So, like that you are going to get n numbers maximum among them that we are denoting by beta. So, we get norm a x infinity to be less than or equal to beta times norm x infinity. From this conclude that norm a x infinity divided by norm x infinity is less than or equal to beta for x not equal to 0 vector take their maximum that is norm a infinity. So, you have got norm a infinity to be less than or equal to beta. And now we want to show the other way inequality that now we want to show that we have proved that norm a infinity is less than or equal to beta. Now, let us show that beta is less than or equal to norm a infinity. So, that combining these two inequalities we can show that norm a infinity is in fact equal to beta. So, now again beta is maximum minus norm summation j goes from 1 to n modulus of a i j maximum over i. So, this maximum will be attained for some i is equal to i 0. Such a i 0 can be more than 1. So, take one of them. So, get hold of i 0 such that beta is equal to summation j goes from 1 to n modulus of a i 0 j. And now let us construct a vector. So, we are looking at elements of the i 0th row. If a particular element is non-zero then you define y j to be mod a i 0 j divided by a i 0 j. If that number is equal to 0 then you define it as mod a i 0 j. So, you define it to be equal to 0. Now, if I define such a matrix modulus of y j for any j is going to be either 1 or 0. And that will mean that norm y infinity should be equal to 1 at least one of the y j should be not 0. Because if y j is equal to 0 that means each entry in the i 0th row will be 0. So, you will get beta to be 0 and that will mean that a is a 0 matrix. So, now I have constructed a vector y j. Now, I look at a y of i 0. So, i 0th component of my vector a y that will be given by summation j goes from 1 to n a i 0 j y j. Now, look at the way we have constructed y j. If corresponding y j is not 0 then a i 0 j multiplied by y j will be mod a i 0 j divided by a i 0 j. So, a i 0 j will get cancelled and you will get modulus of a i 0 j. If the y j is equal to 0 it will not contribute to your summation. So, you will get this is equal to summation j goes from 1 to n a i 0 j y j is equal to 0. So, you will get this is equal to summation j goes from 1 to n modulus of a i 0 j and this is going to be equal to beta. So, we have got beta is equal to i 0th component of our vector a y. I 0th component will be less than or equal to norm of a y infinity norm. Norm a y infinity we know that it is less than or equal to norm a infinity into norm y infinity and then you get or norm a y infinity will be less than or equal to norm a infinity into norm y infinity norm y infinity will be 1. So, you will get norm a infinity. So, we have got beta to be less than or equal to norm a infinity and what was beta it was this maximum. So, this is going to be rho sum norm. So, now we have obtained a formula for norm a 1 and norm a infinity. So, we have obtained norm a infinity. Now, you know why I called norm a max for the term maximum over i and j of modulus of a i j because I want to reserve norm a infinity to the definition of induced matrix norm when the vector norm is infinity norm. So, we have got a formula for norm a 1. We have got a formula for norm a infinity, but unfortunately for norm a 2 we have got only an upper bound. So, let us now calculate that upper bound that upper bound will be we will be again using Cauchy Schwarz inequality. So, the again the method is the same. We will look at norm of a x 2 norm obtain it to be less than or equal to say a gamma times norm x 2. So, that gamma will give us an upper bound. So, we have norm a 2 to be maximum of norm a x by norm x x not equal to 0 vector recall that Euclidean norm 2 of x is summation j goes from 1 to n x j square raise to half. So, look at norm a x it is 2 norm square it will be summation i goes from 1 to n a x i square by definition. Now, ith component of a x is given by summation j goes from 1 to n x j square j goes from 1 to n a i j x j. So, this is a x i and it is square. Now, apply Cauchy Schwarz inequality here. So, square of this will be less than or equal to summation j goes from 1 to n a i j square and summation x j square. So, using the Cauchy Schwarz inequality you get this quantity to be less than or equal to summation i goes from 1 to n summation j goes from 1 to n a i j square summation j goes from 1 to n x j square this is nothing but Euclidean norm of x it is square and then you have got this number. Now, if you recall this is what is known as Frobenius norm. So, you have got norm a f square into norm x square. So, that gives you 2 norm of a to be less than or equal to norm a f. So, now, let us recapitulate our norm. So, we have got our norm a 1 that is the column sum norm norm a infinity that is the row sum norm and norm a 2 is less than or equal to norm a Frobenius. So, this is the best one could do for the 2 norm, but then for the 2 norm we have got the basic inequality norm of a x its 2 norm is going to be less than or equal to norm a Frobenius into norm of x 2 norm. So, that inequality is available if your matrix is symmetric then 1 norm and infinity norm they are going to be the same because for column sum norm that gives us 1 norm if you do corresponding thing for row you get infinity norm. So, for the symmetric matrix both 1 norm and infinity norm they are going to be the same for a general matrix they can be different. So, now, we have proved or we have defined vector norm and matrix norm using these we are going to analyze the behavior of the perturbed system a x is equal to norm a x is equal to b is the original system or the system which we want to solve because of the use of computers we are going to solve system a plus delta a x cap is equal to b plus delta b. So, x is the exact solution x cap is the computed solution and we will like to say something about norm of x minus x cap. So, this perturbation of the linear system and sensitivity of the computed solution to the change in the right hand side or to the perturbation that is going to be topic of our next lecture. So, thank you.