 So, we have been discussing iterative methods for solving linear algebraic equations and then after discussing algorithms, we started looking at convergence of these methods. We also derived necessary and sufficient condition for convergence. Necessary and sufficient condition for convergence was spectral radius of matrix S inverse T should be inside unit circle and we said that well it definitely gives us lot of insight into what happens, but we need something even simpler to calculate and we have started looking at matrix norms to come up with something that is easier to compute in terms of assessing whether a matrix whether A matrix which we used to formulate the iterations, you can apply some quick test to come up with convergence analysis. So, induced matrix norm was defined as norm of A, let us talk about norm in which the norm induced by identical norm on both range and domain space. So, this is max x not equal to 0 norm A x by norm x and the way I try to explain this was this is something like amplification power of a matrix other way of writing the same thing. So, it is 2 norm here and 2 norm if I want 1 norm it will be A 1 norm 1 norm and 1 norm and so on. This is called induced norm because this is induced by the norm defined on the range and a domain space that is why it is called induced norm. Other way of writing this is this is a generic thing. So, I could actually I need not write 2, but we were discussing 2 norm at the end of the last class. So, this inequality can also be written like this norm norm A x divided by norm x is less than or equal to. So, norm A is in some sense a bound upper bound on this ratio of course, this is always greater than 0 for x is not equal to 0 this is always greater than 0 or it could be always greater than or equal to 0. Suppose A is a matrix which is rank deficient and A x is in the null space then A x will be 0, but denominator is not 0. So, this can be greater than or equal to 0 and this is an upper bound norm is an upper bound maximum of this ratio or maximum of the gain maximum of the amplifying power of the matrix whatever way you want to view it. Now, in particular we started discussing 2 norm. So, in 2 norm I just started by putting A x 2 square divided by x 2 square which I wrote as x transpose A transpose A x divided by x transpose x. I can do all this division is because this is a scalar x transpose A transpose A times x is a scalar x transpose x is a scalar. So, this is ratio of 2 scalars. Now, this particular one this particular norm well even though as I said that it is not very convenient for computation, but gives you very nice interpretation. So, this one we showed that we made additional assumption that matrix A, matrix A transpose A this is a symmetric matrix positive definite matrix. We made one more assumption that A transpose A has linearly independent Eigen values. So, we wrote this matrix A transpose A as psi lambda psi inverse actually what I argued was that since this is a symmetric positive definite matrix the Eigen vectors are orthogonal. So, which is same as psi lambda psi transpose where lambda is a diagonal matrix with Eigen values of A transpose A appearing on diagonal on main diagonal and this matrix psi is nothing, but matrix formed by keeping Eigen vectors of A transpose A next to each other. So, I am just keeping Eigen vectors of A next to each other this is a n cross n matrix not only invertible it is a orthonormal matrix which means psi transpose psi is equal to psi transpose is equal to I special property of this matrix and then I transform this ratio using this I transform this ratio to something very interesting. So, I wrote this x transpose A transpose A x as x transpose psi lambda psi transpose x psi psi transpose x and defining y to be psi transpose x we wrote this ratio defining this new vector we defined y or we defined z we defined z. We wrote this ratio we wrote this ratio to be lambda 1 z 1 square plus lambda 2 z 2 square plus lambda n z n square. So, you can see that this is a ratio of 2 positive quantities is a ratio of 2 positive quantities because Eigen values of A transpose A are always positive. So, because A transpose A is a positive definite matrix it is Eigen values are positive z 1 square z 2 square whatever it is z z 1 square z 2 square is always positive. So, this is ratio of 2 positive quantities. Now, if we order the Eigen values as all the Eigen values are real positive for positive definite matrix and then if I say that lambda 1 is greater than or equal to lambda 2 is greater than 0. If this holds if this holds well there is an implicit assumption when we started all this analysis that A is full rank that because we are assuming that when we are solving this we are talking about a full rank matrix. So, that is why you get minimum Eigen value greater than 0 otherwise you may have possibility some Eigen some Eigen values will be equal to 0. Now, if I number my Eigen values such that lambda 1 corresponds to the largest numbering is depends upon me what I call 1 and what I call 2 matrix as Eigen values they do not come numbered we number them. So, I am numbering lambda 1 to be the largest ok. So, it is very easy to see that this ratio will always be less than or equal to lambda 1 z 1 square plus lambda 1 z 2 square plus lambda 1 z n square upon z 1 square up to z n square is everyone with me on this I am just replacing lambda 2 by lambda 1 lambda 3 by lambda 1 lambda 4 by lambda 1 ok lambda 1 is the largest magnitude Eigen value of A transpose A ok. So, I am allowed to do this denominator is same numerator I am replacing by a larger value for every term by term ok very easy to see that you can take lambda common ok. So, this ratio is independent of this is lambda 1 ok. So, this this ratio can never exceed lambda 1 this ratio can never exceed lambda 1 very very nice property ok what did we start with we started with we started with this we wanted to find out maximum of this ratio what is the maximum of this ratio lambda 1 ok this ratio can never exceed Eigen value of maximum magnitude Eigen value of A transpose A ok. So, what is 2 norm what is 2 norm of matrix A lambda 1 this is the upper bound and you can show that this upper bound is attained when Eigen vector is aligned in a particular direction ok what happens when x is equal to V 1 when will when will this be an equality for which direction V 1 transpose A transpose A V 1 upon V 1 transpose V 1 ok, but V 1 can be chosen to be orthonormal. So, V 1 transpose V 1 will be unity ok. So, this will be V 1 transpose lambda 1 V 1 upon V 1 transpose even if it is not chosen orthogonal even then this ratio will be equal to lambda 1 ok. So, when x is aligned along x is aligned along the direction which corresponds to the Eigen vector associated with lambda 1 Eigen vector of A transpose A not Eigen vector of A Eigen vector is defined for a square matrix ok Eigen vector is defined for a square matrix A transpose A is a square matrix A in general need not be a square matrix, but A transpose is always a square matrix. So, of course, right now we are dealing with square matrices we are dealing with square matrices. So, there is no question of non-square matrices we are dealing with solving A x equal to B where A is square. In fact, we also are have a problem where A is invertible otherwise. So, this ratio is becomes equality when x is aligned along Eigen vector of A transpose A first Eigen vector first in the sense that correspond to the maximum magnitude Eigen value. So, this is equality, but this ratio for any other x is not equality for any other x this ratio will not be equality this ratio will be smaller that is why the maximum amplification power of a matrix using two norm is given by this ok. So, I think I have this picture somewhere drawn for a two dimensional case actually when you have defined this z is equal to psi transpose psi transpose x you have actually defined a transformation which is rotation ok. So, suppose this is your this is your x y then this is your z. So, this is so this is x 1 x 2 this is let us say this is x 1 x 2 and this is this will be your z 1 z 2 coordinate space. So, this is the invertible transformation you can go from one to other ok. So, actually you are just rotating your coordinate axis when you are multiplying ok and what happens in the rotated coordinate axis in the rotated coordinate axis this x transpose if you draw this x transpose A transpose A into x that is which is same as if you draw this inside a region where norm x is less than 1 we also had a other interpretation of norm right. Remember we did one more interpretation of norm max of A x cap where x cap is a straight to unit circle ok. If you draw that you know locus of points then you will see here that this actually corresponds to a ellipse actually corresponds to an ellipse ok. And this coordinates will be nothing, but this will be v 1 and this will sorry this will be v 1 have drawn I think wrong the ellipse will be like this the ellipse will be like this this z 1 will be actually aligned along your direction v 1 z 2 will be aligned along direction v 2 ok. It will be an ellipse this ellipse will be major axis will be along the along the Eigen vector corresponds to maximum magnitude Eigen value minor axis will be along the direction which is smallest Eigen value all other axis are in between. So, that you can. So, it will be an ellipsoid in three dimension it will be a ellipsoid in n dimension depending upon what kind of matrix we are looking at. So, I have drawn a picture in somewhere in the you can have a look at it, but again the problem with this two norm is you have to find out Eigen value of A transpose A if A is large does not help us ok. So, we could actually we could actually come up with a criteria which says that convergence will occur if two norm with two norm is nothing but Eigen value of A transpose A which is strictly less than 1, but as I said does not really help. So, actually this what we have found here is ratio of the squares it is square of this by square of this. So, what is the what is the two norm? So, two norm it turns out is that is square root of lambda 1 that is square root of lambda max of A transpose A that ratio that ratio is for square of see this ratio was found for A 2 square ok A 2 square is equal to lambda 1. So, what is A 2 square root of lambda 1 and this will always be positive because A transpose A will always have positive Eigen values ok. So, this number will be positive ok. In fact, Eigen values of A transpose A are called as singular values of A and square root of the maximum magnitude singular value that is what is what do you expect when A is symmetric there is a problem in the problem sheet. If A is symmetric then it will be A transpose is equal to A. So, A transpose A will be equal to A square. What is the relationship of Eigen value of A and A square? Square it is very easy to show that if lambda is Eigen value of A lambda square is Eigen value of A square ok. So, if A is symmetric A transpose A will be and if A is symmetric positive definite then it is much easier A is symmetric positive definite then it is lambda max of A square ok lambda of max of A square is lambda square of A and you can deduce for a symmetric positive definite matrix just look at it is Eigen value and then you can maximum Eigen value will directly give you the norm, but then this norm is again inconvenient because you have to compute Eigen value ok. Now, I am going to state two other norms without deriving the derivations to some extent required for these norms are there in the as a part of your exercises. So, you should look at you should try to work out and then you can see whether you are able to derive that or you end up into some difficulties. I am just going to write these final statements for other norms ok. So, now even two norm though it has some nice geometric interpretation it is not quite convenient for me for computing. So, I am going to talk about one norm. So, one norm is nothing but is one norm is nothing but max over this one norm is nothing but max over this ok well one small correction I realize I had made a small mistake here when I wrote the earlier expression I will just correct it. So, it is not norm 2 square is equal to this equal to max x not equal to 0. So, earlier when I started I had forgotten this max operator is there without max operator you cannot proceed and then I have simplified this quantity ok. So, moving on to one norm. So, this one norm is induced by one norm on the range space and one norm on the domain space ok one norm on the domain space maximum of this ratio. Now, you can show that one norm. So, what is one norm? How do you interpret this? What is this? What is this summation? Is it a column sum or is it a row sum? It is a column sum. So, you take all the elements in one column a 1 1 a 2 1 ok. So, 1 is 1 is summation over 1. So, I am taking summation of more of each row each column ok. So, and max over that. So, I find out first of all I take a matrix which is consisting of only absolute values only positive numbers ok. Then I find the column sums ok I find the column sums max over the column sum is nothing but this ratio max over this ratio that you can show not it is not very difficult to show this. So, max over column sums absolute of column sums that is one norm and what you can show is infinite norm. So, infinite norm is nothing but max over absolute of row sums ok. So, this these norms you can see here computationally is much more easy to see what you have to do when you want to compute one norm or infinite norm. You create a matrix a which is a 1 1 a 1 2 you create this matrix which is absolute of absolute value of each number all of them are positive numbers now ok. If you take all column sums find max over it you will get one norm take all row sums ok find max of the row sums that will give you infinite norm. This is much much easier to compute one norm or infinite norm are much much easier to compute let us close the. See this one norm and infinite norm are much much easier to compute then computing a transpose a and it is Eigen values. So, much more complex business than doing this this is very very easy ok. So, I wanted some easy way of computing norms now where am I going to use this ok why am I computing norms because our condition necessary and sufficient condition was spectral radius. Spectral radius is nothing but Eigen value ok. So, what is the relationship between norm and Eigen value. So, that is the next part of the puzzle is this clear now there are we have three different ways of computing norms two norm one norm and infinite norm among these one norm and infinite norm are computationally preferable ok. And now comes the point is why do why am I talking about norms yeah lambda one was nothing but Eigen value of lambda i we have this a transpose a v i is equal to lambda i v i. So, lambda is are Eigen values of a transpose a ok. Now, I said that I have numbered the Eigen values such that lambda 1 is greater than or equal to lambda 2 is greater than or equal to lambda 3 I have numbered them ok. So, lambda 1 is nothing but another way of saying lambda 1 is lambda max instead of giving a number and remembering it is easier to remember this as a formula lambda max of a transpose a that is why I called it ok. So, this theorem which I am going to state is now the corrects of the matter for any matrix b any square matrix b ok spectral radius sorry any matrix a for any matrix a its spectral radius is always less than or equal to any induced norm ok. The spectral radius is always less than or equal to induced norm. So, can you prove this how will you prove this what is spectral radius spectral radius is spectral radius of a is max over i or let us use the new notation that we have lambda max spectral radius is nothing but max over this. Now, what is what is induced norm of a matrix any induced norm for any induced norm is this is true this is the definition right is the induced norm definition ok. So, this thing also holds for this thing also holds when x corresponds to Eigen vector of a let us say v i is Eigen vector of ok v i is equal to lambda i v i right. So, I am going to write this is equal to norm. So, a v i and if I substitute v i here if I substitute v i here this is nothing but lambda i v i right, but what is this quantity when lambda comes out of the numerator what happens mod lambda mod lambda i norm v i divided by norm v i right mod lambda i norm v i by norm v i. So, this cancels ok what remains is mod lambda i this is hold this holds for every for every Eigen vector. So, this also holds for maximum magnitude Eigen vector right. So, this holds this holds for every Eigen vector ok which means it also holds for that Eigen vector which has a maximum magnitude ok, but what is the maximum magnitude Eigen vector Eigen value maximum magnitude Eigen value is nothing but the spectral radius ok. So, this inequality that is mod lambda i is less than or equal to norm a this holds for all i 1 to n and this implies that spectral radius of a is less than or equal to norm of a. Everyone with me on this ok so far so good. So, now we have we have developed concept of matrix norms we have expressions for computing matrix norms of course, in matlab if you give a matrix and say give a matrix and say 2 norm it will give you 2 norm which is nothing but this if you say i n f infinite norm it will give you infinite norm which is nothing but this and so on. So, computing using a software these days for a any huge matrix is just very very simple of course, computationally for a large scale matrix this is much much easier you know it just has to take absolute absolute sums of rows and find the max very very easy as compared to doing this. But so and we have a very nice relationship here ok. Now, how I am going to exploit this I am going to use this relationship to come up with sufficient conditions for conversions ok. Is this alright we have matrix norms we know how to compute them and now we know what is the relationship between the spectral radius and the matrix a its eigen value. We were at one point analyzing behavior of systems of the time z k plus 1 just about 2 lectures back be linear difference equations we are analyzing behavior of this and we said that if spectral radius of a is strictly less than 1 then what happens then norm z k goes to 0 as k tends to infinity ok as k tends to infinity but spectral radius for a large matrix is difficult to compute you have to compute eigen values ok. But from this inequality what I know is that spectral radius is always less than induced norm ok. Now, suppose I take a matrix ok compute its induced norm I compute its infinite norm ok or not a matrix here we are talking about sorry this should be b matrix here I am really sorry just I stand corrected this should be b matrix here b should be strictly less than 1 I take my b matrix ok if I take my b matrix compute its norm say 1 norm or infinite norm and that norm turns out to be less than 1 what can I say about the spectral radius right. So, if norm of b say infinite norm is less than 1 or I am not going to compute 2 norm I am going to compute only 1 norm or infinite norm. So, this infinite norm or norm of b 1 norm if this is strictly if this is strictly less than 1 either of them are strictly less than 1 ok from this theorem from this theorem what I know is that spectral radius of b should be strictly less than 1 ok. So, which means a sufficient condition a sufficient condition for conversions of this z k sequence to 0 norm of z k sequence to 0 is that take b matrix find its 1 norm or find its infinite norm if that norm turns out to be less than 1 strictly less than 1 I am done ok I know that I know that z k is going to go to 0 irrespective of irrespective of what happens to what is your initial condition it does not depend upon what is your z 0 z 0 can be large z 0 can be small ok z 0 can be arbitrary I know that if this condition holds then the spectral radius is always less than 1 because induced norm gives a upper bound on the spectral radius induced norm gives a upper bound on the spectral radius and if upper bound is smaller than 1 obviously spectral radius is less than 1 and then ok. So, I think we had lot of side stories now let me go back to solving a x equal to b so far so good is this a line of arguments clear ok. So, now let us go back and look at what we were doing finally we are back to what we wanted I wanted to solve a x equal to b using a iteration method x k plus 1 is equal to s inverse t x k plus s inverse b ok we have we have this iteration method and we said that the error behaves according to e k plus 1 is equal to s inverse t e k ok s inverse t e k and then we had this condition that spectral radius of s inverse t now s and t are different depending upon whether it is a Jacobi method or Gauss-Siedel method or relaxation method and so on. Now typically s is a simple matrix s is a diagonal matrix or s is a lower triangular matrix and inverting that matrix is not that difficult well we will come up with conditions which even do not require inversion further, but right now even if you want to do it by brute force by actually inverting s matrix even though it is not difficult, but now we have this condition spectral radius less than 1 for error convergence this is necessary and sufficient condition is necessary and sufficient condition I am now giving you a sufficient condition that if if induced norm of s inverse t is strictly less than 1 then spectral radius is less than 1 and why this is why this is just sufficient why this is not necessary it can happen it can happen that this is greater than 1 and this is less than 1 it can happen see this inequality says it is like saying 0.1 is less than 5 or 0.1 is less than 0.9 ok. So, if you get norm of a to be 5 quite likely that spectral radius of a could be 0.1 you do not know, but if if norm of a is 0.9 I surely know that spectral radius of a is less than 1 ok. If if a norm of a comes out to be 1.1 I cannot say anything about this being less than 1 or this being. Sorry if if induced norm typically will be 1 norm or infinite norm 1 norm if this is less than 1 definitely this is less than 1 ok, but if this is greater than 1 we cannot say anything about what is this ok. Inequality just says that this quantity is you know is less than this quantity we are particularly interested in this number 1 whether this is less than 1. So, this is less than 1 we are sure that this is less than 1 that is the model of this quantity ok. So, I can actually use the norm computation 1 norm or infinite norm computations to to come up with you know a sufficient addition for convergence ok. So, now let us actually apply this I know with more practical conditions because you know I will be talking about 1000 matrix how do I know how do I compute S inverse if it is lower than to the matrix computing S inverse is again not a big idea ok I still want a simpler conditions ok. So, what is this going to be now I am going to define a special class of matrices called as diagonally dominant matrices ok. I am going to define a special class of matrices called as diagonally dominant matrices or strictly diagonally dominant matrices and finding the summation I am taking a matrix ok. I am taking summation from z is equal to 1 to n. So, I am taking rho sum I am excluding one element from the rho sum which is that element. Now, if one of the diagonal element ok if one of the diagonal element is strictly greater than some of the remaining elements absolute some of the remaining elements ok. Then I am taking a matrix and this should hold for every I ok this is for I equal to 1 2 n. So, if this if these inequalities hold for each I then such a matrix is called as diagonally dominant matrix such a matrix is called as diagonally dominant matrix. Now, where is where will it help me diagonally dominant matrix I will go on here I will just I hope you have all these in your group book. Let us go back and see in the Jacobi method what was we wrote a as s minus t ok for Jacobi method for Jacobi method we had s is equal to I will then we also wrote this that a is equal to n plus d plus u then for Jacobi method s was d and t was n plus u minus of minus of n plus u ok minus of n plus u. Now, just think of this s inverse t which is minus d inverse l plus u what is d inverse d is a diagonal matrix all the elements inverse is just small upon diagonal elements ok. Now, if matrix a if matrix a is diagonally dominant if matrix a is diagonally dominant ok then what happens a i i a i i are all greater than the summation of all the row elements absolute number of elements what is this matrix can you this can you this right now what is this matrix for Jacobi method s inverse t will be what will be this matrix 0 minus a 1 2 by a 1 1 minus a 1 3 by a 1 1 minus a 1 n by a 1 1 a minus a 2 1 by a 2 2 0 minus a 2 3 by a 2 2 minus a 2 n by a 2 2 right. So, s inverse t will be a matrix which has 0 on the diagonal elements will be like this infinite norm no sum will be nothing, but what will be each row sum summation a i j j going from 1 to n divided by a i i a i i is divided in each row just look at here ok. But, this is the diagonal the other matrix which means this sum is strictly less than right this sum is strictly less than this if this sum is strictly less than this what does it mean that all these ratios are strictly less than 1 what does it mean infinite norm is strictly less than 1 if infinite norm is strictly less than 1 what can you say about the spectral radius of this matrix ok. So, now I have reduced checking whether J2B method will converge or not just to see whether a matrix whether it is right or not if matrix a is right or not ok my iterations will converge irrespective of where I start from ok irrespective of where I start from my iterations will converge if my matrix a is diagonal element. So, after doing all these jacquery in lot of arguments learning about you know matrix norms and then in spectral radius all that we have come up with a very simple criteria for finding out whether jacquery iterations are converged or not I will give you some more theorems either I can do the proofs of each one of them but I will give you some more theorems which are very elegant and from which you can ascertain whether your iterations will converge or not or you can modify your problem such that the iterations are guaranteed to converge ok. So, that is what we will see next class briefly and then move on to some other analysis, but this is where you can see you know I can value spectral radius and norms everything is action where you take it to analyze the behavior without actually having to solve it just looking at diagonal arguments I can come to the conclusion for any initial guess 5 iterations will converge ok very very powerful