 So, this last few lectures we have been looking at convergence of iterative schemes for solving linear algebraic equations. Starting from the basic equation for where the error evolves, so we had this iteration scheme, we had this iteration scheme to solve A x equal to b and A was written as s minus t and we said that the error which is defined as iterate x k minus the true solution x star this evolves according to e k plus 1 this is a linear difference equation it evolves according to this linear difference equation. Now from this we abstracted from this we abstracted a linear difference equation problem we said essentially we have to look at equations of this type and z0 is initial condition z0 is the initial condition and then we wanted to come up with the way of analyzing asymptotic behavior of the solution as k tends to infinity. So, we came up with with analysis based on Eigen values we came up with a condition that if rho b is nothing but max over i lambda i that means if lambda i are Eigen values of matrix b lambda i are Eigen values of matrix b we find out its absolute value well Eigen values can be complex. So, you find the absolute value and rho b this is called as spectral radius this is called as spectral radius and we said that or we showed that the necessary and sufficient condition if rho b that means spectral radius of matrix b is strictly less than 1 we said that the if the spectral radius of matrix b is strictly less than 1 then the sequence zk norm of that will tend to 0 as k tends to infinity. So, and from this we again connected to our original problem we said which means that if spectral radius of S inverse t is strictly less than 1 then this is necessary and sufficient condition for convergence of error for convergence of error the error will the error of between the true solution and a guess solution will diminish to 0 if this condition is satisfied. I am just doing a recap of what we have done till now. So, from this point we again had some difficulty because we have to compute Eigen values. So, we said well Eigen value computations are difficult and then we used one more result to come up with a sufficient condition. So, we said that spectral radius of matrix b is always less than or equal to any induced norm of matrix b spectral radius of matrix b is less than induced norm of matrix b what is induced norm induced norm this is induced norm induced by the norm defined on the range space and the domain space. So, this norm of a matrix is nothing but amplification power or something like a gain of a matrix if you can think about it as a gain or amplification power. So, if and then we came up with a sufficient condition that if induced norm is less than 1 obviously spectral radius is less than 1 and convergence is guaranteed. If induced norm is greater than 1 we cannot say anything if induced norm is less than 1 we are sure. So, we had a another condition that if induced norm is strictly less than 1 then spectral radius of b is strictly less than 1 and then this implies that asymptotically norm this means asymptotically norm of iterate z k or difference equation z k will go to 0 as k goes to infinity. So, this we can say without actually having to solve it ok. Now in particular we talked about infinite norm or one norm which are more convenient for doing calculations ok and now based on this I wanted to derive some results which are even more simpler I do not probably have to even compute the norm I can compute what is called as diagonal dominance. So, in my last lecture I talked about diagonal dominance. So, I wanted to further catch on this result that if the induced norm is less than 1 then of course, the spectral radius is less than 1 induced norm is very very easy to compute particularly one and infinite norm as compared to computing the spectral radius. So, checking whether a particular iteration will converge or not is very very easy. Now let us move back to the thing that we have done in my last lecture. So, this was overview of the entire stability arguments that we have been giving well I wanted to derive something more specific from the previous result. So, we come back here we are trying to solve for a x equal to b and a has been split as s minus t. So, for Jacobi method I in particular I analyze Jacobi method for Jacobi method s is equal to well we are also writing s as we are writing a as l plus d plus u this is strictly low triangular part of a this is diagonal part of a and this is strictly upper triangular part of a. So, s is equal to d and t is equal to minus minus l minus u and then I defined the concept of strict diagonal dominance. If you take some of absolute values of elements of matrix a in a row except the diagonal element and if that sum is strictly less than the sum is strictly less than you know the diagonal element then the matrix is called as diagonally dominant matrix. Just to give you a simple example well any you just write any matrix in which let us say this is my matrix a this is my matrix a this is the diagonal these are the diagonal elements these are the diagonal elements here. Just have a look if I take absolute sum of this this it is smaller than this I take absolute sum of this smaller than this. I just look at this matrix I look at its diagonal elements this particular matrix will obey this condition this is a strictly diagonal dominant matrix. Just look at this just look at this this is 2 plus 4 plus 3 plus 1 is always less than 15 same thing here 5 plus 3 plus 1 plus 9 is less than 23. So, I am taking absolute values. So, for this particular matrix can you calculate what is going to be what is going to be Jacobi matrix which is what is S inverse T can you calculate that just do it what is S inverse T well mind you again that Jacobi matrix when you actually do computations you never compute S inverse you do row by row calculations. This is for analysis is for getting insights, but what you will realize is that you just look at the diagonal elements you look at the sum of the off diagonal elements you can say whether the iterations are going to converge or not which is very very powerful result you do not have to actually solve it and this is true for 5 cross 5 for 10 cross 10 for 1000 cross 1000 if this condition holds iterations will converge. So, you are guaranteed convergence if you if this condition is satisfied. So, what will be S inverse T what will be Jacobi matrix let us call this Jacobi matrix S inverse T or for Jacobi matrix S inverse T what will it be this will be 0 it will be minus 1 minus 1 1 0 2 divided by minus 5 minus 5 minus 5 then this will be 1 0 3 minus 2 2 divided by 9 9 9 9 9 minus L minus U then what is this 2 4 0 3 minus 1 this divided by 15 15 15 and 15 then the next one is 1 minus 5 minus 3 0 and minus 9 divided by minus 23 and the last row is minus 1 by 5 minus 1 by 5 1 by 5 0 0. Now, what will be the infinite norm of this matrix you take absolute of rho sum you take absolute of. So, absolute of this plus absolute of this plus absolute of this absolute of all these things all these all these numbers is it always going to be less than 1 it is always going to be less than 1 because this matrix is strictly diagonally dominant the denominator some in the numerator this will appear actually for a diagonally dominant matrix what you know is that this divided by norm a i i this will be strictly less than 1 this will be strictly less than 1 you can see here you add absolute of each one of these rows if each one of them is less than 1 the maximum is also going to be less than 1 what does it mean spectral radius of this matrix spectral radius of this matrix is strictly less than 1. So, if a is diagonally dominant the Jacobi matrix which you get by S university has spectral radius strictly less than 1 which means Jacobi iterations will converge without having to solve it for arbitrary initial guess very very important for an arbitrary initial guess. So, any initial guess I give even if it is completely wrong my solution will my iterations will converge to the true solution if diagonal dominance condition is met. So, you can just check diagonal dominance of a matrix very very easily and then you know whether the solution is going to be obtained or not that is that is straight forward. Now, there are many more results of how do you analyze the convergence behavior and all of them I am not going to prove I have stated those results here in the and I am just going to state them and show you how to apply them and the proofs for each one of them or at least most of them some of them you can derive yourself for most of them are included at the end of the chapter that is end of the notes in the appendix. I do not want to go over it in the class you got the philosophy of how it is done and you have to look at the proofs in the appendix to understand more of this because we cannot spend time on this beyond a certain point as long as you get the philosophy it is fine. Now, what are the more results there are some more results which exploit the structure of matrix A one one structure that we exploit is diagonal dominance right. The other thing we will show is that if matrix A is symmetric positive definite if matrix A is symmetric and positive definite then Jacobi and Gauss-Seidel methods Gauss-Seidel method converges also you can show that if matrix A is diagonally dominant Gauss-Seidel method will converge the proof is the proof is little more involved and you should look at the proof given in the appendix I have given details of the proofs in the proof in the appendix. So, if matrix A is diagonally dominant Jacobi iterations will converge it is also true that if matrix A is diagonally dominant then Gauss-Seidel iterations also will converge. So, you just have to check for diagonal dominance you know that Gauss-Seidel iterations will converge and in fact a heuristic is that most of the times Gauss-Seidel iterations converge faster than Jacobi iterations. So, if you know that a matrix is diagonally dominant your preferred choice of using the method should be Gauss-Seidel method not Jacobi method. There are other theorems. So, the first theorem that you should know about convergence of iteration scheme is that if matrix is diagonally dominant A matrix then Jacobi method as well as Gauss-Seidel methods will converge to the true solution. The next result is by the way remember this is the sufficient condition this is not a necessary condition what does it mean that if this condition holds Jacobi and Gauss-Seidel methods will converge. If this condition does not hold even then this you cannot say if this condition does not hold you cannot say anything about convergence you have to go back and check something else you have to go back and check spectral radius. So, this is only a sufficient condition if this happens you are guaranteed convergence will occur if this does not happen we do not know we cannot say anything. So, this is sufficient condition not necessary condition. The other result is so the second result I would say the first result was very very important result. So, this symmetric positive definite line seems to be something very very nice it seems to help us everywhere we go and I will just show you how you can now you might start saying well I have been given this matrix I have been given this matrix A and you know only in very very special cases this A will be symmetric positive definite is not it A is a square matrix A is a square matrix I am not talking about now this square where we had a you know a tall matrix I am just talking about A is a square matrix and my given problem the problem which is given to me is such that A is not symmetric positive definite. But I know that Gauss-Seidel method or Gauss-Seidel method will converge sufficient condition for convergence is that if the matrix in my problem is symmetric positive definite is there something that I do to solve this problem to convert this problem into symmetric positive definite matrix very good excellent I just pre-multiply this equation with A transpose. So, this gives me A transpose A I do not have to solve for A x equal to b I can instead solve for A transpose A is equal to A transpose B I am guaranteed convergence. So, I am using my theory to change the problem in such a way that I am guaranteed to get converge solution. I am going to solve this problem using Gauss-Seidel iterations making use of this theorem how do I make use of this theorem to modify my calculations I pre-multiply both sides by A transpose this becomes a symmetric positive definite matrix. Now if I apply Gauss-Seidel method to this matrix and this problem this transform problem I am guaranteed to get a solution this solution is obviously a solution of if it is a solution of this it is also solution of this you have no problem with that. So, I could solve this transform problem instead of solving this problem I get a symmetric positive definite matrix here I am using theory to modify my calculations to I will just give you an example here I will start with. So, I want to solve for A x equal to b and my A matrix is 4, 5, 9, 7, 1, 6, 5, 2, 9 and my b vector is 1, 1, 1. Let us say I want to solve this by Gauss-Seidel iterations. So, well what I will do is I am now I am not this is not a solution procedure this is analysis from analysis what I know is that if I write this matrix as if I write this matrix as A matrix if I call this s and if I call this as t if I call this as s and if I call this as t then doing Gauss-Seidel iterations is equivalent to then my s inverse t my s inverse t will be 4 0 0 7 1 0 this is my s inverse t if I were to use the raw matrix A if I were to use raw matrix A and in this case the spectral radius of s inverse t turns out to be 7.3 which is strictly less than 1 if I use Gauss-Seidel iterations are not going to converge because if I just use the raw matrix A that matrix is neither diagonal dominant just check it is diagonal dominant it is not is it symmetric matrix it is not a symmetric matrix forget about positive definite it is not symmetric matrix. But if I know this little bit of information if I do this transformation that is A transpose A x is equal to A transpose B then this A transpose A matrix it turns out to be 90 37 and this A transpose this is A transpose A A transpose B becomes 16 8 24 and now if I apply Gauss-Seidel method to this matrix to this equation transformed then then the s inverse t spectral radius of s inverse t it turns out to be 0.96 okay. So for this for the transform problem for the transform problem I am guaranteed conversions of Gauss-Seidel method this is a symmetric matrix just see this symmetric matrix it is a positive definite matrix by definition A transpose A is always positive definite even if A is not positive definite you have seen this several times okay. So this is positive definite matrix symmetric matrix convergence is guaranteed just pre-multiplying both sides by A transpose I can ensure that I will get conversions by iterative method okay. So in the case where obvious things like diagonal dominance are not there if you want to ensure that you get conversions just pre-multiply by A transpose both sides and then use Gauss-Seidel you are guaranteed conversions very very powerful result yeah all the drivers. So this that spectral radius should be less than one is necessary and sufficient condition it necessary if the convergence occurs spectral radius should be less than one if spectral radius is less than one convergence will occur that is not the case with the norm if induced norm is less than one convergence will occur but if induced norm is greater than one convergence may or may not occur you do not know okay. That is not the case with spectral radius spectral radius is the absolute measure which will necessary and sufficient condition for okay. So it is possible to transform there are more results of this type again I am not going to go into the proof for relaxation method well we have this result for relaxation method what you can show again is the proof is again given in the appendix you should go and look at it I will look at it if omega is chosen between 0 and 2 well actually for relaxation method we want to choose it between 1 and 2 because we showed that omega equal to 1 is equivalent to Gauss-Seidel iteration so we want to choose it between 1 and 2 but in general if omega is between 0 and 2 okay we are this is a necessary condition for convergence okay so you cannot choose omega you know how to choose omega you have a guideline here okay so this again remember this is only a necessary condition this is not sufficient if you choose less than 2 that does not mean convergence has to occur but if convergence occurs only when you choose omega is less than 2 and well the this necessary condition the the this is result 3 and the necessary condition becomes necessary and sufficient conditions if extension to this theorem is another result that if this is for an arbitrary matrix okay this is for an arbitrary matrix now if a is symmetric and positive definite so if matrix a is symmetric positive definite okay then this condition necessary condition becomes a necessary and sufficient condition okay now you know how to transform the problem which is originally not symmetric positive definite to symmetric positive definite matrix okay so what I want to do the take home message is that all these results all these theorems are very very useful in shaping your calculations you should know how to make sure that convergence occurs convergence is very very important and then whenever you are not sure in a in a arbitrary large-scale problem you are not sure above a matrix how it's going to be if you want to use iterative schemes for solving a x equal to b it's better to use it's better to use you know a relaxation method in which you transform the problem because in general relaxation method will will converge faster than Gauss-Tiedel method I'll just show you in a very small example that Jacobi method is the slowest to converge typically Gauss-Tiedel method is faster and if you choose omega properly then the relaxation method will even converge faster okay now how do you choose omega such that you get very very fast convergence is very difficult to tell the priory you probably have to compute eigenvalues but that is not desirable you don't want to really compute the eigenvalues so you have to develop some kind of you know experience in beyond the point you have to use all these theorems and understand the theory and then develop experience to tweak with the calculations that's very very important okay so I'll just show you one simple example this is taken from Strang's book but it is very very illustrative very simple problem so I want to solve and such a simple problem of course you don't need any of the iterative methods 2 cross 2 systems you can sans solve it by hand so my a matrix is 2 minus 1 minus 1 2 well we'll say that this is Jacobi and Gauss-Tiedel will converge by symmetric diagonally dominant okay so anyway but that is on the point the point is the Jacobi method S inverse T will be 0 half half 0 and the spectral radius is equal to half for Gauss-Tiedel method S inverse T this turns out to be 0 0 half 1 by 4 and spectral radius is S inverse T okay a spectral radius is given by this actually spectral radius maximum magnitude eigenvalue of S inverse T is an indicator also of the performance okay now there are two aspects it should be less than one okay now how much it is less than one how close it is to 0 that also matters in terms of the rate of convergence whether the convergence is guaranteed or not is decided by whether it is strictly less than one okay that is a stability criteria the performance is given by how much it is less than one okay so this method Jacobi method in which spectral radius is half okay converges slower than converges slower than the Gauss-Tiedel method okay because the spectral radius here is one fourth in fact it you can almost if you start doing calculations you will see that one step of Gauss-Tiedel will be almost equal to two steps of Jacobi okay so the Gauss-Tiedel can move much faster because the spectrum this is you cannot show it for every matrix there is no proof that Gauss-Tiedel always converges but in general in general Gauss-Tiedel converges faster than and the reason is typically spectral radius of S inverse T for Gauss-Tiedel is less than okay that is the that is the reason now what if I formulate the relaxation method so you can almost show that because of this one GS iteration is equivalent to two one Gauss-Tiedel iteration is equivalent to Jacobi iterations okay because spectral radius in this case is even smaller now for relaxation method you know for relaxation method S inverse T turns out to be inverse of this matrix 0 omega minus omega here 2 inverse 2 into 1 minus omega omega 0 2 into 1 minus omega and of course we should choose omega between we want to choose omega between 1 and 2 we want it to be greater than 1 because if it is equal to 1 it is nothing but Gauss-Tiedel method okay we want to be greater than 1 now actually now for this simple case 2 cross 2 matrix you can actually find out what is the best value of omega that will enhance the convergence what is the optimum value okay for different choices of omega you will get different spectral radius okay we can actually find out which value of omega will this is this is just again to tell you a emphasize this is only to get insight okay in the real real problem I am not going to compute optimum omega by doing some I have to tune give a guess for omega okay so if you use some properties of matrices then you know that lambda 1 and lambda 2 if these two are eigenvalues of S inverse T then that is equal to determinant of S inverse T which turns out to be in this case if you take determinant of that it will be 1 minus omega 1 minus omega whole square okay and you know the other property you know this property multiplication of eigenvalues of a matrix is same as determinant and then what is the other property trace so lambda 1 plus lambda 2 is equal to trace of so this will turn out to be 2 minus 2 omega plus omega square by 4 now if you plot if you plot this that is if you plot S inverse T versus omega if you plot spectral radius using these two these two relationships you can find out lambda 1 lambda 2 and a spectral radius and if you plot this you will find that well getting the optimum is not very difficult if you plot this you will find that the optimum value if you plot this the optimum value for which the spectral radius is minimum you know you will get an you will get a point where you will get a minimum value of the spectral radius okay so that value turns out to be omega optimum is equal to 1.07 okay and the spectral radius of S inverse T for this omega is 0.07 okay I am skipping some steps you can see here in the notes that is not important what I want to point out here is that if you are able to choose omega properly then this is 0.07 so we had we had three situations you know Jacobi method then Gauss-Seidel method and relaxation method the S inverse T spectral radius in this case was half this was one fourth and this is you know 0.07 which is almost one fourth of this so one we said one iteration one iteration of Gauss-Seidel was two Jacobi iterations what can what can you say about one relaxation iteration it's like one relaxation iteration is like four Gauss-Seidel iterations okay and almost like eight Jacobi iterations so relaxation method can converge even faster typically values close to one one point one point one one point two are used this is thumb rule and not substantiated I think Strang gives some clue that you can use it close to one point two but it's hard to say generally what what value of omega will make conversions very very fast okay but so the tricks that you should use is first of all make sure that either the matrix is diagonally dominant check whether the matrix is diagonally dominant if it is not okay to ensure convergence you should pre multiply both the sides by a transpose that will make it symmetric positive definite I'm guaranteed convergence okay but I want convergence faster than Gauss-Seidel Gauss-Seidel is better than in general Gauss-Seidel is better than Jacobi so I will apply Gauss-Seidel and I can make convergence faster even going to relaxation method so probably I should all use all these tricks and use relaxation method to enhance my convergence that's how I should proceed with arranging my calculations so this brings us to end of this analysis what is important here is that there are many take home take home messages one of one of the thing is that eigenvalues is one of the prime tools for analyzing behavior qualitatively asymptotically I don't have to solve that is the beauty of this tool I don't have to solve the problem I can just look at eigenvalues or I can in this case it turns out finally I can just look at diagonal dominance I can see whether the convert the problem to a symmetric positive definite matrix I am guaranteed conversions of my iterative scheme very very powerful result in fact eigenvalues are used for convergence analysis in engineering literature in many many many ways okay well most of you I think have done the first course in chemical engineering on chemical process control right and in process control if you well you may not have connected it to the eigenvalues in the first course but actually what you can show is that if you write a differential equation for a local linear differential equation for evolution of the system dynamics then the so-called roots of the characteristic polynomial are nothing but eigenvalues of certain matrix which governs the system dynamics okay and if eigenvalues are on the left half plane and then you know what was the what was nice thing about eigenvalues there or roots of the characteristic polynomial you don't have to solve you just look at the roots whether they are lying in the this half of the plane or this on the plane you can tell how the system is going to behave asymptotically without having to solve okay same thing is here without having to solve I can tell whether my iterations will converge or not okay also using necessary sufficient conditions I can go and modify my problem to make sure that convergence will occur okay this is more important than the algorithm per se algorithm you will learn you will learn to program it or nowadays I think these algorithms might be available on the net you might download it okay algorithm might be very well written that doesn't mean the convergence is going to occur okay you should know why convergence occurs and then make sure that you transform the problem in such a way that convergence occurs that is important okay so this brings us to end of this now there are two more things that I need to do well because we missed one lecture some timetable is disturbed but I will try to make up make up for it we will I will try to cover gradient based or optimization based iterative methods for solving ax equal to b okay so now till now I formulated iterations in one particular way by splitting the matrix and in fact row by row calculations not really splitting the matrix the way the iterations were derived were doing row by row calculations okay my next aim is going to be instead of doing that can I iteratively solve this problem well can I solve if I want to solve ax equal to b okay if I take a guess solution the true solution is let's say x star okay and if xk is my guess solution then obviously ek now my ek has a different definition ek is b minus axk okay this is not going to be 0 when this is equal to x star it will be equal to 0 if xk is equal to x star this is equal to 0 the way I want to solve this problem is is minimize objective function scalar objective function phi is defined as e transpose e that is ax minus b transpose ax minus b with respect to x okay I don't want to solve this I don't want to solve this well you will say that if you apply the necessary condition for optimality you will get you know dou phi by dou x equal to 0 that will give you a transpose ax if you apply this condition dou phi by dou x equal to 0 then you will get a transpose ax is equal to a transpose b I don't want to solve this I don't want to go by this route I want to go iteratively okay I want to guess I want to guess x0 and by some method I want to go to x1 then I want to go to x2 and so on and then I want to see whether this iteration converges we are going to use what is called as the gradient search okay one of the fundamental methods in optimization gradient based search so we look at gradient search and then there is one more method called conjugate gradient search which we will look at next okay that is one thing which I want to do after having done that we have talked about iterative schemes for solving ax equal to b and then we move on to a very very fundamental issue matrix conditioning which problems are inherently ill conditioned which problems are well conditioned how do I classify and say that this is a ill conditioned problem whatever I do I am going to end up into some trouble this is a well conditioned problem if I am getting wrong solution I have made a mistake okay so well conditioned problems you know absurd solutions you have made a mistake ill conditioned problems absurd solutions well you can't do much how do you classify ill conditioned from well conditioned is the next thing that will bring us to end of this module