 We are learning iterative methods for linear systems. In this, we have introduced two main iterative methods namely Jacobi method and Gauss-Seidel method. We have also learnt the convergence theorems for these methods. In fact, in the last class, we have proved the necessary and sufficient condition for the convergence of general iterative method written in a particular form and this condition depends on the spectral radius of the iterative matrix of the method. From this theorem, we learnt that if the spectral radius is very close to 1 but less than 1, then the corresponding iterative method may converge rather very slowly. We have also illustrated this phenomena in Jacobi method and in fact, this can happen with Gauss-Seidel method also. In this lecture, we will learn a slight modification of an iterative method through a relaxation technique. We will do this with Gauss-Seidel method and see how to accelerate the convergence speed of the Gauss-Seidel method. Assume that we are given a linear system Ax equal to b where a is an n cross n real matrix which is invertible and b is a given n dimensional vector. Recall that we can write the Gauss-Seidel iterative procedure to approximate the solution of the given linear system Ax equal to b and component wise the Gauss-Seidel formula is given like this where xi is the ith component of the iterative vector x k plus 1 and k runs from 0 to infinity for k equal to 0 we are given the initial condition x of 0 and from there for each k equal to 0 1 2 and so on we can get x 1 and then x 2 and so on whose components are computed using this formula and this is the Gauss-Seidel method. In fact, we can write this method in the form x k plus 1 is equal to b g x k plus some vector c where b g is the iterative matrix for the Gauss-Seidel method and c is a vector which is derived from the right hand side vector b. We have seen that the Gauss-Seidel method will converge if and only if the spectral radius of the matrix b g is strictly less than 1. If the spectral radius is very close to 1 then the convergence will be very slow in the sense that the sequence may take many terms to achieve the required accuracy of the solution. In such slow convergence situations we are interested in looking for some way to accelerate the convergence say for instance suppose we want the accuracy say our computed solution should be very close to the exact solution in such a way that this error is less than say 10 to the power of minus 5 and suppose it takes a 100 iterations that is k equal to 100 to achieve this accuracy then it may be computationally very costly especially when the matrix is very large in its dimension that is n is a very large integer. In such cases we look forward to have some way to accelerate this convergence that is achieve this accuracy but for some lesser value of k say for instance k is equal to 20 or something like that then that will be very nice when compared to achieving the accuracy with large k right. Now the question is is it possible to somehow modify your original iteration in such a way that you can accelerate the convergence. I hope you understood what I meant by accelerating the convergence. So, there is a nice idea what you do is just denote the Gauss-Seidel formula by z instead of x and now for a given non-zero real number omega we will now define our iterative sequence x k component wise as x i of k plus 1 is equal to 1 minus omega into x i k plus omega into this expression ok. So, this is just a slight modification of the Gauss-Seidel method we are introducing a parameter omega here and writing the iterative formula in this form. Now the question is what is the advantage of rewriting our original iterative formula in this form by introducing a parameter here. Well before answering this question let us just name this parameter omega in literature omega is called the relaxation factor. If omega is less than 1 generally we call it as under relaxation and when omega is greater than 1 we call the method as the over relaxation. You can observe that when omega is equal to 1 then this term goes off and you will have x i k plus 1 is just equal to z i k plus 1 and z i k plus 1 is precisely the Gauss-Seidel formula right. Therefore when omega is equal to 1 our method just reduces to Gauss-Seidel method and in general we call this method as successive over relaxation method. So, here the troublesome part is when the convergence is rather slow. So, our purpose of introducing omega into this formula is to make the convergence little faster. In that way generally we call omega as a relaxation factor ok. So, what we have done so far is we have defined an iterative sequence in this way which is just an extrapolation of the Gauss-Seidel method in this way and it is called the successive over relaxation method. Here we have to choose a omega and based on that omega we define the successive over relaxation iteration process. In short we generally call it as SOR. Therefore for a given initial guess x naught for every k first you will find z k plus 1 plug in that here and then get x k plus 1. Once you have x k plus 1 then again you will repeat this process and in that way you can generate a iterative sequence x k right. Now our interest is to write this formula in the matrix notation for that first what we will do is we will substitute this expression into the first equation and get this equation for x k plus 1 right. Now let us rewrite this equation for this we will first multiply both sides by a i i and then take this term to the left hand side and write this expression in this form. Now you see it is more or less very clear from here how to write this equation in the matrix notation because this is the formula for computing one particular component of the vector x k plus 1 right. So, in order to write it in the matrix form first let us write a as the diagonal part of a plus the strictly lower triangular part of a plus strictly upper triangular part of a. Then you can see that this term is having the diagonal part this term is having the strictly lower triangular part and you can see again this has the diagonal part and this has the strictly upper triangular part right. Once you understand this it is more easy for us to write this expression in the matrix notation well you have d right l again you have d here and u therefore, you can combine these two and write d plus omega l and now we are making everything as vector therefore, this is vector x k plus 1 is equal to 1 minus omega into d minus omega into u into x k plus omega into b vector. So, that is the vector form of our successive over relaxation method you can now multiply both sides by d plus omega l inverse and you can see that this equation now can be written in this form if you recall in the last class we have proved the necessary and sufficient condition for the convergence of any iterative method that can be written in the form x k plus 1 is equal to some t times x k plus some vector c where t is the general form of an iterative matrix right. Now you can see that the SOR method is also written in this form with the iterative matrix as S omega which is precisely d inverse plus l because we are now multiplying both sides of this equation by d plus omega l inverse right. Therefore, S omega is equal to d plus omega l inverse into 1 minus omega d minus omega u that is what is the expression for the iterative matrix S omega in the SOR method and the vector c is given by omega which is already there in this term into you are multiplying d plus omega l inverse right. Therefore, that will come here into b which is already there in the term. So, this is how we can write the SOR iterative formula in the matrix notation next our interest is to see how to choose the relaxation parameter omega. So, that the convergence of the sequence is accelerated well the idea is to look for the spectral radius of the iterative matrix S omega and try to make it as small as possible. That is the idea well the first question theorem towards this is by Cahen the theorem says that if you are working with the coefficient matrix A whose diagonal elements are non-zero then the spectral radius of the iterative matrix S omega will be surely greater than or equal to mod omega minus 1. Well the proof of this theorem is very simple. So, let us prove it go back to the expression of S S omega is given by the product of two matrices right. This is one matrix and you have another matrix and S omega is written as the product of these two matrices. Therefore, the determinant of S omega can be determinant of this first matrix into the determinant of the other matrix. So, this is a very simple property of determinant. Now, once you have this look at the determinant of D plus omega L inverse. Since L is strictly lower triangular you can see that determinant of D plus omega L inverse equal to determinant of D inverse. Once you have this you can plug in this instead of this expression and then again use the property that determinant of the product of two matrix is equal to the product of the determinants. So, once you use that you can write the resulting equation as determinant of S omega equal to determinant of when you take this inside you can see that it becomes 1 minus omega remember now D plus omega L inverse is nothing but D inverse when you take the determinant right. So, when it goes inside you will have 1 minus omega into D inverse into D that becomes identity matrix minus omega into D inverse that is coming here into U which is already there. Now, let us try to understand how this will be when you are taking determinant again you use the fact that D inverse U is the strictly upper triangular matrix. Therefore, you can join these two and look at the determinant of that matrix and that will happen to be simply determinant of 1 minus omega into the identity matrix. Therefore, the determinant of S omega which is given like this is finally reduced to this. Now, since the determinant of a diagonal matrix is simply the product of all the diagonal elements you can see that this is precisely equal to 1 minus omega to the power of n. If you recall the definition of the spectral radius you can see now that when determinant of S omega is given like this which is also the product of all the Eigen values of S omega. You should keep this property of determinant also in mind and look at this you can immediately see that the spectral radius again you have to recall the definition of spectral radius and keep in mind that determinant of a matrix is equal to the product of the Eigen values of that matrix. You use these two conditions together to see that the spectral radius of S omega is greater than or equal to mod omega minus 1. Therefore, this property is proved. Now, what this property tells us? This property tells us that if SOR is converging it means you have chosen your omega in the interval 0 to 2 that is what it says because from the theorem that we have proved in the last class if you recall any iterative method that can be written in the form x k plus 1 is equal to t into x k plus c will converge if and only if the spectral radius of this iterative matrix is strictly less than 1. And we have written SOR in this form and we have used the notation S omega instead of t right. Therefore, SOR iteration is converging means rho of S omega is less than 1, but Keynes theorem says that the spectral radius of S omega is greater than or equal to omega minus 1. You combine these two conditions and you can see that convergence of SOR iteration implies omega belongs to the open interval 0 to 2. Now, what about the converse of this statement? Well, we cannot in general say anything about the converse of the statement however, if a is a symmetric positive definite matrix and if omega belongs to the open interval 0 to 2 then you can say that the SOR iteration sequence will converge for any given initial guess x naught. Well, the proof is little lengthy therefore, we will omit the proof interested students can learn the proof from this classical book. From these two theorems what we understand is when we are working with SOR method it is always important that we should choose omega in this interval. Well, whether it will lead to a convergent sequence or not that we do not know, but definitely you cannot go away from this interval and choose a omega and hope that your sequence will converge that is not possible rather you have to choose your omega in this interval, but then whether the convergence will happen or not still you do not know it will depend on the spectral radius of S omega for the chosen omega. Let us now define what is called optimal choice of omega for that let us first consider the set which we will denote as capital omega and it is the set of all omegas in the open interval 0 comma 2 such that the spectral radius of S omega is less than 1. Let us call this as the admissible set because for all such omegas in the capital omega our corresponding SOR sequence will surely converge right. Now, what you do is you take the spectral radius of S omega for all omega in the admissible set capital omega and take the minimum of all these numbers. We will denote it by omega star and call it as optimal parameter for the SOR iterative sequence why it is so because for this choice of omega we will achieve the fastest convergence right that is why this parameter is called optimal parameter. Now the question is for any given matrix A can we compute omega star well it is in general very difficult to obtain omega star, but often one can just get this omega star with few trial and error it may be an approximation, but sometimes it works dramatically well in terms of speed of convergence than the Gauss-Seidel method. Let us take an example let us consider this system if you recall we have already considered this system in our previous lectures let us take the Gauss-Seidel method if you recall the iterative matrix for Gauss-Seidel method in this example is given like this and the Gauss-Seidel iterative sequence is given like this you can immediately see that the spectral radius of the Gauss-Seidel iterative matrix is equal to 1. In fact, you can also directly see that this sequence will lead to an oscillating sequence just to see this let us take the initial guess as 1 1 1 and apply the Gauss-Seidel iteration for this initial guess you can see that the Gauss-Seidel iteration sequence will be an oscillating sequence and it will never converge. We have seen this behavior in one of our previous lectures itself now let us see how we can choose omega and with that omega how the resulting SOR sequence will behave with the Gauss-Seidel for this particular example. You can immediately see that A can be written as the diagonal matrix D given like this plus the strictly lower triangular matrix L given like this plus the strictly upper triangular matrix U given like this. Once you have this you can immediately form the iterative matrix for the SOR method for any non-zero omega and if you recall we have derived this matrix and its formula is given like this for any omega now is restricted to 0 to 2 because we have seen that if you go away from this interval the SOR method will not converge right with little difficulty you can explicitly write this matrix and it is given like this. Now you see if you want to get the spectral radius of S omega as a function of omega you will have to find the Eigen values of this matrix which itself will be very difficult remember you are only with 3 by 3 system right for this itself the spectral radius expression will become very complicated that is why we told that finding an optimal omega is in general very difficult it may be possible for 2 by 2 case but even for 3 by 3 it becomes very difficult because you have to find all the Eigen values of this matrix and then you have to take the modulus of that and then take the square root of that Eigen values and take the maximum and that will give you a function of omega and now you have to find the minimum of that function right. So, that is theoretically very complicated rather you can just compute the spectral radius numerically and you can plot it and see where the minimum of the spectral radius of S omega is achieved as the function of omega here I am plotting the graph of rho S omega where the x axis represents the values of omega and the y axis represents the values of rho of S omega and the blue line is the graph of the function rho of S omega and the optimal omega that is omega star is achieved at this point which is roughly 0.86 and also you can see that if you take omega is equal to 1 we get back our Gauss-Seidel method right and the spectral radius of Gauss-Seidel method as we have seen explicitly its value is 1. Now, let us apply the SOR method with omega is equal to the optimal value omega star and see how the iteration goes on as I told let us take omega is equal to the optimal value which is roughly 0.86 for this omega the spectral radius of the corresponding iterative matrix of the SOR method is around 0.21 which is pretty close to 0 right. Therefore, we expect a rather faster convergence in the SOR method for omega equal to 0.86 let us see how the iterations go we again take x naught is equal to 1 1 and 1 and see how the iteration goes the first iteration is given like this and the L 2 error involved in the first iteration is around 0.92 and as we go on with the iterations you can see that the error falls down quite rapidly and within 11 iterations we got a pretty accurate solution for our system. If you recall we have also computed Jacobi iteration for this system and we have seen that after 200 iterations the error was around 0.0002 which means we captured the solution up to around 5 significant digits only after 200 iterations whereas, you can see that we captured the exact solution up to around 7 significant digits within 11 iterations using SOR method with omega is equal to 0.86 which is the optimal value of omega in this particular example. Also recall that the Gauss-Seidel method gave an oscillating sequence and therefore, it is not a convergent sequence at all. So, in that way SOR method with a proper choice of omega may often lead to a faster convergence. However, getting the optimal parameter omega star is in general very difficult. However, in certain particular cases we can obtain the optimal value of omega. For instance if your matrix is a symmetric positive definite and tridiagonal matrix then the optimal relaxation parameter omega star can be obtained using this formula that is omega star equal to 2 divided by 1 plus root 1 minus spectral radius of the iterative matrix SOR method. This is from the corresponding Jacobi method. Remember this is the notation we used for the iterative matrix for Jacobi method. So, first you have to compute the Jacobi iterative method and find the spectral radius of that matrix. Once you have this you can get the optimal relaxation parameter for SOR method if the given matrix is a symmetric positive definite and tridiagonal matrix and the formula is given like this. Well the proof of this theorem is very difficult. So, we will omit it in our course. However, interested students can get the proof of this theorem from a classical book authored by J. M. Ortega. Let us take an example. Let us consider this 4 by 4 tridiagonal system. You can see that this system is a symmetric and positive definite system. Let us try to find the optimal omega for the SOR method to this system. For that you first have to find b j and then obtain its spectral radius. How will you get? Well you have to find all the eigenvalues of this matrix b j. Remember we have given the formula of b j in one of our previous lectures. You go back and see that formula then compute this matrix b j and then find all the eigenvalues of that matrix b j and from there you can get the spectral radius. Once you get the spectral radius you can simply use this formula to get the optimal value of the relaxation parameter. For this system the spectral radius of the Jacobi iterative matrix is given by 0.80902. Once you have this you can immediately get the optimal relaxation parameter using the formula given in our previous theorem. It is given by 1.25962. In fact, from the explicit formula you can clearly see that the optimal parameter in this case of the coefficient matrix will always lie in the interval 1 to 2 and the spectral radius is given by 0.2596. That is again pretty small. Therefore, we expect a faster convergence again for this system when we use the SOR method with omega is equal to this. Let us see the plot of the spectral radius of S omega as a function of omega. You can see that for any value of omega in the open interval 0 to 1 the spectral radius of the corresponding S omega is strictly less than 1. That means, for any choice of omega SOR method will surely converges for this particular linear system. Moreover, you can also see that omega star which we have computed using the formula given in the theorem is indeed the minimum of this function rho of S omega. Again we will start our initial guess with 1 1 1 1 and you can see that the convergence is pretty good and we got around 6 significant digits of the exact solution. Remember the exact solution is just the 0 vector and we got the exact solution up to around 6 significant digits in just 12 iterations. So, this is what finally we got in the SOR method when we took the omega as the optimal value of omega which is 1.2596 around and the same when we take omega equal to 1 as you can see in this plot at omega is equal to 1 the spectral radius is way below 1. It is at 0.2596. Around 6.5 therefore, even Gauss-Seidel method will converge, but as you can see that the Gauss-Seidel method is converging rather slowly because its spectral radius is around 0.65 whereas the spectral radius of SOR with this parameter is around 0.25. Therefore, the SOR method converged quite faster that is in 12 iterations we got around 6 significant digits accurately whereas after 12 iterations you can see that the Gauss-Seidel method gave around only 3 significant digits. And in fact, Gauss-Seidel method took around 28 iterations to achieve the accuracy which is almost equivalent to the accuracy achieved by the SOR method. So, in this way we could accelerate the convergence of the Gauss-Seidel method quite efficiently by introducing a relaxation parameter into the sequence and we derived a resulting method called SOR method with this let us finish this lecture and thank you for your attention.