 So in the last lecture we were looking at the iterative methods for solving linear algebraic equations. In particular we looked at two different methods one was Gauss-Seidel method another was Jacobi method. I want to solve linear algebraic equation Ax equal to b we looked at two methods one was Jacobi method other was Gauss-Seidel method and I want to look at one more variation of these two methods in fact there are many variations I am just going to indicate few of them but this is over relaxation or relaxation method is slightly conceptually different from these two in the sense that you are trying to introduce a tuning parameter which will accelerate conversions. So what we are really going to look at is well what I told you that in general Jacobi method is slower than the Gauss-Seidel method Gauss-Seidel method we the difference is that as and when a new iterate is formed we use that in the next calculations that is the difference between Gauss-Seidel and Jacobi method. So Gauss-Seidel method actually tends to converge much faster than Jacobi method now I want to see whether I can do still better and I still further enhance the convergence okay. So this relaxation method this is a variant of these iterative schemes basic idea is like this suppose you know suppose I am at this point y let us call this point y and I want to reach y star I want to reach this point y star I am currently at this point y okay I am currently at this point y and to reach from here to here I apply some method okay I apply some method so let us say my method takes me from here to this point which is y cap okay so I will denote this point as plus okay I am at initial point y I have some method by which I am going to reach y star y star is where I want to go okay but when I have a method of going from here to here okay when I apply this method okay I go from y to y cap. Now if going from y to y cap is taking me closer to y star okay I could move more in this direction okay I could move more in this direction and then go closer even closer try to go in closer okay. So what I can do is I can go to a point y tilde which is y plus omega times y minus y cap so y cap is where my first method takes me y tilde is where if I enhance movement in this direction will take me so let us say this is my this is my y tilde so I am going to move further in the same direction I am going to move further in the same direction so I am going to enhance movement in that direction by moving in so my next iterate is not going to be y cap but I am going to have my next iterate as y tilde which is obtained by moving further in the direction. So the idea here is like this that if I start using Gauss-Siedl method okay I will get let us say I am at y through Gauss-Siedl I reached y cap okay so instead of taking y cap as my next iterate what I could do is I can look at this direction y minus y cap and then you know multiply that and add that to y so I will reach a point which is even further even probably closer to y star this is the basic idea in this relaxation business there are two kinds of relaxations one relaxation is over relaxation so we are going further in some cases it is under relaxation sometimes you know this direction in which you are going might be too far away from y star you may have to contract you may have to come back okay so there are two possibilities one is one is omega is less than one so this is well there should be some limit in this case it turns out to be for linear systems it turns out to be 2 so this is over relaxation okay so basically what I am going to do here is something like this well I am not going to use in the present case under relaxation I am going to use over relaxation I am going to have omega which is greater than one okay omega greater than one so see I am starting from initial guess xk and applying Gauss-Siedl method let us say we get okay see xik xik I am now talking about the ith element of the vector okay from xik if I apply Gauss-Siedl method let us say I get zik plus one I am not calling this x again okay this is my intermediate variable because this is not going to be my next iterate okay what I am going to do is I am going to come up with the next iterate xk plus one okay by enhancing the movement in this direction so my xik plus one okay this is going to be xik plus omega times zik k plus one minus xik where omega is omega is greater than one if omega is equal to one we are not interested in that case because we will reduce to Gauss-Siedl method okay I want to enhance I want to go further than what Gauss-Siedl method is giving me just try to understand this what I am going to do is I am going to start applying Gauss-Siedl method to find a new point but the new point I am going to take it as an intermediate point this new point which I get through Gauss-Siedl I am going to take an intermediate point okay and the next iteration is to be calculated using old guess plus this correction now this correction is developed by enhancing or moving further in the direction of the Gauss-Siedl this is Gauss-Siedl step okay this is the original point so difference between this will give you direction in which you have to move and omega times that okay so this is how I am going to generate the new iterate okay what happens if I put omega equal to one it will be Gauss-Siedl if omega is equal to one it is just Gauss-Siedl method if omega is greater than one it is over relaxation you are moving further than then and hope is that if I do this I will reach my solution faster okay so in general you can even make it you know move faster to the solution if you choose this omega between 1 and 2 so now the question that naturally comes is well how and how to choose this omega and when will it converge and so on so we will of course in a due course I will answer all those questions but is the idea clear we just want to enhance the direction in which we are moving that is Gauss-Siedl direction and move a little further so that maybe we will reach the target faster yeah but when you are reaching towards the solution this will be also small no so you are yeah so there might be some oscillations it depends upon now what whether it will cause oscillations or whether it will cause a smooth decay will depend upon eigenvalues and I am going to talk about that okay so yeah that is a good guess that you may have a problem when you we are close to the solution you might overshoot yeah so that may happen so we have to now relate and we have to understand how the convergence occurs and that is where okay so the actual details you can see here the algorithm I have given here how to implement but if you are the idea is clear algorithm will not be difficult to understand now let us start getting into analysis implementation part you know will be covered when you do programming and I am not worried about implementation part or the algorithm how will you efficiently implement this is given in the tables in the nodes and you can have a look at that when you the critical part is convergence analysis okay so what I am going to do now is rearrange my equations in a matrix vector form okay now this will give me a way to analyze the system of equations or the convergence behavior that is what I want to analyze the vector matrix rotation which I am going to develop here and going to use maybe for next 2 or 3 lectures is not meant for programming programming will be done you know row by row you put a for loop and each row you do calculations that is how you do the programming for this the analysis which I am going to do is going to be using matrices and vectors and that is mainly because analyzing convergence becomes very easy using vector matrix notations okay so do not confuse the 2 things whatever we did till now was meant for actually programming okay now what I am going to do is for analysis okay because getting insights into the analysis of this method using summations becomes very very difficult instead of that if you do everything using matrix notation it is very easy okay so let us get on to the let us say Gauss-Seidel method so what is my Gauss-Seidel method my first equation in the Gauss-Seidel method I can write as a11 x1 k plus 1 is equal to b1 minus summation i going from 2 to n right i going from 2 to n or j going from 2 to n a1 j or let us write it more explicitly let us write this instead of summations it will be better to write this explicitly so this will be a12 x2 k minus a13 x3 k a1 n xn k this is my first equation in the Gauss-Seidel method I have just earlier when I wrote the algorithm I had divided by a11 okay now my intention is different I want to get some convenient vector matrix notation so I have multiplied by a11 so this is the equation which I have to solve right my second equation here is my second equation is actually a22 x2 k plus 1 is equal to b2 minus a21 x1 k plus 1 minus a23 is this fine this is my second equation the iteration in Gauss-Seidel method okay this is Gauss-Seidel method okay this is Gauss-Seidel method the first iterate that was formed is used here the first iterate that was formed is immediately used here okay so I am going to rearrange this I am going to take this on this side okay I am going to take it on this side so I will get a21 x1 k plus 1 plus a22 x2 k plus 1 is equal to b2 minus a23 x3 k everyone with me on this I have taken k plus 1 on one side I have left k on one side okay I have left all xk on one side I am taking k plus 1 on one side not clear I have just taken this which was substituted on right hand side to left hand side okay likewise I can go on doing this so what will be the for the third one can you guess a31 x1 k plus 1 plus a32 x2 k plus 1 plus a33 x3 k plus 1 is equal to v3 minus a34 x4 k up to a4n xn k right I am taking all xk plus 1 on the left hand side I am leaving everything that is xk on the right hand side okay now I want to use I want to put this into a vector and matrix notation it is going to be a long equation okay so this here is a11 a21 a22 do you see this on the left hand side is the lower triangular part of a matrix okay the lower triangular part of a matrix is equal to lower triangular part and the diagonal part okay lower triangular part and the diagonal part what will I get here I will get 0 a12 up to a1n 0 0 a23 a2n right this will be last element here will be 0 and this will be a n minus 1n this into x1 k x2 k xn k plus what will be on this side well I have made a mistake one mistake it should be minus here this should be minus all this is minus minus signs will appear here okay because we have all minus signs coming there right and here will be b vector if you see it is b1 b2 b3 b4 okay the b vector will appear here okay this matrix appears here this matrix appears here okay what is the relationship of these matrices what is the relationship of these matrices to original matrix A LU no what is LU decomposition LU decomposition is completely different don't don't mistake LU decomposition or LDU you transpose that is not this this is this is physically the lower triangular lower triangular and the diagonal part of the matrix this is physically the upper triangular part minus of upper triangular part pardon me you add it so this is if I write this matrix if I decide to write A matrix as sum of three matrices okay see if I decide to write A matrix as sum of three matrices L plus D plus U don't confuse this with LDU transpose LU decomposition is different from this I am just using the notation L and U here this is not LU decomposition this is I mean if I draw a picture this is splitting a matrix into three parts this is diagonal this is strictly upper triangular part of this matrix and this is strictly lower triangular part of the matrix this is diagonal elements this is all elements that are in the okay so this is my U this is my L and this is my D okay so writing this matrix as if I just do pictorially I am just writing this matrix as addition of three matrices strictly lower triangular part the diagonal part and strictly upper triangular part I am just using the same notation as LU decomposition this is not same as this is not same as A is equal to LU that is multiplication that comes through Gaussian elimination this is just simply separating three physically separate parts okay so this and this do not confuse I am just using same notation that is all or also you might see in some books or in one in strang you also have this sometimes you write A as LDU transpose so do not confuse that and this okay only notations are same okay so I have written it like this so this makes me or this particular way of writing this matrix A allows me to do express this method Gaussian method in a vector matrix notation okay so this equation which I have written in matrix form I am going to rewrite this as L plus D into xk plus 1 is equal to minus u xk plus B is everyone with me on this just look at the matrices okay this is L plus D on the left hand side multiplying xk plus 1 okay here it is u strictly upper triangular part of A matrix A is written as L plus D plus u and then I can write this method next iterate is obtained by solving this matrix equation is equivalent to solving this matrix equation actually we are going to do line by line as I told you algorithmically we are not going to write this matrix equation ever we are going to do line by line calculations okay but what you are doing line by line is actually equivalent to this calculation is that okay okay so which means my next iterate is obtained as xk plus 1 is equal to minus L plus D inverse u xk plus conceptually what you are doing is obtaining a new iterate by this method okay sometime back I talked about inverting the matrix or approximately inverting the matrix and so on if you look here if you look here we are trying to say well I cannot invert the full matrix easily but I can invert the lower triangular part it has a nice structure I can exploit that and invert that okay to come up with a new iterate if I do a similar rearrangement of equations in the case of Jacobi method then what will I get if I do this is for Gauss-Seidel method for Jacobi method for Jacobi method what you can show is that doing the iterations is equivalent to D times xk plus 1 is equal to minus L plus u xk plus B xk plus 1 is equal to minus D inverse L plus u xk plus D inverse times B for Jacobi method if you do rearrangement of the calculations you can show actually doing one Jacobi step is equivalent to solving this equation what is nice about Jacobi method inverting the diagonal matrix is very easy just write 1 upon the diagonal elements right that is why we choose to do this so this is very very easy as compared to inverting the whole matrix if all the diagonal elements are nonzero if you arrange the matrix in such a way that all the diagonal elements are nonzero then doing this is very very easy it is okay so Jacobi method is equivalent to doing this okay now what about relaxation method okay relaxation method you can show that you have to do a little more algebra to come up with these matrices for the relaxation method but nevertheless finally you get this equation D plus omega L for relaxation method it turns out it turns out that it is this matrix it is this matrix right hand side is this and it solving this equation iteratively at each point okay so likewise there are other variations what I have taught you is called as actually forward Gauss-Siedl method there is also reverse Gauss-Siedl method okay then there is a reverse or backward Gauss-Siedl method then there is a symmetric Gauss-Siedl method in which you first do forward Gauss-Siedl then you backward Gauss-Siedl so there are all kinds of variations which make conversions faster okay so all of them for all of them you can actually express you can express iterations in terms of these fundamental three components L, D, U okay and so I have listed some of these variations here now let me generalize this let me generalize this so in general if I look at these equation I can have a pattern here what is this pattern this pattern is like this I am every time constructing a new iteration by following formula S inverse T xk plus S inverse B okay what is S matrix what is T matrix changes from method to method but essentially my equation looks like this whichever iteration method I take I take Gauss-Siedl what is S in Gauss-Siedl L plus D what is T here minus U T here is minus U okay so I can actually list this okay so my basic fundamental iteration equation is xk plus 1 is equal to S inverse T xk plus S inverse B now Jacobi method S corresponds to D and T corresponds to minus L plus U okay Gauss-Siedl method S corresponds to L plus D T corresponds to minus U can you guess this is forward back forward can you guess what will be backward Gauss-Siedl see here it is L plus D what will be backward Gauss-Siedl U plus D okay and here it will be minus L backward Gauss-Siedl will be S corresponds to U plus D and T corresponds to minus L in this case relaxation method what what is S this will be S matrix okay this will be T matrix actually you will have to probably divide by omega and then then get that to divide everything by omega then you will get S and T matrices okay to divide everything by omega you will get S matrix T matrix and then same idea okay the fundamental equation fundamental equation is this my fundamental equation is this okay so so basically what I need to do is I need to analyze behavior of this equation right how does this equation behave what is what is this equation have you seen this kind of equations new value is equal to a matrix into old value have you seen this kind of equations these are called as linear difference equations new iterate is equal to matrix into old iterate this is linear difference equation except you might have seen this in time okay here this equation is with respect to the iteration index of iteration okay so if I put it in some standard form okay which I am going to do soon then things will fall in place but you should be able to connect things what you have done some abstract form and when it is going to be applied somewhere okay now now let us say this iterations converge okay we still have not analyzed whether they converge or not but let us say they converge and say x star is the solution okay let us say my x star is the solution so where what will be the final equation what do we might converge what will happen if I what will happen if this converges it will give itself right the final value should give itself so x star is equal to S inverse T x star plus S inverse B is this fine so finally this is where I want to reach okay let me call this as equation 1 let me call this as equation 2 I am going to subtract this equation from this equation can I do that what will cancel this term will be 0 so if I subtract 2 from 1 you know I get this equation x k plus 1 minus x star is equal to S inverse T x k minus x star let me call this as error is this fine this is new error what is error error is the distance from the solution okay distance from the true solution x star is the true solution well I will show you that x star actually indeed is the true solution when you do this method okay we have to show that we should reach finally the point which is solution of A x equal to B that is that also has to be shown okay we will do that of course is everyone with me on this this is a linear difference equation okay and then finally I want to reach the solution of A x equal to B okay so far so good so now let us start analyzing this equation so we started from those complex row summations okay now we have come to a very compact nice form everything looks like this you know one simple equation x e k plus 1 is equal to okay how does this equation behave okay let us say I start with some initial guess say x naught is my initial guess okay what is the error at time x naught it is x star minus e 0 is equal to x 0 minus x star what is this error I do not know if I knew then I would know x star no I do not know what is this error nevertheless I can do analysis without requiring to know the error I am going to analyze the behavior of this equation the beauty of this analysis is that you can analyze without requiring to know what is e naught or what is even okay I am just going to look at properties of S inverse t okay if S inverse t has certain property okay I am guaranteed that this sequence of errors generated by this difference equation will go to 0 if error goes to 0 I am reaching the solution what is the meaning of error going to 0 difference between the iteration and a true is reducing that is what I want to happen right okay so let us start applying this equation so what is even S inverse t e 0 right what is e 2 S inverse t even which is S inverse t square e 0 this is okay I am just substituting for even I am just substituting for even okay what can you say about the error at instant k no at e k at e k can I write this is S inverse t raise to k can I write this well while denoting vectors this e is a vector by the way e is a vector x is a vector x star is a vector okay so x naught minus x star is a vector e is a vector this notation of superscript in the brackets is intentional you should not confuse it with you know raise to something this is not e raise to 0 this is 0th vector okay so in the iterations in the iteration at any point k in the iterations I have to now analyze what happens to this matrix when it is multiplied with itself multiple times okay so what should happen now is do you agree with me what I want to happen is this limit as k tends to infinity if I do more and more iterations I go closer and closer to the solution that is what should happen okay so which means limit as k tends to infinity S inverse t raise to k e naught should be 0 vector is e 0 a 0 vector no it is not a 0 vector so what I want to happen actually is S inverse t raise to k should tend to 0 to null matrix as k tends to infinity what should happen is S inverse t so I should choose S inverse t in such a way that this condition holds okay this condition holds looks quite formidable how do I choose a splitting of a matrix in such a way that this condition holds okay well not that bad as it looks like we will look at the theoretical basis then it will be clear that it is not that difficult to do this let me first show that we are indeed going to go to the solution okay by doing this error is going to go to 0 if error goes to 0 where will you reach okay you will reach the solution actually if I apply the difference equation again and again okay I can show I can start with this difference equation X k is equal to S inverse t or X k plus 1 is equal to S inverse t X k plus S inverse t S inverse b I can start with this difference equation apply it from 0 1 2 3 4 and I can show that actually X k is equal to S inverse t X naught X naught is my initial guess okay I am not deriving this you can derive this very easily what the expression that I am writing finally just recursively write the equations and you will get what I am writing plus S inverse t k minus 1 plus S inverse t less to k minus 2 up to S inverse t plus I into S inverse b okay if you use this difference equation again and again starting from X 0 you will be able to derive this expression okay now this expression has two components if you look here this expression has two components what are the two components one component is this S inverse t sorry S inverse t rest to k I forgot to put k here that is important S inverse t rest to k so this has S inverse t rest to k into X naught okay now what we wanted to happen was S inverse t rest to k should go to null matrix okay S inverse t should go to null matrix so which means if this goes to null matrix this part will be nullified this part will be nullified if I choose S and t intelligently such that this condition holds okay then this part will be nullified where will the iterations go right iterations will go to this now whatever I am writing here is this same as solving AX equal to b we have to prove that are you getting what I am saying see here I started with the argument that well my look at look look here I started with this argument here I said that error should be error should be defined like this then there is a difference equation that governs the error if I apply the difference equation actually I will get that error at time k will be S inverse t rest to k error at time 0 okay error will go to 0 provided this matrix this matrix goes close and close to null matrix okay this matrix that is my next argument this matrix should go close and close to null matrix then I looked at my original equation this is my original equation iteration equation I applied it again and again again and again and then I could derive this it is very easy to derive this expression okay you start with X0 X1 is equal to something X0 this then X1 is equal to or X2 is equal to S inverse t X1 and instead of X1 you substitute for okay you can eliminate and get everything in terms of X0 and then right hand side okay it just repeated application of this equation starting from 0 1 2 3 4 you will get this expression not very difficult to derive now if this part goes to 0 what it means is that as you progress in the iterations okay the effect of initial guess goes to 0 even if your initial guess is wrong if you have chosen S inverse t correctly this part will become 0 and whatever is your initial guess the solution will start going towards this vector on the right hand side so now what I have to show is that this vector on the right hand side is indeed the solution that is first what I am going to show second what I want to do is to give you insights into how do you choose S and t such that convergence is guaranteed okay how to choose S and t such that convergence is guaranteed that is my next mission okay so my claim is that if this condition happens that is S inverse t raise to k if this goes to 0 0 null matrix if this is tending to null matrix the solution that is xk tends to okay if this happens then iterations tend to this vector okay I want to use one matrix identity I suppose you have studied this in your first year or in your 12th standard i minus a inverse i minus a inverse can be written as a series right so this is actually a series where k is increasing so actually I can replace this I can replace this by this okay so which means which means if I use this particular series expansion okay I can say that actually xk is tending to i minus S inverse t inverse S inverse b right see this S inverse b is going to be there I am just replacing this square bracket I am replacing the square bracket by its limit as k tends to infinity this is the limit right this equation is as k tends to infinity as k tends to infinity I can replace I can replace this thing in the bracket okay by i minus S inverse t whole to the inverse is everyone with me yeah so what is this equal to what is this equal to can you expand this further this can be very easily shown to be S minus t inverse just expand this you will get S minus t you take S inverse common you will get S see this thing in the bracket can be written as S inverse into S minus t whole thing inverse right what is S inverse inverse S multiplied by S inverse will give you i okay what remains is S inverse t inverse but what is S minus t S minus t is nothing but a okay so this is nothing but just go back and see this is nothing but okay this is nothing but a so this sequence is going to converge to the true solution that is a inverse b this method of constructing sequence of iterates is actually equivalent to inverting a matrix and solving getting the solution provided you choose S and t correctly what do you mean by S and t correctly S and t should be such that S inverse t rest to k should go to null matrix okay so now the next part of the puzzle is that well under what conditions this will happen under what conditions how do you choose S and t such that you know you will go to under what kind of matrices we will look at class of matrices that will guarantee convergence then we will look at we will come to the point where we will be able to tweak these matrices using relaxation method basically what you may not expect right now but what is going to come is Eigen value analysis now what I am going to do is relate convergence to Eigen values of this matrix S inverse t and what I am going to show is that if Eigen values of S inverse t are strictly less than 1 okay then convergence will occur then the problem is transferred to how to choose S and t such that Eigen values of S inverse t are strictly less than 1 okay the problem is transferred to how do I choose S and t such that again so we are going to relate this to Eigen values okay we are going to relate this to Eigen values very very fundamental and actually analyzing difference equations of this type is where the Eigen value problem arises that is what I want to also highlight when we are taught in the linear algebra course Eigen value problem well the teacher comes and write lambda a v is equal to lambda v I mean why a v is equal to lambda v so you must have seen that Eigen value problem pops out when you are trying to solve linear differential equations I am going to show that it pops out when you try to solve linear difference equations okay same thing happens when you are trying to solve linear partial differential equations you get Eigen function problems okay Eigen functions and Eigen values here you will get Eigen vectors and Eigen values same idea used in different domains okay to understand how matrices behave without actually requiring to solve it so the beauty of that analysis is that we will be able to talk about conversions without actually solving for it we can just look at Eigen values and say whether this will converge or not converge okay that is what we are going to go towards okay. So we will continue that with the next class I will start with analyzing linear difference equations and their behavior and then correlate that to analysis of these equations.