 We are considering solution of non-linear equations. Last lecture we showed that in the Newton's method, we have got quadratic convergence. Now, today we are going to look at the order of convergence in secant method. Then, we will consider a method which is known as regula falsi method. In the secant method, we may not have bracketing of zeros. So, this we modify the method, so that we are going to get an interval in which our 0 is going to lie. Then, we will consider what is the drawback of regula falsi method. Now, after this, we will consider iterative solution of system of linear equations. So, we have already considered direct solution. So, those are the Gauss elimination method and its variants. So, today we will consider two methods which are iterative in nature and those are the Jacobi method and the Gaugoss-Seidel method. We will give sufficient conditions under which these iterative solutions, they will converge to the exact solution of system of linear equation. So, let us look at the error in the secant method. So, in the secant method, this is the definition x 0 and x 1, these are the initial points and then we define x n plus 1 is equal to x n minus f x n upon divided difference of f based on the earlier two approximations x n and x n minus 1. The formula is valid for n is equal to 1, 2 and so on. Suppose that f has a simple 0 at c, that means f of x n is equal to 0, but f dash c is not equal to 0. This f of c is going to be equal to f of x n plus divided difference based on x n x n minus 1 multiplied by c minus x n and the error term. So, the error term is given by the divided difference of f based on x n x n minus 1 c multiplied by c minus x n c minus x n minus 1. So, what we are doing is we are looking at a linear approximation of f x based on interpolation based on x n and x n minus 1 and this is the error and we are putting x is equal to c. Now, you divide throughout by f of x n x n minus 1 and the term f x n divided by this divided difference plus c minus x n you will take it on the other side. So, when we do that on the left hand side we will have x n minus f x n divided by the divided difference of f based on x n x n minus 1 minus c. So, this is the term divided by the divided difference taken on the left hand side and on the right hand side you are left with f of x n x n minus 1 c divided by the divided difference based on x n x n minus 1 and now for c minus x n I am writing the error e n c minus x n minus 1 is the error e n minus 1. If you look at this term this is nothing but our x n plus 1. So, you have got x n plus 1 minus c is equal to if your function is 2 times differentiable then the divided difference in the numerator will be f double dash d n by 2. The divided difference in the denominator will be equal to f dash of r n where both these d n and r n they will lie in the interval bounded by x n minus 1 and x n multiplied by e n multiplied by e n minus 1. So, here on the left hand side what you have is minus of e n plus 1. So, take the modulus of the both the sides. So, you will have modulus of e n plus 1 is equal to modulus of this quotient into mod e n into mod e n minus 1. So, thus modulus of e n plus 1 is equal to modulus of f double dash d n upon 2 times modulus of f dash r n into mod e n into mod e n minus 1. If I call the quotient to be equal to alpha n then limit of alpha n as n tends to infinity will be equal to modulus of f double dash c divided by 2 times modulus of f dash c. So, this is the error e n minus 1. So, this is the error in the secant method. Now, recall the error in the Newton's method there we had modulus of e n plus 1 is equal to some beta n and then mod of e n square. So, this was for Newton's method and for the secant method we have got modulus of e n minus 1 is equal to alpha n mod e n mod e n minus 1. So, what was earlier mod e n square? One mod e n is replaced by modulus of e n minus 1. So, here in the case of Newton's method we could show that limit of modulus of e n plus 1 divided by mod e n square as n tends to infinity is equal to m not equal to 0. So, that is what gave us quadratic convergence. Now, because instead of mod e n 1 mod e n we have got mod e n minus 1 we will not get as high order of convergence at quadratic, but what we will have will be that it can be shown that the order of convergence in the case of Newton's method will be 1.618. Now, this part I am going to skip. So, you are going to have limit as n tends to infinity modulus of e n plus 1 by mod e n raise to p is equal to m which is not equal to 0. So, this p is going to be better than the linear convergence. In the case of linear convergence we have got p is equal to 1 whereas, now you get p to be 1.618. So, not as good as quadratic, but better than the linear convergence. Now, look at this formula for secant method. So, x n plus 1 is x n minus f x n upon f x n x n minus 1. So, I substitute for the divided difference f x n minus f x n minus 1 divided by x n minus x n minus 1 and simplify this term to get x n minus 1 f x n minus x n f of x n minus 1 divided by f x n minus f x n minus 1. Now, nowhere in the secant method we are making any use of the sign of f x n f x n minus 1. So, there is no restriction as such. So, this f x n and f x n minus 1 they can be of the same sign if they are of the same sign. So, see look at here you have got f x n minus f x n minus 1. These both f x n and f x n minus 1 they are going to converge to f of c. So, you will be divide you will be subtracting two numbers which are of the same magnitude and then it can be prone to the round off errors. So, now in order to take care of this or as a remedy what we can do is you start with x 0 and x 1. So, these two x 0 and x 1 you can make sure that f x 0 and f x 1 they are of opposite sign. Then you look at the intersection of the secant past the secant. So, passing through points f x 0 and f x 1 with the x axis whatever is that intersection that is our point x 2. So, now we have got three points f x 0 f x 1 and f x 2 earlier what we were doing was or in the secant method what we are doing is we consider now x 1 and x 2. Now, instead of that from x 0 x 1 x 2 you look at the if f x 1 and f x 2 are opposite of opposite signs then choose your points to be x 1 x 2. If they are of the same sign then you can choose your points to be x 0 and x 2. So, this is known as regular falsely method. So, let me describe the method. So, our assumption is that f from a b to r is continuous f a into a b is less than 0. You set a 0 is equal to a b 0 is equal to b and then you look at w which is a n times f b n minus b n times f a n minus b n times f a n upon f b n minus f a n. So, how do we get this formula? This is nothing, but the intersection of the secant passing through points f a n and f b n with the x axis. So, this is our w. If f a n f w is less than 0 that means if they are of opposite signs then you choose your a n plus 1 to be equal to a n and b n plus 1 to be equal to w. If they are of the same sign then choose a n plus 1 to be w and b n plus 1 to be equal to b n. So, it is similar to the secant method only at every step we make sure that our f a n and f b n they are going to be of opposite sign. So, that means we can say that our 0 is going to be equal to 0 into lie in the interval a n to be n. This is not the case with secant method we have got x n x n plus 1. So, when there is convergence both f x n and f x n plus 1 they will converge to f of c or they will converge to 0, but the interval x n to x n plus 1 is equal to 0 plus 1 it need not contain c. So, we do not have bracketing of 0 in the secant method. Now here in the regular falsely method because we make sure that f a n and f b n they are of opposite signs our interval a n to b n is going to contain our point c. The point c is where f of c is equal to 0, but in this case also there is a disadvantage. So, the disadvantage I want to explain by a graph suppose your function is what is known as concave up. That means when you take any two points on the curve and join them by straight line this secant it lies above the graph of your function. This is satisfied for example, if f dash x is bigger than 0 f double dash x is bigger than or equal to 0. So, let me look at the regular falsely method. So, here you have I start with a 0 and point b 0 I look at the straight line then it intersects the x axis at a 1. So, that is my new point a 1. So, we have a 0 a 1 and this is b is our b 0. If I look at a 0 and a 1 they are of the same sign both of them they are negative. So, what I am going to do is I am going to choose my points to be a 1 and b 1 b 1 is same as b 0. Now you look at the secant joining f of a 1 and f of b 1. Now once again b 1 is the x axis because it lies above the x axis your a 2 point is such that f a 2 and f a 0 they are or f a 2 and f a 1 rather we should compare f a 1 and f a 2. So, they will be of the same sign. So, that means you should choose the point to be a 2 and b. So, you are going to have a 2 b 2. So, which is identical to b then a 3. So, here the point of intersection w it will lie always to the left of c. That means your right hand side is always going to be equal to b n plus 1 is equal to b. So, that means the length of the interval it may not tend to 0. So, in secant method we did not have bracketing of zeros. In case of regular falsely method our a n to b n the interval a n to b n it is going to contain our point c our point our 0. But it can happen like I showed you graphically that one of the points of your interval that keeps changing, but the b n it remained the original whatever was our x 0 and x 1. So, in that if I call x 1 to be equal to b it remains the same. So, the length of the interval it may not shorten. So, it may not tend to. So, you get bracketing of 0, but that length may not be reducing as fast as we wish. In case of bisection method at least we know that at each stage our interval gets reduced by half. This may not happen in case of regular falsely method. So, there are advance advantages and disadvantages. So, let me compare now our Newton's method and the secant method. We have already considered their plus points and minus points. So, I just want to summarize about the two methods. So, in the case of Newton's method we have got x n plus 1 is equal to x n minus f x n upon f dash x n. So, at each stage you need to evaluate the function and the derivative. So, that means we have got two function evaluations per step you have got quadratic convergence that is the biggest advantage of Newton's method. The derivatives are involved. So, if it is complicated to calculate the derivative then Newton's method will not be recommended. In the case of secant method you have got x n plus 1 is equal to x n minus f x n divided by this divided difference. So, in order to calculate the divided difference you need to evaluate f x n and f x n minus 1, but you would have in the earlier step you would have evaluated f x n minus 1 and f x n. So, let me tell you what it is. So, you have got x n plus 1 is equal to x n minus f x n divided by f x n minus f x n minus 1 divided by f x n divided by f x n minus f x n minus 1 divided by x n minus x n minus 1 this is for x n plus 1. So, you have x n is equal to x n minus 1 minus f x n minus 1 divided by f of x n minus 1 minus f of x n minus 2 divided by x n minus 1 minus x n minus 2. So, that means this f x n minus 1 is common to evaluation of x n minus 1. So, you have got x n as well as x n plus 1 and that is why in the case of secant method you are going to essentially have one function evaluation per step except for the first step. In the first step you will need to evaluate f x 0 as well as f x 1. So, when you calculate or when you consider the number of computations for secant method they are half. Here you have two functions here you have got one function evaluation per step. The convergence is not as fast as quadratic convergence you do not need to calculate the derivatives. So, there are some advantages with the Newton's method and some advantages with the secant method. So, now we say that the Newton's method we have got quadratic convergence. This is under the assumption that f has a simple 0 that means f of c is equal to 0, but f dash c is not equal to 0. Now, suppose the function has two 0s or it has a double 0 rather that means you have got f of c is equal to 0 f dash c is equal to 0, but f double dash c not equal to 0. In this case we will see that the quadratic convergence in the Newton's method gets reduced to the linear convergence, but you can modify your Newton's method. So, as to retain quadratic convergence also in the case of a double 0. So, this is what now I am going to explain. We got quadratic convergence in Newton's method because of the fact that when we look at g x to be equal to x minus f x upon f dash x. So, fixed point iteration for this particular function is nothing but the Newton's method and this particular function it has the property that g dash c is equal to 0. So, we are going to relate the Newton's method to fixed point iteration and then that gives us a clue as to how to modify Newton's method in case of multiple 0s. So, as to retain the quadratic convergence I will take the case when it is a double 0. If it is a triple 0 or 0 of multiplicity m then the modification is similar. So, let us look at the order of convergence in Newton's method. So, look at g x is equal to x minus f x upon f dash x, then Newton's method is nothing but fixed point iteration for this particular function. If f of c is equal to 0 f dash c is not equal to 0 then we had seen this before g dash x is nothing but f x f double dash x by f dash x square. We have got f of c is equal to 0. So, that will give you g dash c to be equal to 0. Then you consider x n plus 1 minus c this will be x n plus 1 by definition is g x n c being a fixed point of g it will be equal to g of c write down the Taylor series expansion. So, that will be equal to x n minus c into g dash c plus x n minus c square by 2 g double dash d n. x n plus 1 minus c is e n plus 1 then x n minus c square will be e n square g double dash d n by 2. So, that is what gives you quadratic convergence in case of Newton's method. Now, suppose it has a double 0 c is such that f of c is equal to f dash c is equal to 0 and f double dash c not equal to 0 then you have g x is as before x minus f x upon f dash x. So, we are looking at the case when f of c is equal to f dash c is equal to 0 and you have got f double dash c to be not equal to 0 g x is x minus f x upon f dash x. So, we have got g of c is equal to c then g dash x when we look at it will be equal to 0. It will be f dash x square into f x f double dash x. So, when I look at limit of g dash x as x tends to c this will be equal to limit of f double dash x as x tends to c and limit of f x upon f dash x square as x tends to c. The first part is going to be equal to f double dash c and now here f of c is 0 f dash c is 0. So, it is equal to f of c is equal to f of c is going to be of the 0 by 0 form. So, we have to apply L'Hopital's rule and that will give you limit x tends to 0. So, take the derivative of the numerator that is f dash x and derivative of the denominator will be 2 f dash x f double dash x. So, that means this limit is going to be equal to 1 by 2. So, in case of simple 0 we had g dash c is equal to 0. Now, we have got limit of g dash x as x tends to c to be equal to half. This is what we will make the conversion of the numerator of the numerator to the convergence to be linear convergence if c is a double 0. So, what we want to do is we want to modify this function g. The reason we got quadratic convergence in case of simple 0 was g dash of c is equal to 0. So, under these conditions I want to modify my function g such that g dash c is going to be equal to 0. We have got limit of g dash x as x tends to c is equal to half and hence if I look at function g x which is equal to x minus 2 f x upon f dash x then what will happen will be when I look at g dash x. Here it is 1 then minus 2 and limit of this we have seen that it is half. So, there will be minus 2 into half. So, that will be 1 and then you are going to get limit of g dash x as x tends to c is equal to 0. So, thus if you make this simple modification that if you consider your x n plus 1 to be equal to x n minus 2 f x n upon f dash x n then you are going to have quadratic convergence. If it is a triple 0 then instead of 2 here you put 3. If it is a 0 of multiplicity m then you put here m. So, now this was about the solution of non-linear equation, one single equation. One can consider system of non-linear equations, but let us first look at system of linear equations. So, let us go back to our system of linear equations and consider iterative methods. So, our setting is going to be n equations in n unknowns. The coefficient matrix A is invertible. So, we have got A x is equal to B A is equal to A i j n by n invertible matrix. This is our additional assumption that A i i the diagonal entries of A they are going to be not equal to 0 for i is equal to 1 2 up to n. This A x is equal to B it is nothing but summation A i j x j j going from 1 to n equal to B i. I goes from 1 2 up to n these are the n equations. The right hand side B is given to us the coefficient matrix A is given to us x the vector x 1 x 2 x n that is unknown. Since A i i is not equal to 0 we can write it as x 1 x 2 x n. So, we write this as x i is equal to B i minus summation j goes from 1 to n A i j x j j not equal to i divided by A i i i going from 1 to up to n. So, it is the same system of linear equations I am writing in a different manner. So, now in the case of iterative methods what we are going to do is we will start with some initial approximation. So, x 0 is going to be our initial vector you can choose that vector to be equal to 0 0 0. So, start with a 0 vector. So, we have written the equations which exact solutions are not equal to 0. So, we have to satisfy that we had x i is equal to B i minus summation over j A i j x j except j not equal to i divided by A i i. So, now on the right hand side you put values for x j to be from the earlier one. So, like consider the starting iteration x 1 0 x 2 0 x n 0. So, put it on the right hand side and whatever new value you get that is going to be your x i 1. So, this is known as the Jacobi method and for this method we will give a sufficient condition under which the iterates will converge to the exact solution. These iterative methods become important when solving the system of linear equations by direct methods become expensive because we have seen that the direct solution is of the order of in cube by 3. So, if n is very large then it can be very expensive. Also in these direct methods we do not really make use of the fact that if your coefficient matrix A happens to be sparse if it has lot of 0 still your Gauss elimination method is going to cost in cube by 3 except if your 0s are structured. That means if your matrix is a coefficient matrix is a tridiagonal matrix then instead of the operations of the size n cube they will reduce to the operations of the size n, but the methods which I am going to describe now they are useful when you have lot of 0s, but they are they do not have a pattern like tridiagonal or something like that, but still they we have lot of 0s in that case these methods they can be useful. And of course, there will be sufficient conditions we have to have if the some conditions are satisfied then we are going to get convergence. So, essentially if your matrix is diagonally dominant then the Jacobi method as well as the Gauss Seidel method they will converge. So, here is the description of the method that A x is equal to b this is the equation satisfied by the exact solution start with the initial approximation and then x i k is equal to b i minus summation j goes from 1 to n j not equal to i A i j x j k minus 1. So, these you have obtained from the earlier one divided by A i i i goes from 1 to up to n. So, you calculate x i 1 or rather x 1 k x 2 k x n k once you have calculated all of them then you go to the go to calculation of x 1 k plus 1 because now on the right hand side you will know all x j k. So, this is the Jacobi method and now let us look at the error. So, when I consider the x i minus x i minus k when I subtract these two the b i's will get cancelled. So, you are left with summation j goes from 1 to n j not equal to i A i j by A i i x j and here similar thing with instead of x j x j k minus 1 and summation over j A i j by A i i j not equal to i. So, I subtract the two equations and I get x i minus x i minus k to be summation j goes from 1 to n A i j upon A i i x j minus x j minus x j k minus 1. Here this is the equation where there should be a minus afterwards because we are going to take modulus it will not matter, but here there is one extra minus sign. Then we look at the maximum norm. So, norm of x minus x k it is infinity norm is maximum of modulus of x i minus x i k 1 less than or equal to i less than or equal to n x is the exact solution it is a n by 1 vector. x k is the k th iterate it is also a n by 1 vector. Now, from here you will get modulus of x i minus x i k to be less than or equal to let me dominate x j minus x j k minus 1 it is modulus by norm of x minus x k minus 1 it is infinity norm. So, if I dominate by this norm it will come out of the summation sign. So, you are left with summation j goes from 1 to n j not equal to i modulus of A i j by A i i. Now, norm of x minus x k infinity will be less than or equal to maximum of this summation the maximum is over i. So, summation is over j goes from 1 to n j not equal to i. So, this number depends on i and then you are taking its maximum into norm of x minus x k minus x k minus 1 infinity. Let me call this number as mu. So, we have got the error in the k th iterate to be less than or equal to mu times error in the k minus first iterate. So, then replacing k by k minus 1 this will be less than or equal to mu times the error in k minus second iterate and so on. So, you will get norm of x minus x k infinity to be less than or equal to mu raise to k norm of x minus x 0 it is infinity norm. If your mu is less than 1 then mu raise to k will tend to 0 as k tends to infinity and the vector x k will tend to x as k tends to infinity x is the exact solution. So, this mu less than 1 it means summation j goes from 1 to n modulus of A i j j not equal to i should be less than modulus of A i i. These are the diagonal entries these are the off diagonal entries. In the same row so mu less than 1 that means strictly domin the strictly diagonally row dominant. So, if this is the case then your Jacobi method is going to converge. So, in case of Jacobi method what we do is we start with a initial vector. So, you have got x 1 0 x 2 0 x 2 n 0 0 is the superscript. Then using the formula you calculate the first iterates. So, you calculate x 1 1 x 2 1 x n 1. So, you are going to calculate do the calculations in the order. So, we have x 1 0 x 2 0 and x n 0 this is our initial approximation. Then you calculate x 1 1 x 2 1 and x n 1. This is your next step in the Jacobi method. So, when actually you calculate x 2 1 you have values available of x 1 0 and x 1 1. But we do not use this value we calculate all of these x 1 1 x 2 1 x n 1. So, in the Gauss-Seidel method what one does is when I want to calculate x 2 1 then my x 2 1 is going to be b i minus summation j going from 1 to n j not equal to 2 a i j x j 0 divided by a i i. So, I have got a term b i minus b i minus b i minus b i minus a here i should be equal to 2 and here it is a 2 2. So, it is going to be minus a 2 1 x 1 0, but I have I am going to go in order. So, for this x 1 0 actually I have calculated x 1 1. So, why not use that more recent value. So, if you do that x 1 1 x 2 1 x 2 1 2 that then you get the Gauss-Seidel method like when you are calculating say x n 1 the last one. Then for the x n 1 it will be in the case of Jacobi method what we do is we use the values x 1 0 x 2 0 x n minus 1 0. But now actually hopefully you will get the better approximations are available. So, why not use those approximations. So, at any stage whatever are the recent values of the approximations whatever are available like when I consider x 2 1 it will also need x 3 0 and then x 3 1 is not available. So, whatever is whatever recent values are available use those values and then one hopes that that should give you better approximation. Now, one can construct pathological examples where you would not have any improvement, but in general you will have improvement and what I am going to do is I am going to show that we had a sufficient condition for convergence of Jacobi method. So, that was mu should be less than 1. Now, in case of Gauss-Seidel method we will have another sufficient condition satisfied for the condition for the convergence of the iterates. So, we will show. So, in that case suppose eta is less than 1. So, what we will show is mu less than 1 implies eta less than or equal to mu less than 1. We are not saying that whenever Jacobi method converges Gauss-Seidel method has to converge that is false. What I am saying is we are going to obtain two sets of sufficient condition one for the Jacobi method another for the Gauss-Seidel method. If you compare these sufficient conditions then if the sufficient condition in the Jacobi method is satisfied it will imply that the sufficient condition in the Gauss-Seidel method also will be satisfied. So, let me first describe what is Gauss-Seidel method obtain a sufficient condition for the convergence and then compare the sufficient conditions in the Jacobi method and in the Gauss-Seidel method. So, as before our exact solution satisfies x i is equal to b i minus summation over j j not equal to i a i j x j divided by a i i. i goes from 1 to up to n. So, let me split this summation as j going from 1 to i minus 1 and j is equal to i plus 1 to n. Our summation is from 1 to n except for the term j not equal to i. So, except for the term j is equal to i. So, this summation I am splitting when we define the iterations for this sum we will use the recent value available and for this we will use the earlier value from the iterate. Now here the convention which we are following is if i is equal to 1 then this term will not be there it is from it will become from j is equal to 1 to 0. So, our convention is this term will not be there if i is equal to n then this term will not be there like when i is equal to 1 then you are going to have x 1 is equal to b 1 minus summation j goes from 2 to n. If i is equal to n it will be x n is equal to b n minus summation j goes from 1 to n minus 1. So, here is the definition these are the k iterate. So, you have x i k is equal to b i minus summation j goes from 1 to i minus 1 a i j x j k. So, we have already calculated x 1 k x 2 k x i minus 1 k. So, use those recent values here and then minus summation j is equal to i plus 1 to n a i j x j k minus 1. So, at this stage when you look at x i k x 1 k up to x i minus 1 k are available. So, use those values and here you have no choice, but you have to use the values from the earlier iterate. So, if it was a Jacobi iterate then here also it would have been x j k minus 1. So, here this is the Gauss-Seidel iterate look at x i minus x i minus k subtract the 2 you are going to have minus summation j goes from 1 to i minus 1 a i j by a i i x j minus x j k minus summation j is equal to i plus 1 to n a i j by a i i x j minus x j k minus 1. So, this is e i k this will be e j k and this will be e j k. So, e i k is equal to minus this summation minus this summation. So, let me define alpha i to be summation j goes from 1 to i minus 1 modulus of a i j by a i i beta i to be summation j going from i plus 1 to n modulus of a i j by a i i define alpha 1 to be 0 minus 1 and beta i to be 0. So, you will get modulus of e i k to be less than or equal to alpha i times norm e k infinity plus beta i times norm of e k minus 1 infinity. So, now as we did in the case of Jacobi method we will be trying to relate norm e k infinity with norm e k minus 1 infinity. Here it becomes slightly more complicated, but not much more. So, we will continue next time and we will obtain a sufficient condition for convergence of Gauss-Seidel method. So, thank you.