 In the last lecture, we have discussed Newton-Raphson method for approximating an isolated root of a non-linear equation. We have also stated the convergence theorem of the method in the last class. In this lecture, we will prove the convergence theorem. As a first reading, you may skip this lecture. However, I advise you to carefully understand the proof of the convergence theorem of the Newton-Raphson method, because the mathematical tools used in this proof is very important for you, especially if you want to train yourself as a numerical analyst from the research point of view. Let us get into the theorem. Let us first state the theorem, which we have already stated in the last class. Let F be a C 2 function. Recall, to set up the Newton-Raphson iteration, we just need F dash of x n. That is, we just need the first derivative of the function F, but for the convergence, we need one more order of smoothness of F. That is why we have assumed that F is a C 2 function. And also, we will assume that the root R, which we are interested in is a simple root. It means what? It means F dash of R is not equal to 0. If these two conditions hold for our function F, then the theorem concludes that there exists a small neighborhood of the root R. That is what we meant by saying that there exists a delta such that R minus delta to R plus delta is a small neighborhood, delta neighborhood of R. And if you start your iteration, that is, if your initial guess is taken in this small neighborhood, remember, we have to find this neighborhood delta. There exist means we have to find this delta neighborhood such that if you start your iteration with your initial guess x naught in this neighborhood, then everything will go nicely. That is what the theorem says. What is mean by everything will go nicely? Let us see. First thing is each term of the Newton-Raphson sequence is well defined. What is mean by this? Well, what happens if you started with a x naught, then you went to x 1, then you went to x 2, like that you kept on going. At some x n, you see that your x n plus 1 is infinity. Then what happens? Well, then the Newton-Raphson iteration sequence is not well defined. The theorem says that if you start your initial guess pretty close to the root, then this will never happen. That is what it says. Well, the question is when such situation will happen? Let us see. Suppose, we are working with a function f whose graph is like this. You can see that it is a nice smooth function. At least visually, you can see that and suppose we are interested in capturing this root r. Now, you started with some x naught, say and you started computing x 1, x 2 and so on. At some n, say x n is this, then what happens? Your f of x n is this and now x n plus 1 is nothing but the point of intersection of the tangent line with x axis. What is this tangent line? Tangent line is parallel to the x axis and therefore, it never intersects x axis. That means, you can never get x n plus 1 at all if you face such a situation. This is a geometrical illustration. What is happening analytically? You see in this case, you will have x n plus 1 is equal to x n minus f of x n divided by f dash of x n. What is f dash of x n here? That is equal to 0. So, this becomes 0 here and that will give you infinity. This is how the Newton-Raphson iteration sequence may fail to exist. The theorem says that I can find a small neighborhood of r as long as I start my initial guess within that neighborhood, then such a situation will never occur. That is what it says. Suppose, you see if you restrict yourself to this neighborhood and start your x n here, x naught here, then you see your x 1 will be somewhere here and your x 2 will be somewhere here and your x 3 will be somewhere here. Everything will lie in this interval only. That is, all your x n's will lie in this interval only and they all will be well defined. So, that is what we are going to see. So, as long as you start your iteration with the initial guess x naught in a small neighborhood like this, then your Newton-Raphson iteration sequence will be well defined. The sequence x n will remain within this interval. This is what I have shown geometrically in the previous slide and the important point is that the sequence will also converge to the root and finally, this gives us the quadratic convergence of the Newton-Raphson method. Well, it means what? This quantity is equal to constant. Some finite number, that is what we have to show in order to conclude that the sequence converges with this order. You can see now here it is 2. Therefore, the order is 2. Why this term is finite? Because we assumed f dash of r is not equal to 0 and f double dash is a continuous function. That is what we assumed here that f is a c 2 function and we are always restricting to a closed and bounded interval. Therefore, this is a finite quantity. This is also non-zero finite and therefore, this entire thing will be finite. So, if you prove this expression, it precisely means that the Newton-Raphson iterative sequence converges quadratically. Let us see how to prove this theorem. The proof of this theorem is much simpler than the proof of the secant methods convergence theorem, but the idea goes more or less the same. First, we have to establish this relation for some xi which is very close to x n and this we have to establish for each n. Let us see how to prove this expression. It is not very difficult. You start with the Newton-Raphson method formula and now what you do is you take r minus on both sides. So, you will take r minus this equal to r minus this term. Now, this right hand side can be rewritten like this. It is very easy to see that the right hand side with r minus x n minus f of x n divided by f dash of x n can be rewritten like this. Now, let us see how to deal with this expression for that. First, we will go to the Taylor's theorem and try to expand the Taylor's theorem around x n at the point r. If you recall, you are taking a equal to x n and you are writing the Taylor's formula for some x equal to r here. What is r? r is precisely the root of the equation f of x equal to 0. Therefore, the left hand side is 0, right hand side is precisely the Taylor formula and this is the reminder. Now, you can see that this term that is the linear term is precisely what you have here. This is quite natural because in the Newton-Raphson method, we precisely did the linearization and took that expression, right. That is why this and this are just coinciding. Now, what I will do is this term is equal to minus of this term, right. So, I will take this reminder term to the other side and I can replace the numerator in this expression by minus of this, right. So, that is what I am going to do. I will replace this with minus of this one. I hope you understand this where xi n lying between r and x n, which is coming from the reminder term of the Taylor's theorem. Once you do that, you get this expression. This is precisely what we wanted to show, right. Therefore, we have obtained our claim. That is, we wanted to prove this and we have derived this expression right from the Newton-Raphson method. Now, we can use this expression to prove all our conclusions. Let us see how to do that. First of all, if you recall, we have assumed that the root r is a simple root and also if you recall, we assume that f dash is continuous. In fact, we assume that f double dash is also a continuous function. Why we assumed? First and foremost, we want this Taylor formula to hold. For that, we need f to be c 2, right. We already used that hypothesis that f is a c 2 function. Now, we are only using a part of it. That is f is a continuous function. Therefore, we will combine these two results. That is, f dash is not equal to 0 at the point x equal to r. It means what? When f crosses the x axis at r, f dash of r is not equal to 0. Therefore, it will either cross like this or it may cross like this and so on, right. At the point r when it crosses, it has some non-zero slope. That is what we meant by saying r is simple. Once f dash of r is not equal to 0, then we can find a small neighborhood of r in which f dash remains non-zero, right. So, in this interval f dash of x is non-zero. How we get this? This is by the intermediate value theorem. So, intermediate value theorem says that if the continuous function f dash is non-zero at r, then you can find a small neighborhood say delta naught neighborhood. That is r minus delta naught to r plus delta naught such that f dash will remain non-zero for all psi in that neighborhood. That is what the intermediate value theorem says. Remember we have to capture a delta neighborhood, right. But this is not the neighborhood that we want. We want further a small value delta from where we pick up. We should choose this delta which is less than delta naught such that this quantity is less than 1. What is this? Well, capital M is the maximum of f double dash and small m is the minimum of f dash. So, minimum of f dash is always greater than 0 because we are restricting ourselves to this neighborhood where f dash is not equal to 0. Therefore, this will always be positive. Well, why should f double dash and its maximum should be positive? Well, that can be assumed without loss of generality. This may not be true, but we can assume this without losing any generality. Why? Just see if m is equal to 0, it means f double dash of zeta is 0 for all zeta in the neighborhood. That is this delta neighborhood. It means what? It means f dash of zeta is constant for all zeta in this neighborhood. It means what? f is a linear polynomial. It means its graph is a straight line. f graph is a straight line. So, what it means? If you take any x naught and apply the Newton Raphson method, if you recall Newton Raphson method, we will take the tangent line at the point x naught and see the point of intersection of that tangent line with the x axis. In that sense, you can see that if f is itself a straight line, then the Newton Raphson method will capture the exact root at the first iteration itself. Therefore, there is no question of convergence and there is no question of the existence of the Newton Raphson iteration or anything. Everything becomes trivial. Therefore, we will not consider the case that f double dash of zeta is equal to 0. It means we will assume that f double dash of zeta is not equal to 0. Therefore, without loss of generality, you can assume that M is greater than 0. We choose our delta which is less than delta naught. You should also think and observe that such a choice is possible. I will leave it to you to see why such a choice is possible. Therefore, you have to choose like this and the corresponding delta you take. Now, let us see that this delta will work very nicely for us. How? Let us see that choose x naught in that smaller delta neighborhood, not the delta naught. Delta naught we are not going to take. We are going to take the delta which is smaller than delta naught such that this holds and this will work very nicely for us. Let us check that you choose x naught in this neighborhood. Remember, in this neighborhood f dash is not equal to 0. That is something which is already there because it is there for the bigger neighborhood itself. Therefore, it will be there for this smaller neighborhood also. Now, let us take this formula which we have derived as the first claim of our proof. We will use this how? Take modulus on both sides. Take modulus on both sides. You can see that this term is less than or equal to. Now, what I am doing is I am taking modulus and putting maximum for this that is this one and minimum for this. Therefore, this will be less than or equal to m divided by 2 small m into this I am keeping as it is. So, I am not taking modulus here because I am squaring it. So, we got this inequality. Let us see what we can do with this inequality. In this inequality you take n equal to 0. Now, what you will get? The first left hand side will be mod x 1 minus r and that is less than or equal to m by 2 m into mod x naught minus r square. That is what I am taking like this. Now, this is less than delta and this is less than 1. Why? That is how we have chosen our delta. Therefore, as long as x naught belongs to this interval, this first term will be less than 1 and the second term obviously will be less than delta. Therefore, the both together will be less than delta. What it means? It means that you see x 1 minus r is less than delta means x 1 belongs to r minus delta to r plus delta. So, it means if you start your x naught from this neighbor root, we have shown that x 1 will also belong to that neighbor root. So, that is what we have shown. Now, you can use an induction argument to show that all x n's belongs to this neighbor root. That is very easy for you to show. So, what we have shown? We have obtained a delta neighbor root. Remember how we obtained? We obtained this delta such that this inequality holds. That is always the demand. So, once you choose such a delta, we have seen that all the x n's belong to this neighbor root. That is what we have seen and also you can see that as long as x n belongs to this neighbor root, all x n will exist. Why? Because the only way they do not exist is when x dash of x n minus 1 is 0. But that can never happen because in this interval, in this neighbor root, basically we have chosen this delta such that if dash is not equal to 0, therefore all the x n's will be well defined. Therefore, we have also shown this property. Let us now prove the convergence. It is not very difficult. Let us start with the inequality that we have already derived in our previous slide. First thing is you can see that x n minus r is less than delta because x n belongs to the delta neighbor root. Therefore, one term in this square I will take and put it. In fact, this can be put as strictly less than delta and remaining m by 2 m into x n minus r is kept as it is. Now, let us go to apply the same inequality for x n minus r. See, this kind of recursive idea is used in many places. So, once you use this inequality for x n minus r, you will get delta m by 2 m square into mod x n minus 1 r. Again, you use this inequality for mod x n minus 1 minus r. That gives you less than or equal to, well it is strictly less than delta into m by 2 m cube into x n minus 2 minus r. Like that, you can keep on going till what? Well, you will go till you hit x naught minus r. At that level, you will have this constant as delta m by 2 m to the power of n plus 1. Now, you see what is this constant? You can show that this constant is less than 1. Why it is so? Well, we have chosen our delta such that m by 2 m into mod x naught minus r is less than 1. What is mod x naught minus r? That is less than delta. Therefore, this implies delta into m by 2 m is actually less than 1. Now, you see you have this constant which is less than 1 and that constant is given with power n plus 1. Now, what can you say about this right hand side as n tends to infinity? That is the question. Well, as n tends to infinity, this term goes to 0 right, because this constant is less than 1. That is precisely what we want to show because once this side goes to 0, then left hand side will also tends to 0 and that is precisely the convergence of the sequence x n right. So, we have proved the convergence also. Now, all reminds is to prove this order of convergence that is not very difficult. Again, you start with this expression which we have derived as the first step of our proof. Now, you can see you want to show limit n tends to infinity mod x n plus 1 minus r. You see that is there already with you divided by mod x n minus r square that is already there. So, you bring it to the left hand side and that will be equal to limit n tends to infinity f double dash of xi n divided by 2 into f dash of x n right. That is what is given here. Now, you see x n tends to r. We have just now proved the convergence. Therefore, this is converging to r. Similarly, xi n lies between r and x n. Therefore, you use the sandwich theorem to conclude that xi n also converges to r as n tends to infinity because x n is converging to r and xi n is lying between r and x n. Therefore, xi n will also converge. Therefore, your numerator will converge to f double dash of r. This is what precisely we wanted to show in the order of convergence and that is also achieved. By this, we have proved the convergence theorem of Newton-Raphson method. The Newton-Raphson method converges if you start your initial guess pretty close to your root. That is what the theorem says and the important take away of this theorem is that if the sequence converges then it converges quadratically. With this, we will end this lecture. Thank you for your attention.