 In this lecture we will continue our discussion and prove the convergence theorem for fixed to point iteration method. We will also continue our discussion on some modified versions of Newton-Raphson method in order to achieve quadratic convergence in the Newton-Raphson method when the root is not a simple root that is if it has multiplicity something greater than 1. Let us start our discussion with a convergence theorem of the fixed point iteration method. If you recall in the last class we have listed three assumptions that we want to make in order to choose a good iteration function. We will put those assumptions as the hypothesis of our theorem. The first assumption is that the function g which we want to choose as an iteration function should be a self map on an interval of our interest and also the function g should be continuously differentiable. That is g should be continuous and its derivative g dash should exist and it is also continuous on the interval a b and the third assumption was the contraction principle that is the maximum of the absolute value of g dash of x over the interval a comma b should be strictly less than 1. So, these are the three hypothesis that we will impose and the theorem says that if we impose these three conditions on our iteration function then the iteration function g will have a unique fixed point in the interval a comma b. The existence of the fixed point comes from the assumption that g is a self map and g is continuous on the interval a b whereas, the uniqueness comes from the assumption that the function g is a contraction map. This is not very difficult for you to see I will leave it to you to prove this. The second conclusion of the theorem is that if you choose your x naught inside the interval a comma b then all the terms of the sequence will belong to the interval a comma b and the sequence x n will converge to the fixed point r as n tends to infinity. The third conclusion is rather interesting the conclusion says that you can write mod x n minus r this is the absolute error is less than or equal to lambda to the power of n into x naught minus r observe that if you prove this inequality then you will achieve the convergence because lambda is less than 1 therefore, lambda to the power of n will go to 0 as n tends to infinity right. Therefore, convergence will come immediately once you prove this inequality and not only that there is another interesting inequality that can be derived from here and that says that the absolute error is in fact, less than or equal to lambda to the power of n divided by 1 minus lambda into mod x 1 minus x naught. What is interesting in this well you can see that the right hand side involves all the terms which are known to us at least after computing x 1. You see lambda is also known to us here and therefore, this term is completely known to us. Suppose if we want our iteration term x n to be very close to r for instance we want the accuracy of our approximation in such a way that the absolute error is say less than or equal to some epsilon where epsilon is some tolerance error which we choose then what you do is you can impose that condition here that is this term is less than or equal to epsilon in that way you can achieve this accuracy and now you see in order to achieve this accuracy you can see how many iterations are needed for us to compute and that information you can get without explicitly computing the iteration terms. If you recall such a result was obtained in the bisection method also how did we get such an n well you can take log on both sides right and from here you can get an inequality for n that is we will get the integer n greater than or equal to some expression which can be computed fully without explicitly computing the iterations. That is the interesting part of this inequality you should recall how we did it in the case of bisection method come back you can also do it here and get a n which says that we have to perform at most that many iterations in order to achieve this accuracy that is very interesting and finally, the theorem also concludes that limit n tends to infinity r minus x n plus 1 divided by r minus x n is equal to G dash of r. If you recall this is one way of defining linear convergence right as long as this is a finite number note that r belongs to the interval a b which is a closed and bounded interval and G dash is a continuous function on this closed and bounded interval therefore, G dash of r should be a finite number. Therefore, this expression in fact tells that the fixed point iteration method will in general have at least linear convergence. Let us go to prove this theorem the proof of this theorem is not very difficult. In fact, as I told the first part is very easy it is a very simple exercise in your calculus course therefore, I will leave it to you as an exercise. Let us try to prove the second result. In fact, the second result can be achieved if we prove the inequality in the third part of the conclusion. Therefore, we will try to prove this inequality for that we have to use the mean value theorem. Before using mean value theorem first let us start with the absolute error r minus x n plus 1. Since r is the fixed point you can write r is equal to G of r that is what I am writing in the first place here and the way the iteration method is defined you can see that x n plus 1 is equal to G of x n. That is why I am writing G of x n here therefore, your absolute error is precisely equal to modulus of G of r minus G of x n. Now, I will use the mean value theorem which says that I can find a xi n such that this term is equal to G dash of xi n into r minus x n right. So, that is what I am going to use and of course, there is a modulus here and now go back to our statement we have taken the maximum over the interval a b of modulus of G dash of x and we call that as lambda right. Therefore, I can replace this by lambda and I will get this to be less than or equal to lambda into r minus x n and that is what I am writing here this term is less than or equal to lambda times r minus x n modulus right. So, we got this now what is the next idea whenever you see such an inequality you immediately go to apply this inequality recursively that is what you do is you can write this as less than or equal to lambda into lambda times mod r minus x n minus 1 that is applying the same inequality for mod r minus x n will give you this that is equal to lambda square into mod r minus x n minus 1 again apply this inequality to mod r minus x n minus 1 that gives you lambda cube into r minus x n minus 2 and you can keep on going like this till you hit the last term that is r minus x naught. And therefore, you can get this inequality mod r minus x n plus 1 is less than or equal to lambda to the power of n plus 1 into mod x naught minus r. If you recall that is what we wanted to show as the first part of the inequality in the third conclusion. Let us now try to derive this part of the inequality again that is not very difficult before going to derive that we will also observe that lambda is less than 1 that is the contraction property of the iterative function g and that says that lambda to the power of n goes to 0 as n tends to infinity and that gives us also the convergence and that also completes the second part of the conclusion. So, let us prove the last part of the inequality for that let us take the right hand side mod x naught minus r and what we do is we will add and subtract x 1 in that and then we will use the triangle inequality to write like this and now you see this is already there in our right hand side whereas, this term needs to be rewritten, but that is not very difficult because we already have an estimate of this term from here. So, we will try to use this idea into it with n plus 1 replaced by 1 then what we will get this is less than or equal to lambda times mod x naught minus r right. So, that is what we get here now you see you have mod x naught minus r on the left hand side you have lambda into mod x naught minus r on the right hand side you take it to the left hand side you will have 1 minus lambda into mod x naught minus r right. So, that is more or less what we wanted to show therefore, that can be brought to the right hand side why because lambda is less than 1 therefore, when you bring it to the right hand side the inequality will not get disturbed right. So, we got this now what you do is instead of this you put the right hand side estimate for this that gives us lambda to the power of n which is already there now mod x naught minus r we will write as less than or equal to 1 by 1 minus lambda into mod x naught minus x 1 and this is precisely what we wanted to show in the third conclusion you can see this is what we wanted therefore, all remains for us to now show the order of convergence. Let us see how to prove the order of convergence that is also not very difficult again you go back to our definition of the fixed point iteration that gives you the first equality and from this to this we will use the mean value theorem this is what precisely we have done at the beginning of this proof the same exercise we are doing once again and we get this unknown xi n lying between r and x n right. Once you get this you can see that r minus x n plus 1 divided by r minus x n is equal to g dash of xi n we have already proved that x n converges to r right and xi n lies between x n and r now you use the sandwich theorem and that will tell us that when we take limit n tends to infinity we will have g dash of limit n tends to infinity xi n remember g dash is a continuous function and now this converges to r right because of the sandwich theorem and that will immediately give us what we want in the order of convergence part right. So, this also shows that the order of convergence of the fixed point iteration method is in general 1 that is it will have at least linear order of convergence well let us try to see some examples. Let us take this equation sin x plus x square minus 1 equal to 0 you can see that there is a root of this equation in the interval 0 to 1 if you recall we have computed a root of this equation in the interval 0 to 1 using bisection method secant method and also Newton-Raphson method. Now let us try to formulate a fixed point iteration method for this equation and see whether it converges or not, but now for the fixed point iteration method we have to choose a good iteration function right. Let us propose these 3 iteration functions for this equation and see which one will be good for us to capture a root of this equation in the interval 0 to 1. Let us take the first choice g 1 of x and we are interested in this equation. Now in order to see whether g 1 is a good iteration function or not we have to check 3 properties right. One is the self map whether g 1 is a self map or not well you can see that definitely g 1 is not a self map because you choose any point in this interval and put that into this expression you will get a negative value right. Therefore, g 1 is not a self map however that is not a very serious problem in going for g 1 because you can always extend the interval on the negative side little bit in order to make it a self map, but then you have to also check whether it is continuous function which is also not very difficult for you to check and also you have to check whether it is a contraction map or not. I will leave it to you to check that. Let us go to the next choice g 2 and again we want to check all the 3 assumptions. Let me directly go to check whether it is a contraction map or not. Remember in order to check whether g 2 is a contraction map or not we have to first differentiate g 2 with respect to x and the derivative of g 2 is given like this and you can clearly observe that g 2 dash of x is greater than 1 that is absolute value of g 2 dash of x is greater than 1 for x in the interval 0 to 1 right. That shows that g 2 is not a contraction map in the interval 0 to 1. Let us still go to compute few iterations with g 2 as the iteration function. Let us start our iteration with x naught equal to 0.8. I have shown here first 3 terms of the iteration sequence. You can see that the error is gradually increasing as you go on computing the iterations. In fact, if you go on computing the terms greater than 3 you will see that the error increases more rapidly and that gives us a feeling that the iteration sequence seems to be not converging to the fixed point of the function g 2. It means the corresponding fixed point iteration method is not converging in this case that is not surprising because g 2 is not a contraction map. Let us now consider our last choice g 3 and see whether this is a good iteration function for us or not. You have to check first whether it is a self map or not well you can see that it is a self map and also it is a continuous function. You can easily check that and finally the question is is it a contraction map on the interval 0 and 1? Well for that we have to first compute g 3 dash of x and that is given like this and from here we can see that mod g 3 dash of x is less than or equal to 1 by root 2 for any x in the interval 0 to 1. That shows that g 3 is a contraction map in the interval 0 to 1 therefore, it satisfies all the hypothesis of our convergence theorem which implies that the fixed point iteration sequence with g 3 as the iteration function will surely converge if you choose any x naught in the interval 0 to 1. Let us take our x naught as 0.8 and you can see that the error is gradually decreasing at least up to 3 iterations that I have shown here. In fact, I have computed it for more iterations and I have observed that the error is gradually decreasing that gives us a feeling that the sequence in this case is converging to a fixed point of the iteration function g 3 in the interval 0 to 1. With this our discussion on fixed point iteration method is over. Let us list some of the key takeaways of the fixed point iteration method. One thing is fixed point iteration method in general will provide you at least linear convergence especially when g dash of r is not equal to 0 right. What happens if g dash of r is equal to 0 then well we can expect a higher order of convergence. Why it is so? Again take our derivation using mean value theorem. Now you use the Taylor expansion instead of applying the mean value theorem that is you take the error r minus x n plus 1 and that can be written as g of r minus g of x n and from here you can see that g of x n can be written in the Taylor's formula with Taylor polynomial of degree 1 plus the reminder term right. Once you do that you can see that when g is not equal to 0 then the error is actually dominated by this term and hence you have linear convergence right. On the other hand if g dash is 0 then what happens you can see that this term vanishes and then this term will remain as the dominating term in your truncation error which will show that the error is actually having a quadratic convergence. Therefore in your fixed point iteration method if your g in addition to the 3 assumptions that we made if it is also happens to be that g dash of r is equal to 0 then we will have at least quadratic convergence that is what we are seeing from this expression. Now the question is how to obtain a g such that g dash of r is equal to 0 let us see how to do that let us start with our basic equation f of x is equal to 0. Now what you do is you add x on both sides. Now this will define an iteration function g of x right but what I want I am not now interested in any g of x but I want that g such that g dash of r equal to 0 right. For that what I will do is I will introduce a function phi of x here and then I will try to choose this function in such a way that g dash of r is equal to 0. Let us see how to do that I want to choose my phi of x such that the iteration function g of x will give me the condition that g dash of r is equal to 0 why that will give me quadratic convergence in the fixed point iteration method right. So, that is the interest for us let us see how to achieve that for that first we will differentiate this function g with respect to x and then put x equal to r in that expression and equate it to 0 that will give us this expression what I am doing I am just differentiating g with respect to x then putting x equal to r and then equating to 0 in order to achieve this condition right. In this you can notice that this term will be 0 because r is a root of the equation f of x equal to 0. Therefore, this third term will vanish and we will have 1 plus phi of r into f dash of r is equal to 0 right and that will give us phi of r to be minus 1 by f dash of r remember this will go well only when f dash of r is not equal to 0 that is when r is a simple root of our equation f of x equal to 0 that we have to keep in mind, but when you take phi of r is equal to minus 1 by f dash of r then we will get g dash of r is equal to 0 which is of our interest. Therefore, this will motivate us to define our iteration function like this that is x plus phi is now minus 1 by f dash right. Therefore, it is minus 1 by f dash of x into f of x if you define your iteration function like this and if f dash of r is not equal to 0 then you will get quadratic convergence well this is another point of view where we made our fixed point iteration method to converge quadratically by appropriately choosing the iteration function remember the convergence will be quadratic provided the sequence converges that is more important this is not something new to us because this is what precisely we learnt as Newton-Raphson method right. In fact, in the convergence theorem of Newton-Raphson method we have derived this expression which also tells us that the sequence with iteration function like this that is the Newton-Raphson method will converge quadratically you can see that this is what we have derived in that theorem. We are just putting a different point of view of what we studied in our previous classes that is all we are not doing anything new this is just another point of view but why am I putting this point of view well there is a purpose for that. So, what we conclude therefore, is that if f dash of r is not equal to 0 that is if r is a simple root of our equation then the Newton-Raphson method converges quadratically we know it very well the question now is what if f dash of r is equal to 0 that is the question. Now that is to say that if r is a root of the equation f of x equal to 0 with multiplicity something m some integer m which is greater than 1 then what happens will the Newton-Raphson method converges quadratically no it may not converge quadratically therefore, now our next question is can we get an iterative method with higher order convergence that is can you define another iterative method that can give you quadratic convergence when our root is with multiplicity something m greater than 1 that is our next question well there is a nice idea to do this the idea is look for the root of this equation capital F of x equal to 0 what is capital F capital F is given by F of x divided by f dash of x. Now when you define your function capital F like this what is the advantage let us see first you can see that r is also a root of the equation capital F of x equal to 0 right that you can easily see now the next question is what is the multiplicity of r for the function capital F of x. Let us recall what is the definition of multiplicity a root r of the equation F of x equal to 0 is said to have multiplicity m if we can find a function q such that F of x can be written as the product of this multiplicity remember this is the multiplicity part and the rest of the part of the function F of x is written as q of x where q is such that q of r is not equal to 0 and also q is continuous at r. So, this is the definition of multiplicity of a root what is interesting now is to check what is the multiplicity of r for this function F that is our aim for that what we will do is we will take this expression and substitute and substitute in this place and also we differentiate F and substitute that in this place and see what happens well then capital F of x can be given by remember we are taking F of x equal to x minus r to the power of m into q of x right that is what we have put therefore F of x is given like this. Now I am differentiating this and putting that expression in the place of F dash of x in the definition of capital F and that gives us this expression you can see that you can pull out this x minus r to the power of m minus 1 and then cancel it with the numerator term and you will get capital F of x equal to this expression now from here you can see that capital F of r equal to 0 that is r is a root of this equation F of x equal to 0 and what is the multiplicity well the multiplicity can be seen very easily since q of r is not equal to 0 remember this is coming from the definition of multiplicity of r and that tells us that this term is not equal to 0 therefore you can see that F dash of r is not equal to 0 and that says that r is a simple root to the equation F of x equal to 0 right. So, what we achieved here we assume that r is a root of the equation F of x equal to 0 with multiplicity m then you define capital F of x equal to small f of x divided by F dash of x then we have seen that r will also be the root of the equation capital F of x equal to 0 and this will be a simple root of that equation now instead of applying the Newton Raphson method to F of x that is small f of x you can also apply the Newton Raphson method to capital F of x right. So, that is the idea therefore, you take the iteration function now as like this remember this is the Newton Raphson iteration function applied to capital F of x now that is the idea and that can be in fact, written like this that is I am putting capital F of x is equal to small f by f dash and then just writing it here to get the precise expression for the iterative function in terms of small f that is what I am trying to do. Therefore, I will now define my first modified Newton's method as this is my modified Newton's method remember we have not done anything we have just applied the Newton Raphson method to capital F instead of small f and that is how we have defined it here since r is a root of capital F and it is a simple root you can immediately see that this iterative sequence if it converges it will converge to r with quadratic order of convergence right. Now, let us see what is the advantage of this well the advantage is that this sequence will converge quadratically even if the root r has some multiplicity greater than 1, but there are some disadvantages what are they first thing is you have to give f double dash of x as a input to your code because that is involved in your formula in addition to f dash right and second thing is it also involves more operations than the Newton Raphson method. You can see that Newton Raphson method has one subtraction and one division in addition to the function evaluation of course, function evaluation is something extra excluding that you have one division and one subtraction whereas, here you can see that there are two subtractions two multiplications and one division right. So, it involves more operations and more importantly you can see that as n increases this term also goes very close to 0 at the same time this will also goes very close to 0 remember why this term goes to 0 because r is having multiplicity something greater than 1 therefore, f dash of r is 0 right. So, if the convergence happens then as you go on increasing n x n will go very close to r therefore, f dash of x n will go very close to 0 similarly f of x n will also go very close to 0 because r is root of the equation f of x equal to 0 therefore, you have two terms which are going very close to each other in fact, both of them are going close to 0 and you are subtracting them therefore, you have the risk of having the loss of significance here. So, that is going to be dangerous as we have seen in our first chapter when we do the computation with such expressions right. So, that danger is inbuilt into this formula. So, these are some of the disadvantages of using this formula as a iteration sequence of our method. There is another idea that can avoid these disadvantages, but for this we need to know what is the multiplicity of r. Therefore, in this approach we will assume that the multiplicity of r is known to us and let us take the Newton-Raphson iterative function and since the multiplicity is m in the previous slide we saw that g of x can be written like this right and now you differentiate g and see whether it is going to be 0 or not. If you differentiate g you get this expression and now put x equal to r in this expression you will see that g dash of r is equal to 1 minus 1 by m. Therefore, if m that is the multiplicity of r is strictly greater than 1 then g dash of r is never going to be 0 right. So, this is also a nice way to see that if the multiplicity of r is strictly greater than 1 then the Newton-Raphson method is not going to have quadratic convergence right. So, g dash of r is not equal to 0. Therefore, we can only say that the Newton-Raphson method in this case will converge at least linearly we cannot say it converges quadratically. So, that is the problem when you are working with root with multiplicity greater than 1. Now the idea is very clear how to rewrite our Newton-Raphson method you see just multiply m here. Since we have assumed that m is known we can just multiply m here that will make g of x in this expression as m into this term and that will bring a m here and that when you take g dash of r it will bring a m here as well and that will get cancelled with the m that is already there in the denominator and that will make g dash to be equal to 0 ok. Therefore, the idea is to consider this expression as the iterative function rather than the classical Newton-Raphson iterative function that will again lead to quadratic convergence if the root r has multiplicity greater than 1 ok because in this case you can see that g dash of r is equal to 0 and again if you recall g dash of r is equal to 0 therefore, we will have the quadratic convergence in this case. Therefore, another way of defining the modified Newton's method is to take the iteration sequence with this formula where you are multiplying this m in the second term that is the only difference when compared to the Newton-Raphson method, but this can be done only if you know what is the multiplicity of r right. So, therefore, in the second part of our lecture we have introduced two approaches to modify our Newton-Raphson method in order to achieve quadratic convergence for a root having multiplicity greater than 1. The first approach is to work with capital f of x which is given by small f of x divided by f dash of x the advantage is that we do not need to know what is the multiplicity m. This idea will automatically take care of it now the idea is to apply the Newton-Raphson method for capital F that is the idea and the resulting iterative method will converge quadratically if it converges irrespective to whatever may be the multiplicity, but it has its own disadvantages that we need to know f double dash also it involves more operations than what is involved in Newton-Raphson method and the last disadvantage is that there is a danger of loss of significance in the denominator expression. Another approach is if you know m then the best way to achieve quadratic convergence is to take the iterative formula like this and that will give you quadratic convergence if the sequence converges. These are the few modifications that one can do if your root is known to be of multiplicity strictly greater than 1. There are also other modifications that one can do to achieve higher order convergence, but we will restrict our discussions only to this. You can refer some of the books like Atkinson's numerical analysis or Kincaid and Cheney's numerical mathematics or Burden and Faire's numerical analysis books. These are some good books where you can find more modified Newton's method to achieve quadratic convergence for root with multiplicity strictly greater than 1. With this I will stop my discussion on non-linear equations here and thank you for your attention.