 Hi, we are discussing numerical methods for capturing isolated roots for non-linear equations. In today's class, we will extend our studies to system of equations. We will restrict our discussion only to system of two equations. Once we understand this, extending the idea to any number of equations is straight forward. So, with that in mind, let us consider two non-linear equations given by f 1 of x 1 comma x 2 equal to 0 and f 2 of x 1 comma x 2 equal to 0, where f 1 and f 2 are functions from r 2 to r, right. We can write this system in the vector notation as f of x equal to 0, 0 is vector and f is given by f 1 comma f 2, right and x is the vector in r 2 and f is a function from r 2 to r 2. Now, we are interested in finding an isolated root of this non-linear system. Let us call this isolated root that we are interested in as the vector x star and it is given by x 1 star and x 2 star. Remember, right from our chapter on numerical linear algebra, we always considered vectors as column vectors that is why whenever I write a vector, I write it in the row wise and then put a t there. It means it is written in the column wise. I am doing this just to save the space. If I write it in the column wise, it may take more space, right. So, you should understand the vector for instance this as x 1 star and x 2 star. This is what we meant by writing like this and putting a transpose here. The idea is to first set up a fixed point iteration sequence. If you recall in our last two lectures, we have introduced this general framework of iterative methods with one initial guess in the form of a fixed point iterative method. So, we will apply that idea component wise to each equation in our system and thereby, we have two equations. In the vector notation, this fixed point iteration method can be written like this where G is the vector valued function from R 2 to R 2 and x n is a vector in R 2 and this expression generates the iterative sequence. We will provide the value of x naught as the initial guess and then we will generate this sequence using this expression for x 1, x 2, x 3 and so on. Remember the function G is called the iterative function and this has to be specified as the part of our method itself. All these ideas are familiar to us therefore, I will not explain them once again. Now, to analyze the convergence of this fixed point iteration method, if you recall we have certain hypothesis proved in our last class when we worked with single non-linear equation. The same idea will also go with system of equations. Recall that you have this system f of x equal to 0, you will write it in an equivalent form as x is equal to G of x that is the main idea of the fixed point method. So, you will choose this iteration function in such a way that isolated root of the non-linear system f of x equal to 0 is also an isolated fixed point of the iterative function G. Therefore, we will denote the isolated fixed point of G as x star. Now, we will use the Taylor approximation to write each component of the vector G around an iteration vector x n with Taylor polynomial being degree 0. Then the reminder term is given like this. Remember this is the Taylor formula in two dimension. You should go back and recall how we write Taylor's formula in 2D. We have only introduced it for one variable. You can similarly write the Taylor formula for two variable functions. Here I am just writing the function G at the fixed point and that is taken around the point x n and we have only taken the Taylor polynomial of degree 0 and therefore, the reminder term is written like this. Here you will get two xi n's one for each component that is for G 1 when you expand you will have a xi 1 and since we are expanding this G around the point x n therefore, we will also use the notation xi n for it and this is a vector and it is given by xi 1 comma n and this is for the first component of the vector G and xi 2 n and this lies on the line segment joining x star and x n. Remember you are writing G at x star and using the Taylor formula at a is equal to x n that is what you are doing and the Taylor formula is written with polynomial of degree 0 that is Taylor polynomial of degree 0 and you have the reminder term. So, you have to do it component wise we have two components for G. G 1 will give one such expression and G 2 similarly will give another expression like this with xi n of 2 now equal to xi 1 n of 2 comma xi 2 n of 2. This is the notation that we will use now you can see that this is equal to x 1 star right this is what we have given here G is chosen in such a way that x star is a fixed point. Therefore, this is equal to well xi for i equal to 1 G 1 of this is x 1 star and G 2 of this is equal to x 2 star that is what we have written here. Similarly using the iteration formula you can write this as x i comma n plus 1 right and then I will take this to the other side to write x i star which is on the left hand side minus x i n plus 1 which I have taken from the right hand side to the left hand side and that is going to be equal to the reminder term that is already there on the right hand side right. So, that is what we get here and this is obtained component wise that is i equal to 1 and 2. Just to write these two equations in the matrix form we get this equation I am just putting i equal to 1 and that gives me the first row of the system and put i equal to 2 that gives me the second row of the system ok. So, now we got this now we will give a notation for this matrix remember this is nothing, but x star vector minus x n right and this I will give a notation g n therefore, this will be g n into x star minus x n sorry this is x n plus 1. So, this is what we can write it in the vector notation for that I will define the matrix g n like this. Now, what is this g n if you recall the Jacobian of the function g remember g is a vector valued function right therefore, you can write the Jacobian matrix of g and it is given like this. Now, this is not exactly the Jacobian matrix because for having the Jacobian matrix you have x applied in all the elements of this matrix whereas, here we are applying xi 1 for the first row this is also xi 1 and this is actually xi 2. So, this g n actually resembles the Jacobian matrix of g, but it is not exactly g, but one interesting thing that we can observe here is as n tends to infinity if your sequence x n converges to say x star then you can see that xi n 1 will also converges to x star and xi n 2 will also converges to x star. Why it is so? You can see that xi n 1 and 2 are lying on the line segment of x n and x star right. So, that is what we have seen when we were expanding g in the Taylor formula therefore, as x n tends to x star by the sandwich theorem you can see that xi n will also converges to x star. Therefore, in the limiting case everybody will have x star as the argument that is all the elements of this matrix will be evaluated at x star in the limiting case right. So, in the limiting case this g n will be precisely the Jacobian of the matrix small g evaluated at the fixed point x star of this function g. And as I told in the previous slide we can write our equation in the vector notation like this remember this is the error vector at the n plus 1th iteration that is equal to g n into E n. Remember this is a matrix and here you see this is a matrix and this is a vector. Therefore, the right hand side is the product of matrix and vector and that is actually a vector. Now to see the convergence of the iterative method we have to take the norm of this equation on both sides and then we have to see how this matrix behaves right. Let us state the convergence theorem of the fixed point method for a system and we will omit the proof of this theorem. The theorem says that let D be a closed bounded and a convex set in R 2. So, if you recall what is mean by a convex set by convex set we mean that the set D should be in such a way that you take any two points in the set D and draw the line segment joining these two points that line segment should entirely lie in that set and this should happen for any pair of points right. So, suppose if you take a set like this this is not convex because if you take two points like this and draw the line segment that line segment will not entirely lie in the set D. So, this is used in the proof of this theorem that is why we are assuming anyway we will omit the proof for this course. The second assumption is that we will assume all the components of the vector G are as continuously differentiable functions at all the points in the domain D and further we assume that the iteration function G is a self map. If you recall we now know why we are assuming all this conditions on G at least in the case of single non-linear equation we have studied all these properties in full details and then if you take the maximum of the matrix norm remember this is the subordinate matrix norm and we are just restricting it to the L infinity norm that does not matter you can use any norm for easy handling in the proof it is better to have the L infinity norm that is why we have taken this and if you take the maximum of the L infinity norm of G then that should be less than 1 remember this is the contraction condition right. Then the theorem says that G has a unique fixed point in the domain D remember the existing existence comes from the continuity of the function G and the self map and the uniqueness comes from the assumption that G is a contraction map. Then for any initial condition x naught in D that is if you start your iteration by taking x naught in the domain D then the iteration sequence x n will be well defined and all the terms of the iterative sequence will lie in the domain D and further x n will converge to the fixed point x star in D and further we also have this estimate which tells us the order of convergence of this fixed point iteration method is actually 1 that is it has a linear order of convergence right. So, you just keep your knowledge that you had in the case of single non-linear equation and just see this it is only the vector notation, but idea is the same as we have studied in the case of single non-linear equation. However, proof will use some multivariable calculus ideas therefore, we will omit the proof of this theorem. Now the question is can we improve the order of convergence by suitably choosing our iterative function G. Again if you recall we have done such an exercise in the case of single non-linear equation. Again the idea goes very much similar to that let us see we will now see how to choose G for a given system of non-linear equation. So, as to faster the convergence for that we will now take G that is our iteration function as x plus a into f of x and look for the fixed point of this function. So, this is your G of x and you are looking for the fixed point of G of x that is you are looking for the root of this non-linear equation where a is a matrix that needs to be determined in order to faster the convergence. If you recall how can we make a faster convergence in the single equation let me quickly recall that we had limit n tends to infinity mod x n minus x star this is the fixed point divided by mod x n minus 1 may be we will put plus 1 here x n minus x star is equal to G dash of x star that is what we have obtained. Now in order to have higher order convergence here we need to choose G such that G dash of x star is equal to G dash of x star 0 that is the idea the same idea here instead of G dash in the single equation. Now we will have the Jacobian of the vector valued function G. So, that is the only difference that we will have in our present case. Let us see we will find the Jacobian of this iterative function and see how it looks like you can see that the Jacobian of this function is G of x and it is given by the n cross n identity matrix i plus a into f of x where f is the Jacobian of the function small f of x. So, that is how the Jacobian will look like and if you go back recall that the Jacobian of the function f of x is given by this matrix right. Now how to get the faster convergence precisely we have to look for that iterative function G such that this is equal to 0 that is the idea which is similar to what we had in the single equation case right. So, therefore, we will look for the iteration function small g such that norm of G n infinity is equal to 0. Let us try to achieve that for our initial guess x naught. Now how you will get remember G of x naught is given as i plus a into f of x naught right. Now we want norm of this to be equal to 0 that should be the case when the matrix itself is the 0 matrix. Remember this is one of the conditions of the matrix norm right. So, I am just using that in order to have the matrix norm remember this is the subordinate matrix norm ok. So, in order to have the matrix norm equal to 0 I should have my matrix itself as 0. So, if this matrix has to be 0 it means my a should be equal to minus f of x naught inverse right. So, that is how I will get this expression to be 0. So, therefore, in order to have this which will actually make my fixed point iteration sequence to have a higher order of convergence in order to have this I need to choose my a as minus f of x inverse and this I will do for each iteration that is I will propose my a just to have the notation a n here because I am going to change my a in every iteration. Therefore, I will have a n which is defined as minus f of x n inverse where capital F is the Jacobian of the victor valued function small f. Therefore, my actual iteration formula that is fixed point iteration formula is given like this. Remember we have taken it as x n plus 1 equal to x n plus a into f of x n right and now I have taken a as this expression. Therefore, this is my iteration formula given x naught now I can generate my iteration formula for n equal to 0 which gives me x 1 n equal to 1 that gives me x 2 and so on. You can just see that this is very much similar to what we defined as Newton-Raphson method in the case of single non-linear equation right. In the case of single non-linear equation the Newton-Raphson method looks like this x n minus f of x n divided by f dash of x n and this seems to be a natural extension of this formula to a system of non-linear equation and that is why this method is called Newton's method. Let us do an example consider this system of equations and we are interested in finding an isolated root of this system for that we will use the Newton's method with x naught given by this vector. As a first step we have to compute f of x inverse right because that is involved in our formula. So, first we will find the Jacobian of the matrix f remember the vector f is nothing but f 1 comma f 2. Therefore, the Jacobian of this function small f is given by this formula and now f 1 is given like this and f 2 is given like this from there you can compute each of the elements of this matrix and they are given like this right. So, we found f that is capital F now we have to find the inverse of that matrix and that is given like this. Once you find the inverse of the Jacobian matrix capital F you can now go to set up your formula for the Newton's iteration. Now for each x n therefore, you have to find this matrix for x naught the matrix is given like this. Therefore, to compute x 1 which is component wise written like this and that is given by x naught which is this into f of x naught inverse right. So, remember this is obtained from the inverse of the Jacobian matrix times f of x naught. So, that is what is this and therefore, the first iteration formula is given like this. Now once you have this idea you can compute your x 1 and that is given like this. Now once you have x 1 you then have to find f of x 1 once you find this matrix then you have to find the inverse of that. Once you find the inverse of that then you can set the expression for x 2 that is given by x 1 minus this inverse matrix into the vector f of x 1 right. Like that you can keep on going as long as you want to go right. So, this is how we can implement the Newton's method for a system of two equations. Let us take a small application of this Newton's method. Here we would like to find an extremum for the given real valued function f defined on R n. So, this is what is called the unconstrained optimization problem. We will assume that the function f is continuous function and recall that a point x star is called a stick to local minimum. Similarly, you can also define for local maximum. A point x star is called a stick to local minimum of f if f of x is greater than f of x star. Suppose you have the function like this just I am imagining it in the case of a I am just showing it in the case of a function in R then this is a local maximum and this is a local minimum and this is local maximum and so on right. Because you just have to have this condition in a small neighborhood of x star. Suppose this is your x star then you can see that in a small neighborhood of this x star this is the maximum right. So, therefore this is a local maximum and similarly you can see that this is a local minimum and so on. So, similarly you can also define what is mean by local maximum. Now from the calculus course you can see that a necessary condition for x star to be a stick to local minimum or maximum is that this partial derivative should be 0 right. Thus we have to set a non-linear system of equation like this. So, that an isolated root of this system will be a local maximum or minimum of the given function. So, let me write it in the vector notation if you see this gives you the gradient vector right. Therefore I am writing my non-linear system as gradient of f of x equal to 0 vector where the gradient is given like this. This is precisely what I am writing this system in the vector notation. Now we have to find an isolated root of this system for that we can use the Newton's method to obtain an approximation to it. And the Newton's method is given by this formula where H is the Hersian matrix for the function f right. If you recall in our first discussion we had a vector f that is the function from R 2 to R 2 and we had Jacobian of this and H is the function its inverse is used in the Newton-Raphson method. Now what we are doing is there is a slight confusion in the notation that we are still using the notation f now which is a function from R 2 to R and we want to find a local maximum or minimum for this function. For that what we are doing is we are setting up the system grad f equal to 0. So, that is what in the unconstrained optimization we are doing. In unconstrained optimization we are given a real valued function defined on R 2 in general it can be R n and then to find the local maximum or minimum of this function we have to set this non-linear equation. Therefore you have to apply the Newton-Raphson method for this equation just like how you applied the Newton-Raphson method for this equation f of x equal to 0 in our previous discussion. Similarly you have to now apply this Newton-Raphson method for this equation and therefore you have to find the Jacobian of grad f. Jacobian of grad f is nothing, but the Hessian matrix of your given function f. So, that is why we are having this and that is given by this formula. So, once you have the Hessian matrix then you know how to set up the Newton's method iterative sequence and it is given like this. Just follow the Newton's method that we have introduced in our previous slide. Note that if x star is a stick to local minimum of f then the Taylor formula can be used to show that the Hessian matrix is non-singular. Therefore you can always invert this matrix and therefore h is non-singular in a small neighborhood of x star and therefore you can always invert this matrix right. Again let us quickly see an example. Now we are given this function this is a real valued function defined on R 2. Now we want to find a local maximum or minimum we will call it as local extremum of this function. For that first you have to set up your non-linear equation for that you have to find the gradient of this function f and it is given like this. Therefore you have to set up the Newton's iterative formula for this system right. So, you have to find the Jacobian of this vector and that will be the Herschian of your given function f right and that is given by this matrix. Once you have this you can find the inverse of this matrix and that is given like this. Once you have the inverse of the Hessian matrix you can set up the Newton's method and that is given like this. Now once you have this then you can start with the initial guess and you can go on computing the Newton's iterative sequence and if that sequence converges it will converge to an local extremum of the given function f of x 1 comma x 2 equal to this expression. So, that is the idea with this our discussion on non-linear equation is complete I thank you for your attention.