 We have been looking at non-linear algebraic equations and we looked at three different classes of methods. One was derivative free method, the other was slope or derivative based methods and the third was optimization. So which was numerical optimization and we have looked at the algorithmic aspect of non-linear algebraic equations. Today I am going to touch upon the convergence aspect, so very, very important aspect of equations but I am just going to give a very, very brief introduction. I am not going to go deep into this, I just want to sensitize you that there exist lot of work, lot of literature on convergence of non-linear iterative schemes. For convergence of linear iterative schemes like Gauss-Seidel method, Jacobi method, we could actually derive in the class necessary and sufficient conditions whereas for non-linear case it is much more difficult and the machinery that you require is fairly more advanced than what we have been covering in this course and also many times you only get sufficient conditions, you do not get necessary conditions. So nevertheless these, the tools or the theorems that actually give you sufficient conditions give lot of insight into how solutions of non-linear algebraic equations behave. So I am just going to touch upon it today, not really go into deep of this subject. So one thing that we need to talk about, see if you look at the development that we did for the linear algebraic equations, we had, there is some sense parallel between what we have done there and what we have done here, there too we talked about non-iterative schemes, iterative schemes and then we talked about optimization based schemes, then we talked about a very important issue called condition number. So we said that a linear algebraic equation, set of linear algebraic equations is well conditioned or ill conditioned depending upon some properties of matrix A and it was possible to do analytical treatment quite easily with whatever we have learnt till now. Can we extend this to non-linear algebraic equations? I am just going to briefly touch upon this idea that what is the condition number of a non-linear algebraic system and then move on to the convergence properties of or how do you analyze the convergence of non-linear algebraic equations. So what was in the case of, so first thing I just want to touch upon is condition number. So in linear algebraic equations, we had defined condition number as when you have A x is equal to b, one way we defined this condition number was sensitivity of the solution x to a small change in b. So if you look at this as a input and x as a output, if you look at this system as b as the input and x as the output where A is the operator, one way we defined the condition number was norm delta x by norm x. So we showed that this ratio that is fractional change in the solution to fractional change in the input is bounded by this condition number which is multiplication of norm of A into norm of A inverse. Now to draw a parallel, I am going to consider non-linear algebraic equations of the form f of x u is equal to 0, well this kind of equations very routinely arise in chemical engineering when you are solving steady state behavior of a say CSTR, x are no states concentration temperature inside the reactor, u are inputs as in input flow rate, inlet concentration, inlet temperature all these free parameters, input parameters. So if you fix yourself to one input condition, you will get one steady state, you fix yourself to one input condition, you will get one steady state of the reactor or let us say you have a distillation column, you have this kind of equation, u there is nothing but feed composition, feed flow rate, feed temperature, reflux rate, heat input, all these free inputs which in parlance of control you call as disturbances or manipulated inputs, all these inputs are u, x are all the dependent variables like tray temperature, tray concentrations, vapor concentration, liquid concentration everything. So when you fix one u vector, for a particular u vector let us say u equal to u bar, u equal to u bar, you get f of x u bar equal to 0, this is what you have to solve. Once you fix u bar, say typically x belongs to R n and u belongs to R m and for every u you fix and f is a n cross 1 vector, f is a n cross 1 vector, this particular vector n cross 1 vector, so set of non linear algebraic equations, you have to solve them simultaneously for a given u, for every u, if I change the feed condition, if I change the feed composition, the concentration or temperature profile in the distillation column is going to be different. For every value of this input conditions, you get one set of steady state solution. So in some sense this u is parallel with b on the right hand side, if you change the number at hand side b, you get different x. So now we define condition number as with respect to solution of f of x plus delta x and u plus delta u is equal to 0. So when I change u from u bar to say u bar plus delta u bar, when I introduce a perturbation in u, what is the corresponding perturbation in the solution x? So I am going to define sensitivity of this equation or sensitivity of the solution with reference to perturbation in u as my condition number. Same idea fractional sensitivity of the solution to fractional sensitivity of fractional change in the input, that is going to be my condition number. So for a non-linear system, we define Cx as supremum over delta u. We define this supremum over all perturbations delta u. This of course non-zero perturbations delta u. So in other words, this delta x not C of x, C of f. So this should be C, well this in general will not be a constant number like matrix. You get matrix is a operator which only consists of, we have considered a matrices which are of real numbers. So you will get matrix norm of matrix into norm of matrix inverse as your condition number. Here that is not going to happen, you are non-linear algebraic equations. So the conditioning of a non-linear algebraic system could be different in different regions of the state space. Suppose you are solving non-linear, this is the abstract way of putting it. I will put it in the simple words, let us say distillation column. You are trying to solve set of algebraic equation for a binary distillation column in a low purity region as against in the high purity region. Conditioning of these non-linear algebraic equations in low purity region will be different from conditioning of these non-linear algebraic equations in the high purity region. It might be more difficult to solve for example in high purity region. I am not saying it is always difficult but it might be more ill conditioned let us say and it is well conditioned when you are away from the high purity region. So if you are trying to solve equations when the purity is 0.99 as against the top purity as against top purity is 0.9, you will have different behavior of the non-linear algebraic equations. So the sensitivity of the solution to a small change on the right hand side might be different in different regions. It depends upon where in the state space you are solving this set of equations that is critical. So this actually gives you an upper bound on this ratio change in the or a fractional change in the solution to fractional change in the input condition. So analogous to the non-linear to the linear case one can define something called a condition number here. You can talk about well conditioned non-linear systems, ill conditioned non-linear systems. You can talk about local you have to understand this is local for a same non-linear system could be well conditioned in some region it could be ill conditioned in some region. So non-linear algebraic equations are much more difficult to handle in terms of conditioning than linear algebraic systems. So sensitivity what is what did actually condition number tell you? Sensitivity of the solution to errors for example. So if non-linear algebraic equations are in some cases if the condition number is high which means a small change in u will cause a large change in the solution x. A small error in representation of u will cause a large change in the solution x. And just imagine when we are solving many of these non-linear algebraic equations arise because we are doing discretization of some non-linear boundary value problem or some partial differential equation when you are doing that you are approximating. So in some regions a small perturbation in the input condition can leads to a large change in the solution because of sensitivity of the equations in that region. But this is again as I said it is much more difficult to analyze this than the linear case. The next concept is we just touch upon this existence of solution and convergence of iteration schemes. So you have seen that all the methods that we have for solving non-linear algebraic equations are iterative quadratic multidimensional equations can be solved analytically but I am not aware of solution for the cubic case. So majority of elements in this set of non-linear algebraic equations cannot be solved analytically you have to solve them using some numerical procedure. Invariably any numerical scheme that you come up with can be written in this form any numerical scheme that you come up with. You start with a guess generate a new guess start with a guess x is equal to where you want to reach finally I want to reach finally to what is called as a stationary point I want to reach to a stationary point x star is equal to g of x star this x star is called as stationary point it is called as stationary point it is called as fixed point. So we want to actually reach here see for example when you are solving f of x equal to 0 is equal to 0 if you are solving using Newton Raphson method or Newton's method Newton's method was x k plus 1 is equal to x k minus dou f by dou x at x is equal to x k inverse f of this was my Newton's method I wanted to solve for f of x equal to 0. Now g would be here in this case g is equivalent to x minus dou f by dou x inverse f of x this is g of x this is my g of x and ultimately you are solving for x equal to g of x x equal to right you are solving for x k plus 1 is equal to g of x k from the previous guess you construct a new guess. So any method that we have looked at till now for solving non-linear algebraic equations iterative method can be put into this generic form and you are looking for x star x star is the fixed point well I think the word stationary is not really used here mostly stationary point is used in the case of optimization it is the fixed point. So literature on analysis function analysis will be full of fixed point theorems so doing analysis of iterative equations okay so how do the okay so now I am going to just revisit some of the terms that we looked at right in the beginning Banach space and operator mapping from a Banach space to Banach space and so on okay why I am worried about Banach space what is a Banach space a Banach space is one in which every sequence has a limit within the space is converges why about why am I worried about every sequence look here what is this if I start from some x naught if I start from some x naught okay I will get a sequence of vectors x 1, x 2, x 3, x 4, x 5 and so on. This iterative process will generate a sequence of vectors right generate a sequence of vectors now if I give one particular problem okay and if I ask him to solve the problem he will start with one x naught she will start with another x naught she will start with another x naught okay what is important is that if they are starting from different initial guesses okay will those sequences converge to the same solution under what condition first of all one condition or one primary condition is that the sequence should not go to a limit which is outside the space right the sequence should remain within the space that is the first condition second condition that is important is that we want to know is that whether the sequence will converge to a solution is the solution unique so does the solution exist okay and is the solution that you get is it unique all these questions are very very important okay so I am just going to give you some hint about how these are handled in the so in some sense this would connect to the theory that we had done in the beginning abstract you know abstractions of one x space and Hilbert space and so on. Now g is a mapping from x to x where x is a one x space or a complete non-linear space which means moment I say this I am ensuring that the sequence generated from any initial guess x naught will never leave the space will always be within the space that is what I mean here okay the sequence will never leave the space and important concept here is contraction mapping okay a very important concept here is contraction mapping now when I am writing here an operator g implicitly one which was defined we have just defined it is also x is one x space to one x space all these things are implicit I am not writing them on the board okay I have just have to complete this definition but before that let us look at what I have written here an operator g is called as a contraction mapping of a closed ball okay a closed ball is set of all x belonging to the vector space x such that x minus x naught is less than r r is some radius okay how do you what is the relation of this radius and convergence all that will come to soon but right now I am defining a idea of contraction okay on a small region in the neighborhood of x naught this is the way of defining a neighborhood of x naught some region around x naught okay so I am which norm you use depends upon you one norm two norm infinite norm does not matter okay any norm that of your choice but I am defining a region in the neighborhood of a initial guess okay what is x naught here because we are solving non-linear algebraic equations we can look at x naught as my initial guess okay it is not a fixed point as I said x naught can vary from person to person everyone can take a different guess okay just pay attention to these concepts because these are little difficult and then you are not the other things which I have been teaching at least you know something about it okay whereas these are little advanced concepts so you have to understand them carefully now I want to call this mapping into a contraction mapping if there exists a real number theta which is strictly less than one which is a positive number strictly less than one such that so this completes my definition so when do I call mapping g to be a contraction mapping okay if I pick up any two points if I pick up any two points x1 and x2 in this region okay and take difference between g of x1 and g of x2 that is always smaller than that is always smaller than x1 minus x2 which means if I draw it pictorially let us say this is my x2 and x1 and this is my x naught this is my x naught initial guess and let us say this is the region this is the ball in which this is the ball in which I am defining the contraction mapping okay what I am going to do what I am going to do is I am going to randomly pick any two points say here and here any two points okay now what is g g is an operator which gives you element in the same set right g is a mapping from x to x so if I apply g on one element it will give me another element so let us say this is my x1 and this is my x2 okay so when I apply g x1 see what is g is equal to x is equal to g of x right this is the kind of equation we are solving so when you apply g on x you get another x okay so let us say this gives me some x3 element x3 okay I pick up x2 and apply g of x2 this gives me say x4 okay now we are concerned about we are concerned about this we are concerned about this ratio that is x3 minus x4 upon x1 minus x2 we are saying if this is less than theta which is less than one see I get I get two points let us say when I apply this when I apply this I get I get x3 and when I apply g on this I get x4 what we are saying is that this distance between x2 and x x1 and x2 is larger than x3 and x4 so sorry I should put norm here it is not we are working in multiple dimensions I should put norm okay what I am saying here is that the distance between any two points x1 x2 okay let us say x1 x2 this distance is always larger than this distance this is this is x3 which was obtained by applying g on x1 this is x4 which was obtained by applying g on x2 okay so this is my x3 this is my x4 okay so if this condition holds for any two x1 x2 inside this region okay which means which means when you apply g on x on any two separate points okay then the relative distance contracts it comes closer then it is called as a contraction map is this clear okay yeah yeah that is a good question we will come to that okay so that will depend upon how you are chosen this radius okay and that is a very good question leading question I will answer this question too okay that is a that actually forms the crucial it is very crucial to the solution procedure or the the convergence of solution method so now let us for the timing assume that it lies within the same ball okay let us assume for the time then every time you apply on any two points the the the new two points that you generate x3 and x4 are closer than the initial two points you take any two points apply g on the first point apply g on the second point you get two new points okay there should be closer than x1 x2 it should happen for any x1 x2 in this region then g is called as a contraction mapping on this ball u so this is my this is my u x0 r and as she has rightly guessed this it is critical point is what is this r we will come to that now in general when you are solving for x k plus 1 is equal to g of xk it is quite likely that g is not a continuous operator not a differentiable operator it could be continuous operator but not a differentiable operator so actually this theory that has been derived is not necessarily for all differentiable operators but if g is differentiable which is the case in most of the chemical engineering situations then we can derive some nice conditions so this is a result well all the other things hold that is g is a operator from one x space to one x space and it is differentiable on this ball okay if it is well this makes it easier for you to understand because derivatives is something which you are more comfortable with okay so if the derivative of g if norm of derivative of g is strictly less than 1 for every x belonging to well this is a very nice result it says that if the operator is differentiable okay then it is a contraction mapping if and only if necessary and sufficient condition if and only if the norm of the derivative is strictly less than 1 okay so if the norm of the derivative is strictly less than 1 in some region then it is well I have to check whether it is necessary and sufficient I will confirm this if it is strictly less than 1 it is definitely a contraction but if it is a contraction does it necessarily mean that norm has to be strictly less than 1 that we have to check I am definitely sure that if part of it I will confirm this result so only if part is in doubt so if this is strictly less than 1 okay then it is surely a contraction so the derivative is if derivative is has norm strictly less than 1 we are guaranteed that so this this part I am not too sure right now I have to confirm okay how are you going to use this contraction mapping business the literature on theoretical numeric analysis is full of what are called as fixed point theorems they are worried about under what condition the solutions to x is equal to g of x exist under what conditions iteration sequences will converge to the solution the solutions are local first of all you understand that unlike linear algebraic equations when a is non-singular you have a unique solution right a is non-singular you have unique solution that is not the case non-linear algebraic equations you can have multiple solutions to same set of non-linear algebraic equations simplest example is I have given you is you know from abstract this thing is eigenvalues when we looked at eigenvalue problem they were it was a set of non-linear algebraic equations okay and then you have multiple solutions to that problem other example of course is CSTR you know that the CSTR can have multiple steady states under the same input conditions it can have a steady state operating point an unsteady state operating point depending upon how the heat removal and heat generation terms are so same set of non-linear algebraic equations under identical input conditions can have multiple solutions okay so we are talking about local conversions to a local solutions we are not we are talking about conversions inside a ball okay this ball which is in the neighborhood of the initial guess okay now let us try to understand this theorem this is the contraction mapping principle one of the fundamental results in now probably you can already guess norm of an operator strictly less than one okay then you get conversions we have seen something similar to this where was that linear algebraic equations we are we were analyzing conversions of iterative schemes and we said that norm induced norm is a upper bound on the set of you know is a upper bound the lower bound lower bound of that is the spectral radius and so if norm is less than one norm of the operator so norm of operator there was a okay norm of the operator not not a sorry s inverse t now the operator there was s inverse t and if now the operator s inverse t was less than one we were ensured conversion so this is something like generalization so try to compare draw parallels then you will understand this things better okay now i am going to assume something which she was suspecting okay the theorem assumes that g is a map which maps u into itself so which means you take any point inside this u take any point inside this u okay and apply g on it the resultant will also be inside u that is the first assumption okay so to so actually how do you choose r becomes very very critical because you know g has to map into itself okay now here i am coming up with a condition on how do i choose r okay okay now see carefully you have this map g okay which is a contraction map first of all g maps u into itself if i take any element in the set u g will map it into itself so you will find a new element also inside u it is not going to be different second thing is it is a contraction map okay which means you take any two points in u and apply g to it okay the new point generator are going to be closer than the two initial points any two points okay the second thing how should what should be the size what should be the minimum size of this ball okay look carefully it is related to the first it is related to the first x that you produce okay is the first x that you produce why why this is related to first x that you produce see because see what what what what should happen is that if you take x1 x2 and x2 x3 okay x2 x3 will be shorter than sorry x0 x1 if you take x0 x1 and x1 x2 x1 x2 because it is a contraction x1 x2 will be shorter than x0 and x1 the first the very first x1 that you produce by applying so this radius okay should be greater than should be greater than this is in in some way it is related to this this distance x1 minus x0 how this factor comes you have to read the proof why this 1 minus theta comes okay but you can appreciate that the radius is related to the first if you start with x0 the first x1 that you generate okay that should be within the ball after that whatever you do will be within the ball because it is a contraction okay it will it will stay within the ball okay so okay what next then now if these conditions are satisfied okay first thing that this theorem guarantees is that g has a unique fixed point inside the ball okay there exists a unique solution inside the ball what is the solution of the problem the fixed point you want to reach x star is equal to g of x star okay so there is a unique fixed point inside this ball okay when the radius of the ball is chosen according to this condition okay this minimum radius and then and when g is a contraction on this particular ball okay then we are guaranteed that a solution exists inside the ball there exists one point okay this condition is satisfied okay moreover within this ball is only one such point there are no two points okay there is only one such point in which a unique solution that is also so now the second part is very very important I will just continue here second and third part there are three parts to this result second part says that then it says that if it is a contraction and if these conditions are met then applying g repeatedly on the sequence will take you to the solution okay that is guaranteed at what rate you will go to the solution okay the distance between x k minus x star this will reduce with theta to power k again look at this result it says that the distance between x k and x star will be shorter than distance between x not and x star this is my x not this is the initial distance you started with x star is let us say this is my x star this is the solution I am starting with some x not here I want to reach here okay in doing so I might move around you do not know how it is going to happen it is a non-linear map you might move around all over this all over the set and then come back to the solution okay how is there going to be path how the path is going to be you do not know but what you know is that the initial distance okay now how is this distance going to shrink rest to theta to power k theta is a fraction theta is a fraction so theta to power k as k increases this distance will reduce okay as k increases distance will reduce if theta is you can appreciate if theta is 0.99 okay rate at which you will go to x star will be slower if theta is 0.1 0.1 rest to k will go to 0 very very fast iterations will converge very fast so what is the contraction constant will decide how fast you converge to the solution okay so that is another message this theorem gives the last message is very very important this is very very important message it says that I do not have to start from x not see we were talking about here so this is my x star and say this is my x not okay I do not have to start my iterations only from x not if I happen to start my iterations from some other some other x tilde x tilde not in the same ball okay so as I said you know she might take a different case he might take a different case he might take a different guess okay as long as those guesses lie within this ball all those sequences will converge to the solution very very important okay there is no unique guess initial guess if you are in the region of convergence any initial guess in that if you give a good initial guess in that region you will you are ensured to converge okay so sequence x tilde k generated by x tilde k plus 1 is equal to g of x tilde k starting from any x not belonging to this region belonging to this region okay well I will just continue this I will just continue this here okay so if I were to start from any other initial guess then x not okay as long as as long as g is a contraction in this region okay I am guaranteed that the sequence will converge okay all the concepts are important why 1x space any sequence that you start from any initial guess should remain within the space very very important okay next thing is we have this operator which maps the this ball into itself then it should be a contraction okay if it is a contraction if all these conditions are met these are sufficient conditions if these sufficient conditions are met we are guaranteed to get convergence to the solution okay so this is the famous theorem called contraction mapping principle or contraction mapping theorem there are many many variants of this and I have just presented you one one particular variant which is easy to understand and very very very powerful we will just look at one or two examples briefly in the next lecture and then move on to the next topic we cannot spend too much time on this because I will have to take many lectures if I have to really go into proving these theorems getting more insights but but what I want to do here by this one lecture is to just sensitize you that you know how do you look at the convergence properties of non-linear algebraic equations okay one simple message that you can carry is that look at the local norm of the local Jacobian of g of x okay if that is not less than one maybe you should try to make it less than one so that you know you can ensure convergence and so on.