 In this class we will start a new topic. This is on numerical differentiation. We will learn how to develop numerical formulas for approximating derivatives of a given function. These are called finite difference formulas. Let us start with a very basic idea. We all know the definition of the first derivative of a given function. We can recall it from the calculus course. It is given by f dash of x is equal to limit h tends to 0 f of x plus h minus f of x divided by h. You can also define it as limit h tends to 0 f of x minus f of x minus h divided by h or limit h tends to 0 f of x plus h minus f of x minus h divided by h. These are the three ways that we can define the derivative of a given function. Let us take this form of the definition. The idea behind getting a formula for approximating the first derivative of a given function is to just forget this limit argument and simply take this expression as the approximation to the first derivative of the function. You can see that if you choose your parameter h to be very small then the value that you obtain from this formula will be almost the same as the derivative of the function at the point x. That is very clear. In fact, you take h more and more closer to 0 you tend to get more and more accurate value for your derivative f dash at the point x using this formula. Therefore, we can propose this as an approximation formula for the first derivative of the function f. We will use the notation d plus h f evaluated at the point x and it is called the forward difference formula for the first derivative of the function f at the point x. Now, the question is what is the error involved in this formula? Well, you can derive an expression for the mathematical error involved in the forward difference formula for approximating f dash of x and it is given by this expression minus h by 2 f double dash of eta for some eta lying between the point x and x plus h. How will you derive this expression for the mathematical error? Well, it is not very difficult. You take the Taylor's formula for f of x plus h. It can be written as f of x plus h into f dash of x plus h square by 2 into f double dash of eta where eta lies between the points x and x plus h. You can see that this is the reminder term that we have taken and we are approximating the value of the function f at x plus h by the Taylor polynomial of degree 1. Now, recall what is the formula for d plus h f of x? It is nothing but f of x plus h minus f of x divided by h. You can see that you have f of x plus h here. You can bring f of x to the left hand side and write f of x plus h minus f of x that is what is there in the numerator of the forward difference formula d plus h. Now, divided by h therefore, you divide both sides by h thereby it will get cancelled with the h in the second term. Also, it gets cancelled with h square and it will leave h in the reminder term. That is what you will get. What I am doing is f plus h is written as f of x plus h minus f of x divided by h. f of x plus h is just replaced by this expression here minus f of x divided by 1 by h. So, this is what I am writing precisely the formula for the forward difference operator. Now, you can see that this simplifies to f dash of x that is coming from here plus h by 2 into f double dash of eta. That is what precisely you want to show here. So, this is the expression for mathematical error. You can observe that it cannot be explicitly computed. Why? Because this eta is just an unknown. Let us see what is the order of accuracy or order of convergence of the mathematical error. You just go back to our first chapter where we are discussing the big O and small O notations and then you can come back and continue this lecture. So, that you will understand what I am talking about. I am interested to know how fast the mathematical error converges to 0 as h tends to 0. So, that is what our interest is. For that we will use this mathematical error expression. What we will do is we will take the left hand side as the function of h. Of course, it is a function of h. You can see that it is given by this expression. Therefore, there is nothing wrong in considering the left hand side that is the mathematical error as a function of h. Let us denote this function by g of h just for the clarity and now we will look for mod g of h divided by h. You can see from this expression that g of h divided by h is nothing but minus 1 by 2 into f double dash of eta. Now, we will take modulus on both sides and thereby you get mod g of h divided by h is equal to 1 by 2 into mod f double dash of eta. Now, we have assumed f is a c 2 function. It means f double dash exists and it is a continuous function. We will restrict ourselves to a closed and bounded interval. Therefore, f double dash will be a bounded function in the closed and bounded interval a b. It means what? We can find a constant m greater than 0 such that mod f double dash of x is less than or equal to m for all x in the interval a to b. Now, we will try to replace f double dash of eta in this expression by m. To do this, you have to make this sign as less than or equal to because you are replacing this by m. Therefore, we will get mod g of h divided by h is less than or equal to m that is coming from this term divided by 2 which is already there. Well, we got this inequality. Now, what it means? If you recall in our first chapter when we were introducing big o and small o for functions, we have seen that suppose you have 2 functions f and g and you want to see whether f of x is big o of g of x as x tends to some x naught. We have to check that there exist a constant c such that f of x is less than or equal to c times g of x modulus for all x in a small neighborhood of x naught. Say it is x naught minus h to x naught plus h. So, this is what we have defined for the big o notation. This can also be viewed as mod f of x divided by g of x. If mod g of x is not equal to 0, then this inequality can also be viewed like this. Now, just compare this inequality in the definition of big o with what you have in hand now. You can see that in the numerator you have the mathematical error that is something like you are having f and in the denominator you have h. Now, you can say that g is big o of h as h tends to 0. You just compare it with the definition of big o. You can see that this inequality precisely means g is big o of h as h tends to 0. In such situations, we can see that g is we say that the forward difference formula is of order 1. Suppose if it happens to be h square, then you will say that that formula is of order 2 and so on. Here, since the h is appearing with power 1, we are saying that this formula is of order 1. It is also sometimes referred as order of accuracy or even order of convergence. So, when we give a finite difference formula and ask you to find the order of accuracy or order of convergence, what you have to do is you find the expression for its mathematical error. Somehow you have to find it. Mostly you have to use the Taylor formula for obtaining the expression for the mathematical error involved in a finite difference formula and then consider that mathematical error as a function of h and you can then do an analysis like this to see what is the order of accuracy of the method. In fact, once you are familiar with this idea, you can in fact see from here itself whatever the power appears here, that is going to be the order of convergence. Here, the mathematical error expression involves h with power 1. If it was involving h with power 2, then it is order 2 and so on. This is the idea behind finding order of accuracy of a finite difference formula. Let us see an example. Let us consider a function f of x equal to sin x. Therefore, we know what is f dash of x. It is given by cos x. Now our interest is to find the value of f dash of x at the point x equal to 0.5. That is we want to find the value of cos 0.5. Generally, in our course when we give just x like this it means it is considered in radiance and its value is given by 0.875 and so on. I am just taking it with 10 digit rounding. Now, let us use the forward difference formula and see what the forward difference formula gives as the value of f dash of x at the point x equal to 0.5. Remember, the forward difference formula is f of x plus h minus f of x divided by h. Here, you have to take f of x as sin x. Therefore, the forward difference formula is given by sin x plus h minus sin x divided by h and that has to be taken as a approximate value of the derivative of the sin function at the point x. Remember, we want to take x as 0.5. Therefore, you have to put x is equal to 0.5 in the formula. We also need to know what is h. Let us take h also equal to 0.5. Just for an example, generally it is better if you take h as a very small number. Then, the approximation is good. Now, let us compute the value of the right hand side which is the finite difference formula. Now, when you take x equal to 0.5 and h is equal to 0.5, you have to take this expression. Now, you can use a calculator with radians and you can find the values of all these terms, sin 1 and sin 0.5. Then, you can see finally, the forward difference formula gives the value something like 0.7 to 4 0 9 0 and so on to 10 decimal places I am taking. Let us see what is the relative error involved in the forward difference formula in evaluating the derivative of the sin function at the point x equal to 0.5. Well, this is the exact value minus this is the value given by the forward difference formula divided by the exact value. This is precisely the relative error and that happens to be something like 0.1749. It means the forward difference formula in this particular example has around 17.5 percent error when compared to the exact value of the derivative of the function f. So, that is pretty bad, but it is also expected because you have taken a relatively bigger value for h that is why you got such a poor approximation. Perhaps, if you would have taken h to be 10 to the power of minus 1 or minus 2 or minus 3 like that it may be that your approximation may be better, but at the end of the class we will also see that you cannot go on taking h smaller and smaller. Your arithmetic error will also start playing an important role when you go on taking h smaller and smaller. We will see that later, but at least in this case a poor approximation is because you have taken h to be a bit larger value. Next let us see how we can get backward difference formula for the first derivative of a function f. Again go back to the definition of the first derivative. Now with backward difference form it is given by f dash of x is equal to limit h tends to 0 f of x minus f of x minus h divided by h right. Again to obtain a finite difference formula what you have to do is you simply forget this limit concept here and consider f of x minus f of x minus h divided by h as an approximation formula for the first derivative of the function f and that is what we call as the backward difference formula and it is denoted by the minus h f of x. Again you can derive the mathematical error involved in the backward difference formula from the expression of the mathematical error you can also try to see what is the order of accuracy of this formula. I will leave it to you to see all this it is a very easy exercise. Let us pass on to one more type of finite difference formula called central difference formula. Again recall from our calculus course that the definition of the first derivative of a function f can also be taken as limit h tends to 0 f of x plus h minus f of x minus h divided by 2 h right. From here again you forget this limit part and then consider this expression as an approximation to the first derivative of the function f and that is called the central difference formula and we use the notation d 0 h f of x to denote the central difference formula for approximating the first derivative of a function f at a point x. Let us try to derive the mathematical error formula for the central difference approximation. The mathematical error by definition is the exact value minus the approximate value right and you can show that that is equal to minus h square by 6 into f triple dash of eta. Remember in order to get this expression for the mathematical error you have to assume that f is a c 3 function that is f triple dash exist in the interval a b and it is a continuous function that is what we meant by saying that f is a c 3 function in the interval a b. Remember again this is not something that can be explicitly computed because we have an unknown parameter eta involved in this expression and that eta is some number lying between x minus h and x plus h. Well we will see how this eta comes it will come again from the reminder term of the Taylor's formula and also we will see how to derive this expression for the mathematical error. Using the Taylor's theorem we can see that f of x plus h can be written as f of x plus h f dash of x plus h square by 2 factorial f double dash of x plus I am writing the third degree part as the reminder and thereby it involves an unknown eta right. We will denote this eta by eta plus and it lies between x and x plus h. Similarly you can also obtain the Taylor's formula for f of x minus h right and that again involves the formula involves an unknown eta minus lying between the points x minus h and x. Here we will assume that h is positive. Now if you recall the central difference formula is nothing but f of x plus h minus f of x minus h divided by h right. Therefore let us see how this term that is the numerator part of the central difference formula will look like you can now substitute the Taylor's formula for f of x plus h and the Taylor's formula for f of x minus h right. And you can see that when you subtract these two then the first term gets cancelled the second term survives again the third term gets cancelled and the fourth term survives right. So that is how we are getting this expression. Now you see if you divide this by 2 h that is precisely what is the central difference formula right. Therefore the mathematical error which is f dash of x minus h minus d naught h f of x can be represented in terms of the reminder terms right. That is what we are seeing from this derivation for that you have to divide both sides by 2 h. So when you divide by 2 h on both sides this goes off and you will have h square surviving here in the reminder term. And that will come as the expression for the mathematical error right. So that is what we will be having and further you can also use the intermediate value theorem to show that there exist a eta in the interval x minus h to x plus h such that this term can be written as f triple dash of eta. How can you use this? Well you go back to our lecture number 7. We have discussed certain tutorial problems in that video in that you take the second problem. You will see that such a formula is derived using intermediate value theorem. We are using that formula to write this expression in a rather nice way like this with some eta lying in this interval. Now once you use this expression you can see that the mathematical error can be written like this. Now the question is what is the order of accuracy of the central difference formula? Well you can directly look at this mathematical error expression and see what is the order of accuracy of this formula. It is 2 right. So I leave it to you to write it rigorously like how we did it with the forward difference formula right. So if you want to write it more rigorously in the mathematical language you can just follow these steps. So you can follow these steps and show that the central difference formula is of order 2 that is the central difference formula is big o of h square that is what you can show. Let us take an example again we will consider the function f of x equal to sin x. I am interested in finding the derivative of the sin function at the point x equal to 1. With the step size h is equal to 0.003906. I want to use all the 3 formulas that we have derived that is the forward difference formula, backward difference formula and central difference formula. These 3 formulas are generally called as primitive difference formulas. So for these formulas we need the value of the sin function at x minus h, x and x plus h. We know what is x, we know what is h. Therefore you can find the values of the sin function at these 3 points and they are given like this. Once you have this you can just plug into the formulas and get the approximate value of the first derivative of the sin function using these 3 primitive difference formulas. For instance the backward difference formula gives us the value 0.541935 and so on I am only giving you some approximate value right to 6 decimal places. Therefore there is some rounding error involved in this also. Similarly the central difference formula gives us this value and the forward difference formula gives this value. What is the exact value? Well the exact value is something like 0.5403. You can see that the central difference formula give a better approximation when compared to the backward difference formula and the forward difference formula. In general one may say that if the order of accuracy is higher the approximation quality will be better. That is what in a rough sense you can infer you can see that the central difference formula is of order 2 right. Whereas backward difference and forward difference formulas are of order 1. Therefore for a given h you may generally expect that the central difference formula will give a bit better accuracy than the other two. It may not be always true but mostly this can happen. This is what we have derived using directly the definition of the first derivative. However using this idea to obtain formulas for higher derivatives is little difficult. Therefore we can go for an alternate idea that is you are given a function f. You can first obtain a polynomial interpolation for that function at some given nodes and then differentiate that polynomial and consider the resulting expression as an approximation to the derivative of your function f. That is the idea. If you adopt this idea in fact you can find the approximation formula for any order derivative of a given function not only for the first order derivatives right. Let us just try to understand this idea through approximating the first derivative of the function itself. What you will do is for a given n and for a given set of nodes first you will find the corresponding interpolating polynomial and then you differentiate this interpolating polynomial and consider that expression as an approximation to the exact value of the derivative of the function at the point x. So, that is the idea. You can see that once you understand this idea you can in fact derive many formulas for f dash similarly for f double dash f triple dash and so on. Not just central difference forward difference and backward difference formulas that we have derived in the previous slides you can in fact derive many more formulas by supplying some value for n and also choosing the corresponding nodes as you want you can generate different formulas right. So, that is the idea. Let us just illustrate this by taking n is equal to 1 then you can write the interpolating polynomial for the function f well I am taking the Newton's form of the interpolating polynomial and now you differentiate p 1 of x and that gives you the first order divided difference. If you recall this is the first order divided difference and that is precisely the approximation for the function f dash at the point x. Now, you see choosing different values for x naught and x 1 will lead to different formulas for f dash. For instance if you take x naught equal to x and x 1 equal to x plus h you can see that this idea also leads to the forward difference formula that we have derived directly from the definition. Similarly, if you take x naught equal to x minus h and x 1 equal to x then it will lead to the backward difference formula and similarly if you take x naught equal to x minus h and x 1 equal to x plus h that will lead to the central difference formula. Now, you can also approximate the function f by quadratic polynomial you can differentiate that quadratic polynomial and thereby also you can get different formulas for f dash. Similarly, you can also get different formulas for f double dash also for f triple dash well you have to at least go for the cubic interpolating polynomial and so on. Let us take another example where we are interested in obtaining a finite difference formula using quadratic polynomial interpolation. Well if you recall the quadratic polynomial interpolation for a function using Newton's form is given like this and now our interest is to obtain an approximation formula that is a finite difference formula by differentiating the quadratic polynomial interpolating the function f thereby you will have this expression as the approximation for the first derivative of the function f. Similarly, if you take the second derivative of the quadratic interpolating polynomial you can take that as the approximation to the second derivative of the function f that is f double dash and that is given by 2 into second order divided difference formula right. Now, we will see how these formulas will look like by choosing different positions for the nodes x naught x 1 and x 2 recall the divided difference formula of order 1 is given by this and the divided difference formula of order 2 is given like this. Therefore, if you want to obtain an approximation formula for f dash using the quadratic interpolating polynomial which is given like this then it will look like this. I am just putting this expression into the first order divided difference and similarly the definition of second order divided difference in the second term and then you can write this expression in this form. So, therefore, this is the formula for the derivative of the function f at the point x let us see how this formula looks like by choosing different positions for x naught x 1 and x 2. For instance if you take x naught equal to x minus h x 1 equal to x and x 2 equal to x plus h generally if you choose the nodes like this that is called the central difference formula previously what we obtained is the central difference formula using p 1 of x. Now, we are trying to obtain the central difference formula for f dash using p 2 of x and that is given by this expression and that can be further simplified to this expression and this is the central difference formula for f dash of x when we use the quadratic interpolating polynomial approximation. Now, let us take x naught equal to x x 1 equal to x plus h and x 2 equal to x plus 2 h and that is called the forward difference nodes and let us see how the formula for f dash using this forward difference nodes in the quadratic interpolating polynomial looks like and it looks like this well if you want to find the mathematical error involved in it what you have to do is you have to write the Taylor formula for this similarly find the Taylor formula for this and then substitute the Taylor's formula into these expressions and then choosing appropriately the reminder term you can see that the mathematical error involved in this formula is given by this that shows that the order of accuracy of this formula in approximating the first derivative of the function f at a point x is of order 2 right. Similarly, you can obtain the divided difference formula for backward nodes in that case you have to take x naught is equal to x minus 2 h x 1 is equal to x minus h and x 2 equal to x that will lead to backward difference formula for f dash using the quadratic interpolating polynomial you can also find the mathematical error from where you can see what is the order of accuracy and leave it to you this as an exercise with this we will finish this lecture. Thank you for your attention.