 So, today what we are going to do is, we are going to look at approximating derivatives, so far we looked at approximating functions. And we have seen there are lots of ways by which we can approximate functions. All I have done by way of either box functions, har functions, hat functions, all of cubic, quadratics, cubic splines, quadratics and so on is just one class. There are many, many ways by which you can approximate functions and represent them on the computer, which is the most important part that we are interested in. Now, we saw in the last class that if you had a function f of x and you wanted to find out what was its value at f of x plus delta x, that we could use Taylor series. We could use Taylor series to write f of x plus delta x in terms of f of x, okay. So, for the sake of this discussion like I did earlier, I will say h is approximately xi plus 1 minus xi, I know I have introduced delta x because that is what you are used to seeing in your Taylor series formulas and so on. So, I will write this as f of x plus h is f of x plus h times the derivative f prime of x plus h squared by 2 factorial by 2, second derivative of x and so on, 3 factorial, third derivative, is that fine? So, I am just writing a Taylor series expansion of f at x plus h expanded about the point x, right, that is usually what we call it. And it is possible for us, one thing that we saw was it was possible for us to approximate x at f at x plus h in terms of these terms, the more information that you have, the more terms that you can add. So, if you know the first derivative, you can get a representation whose error is of the order of h squared and so on, right. The other possibility that we saw was that we can actually use this to extract out the first derivative, that minus f prime of x can be written as solving for f prime f of x plus h minus f of x divided by h. Then we have to take all of these infinite number of terms to the other side and what will that give us? Plus, excuse me, minus h by 2 f double prime of x minus h squared by 3 factorial which is 6, the third derivative of x and so on, okay. If you go back, if you think about one of the definitions at least that you have seen of the, of getting the derivative, of actually calculating the derivative, right, from first principles using limits, you would have seen that f of x plus h minus f of x divided by h, right, limit h going to 0, right, would be the definition of the derivative and all of these terms will go to 0 and you will get the derivative. So that is an infinite process, right, it is a limiting process, it is an infinite process. We do not go through the infinite process, so we stay with the finite process and therefore this is called the finite difference, okay. So we have a finite difference approximation for the first derivative. I will truncate the series at that point, I will truncate that series at that point. This error that you see here, all the terms that I have truncated, it is called the truncation error, truncation error and the order of the truncation error is indicated typically by the leading term which is minus h by 2, the second derivative of x, is that fine, okay. So the representation that we have for the derivative f prime of x, right, is f of x plus h minus f of x divided by h, if I am going to truncate it then I have to replace this, it is really more of an approximation rather than, right, rather than being exact. So you could call it f prime, we can indicate an h up there, prime of x, h indicating that, right, equals and as I told you earlier, I will use this h to indicate that it is an approximation and after a point maybe I will drop it where it is, there is clarity that I am actually talking about an approximation. So this is f of x plus h minus f of x divided by h and the truncation error is minus h by 2, second derivative of x. What is the nature of the truncation error? In the limit h goes to 0 as h gets smaller and smaller, the truncation error is going to go to 0 in a linear fashion, okay. So the convergence is basically first order convergence, okay. For what polynomial, what degree polynomial is the truncation error 0? For a function that is linear, right, for a function that is linear, for a function that is linear. So this sounds like the hat function, it sounds like the hat function. Remember what was the derivative of the hat function on the interval xi minus 1 or xi plus 1, so that was fi minus fi minus 1 divided by h, it is the same thing, okay. So we are essentially representing, in a sense we are representing the function when we do this, when we go through this process we are representing the function using hat function, linear interpolates, okay. So and of course we will ask ourselves the question, can we do better than this? And we can obviously do better than this because we know that we can represent the function using quadratics qubits and so on. Because this function, because this approximation involves x plus h and x and the derivative is at x, right. We are using a point that is ahead, we are using a point that is ahead. So I will, again, so this is xi, this is xi plus 1, right. So if I call this point the function value here is fi, so I am saying that f prime, I have already dropped the h to indicate, so I know it is f prime at the point i is fi plus 1 minus fi divided by h. Since I am using a point that is ahead, this is called a forward difference. This difference is called a forward difference, a forward difference, right. So it is a forward difference, approximation is that fine. So if you can have a forward difference, the natural question that you have is, can we have a backward difference? So in order to do that, just like we did earlier, we say f of x minus h is f of x, we just follow that process, same process, plus h times f prime of x plus h squared by 2 factorial, second derivative of x plus cube 3 factorial, I am sorry, am I making a mistake here? I have to pay attention, thank you very much, h cube f triple prime of x and so on, the sign change is important, right. So and therefore solving f prime, I will just write it in the same notation, f prime at i is f at i minus f at i minus 1 divided by h and what is the truncation error? h by 2 times f double prime at i, this is minus h by 2, this is plus h by 2, is that fine? Minus h by 2 and plus h by 2, the signs are opposite. This is called a backward difference, backward difference, so that is a backward difference approximation, is that okay? Okay, so we have a forward difference approximation, we have a backward difference approximation, look at this. So at the same, across the same interval, across the same interval, I have these two across, let me not go there, just a second, so I have i minus 1, i plus 1, so I have one representation for f prime i which is a forward difference, one approximation for the f prime which is forward difference, right, at the point i, at the point i which equals f i plus 1 minus f i divided by h. I am just rewriting that, minus h by 2 f double prime of x, then I have another approximation backward, I will just write back at i which is f i minus f i minus 1 by h and the error here, truncation error here is okay. Is it possible for me to get a better approximation, if I give you these two, to get a better approximation? It looks like if I add this estimate and that estimate, this error will just cancel, you understand? It looks like if I add these, so if I take the average of these two, so this is just by a way of suggestion, just from observation, just from observation, f prime some approximate i is, what is the average of these two, f i plus 1, the minus f i and plus f i will cancel, minus f i minus 1 divided by 2 h, this goes away, so I will have to go back and calculate, recalculate my truncation error. So what would be my truncation error? What is my truncation error going to be like? I have a minus h cubed by 3 factorial which is plus h cube by, truncation error is 2 h squared by 3 factorial, of course we can just do it. There is a minus h squared by 6 here and what are we going to get here? You have a plus h squared by 6, anyway what we will do is, we will do this the legal way, let me not, so you can try going through this process, you will see that we will actually go through the legal straight forward fashion instead of this intuitive. So here all I am saying is, if I look at this, this gives me a clue, so I have a forward difference approximation, I have a backward difference approximation in this fashion and it turns out that these terms cancel. So when I say let us go about this legal way, what I am saying is, if you want to find the truncation error, there is only one way to do it, you have to get the series and actually do the truncation error. So we have to get it from Taylor series. So let us get back to the first one, we get it from Taylor series, so it involves the points x plus h and x minus h, so I will rewrite my Taylor series for x minus h, f of x minus h is f of x minus h times f prime of x plus h squared by 2 factorial f double prime of x minus h cube by 3 factorial f triple prime of x and so on, is that okay everyone? So what I am going to do now, I want to solve, so this is how you do it, so please do not go around taking averages and all, that was just a question to lead us intuitively to the spot, so this is how you would do it, what do I want from these two equations? I want to get f prime of x, that is my objective, I want to solve for f prime of x, so which means that from the first equation if I subtract the second equation, I will get 2 f prime of x, 2 f prime of x is f of x plus h minus f of x minus h divided by, well so I will remove the 2 f prime of 2 h, so I am dividing through by 2 h, what happens to this term? It cancels, there are 2 of these but I divide by the 2 okay and therefore I get and we have to make sure that the sign is right, we have a minus h square by 6 f third derivative of x and so on and other terms, that would be the truncation error okay. So in this particular case as h goes to 0, we converge quadratically okay, we go, the error goes to 0 quadratically, h square and what about the polynomial that we are using? That is also quadratic, that is also quadratic okay, that is also by chance, it happens that it is also quadratic, it is also a second degree polynomial, we can represent a second degree polynomial, we use a second degree polynomial for f, this term will be 0 and f prime we will get the right answer, is that fine okay, so essentially we are using parabolic to represent the f in this case right, though we are not actually constructed the quadratic right, all we have is the, all we have is those points. If you go back and try to find the coefficients right for the quadratics and qubits, if you go back right, we did, you have done, I have just said I repeat, you have done hat functions, quadratics, qubits, I have not used quadratics and qubits to represent functions right, I would suggest that you try it, if you go back and try to do it, when you look at, you will get a system of equations for the coefficients and when you look at the system of equations for the coefficients and solve for the coefficients, you will be surprised that, you should not be surprised that you will get terms like this, just try it out right, there is a reason why I ask you to do it, if you have done it this should look familiar right, you should, there is supposed to be an aha moment for you saying that aha I have seen this right, when I did the quadratic I got exactly the same expression okay, so there is a connection between this and that the quadratic, the coefficients of the quadratic okay, well so this is what we have, let me make an observation here and then we will see what is it that I am going to choose an interval to see what it is that we have, this is i, this is i plus 1, so I can have some function in between right, there is some function that I am sampling, this happens to be F i, that happens to be F i plus 1, what I am doing is, if I look at F i plus 1 minus F i divided by h, so that you get a clear radius to what it is that we are doing here right, F i plus 1 minus F i divided by h, this length is h right, just for clarity, I am actually calculating the slope of that line, I am calculating the slope of that line, if I were to take the slope of that line and assign it and assume that it is the derivative at the point i, I would be doing forward difference right and the error that we have, the truncation error that we have is minus h by 2 F double prime at i, if I were to take the same slope, the same number, the actual number that you calculated, if I were to take the same slope and assign it as a derivative at this point at i plus 1 right, then I would have backward difference, the same number right, the same actual number that you calculate is a backward difference and the truncation error would be h by 2 F double prime, be careful with this i plus 1 okay and interestingly, if I take that and assume that it is the derivative at some midpoint right, derivative at a midpoint, now my h is not the same h there, this is right but it is still I mean it, so 2h actually becomes h in that formula, so it is the same expression, the same expression you just have to be careful with the truncation error okay, then the truncation error if I take it as a central difference for this would turn out to be h squared by 24, third derivative you can think about that 24 right and this is at some intermediate point, we do not know what is that value but it is some intermediate point i plus 1. I will just call it i plus 1, it is a midpoint, the same number right is first order approximation there, first order approximation there, forward difference, backward difference, this is called central difference okay, so central difference, same number central difference is a better approximation, so that basically actually if you think about it, this should remind you of mean value theorem right, mean value theorem simply says that if you are going on a continuous path, if you are walking on a continuous path right from one point to another point, if you are walking on a continuous path from one point to another point, then somewhere in between in that trajectory, somewhere in between in that trajectory, what does mean value theorem say, you are walking parallel to the line joining those 2 points, if you start, if you start at a point and go to another point, you understand, if you start at a point and go to another point, it does not matter, you are following a continuous path, somewhere along the line, you must have been walking along the line joining those 2 points, that is all mean value theorem says, it does not say where, that is our problem okay, it does not say where, so this line joining them, this line joining the function is tangent to it at some point, the function is tangent, the problem is we do not know where, if you give me the point at which it is tangent, then I have the exact derivative at that point, somewhere in between it is the exact derivative, here it is an approximation which is first order, here it is an approximation which is first order, in between it is an approximation which is second order and at some unknown point it is an exact representation, somewhere in between for a continuous function, F i plus 1 minus F i divided by h is the derivative, is that clear okay and it is a fact that we do not know it that gives us this truncation error, the fact that we do not know at what point it is a tangent that gives us this truncation error. So it is possible for us to calculate the derivative, to approximate the derivative, to estimate the derivative at any given point, possible for us to estimate the derivative at any given point, using forward differences, central differences, backward differences and the forward difference, backward difference and central difference have the appropriate truncation errors, associated truncation errors right. So in each one of them you will see as we go along there is a reason why you would use one or the other okay, are there any questions? So then there comes the natural next question, if I could use x plus h and x minus h to get a second order right, we are a little greedy, why do not I get, how is it possible for me to use a third equation and get one more right, a higher order approximation, is it possible for me to add one more term, one more equation to that and get a higher order approximation and it is possible okay. So I will go back here, so what happens is something, no I want to solve for, I want to solve for f prime, I want to solve for f prime okay, so just to be careful I will add one more term and you allow me to change the notation slightly, h to the fourth factorial, f fourth derivative at i and plus h to the fourth by four factorial, f fourth derivative at i, is that okay, fine. So I am going to go with this subscript i, though I have left the other terms as x, I am not going to rewrite the rest of those equations but you allow me to change to this notation, so that f of, you want to take one behind one ahead, we have to decide where to take it, if I take i-2 that corresponds to f of x-2h okay, f of y-2 is f of y-2h, second derivative, first derivative at i, what are the mistakes, plus 4h squared by 2, second derivative at i minus 8h cube by 6, third derivative at i, plus 16 by 24, it is four factorial, fourth derivative at i and so on, it should be possible now for us to, what shall we do, how can we go about this, what we want our objective is to make sure that we retain the first derivative term, the f prime term, we retain the f prime term okay, so when we had two equations, we essentially eliminated the f of x and retained this term okay and eliminated the f double prime term right, we do not mind having the f of x actually, the f of x got eliminated as a right, that was just one of those side things that happened, what we really want to do is get rid of this, now in this case I want to get rid of the third derivative term also, am I making sense, it is clear, so you have basically three equations and I want to get f prime of i at i out of it, how can we do this, what is the deal, how should we do this, I have a double prime here, I have a double prime here okay, so I want to knock this out, what do I get here, so if I take this equation and this equation, so this is basically 2h squared or 4 by 2 is fine, so I need to multiply the first equation, if I multiply this equation by 4 and subtract the second equation from the first equation right, that should get me something, so if I take this and multiply it by 4 and subtract this equation from there, what am I going to get, 4fi-1-fi-2 equals 4fi-1-fi-2 equals 3fi and I have a – and plus – 2, no not – 2, – 2, h f prime at i, then – I say – but I write plus okay, – 2 h f prime at i, plus this term and that term are going to cancel, that is the whole objective, then I have 4 of these, I have 8 of those right, I have 4 of these, I have 8 of those plus 4 by 6, 2 thirds h cubed f triple prime at i, then what else, I want to retain the fourth derivative, I have added the fourth derivative because it is likely that I suspect that my truncation error is going to be like the fourth derivative right, so there are 4 of these and 16 of these right, there are 4 of these and 16 of these, – 12 by 24 fourth derivative of i, so on, h power 4 most important, h power 4 okay, so that is taking these two and of course earlier we had knocked out the h square term by subtracting these from each other which gave me f i plus 1 – f i – 1 equals, I will just repeat that no f i is plus 2 h f prime at i, this quadratic term cancels out, plus 2 h cubed by 6 f triple prime at i and what happens to the fourth derivative? Fourth derivative cancels okay, let us move on, what does this give me? What do I want to do now? I have a 2 h f prime i here which I want to retain but I have a third derivative term here that I want to eliminate, I have a third derivative term here that I want to eliminate, so if I multiply the second equation by 2, if I were to multiply this equation by 2, if I were to multiply that equation by 2 and then do what? Subtract one from the other okay, so from maybe the second equation from this equation I subtract that equation, so I get 2 times f i plus 1 – f i – 1 – 4 times f i – 1 – f i – 2, 4 times, oops I made a mistake, I have to be a bit careful here, 4 times f i – 2 – f i – 1 – f i, this gives me – 3 f i plus there is a 2 h f prime and the 4 h f prime 6 h f prime at i, what else? The triple prime goes away and we have multiplied this so it will be plus 1 by 2 h to the 4th or that is plus f i – 2, h to the 4th, 4th derivative at i, is that okay, fine. So what does this work out to? I want to solve for the f prime, so I will take everything else over to the other side, swap the equations around because I want the f prime, so f prime at i equals I have a 2 f i plus 1 – 2 f i – 1, let me leave the 6 h here, I do not skip any steps – 4 f i – 1 plus f i – 2 – 3 f i plus 3 f i plus 1 half h 4th derivative at i – so clearly I am going to make a lot of sign on this today, okay. This gives me 2 f i plus 1 plus 3 f i – 6 f i – 1 divided by 6 h – 1 by 12 h 2 h cubed f to the 4th derivative i, yeah f i, I am sorry I should not say f i – f i plus 1, where did I get an f i plus 1? f i plus 1 f i f i – 1 f i – 2 f i – 2 – h cubed by 12 h to the 4th derivative, let me add them up and check that I am performing, I am performing a test here, may not be obvious, the test that I am performing is I am adding up the coefficients in the numerator, I am adding up the coefficients in numerator and in fact they do add up to 0, okay, in fact they do add up to 0. So here you have some mechanism by which we have got something that has third order converges as third order, okay and the representation is third order cubic, is that fine, are there any questions, okay, right. So it is possible, so one thing is for the representation of f prime at x, right, can be represented or approximated, I will slowly switch from representing on the computer to approximating on the computer, represented slash approximated, right, using various points, the sample values at various point, various points, so you can use f i plus 1, i plus 2, i minus 2 and so on, it can be represented to apparently any order that we seek, any order, it is just a matter of adding points, as you add points, as you add more and more points, right, as you add more and more points, so if you have i plus 1, i, i minus 1, i minus 2, then apparently you can get, right, very, very, very, very higher order, as high an order as you want, so it is possible that you can actually get higher and higher order representations or approximations, okay, so right now I just make that as a broad statement, what we need to do is we need to find out and we have this truncation error, everything seems fine, right, everything seems fine, but you have to be a bit careful, okay, so what I do is I will give you an assignment, I ask you to, you try this out and see what you get, so we can use this representation that we have to check whether we are actually getting it, so pick a function, since we are talking about trig functions in the last class, I will use sin x, so you try out sin of x plus h or x plus delta x or whatever, minus sin of x divided by h, calculate, okay, so this is a candidate derivative, this is a candidate derivative for sin, let us assume it is forward differences, so at x, right, so this is a candidate derivative f prime at x, you also know that the derivative is actually cosine x, pick a value of x, right, pick a value of x say x equals pi by 4 and pick a value of h, pick a value of h say h is pi by 4, okay and evaluate this for different values of h, evaluate this for different values of h and what I want you to do is look at cosine of x minus f prime of x divided by cosine of x, you can take either, right, take the modulus if you want to keep it positive and what is this, this is some kind of a relative error, okay, so far so good, now this is what I want you to try to do, you try to make h smaller and smaller, so in your program you keep having the value of h, so you say h is 0.5 times h in your program, you keep having the value of h and keep calculating this relative error and I want you to plot, on a log log plot, on a log log plot I want you to plot the relative error versus h, is that fine, okay, so now that that is clear you tell me what you expect, now that the assignment is clear what do you expect, I want you to try it for different schemes that is I want you to do it for forward difference, backward difference, central difference, what is it that you expect, remember h grows larger in this direction, that means you are starting somewhere here that is 5 by 4 and you are going to work your way towards 0, okay, how many times are you going to divide by 2, how many times are you going, for how many values of h are you going to try this, h well whether h should be below machine epsilon or whatever it is you figure out whether it should be, does it have to be below machine epsilon, you do it just see what happens, right, so what is it that you expect, say for a first order, for a first order scheme, for a forward difference, this is forward difference, what is it that you expect, what do you expect the graph to look like, you expect a linear curve, linear curve like that, something like that, try it out and see what you get, right, and if it is second order, if it is second order, you have to be careful, it is second order it is still going to be linear, this was a 45 degree line, second order will be, you expect the steeper line, it will go through the origin I do not get it, some problem or maybe it starts somewhere else, I do not know if I am making sense, you are taking log on both sides, you have log error, relative error equals whatever, look at the truncation and work this out, you try to work it out, we will get back, we will take a look at it, try to work it out, okay, try to work it out, and as I have warned you before, always some little trap in something that I ask you to do, so keep your eyes open, okay, so keep having this, maybe try 64 times possibly, that is a lot, rate or 50 times, try having it 50 times, okay, that should not get you into trouble, try having it 50 times, try it for float and double, float double, you can try long double, okay, try a combination and see what happens, is that fine, okay, right, so let us get back to this, as it turns out, we are able to get derivatives of, we are able to get, we are able to get derivatives of various orders, approximations to a derivative, first derivative of various orders, so can we get second derivatives, right, so we look at these two equations and in fact it turns out we can get a second derivative, if we just add the two equations, it is clear now that these terms will cancel, so we have f of x plus h which is f of y plus 1 plus f of x minus h which is f of y minus 1, as 2 times f of x which is f of y, f of y, this term cancels plus h squared by 2 goes away, f double prime at i which is what we want, this term cancels, leaving us two of these plus 1 by 12 h to the fourth derivative at i and there are higher order terms, we can solve for the second derivative and therefore the second derivative at i, equals f i plus 1 minus 2 f i plus f i minus 1 divided by h squared and the truncation error is minus 1 by 12 h squared fourth derivative at i, okay, that is the truncation error, is that fine, I wanted to, so it is pretty, actually one, I wanted to do it today because it is straightforward, the expression is here, we can just subtract, get the second derivative, it is possible to get the second derivative in terms of i, i plus 1, i minus 1 and so on. I also wanted to do it because I wanted to make one thing clear, what is the rate at which the truncation error goes to 0, h squared, it is quadratic, it is second order, right, second order, what is the polynomial that we can, for which we can get the answer, that is cubic, right, so I do not want you to get in your mind that the function representation is quadratic, convergence is quadratic, right, that was only for first derivatives, when you go to second derivatives, the error term converges as h squared but the order of the representation of the function, you understand, the order of the representation, right, is the fourth derivative, the fourth derivative and therefore you can represent the cubic exactly, is that, does that make sense, right, so only if you plug in a fourth degree polynomial will this be non-zero, okay, so even for a third degree polynomial it is going to work but this convergence is second order, okay, just because for the first derivative approximations they seem to match, I do not want you to get in your mind, they match because we were dividing by h but here we are dividing by h squared and I suspect looking at this that maybe for third derivatives we end up dividing by h cube, okay, so that is one thing to remember, so you have a h squared order of convergence, representation is cubic for the function, is that fine, okay, so it is clear using the values of the function at various locations that we can actually approximate derivatives, right, so we are now in a position to actually approximate differential equations, is that fine, right, so once we are able to approximate functions, once we are able to approximate derivatives, we are in a position to approximate differential equations, we can approximate the differential equations and actually represent our functions on the computer and systematically hunt for the solution, okay, so this we will do in the next class what we are going to do is, we will look at a equation in two dimensions, so all my derivations are in 1D, I am going to look at the equations in 2D but before I finish I just want to emphasize some terms that I am using, so these are if i-1, i, i-1, i-2 and so on, you will see me refer to these, this whole interval being broken up into a mesh or a grid, I will explain these terms more detail later and these points I will typically refer to them as grid points but you may hear people referring to them as nodes, okay, so I will typically refer to them as grid points, so given function values at grid points we are able to represent functions, we are able to represent derivatives and therefore we will be able to represent on those, on that mesh, a differential equation, right, because we are able to represent derivatives and also extract out a solution which we are able to represent on that mesh, is that fine, okay, so that is our objective next class, fine, thank you.