 Are we ready? Set to go. I want to take the opportunity, I guess, to tell you guys today about an idea that is really something that was surprising as I was writing the book that you guys have for your textbook. It's something that came out of the equations that I was describing. The book didn't start out with this idea, but as I was writing the equations, I realized there was something interesting in them and it was worth thinking about, which is really the beginning of this idea that links the way individuals consider time in the way they make decisions, potentially in how they make movements. The story of it is a little bit of a personal story. When I was a kid, I lived in an apartment in Tehran and my parents would go out for walks in the evening. Most parents do that. When they live in a city, your apartment is small and you want to get away from the kids, so they would go out for walks. My mom would often come back kind of frustrated at the fact that my dad would walk so fast. My dad was a fast walker and my mom was not a fast walker. When I began to study motor control, I thought it was interesting why was it that some people were just fast in the way they spoke and I wondered whether those individuals also were fast in how they walked and maybe they were also fast in the reaching movements and maybe their eye movements were fast as well. I didn't know the answer to that and we still don't know the answer to that, but it was interesting to me that I had met throughout my career people who just were like vigorous in the way they moved, meaning that when they spoke, when they walked, they were fast walkers and it was like, why? Why did they walk so fast? What's so special about this way of walking than in other ways? If you look at the literature on locomotion and controller movements, people who study biomechanics, they imagine that there's some energetic cost associated with movements and that energetic cost is somehow something that our brain would care about. If you wanted to move in some way that would be at least effortful, then you would tend to have a particular pattern. But if that was true, then it wasn't clear to me why did my dad walk so fast? He was like, they do not care about energetics as much as other people do. What was the relationship here? That was one aspect of the story. Then there was this other aspect of the story that had to do with diseases that affect the brain. So, you know, there are some diseases that make individuals just move slowly, like Parkinson's disease. It just seems to have very little to do with energetics. It seems like why would they not care about energy of their movements like other people do, or care more about it than other people do? That wasn't very clear. So, when I was writing the book, I got to this chapter which describes costs of movements. And what I'm going to show you is that our understanding about what kind of cost there are in the movements was missing, in my view at least, the critical component. That's the concept of time and how time discounts reward, meaning that you make movements in order to acquire a rewarding state, but that rewarding state is more valuable if it has arrived that sooner, right, and later. So, if we play with the mathematics, we'll see that discounting of reward will affect the vigor of movements, meaning that if your brain discounts reward more steeply than my brain, then the theory says that you move in general faster than me as well, as make decisions based on reward and time that are more impulsive than me. So, this led to the hypothesis that maybe the way people move is in some way a reflection of how the brain represents reward and the cost of time in those rewards. So, that's what today's lecture is going to be about. We're going to set up the mathematical problem of this cost function. I'm going to show you what the standard way of thinking about it was until about the year 2009 or so, and then how that changed with this idea that there may be a cost of time that describes things. But to motivate our behavior, we need to have an action that we're going to be studying. We need to have some data, and that data that we're going to look at are the simplest of all movements, which are saccades, how you move your eye from here to here. There is a regularity in how movements are taking place. So, we're going to do chapter 10 today. So, saccades are movements of your eye. So, when you move your eye from this location to this location, if you look at the time and how it relates to velocity and displacement, it looks something like this. So, time is on the x-axis, and if this from here to here is something like maybe 15 degrees of motion of your eye, what your eyes will do is that you have a velocity profile that looks like this where it ends at around 40 milliseconds or so. Maybe, and has a velocity of about, say, 400 degrees per second. So, this is what a typical movement looks like. If you look at parameters like duration in terms of milliseconds versus amplitude, what one sees is that as saccades go from small to large, 70 degrees, the duration increases, and then it kind of grows a little faster than linearly as amplitude increases. And these milliseconds are, this is on the order of about 200 milliseconds. This is what the typical data looks like. And until, really, to be honest with you, until I began thinking about this stuff, if you looked at the literature, you saw that people had noticed that there are some people who have fast saccades and some people who have slow saccades, but it wasn't really something that was of interest because oftentimes what happens is that the data is available to tell you there's something interesting there, but unless there's a theory that says, here's why there's something interesting there, nobody pays attention to it. Let me give you an example. So, when scientists were looking at the flowers and animals that lived on the western coast of Africa, they noticed that there's a great similarity to the flowers and animals and fossils that exist on the eastern coast of South America. And in fact, if you look at the map, it really looks like those two things could fit in there. So, there was no question that was data that suggested that these two things some long time ago used to be close to each other. But, you know, how can you move a continent across water? What kind of force could push that? Unless there's no theory that tells you that here's a way that these two things can separate, the evidence that suggests that they used to be neighbors is irrelevant. So, unless there's a theory that tells you that there's a way to explain the data, the data is, you know, by itself doesn't mean anything. It just sits there. As an aside, I'll tell you a story. So, when Galileo described this theory that the sun is at the center of our solar system, the primary problem that he had was really with people who were other scientists and they asked if that's true what kind of force is out there that moves this planet earth with all its mass across the heavens? Who's pushing the earth? So, his problem wasn't so much with the fact that there's data that says it fits this theory better. His problem was with the fact that he didn't have force as a function of gravity. He couldn't provide you with an answer of how the earth is moving around the sun. So, what did he say? He said, well, God in all its wisdom has found a way to move the earth around the sun. And in fact Tycho Bra, who was the prominent physicist at the time said, well, really the way things work is that here is earth in the center of our solar system. Out here there's the sun and everything else goes around the sun and then the sun goes around the earth. And in his theory, much of the data fits because it doesn't matter if the sun goes around the earth or the earth goes around the sun and he had the advantage of their heavenly bodies. I don't need force to push them around. This is a real rock. We know that it's no heavenly body and so if you can't tell me the force that's going to push it around it better be stationary. So, data by itself often means nothing unless there's a theory that provides a framework to explain it and that's really been the history of science. Alright, so let's go back to our observations. So, it turns out that if you look at some people when they make these saccades some people do it like this and some people do it like this. Okay? And this data had been around for a while but no theory to say, okay, why where does this come from? So, the major idea regarding cost functions and how they can be used to explain movements came in 1997 or eight or something like that it was a paper that appeared in Nature and the way it described the cost of movement was based on the notion that the reason why you move is because you want to get accurately to the destination. So, in that theory what mattered was the variance of your movements. So, the idea was let's say I don't know why your 15 degrees saccade takes 40 milliseconds I don't know why but what I'm going to assume is that I have 40 milliseconds to make a movement what's the best movement that I can make? So, here's the idea that's asking if this is if this is 40 milliseconds to make an eye movement should I have a velocity that looks like this or should I have something else? So, why is it that velocity looks like something else? So, suppose that God made your brain so that you could only make 40 milliseconds long 15 degree saccades if that's the case why do you make a saccade that has a particular profile like it does? So, the cost that was suggested is that if you make your motor commands so that they produce a velocity that looks like this at the end of your movement you will have the minimum variance that you could have. The variance of your movement would be as small as possible. So, the cost that was suggested is as follows j is going to be equal to variance of x position of your eyes at the end of your movement p it's going to minimize this function where this is end of movement x is state of your eye and for now let's assume that state just means position in a little while we'll make it a little bit more interesting. So, we're going to minimize the end point variance and now for this to happen there also needs to be a constraint. That constraint is that the expected value of x at time point p is going to be equal to our goal g so meaning that I don't just want to have an end point variance I actually want to have a mean that's centered at where I'm supposed to go right so okay I have a cost function and I have a constraint this is my cost this is my constraint I want to minimize the cost given the constraint so in the class of functions that we're going to be considering we have the eye and the eye is moving unfortunately for us the eye is really well represented as a linear system and it looks like this and I'll show you how to derive the linear system but for now let's suppose that we that I have shown it to you that indeed x at time point n plus delta is equal to a times x at time point n plus vector b times the motor command u of n but then these motor commands have some noise associated with epsilon u of n plus an epsilon x and the characteristics of these noises are as follows epsilon u is normally distributed with mean zero variance k squared u squared and epsilon x is normal variance zero sorry mean zero and variance sigma squared so the signal dependent noise appears in epsilon u so the noise in this equation this part of the equation depends on the motor commands okay so the idea of that paper was that if you have a system in which the noise depends on the magnitude of the motor commands that's going to influence the variance of your n point so the idea is to find u star the best motor commands that are minimizing j is cost given the constraint that the expected value of x at the end of my movement was equal to my goal so well to do this what we need to know is that what is the variance at the end of the movement and then once I know that variance I'm going to have to show you how do you solve a cost minimization problem that's dependent on a constraint so the nature of our problem is as follows we're going to find the motor commands that minimize n point variance given this constraint okay yeah yes yeah okay so that's what's nice about people that worked in the field of ocular motor control they had measured these parameters what that paper did is to say let's assume there isn't signal dependent noise and if we made this assumption and put this number in there associated with how much noise there is then indeed it turns out that minimizing this function gives you a velocity profile that is like this has this form to it so that was the big idea of that paper that minimizing n point variance is something special about it because it seems to produce the kinds of movements that people do it didn't produce this concept of why is it that the difference is between four and why is it that you should have only 40 milliseconds you should make it, why should you move it 45 milliseconds what's different than that but we're going to get to that in a while but the first one I want to show you is how do you in principle solve these kinds of problems constraint minimization problems in which you have a cost and you have a constraint so to show you how to do that let me solve a simpler problem for you, suppose you have the following problem there's a line described by the function y is equal to mx plus b and there's a point here identified by x0, y0 and what I want to know is that where is the point on this line that is closest to this point x0, y0 so to do that I want to show it to you in terms of a cost and a constraint so clearly our point x and y, xy is the point that I'm looking for it must belong to the following constraint y minus mx minus b is equal to 0 if you give me xy it has to be in the line so that's my constraint the cost that I want to minimize is one that says find the distance between x and x0 y and y0 and minimize this cost minimize this distance so what that means is that there is a circle that represents my cost that's equal cost away from my data point and what I'm looking for is the point where my circle really it's going to touch that line that point where it touches it is the point where I'm minimizing the cost given the constraint so what Lagrange did is to represent this problem in terms geometrically in terms of the normals to these functions and he came up with the term that we use now it's called a Lagrange multiplier and what we're going to do we're going to change our cost function so that it incorporates the constraint that we have and so when we minimize the cost function we're also meeting the constraint and we're going to do using this concept of a Lagrange multiplier so what's his idea Lagrange says that what you need to do is notice that there is a vector that's perpendicular to your constraint so here are these vectors these vectors are all perpendicular to my constraint normal to the constraint and there's also vectors that are normal to your cost so here they are there's also this point out there there's one place where the vector associated with the constraint is parallel with the vector associated with the normal to the cost so let's write down these quantities so this is J this is J my cost and there's a vector that I can describe dJ dx dJ dy and this is equal to 2x minus x0 2y minus y0 and similarly I can describe the vector associated with my constraint normal to the constraint let me call the constraint function g don't confuse this with the goal up there this is a function it's a function of x and y so dG dx dG dy dG dx is minus m dG dy is 1 that's this function it's derivative with respect to x and y and the point that we're looking for is the one that has the following property dJ dx dJ dy is equal to lambda times dG dx dG dy and g of x and y is equal to 0 so if you look at what I just wrote I wrote 3 equations with 3 unknowns dG dx is 2 times x minus x0 is equal to lambda times minus m the unknowns here are x, y and lambda right, that's what I'm trying to find I want to find x and y and I've introduced a new unknown, lambda so I have 2 equations that I have here I have 3 unknowns x, y and lambda but here's my third equation y minus mx minus b is equal to 0 so I have 3 equations, 3 unknowns I want to find x, y and lambda that meet this constraint okay so what he's doing is the following he's saying that you have a cost j that's equal to x minus x0 squared plus y minus y0 squared you have a constraint g say that you write your cost in the following form in the augmented form jA is equal to the old cost as before plus lambda times g so if I write this like this the minimum djA dx djA dy is equal to 2 times x minus x0 plus the derivative of this this dg dx I don't know if I should do it this way let me, don't do it in that form let me write it in vector form so djA with respect to the vector x where x is equal to x, y that's equal to dj dx plus lambda dg dx and what we do is that we we find the x that minimizes we set this equal to 0 so to minimize this cost we find this derivative with respect to x and we set it equal to 0 which means that we have dj dx is equal to minus lambda dg dx and if you look at what I did here that's what this is if I just take my cost and augment it as follows then effectively what I'm doing is setting the normals associated with the cost function and the constraint equal to each other with an unknown called a Lagrange multiplier of lambda and how do I find that lambda? here I have whatever number of unknowns I have two unknowns I have three unknowns here lambda I have two equations my third equation is going to be the one that I get from g of x and y is equal to 0 so I get three equations, three unknowns in principle if you have a constraint that's not just a scalar function so this is a scalar function if I have a vector function that's a constraint then what I do is I want to write my cost function the way I write it is as follows I write it as follows so I say j augmented is equal to j plus lambda transpose the constraint so you may have a constraint that's not just one line you have to meet many constraints in that case you have a vector and so what you have now is you have as many Lagrange multipliers as you have constraints yeah so I have two solutions I have it here and I have it here as well they're collinear there could be anywhere on this on the circle and there's two places but then the constraint makes it so that there's only one place which is on the line the normal so the normal is just saying show me the vector that's perpendicular to your function at any point x, y that's what this is this equation for the normal so you know I have also a normal here but this is not a solution because of the third equation so you have anywhere you have a normal that's in the same that's collinear with the normal to the line you have a potential solution in fact that that point has to belong to the line why collinear yeah yeah I think it was because of derivatives are being easy to find and that's the geometric meaning of a derivative it's that it's perpendicular but you could have any point so you know suppose you could have it perpendicular right but I don't know that wouldn't work so the thing to keep in mind is that the norm that we are considering that makes it perpendicular to the cost function gives us any point along this line here but the point that matters is the one that also belongs to the line which is what the third equation gives us alright so let me now get back to our original problem minimizing the end point variance so this is what we had up here yeah yeah sure which one it only changes what lambda is whatever you call lambda if it has a minus in front of it lambda is going to be negative of that it's just a constant yeah yeah yeah right you're right this is different this lambda is different than this lambda this has a minus on it but you just find that when you solve the equation you find the value for it sure no problem yes yes well and it does right yeah yeah yeah that's fine that's fine so something like this right that's g of x y is equal to zero that's the equation for this and so now we have you know something like this right so it is possible that I'm going to get two points that have minimum cost and what this means is that this is going to be a quadratic equation or some function that has multiple solutions that won't give me one yeah you don't have enough constraints now well it means that you're going to have not three linear equations you're going to have two linear equations and a quadratic equation in which case in which case gives you multiple seconds yeah but in principle if you have constraints that are many equations so this is just one equation if I have constraint that is many equations like I wrote over there where g is not just one equation but like five equations then what you have is you have to meet five constraints simultaneously and the way you do it is by writing your cost function so that the Lagrange multiplier becomes a vector and you just have vector times that the result is the same you're going to get a solution that minimizes the cost meeting those constraints alright I want to go back now to our original problem I want to find and minimize this cost function given this constraint okay so let's see how we can do that so first let's write down the the physics of our problem so typically what we would have is that we have something that looks like this so force produces a displacement and if you're talking about a movement of the eye that's angular displacement so x is like theta that you're considering this is force that's produced by some muscle and you can imagine that a very simple muscle model might be as follows df dt say alpha one plus alpha two times f is equal to u this is the input that you give to that muscle this is the force that it would produce and that force moves the system and this is a low pass filter which is typical of a muscle if you give it a bang it gradually produces that force so it's a differential equation describing the low pass relationship between input to the force that the muscle produces and then we have the physics of the eye if you produce this force here's how it's going to move mx double dot plus bx dot plus kx is equal to the force okay so this is the dynamics of our system now the first step is how do we go from continuous time which is what I wrote which is what we've been talking about so this is continuous time and we need to go to discrete time so let's first write our problem in terms of a state space model in continuous time so let me solve the equations here so I have x double dot is equal to f one over m minus one over m b over m x dot minus k over mx and then I have the relationship between f and df dt so I have df dt is equal to one over alpha one times u minus alpha two over alpha one times f so suppose that I define x one to be x x two to be x dot x three to be force these are how I'm going to define my state vector and if I want to write now the following the state update equation x dot x double dot is going to be equal to some matrix times x x dot f plus some vector times u I'm going to take my these two equations and write them as a you know canonical state space model so x dot is equal to zero one zero times this is equal to minus k over m minus b over m and one over m times this and then f dot is equal to zero zero minus alpha two over alpha one times f plus zero zero one over alpha one times u so third order dynamics that are being described by an input u to a low pass filter model of a muscle that moves an eye that has second order dynamics so how do I now write this in discrete time let's call this vector x a continuous this is going to be a matrix I'm going to call it a in continuous time times x of t plus b in continuous time times u of t so that's x dot that's equal to x at time t plus delta minus x at time t divided by delta right approximately this is not equal but almost equal right so I just wrote that using rectangular differentiation I can define something very close to x dot as being the difference between x at time t delta minus x at time t divided by delta so if that's true then what I have is that x dot of t times delta that's the time that I have in my discrete time that's going to be equal to x of t plus delta minus x of t so x of t plus delta is equal to x dot of t times delta plus x of t which means that x of t plus delta is equal to what's x dot that's a continuous times x of t plus times b continuous times u of t the whole thing times delta plus x of t which is equal to a continuous plus delta sorry plus identity matrix times delta times x of t plus b continuous times delta times u of t so delta is just a scalar just a number to tell you how big is your step in discrete time and a continuous is what we just had before this is called a discrete this is called b discrete yeah yeah yeah you're right that's because good okay so given our model in continuous time we can write the equation in discrete time alright so let's go now back to here this is discrete time version of that model so delta here is just time some small amount of time in space alright so now what we're going to do is write the variance of this equation and to do that I am going to let me do it here so what I have is excuse me I have the following issue I have I don't know if you can see this let me clean this okay so what we're going to do is write variance at the endpoint as a function of all the motor commands we've given from the beginning to the end and to do that what we need to know is how does the variance change as a function of the motor commands so let's see oh this was the my pen alright so I have x at time point one is equal to a times x at time point zero plus b times x at time point zero plus epsilon u at time point zero plus epsilon x at time point zero x at time point two is equal to a times x at time point one plus b u at time point one plus epsilon u is epsilon x which is equal to a squared at time point zero plus a times b u zero plus epsilon u zero plus a times epsilon x at time point zero plus b times u at time point one plus e u at one plus e x at one. So in principle I can write my x at time point p the final time point in which a movement takes place is equal to a raised to the power p x at time point zero plus sum of k going from zero to p minus one a raised to the power of p minus one minus k times b times u of k plus epsilon u of k plus a raised to the power p minus one minus k epsilon x of k. Let me check to see if I did that right. I guess I can just put this inside the parentheses. No, the b's need to go inside. Okay so what do we mean when we say variance of x of p the only noise terms here are epsilon u and epsilon k sorry epsilon x so I have sum k is equal to zero to p minus one the let me bring so this noise here epsilon u has mean zero variance k squared u squared this is mean zero variance sigma squared so I get k squared u of k quantity squared times a p minus one minus k times b times this quantity transpose bt a p minus one minus k quantity transpose plus sum same sum times the noise there which is sigma squared times a p minus one minus k a p minus one minus k transpose. Let's see if that's right. Okay so x however is a vector right x is a vector it's a three-dimensional vector so what I'm really interested in is the variance of the position not the variance of velocity variance of acceleration so what I'm really interested in is not variance of x I'm interested in variance of s transpose x at time p where s is just a selector one zero zero I'm just interested in the position I'm interested in the variance of you know where you are not the variance of your velocity or your acceleration just the variance of position so what this means is that the variance of s transpose x p variance of position it's just going to have when I have s transpose times this what it does is that it takes the s's inside this matrix so what I get is basically let's see I get the sum times k squared u of k squared then I get s transpose a p minus one minus k b b transpose a times s plus s times sigma square times a p minus minus k transpose a transpose times s so it's just a selector when it when there's a selector matrix that acts on it just adds an s transpose here s here s transpose here s here making these a scalar quantity that's the variance okay so I also have a constraint my constraint is the expected value so my constraint is that the expected value of x at time point p should be equal to this goal and what that goal is is that you know I should be at some location g with zero velocity and you know zero acceleration change in force so I'm going to have three constraints and what is the expected value of x of p well I have the equation for x of p it's there this expected value is going to be equal to a raised to the power of p times x zero plus the sum k is equal to zero to p minus one a p minus one minus k times b times u of k that's it the rest of them have expected value of zero so that should be equal to this this is my constraint you have three equations here this is my cost I have one equation it's a scalar quantity so what I want to do is write my problem as follows j a the augmented cost function is equal to the variance of s transpose x at time point p endpoint variance plus lambda transpose times my expected value here minus g minus this this goal here that's going to be my my constraint is this minus this should be zero right that's my constraint that for this to be equal to this this minus this has to be zero which is expected value of x minus g zero zero is my constraint that's my augmented cost yes what it does to a is much more complicated I don't have a I don't have a I don't see precisely what it does to a yeah as transpose a will pull out the first row uh-huh yes yes right all right yeah go ahead it's like substitute s transpose times x into the equation up there and then take the variance of it and it just falls through I was actually looking at the the last the sigma squared s transpose a mm-hmm a transpose mm-hmm zero no it won't it won't why would it be zero oh it definitely won't be zero okay no no it's a squaring it yeah so so what that says is that that this noise here this noise is is growing as k increases because the longer this this goes on the more you're going to get this noise that's associated with with this x yeah yeah yeah this yeah yeah this is not an index this is being raised to a power right right indexes aren't in parentheses yeah yeah good good okay so alright let's come back to our cost here's our cost and we we added our Lagrange multiplier times our constraint here right so now what we're going to do is to find the derivative of our cost with respect to the motor command u that we're looking for so we're looking for this right arg min u of j given this constraint and we said that we can do this by finding u star is going to be equal to arg min u of j augmented this augmented j that I wrote over there so I'm going to minimize this function with respect to you so what does that mean so I have a cost that's a function of you and what I need to do is find the derivative of that function with respect to the vector u so u runs from x 0 from time 0 to time p oh man this blackboard is this whiteboard is in bad shape alright so we're going to do is find the derivative with respect to with respect to you let's see what do I mean by that we have we have this this function here this sum and I want to find the derivative of this with respect to the vector u where what I mean by the vector u is as follows so dj a du what that means is derivative of j a with respect to u 0 derivative of j a with respect to u 1 du time point p said that equal to 0 and I have my constraint which is expected value of x of x time point p minus g 0 0 is equal to 0 so this is going to give me three equations this is going to give me p equations right how many unknowns do I have so my unknowns is u 0 u p and lambda 1 lambda 2 lambda 3 the three Lagrange multipliers these are my three unknowns plus the use so I just got p plus 1 equations here I have three more equations here I have enough equations to find all of them all right so just to give you a sense of what this means okay so let's look at let's look at this equation what is what it's a derivative of our augmented costs with respect to u 0 well u 0 appears in only one location the first element so the derivative with respect to u 0 is going to be 2 times k squared times u at time point 0 times this plus this derivative with respect to so right lambda t times this so the derivative of this with respect to u 0 which is going to be lambda transpose times this times u 0 so just one more time the derivative of j a with respect to u 0 is a derivative of this with respect to u 0 which is going to be 2 times u 0 times k squared times this the derivative of the second term with respect to u 0 is going to be lambda transpose times a to the p minus 1 minus k times b times u 0 times b no more u 0s yes my exponent of a is going to be a times p minus 1 k is equal to 0 yeah a raised to the power of p minus 1 so the first equation there let me let me just write it down this term here it's going to be 2 times u 0 k squared oh I'm so sorry I I'm using this is stupid here so this k of course is very different than this k right this is a this is an index this k inside of that is referring to the noise of the of epsilon u sorry about that the one is kappa the other one is k I apologize for that dumbness all right so this is going to be s transpose a raised to the power of p minus 1 b b transpose a raised to the power of p minus 1 transpose times s plus lambda transpose times a minus 1 that's equal to 0 that's our first equation I'm going to get p more equations just like this and in addition to that I'm going to have three equations here so I have as many equations as I have unknowns I can find all the use I can find all the Lagrange multipliers that gives me u star the motor commands that minimize that cost so when you do that you get the optimum motor commands and what this says is that for p some duration of a movement you have optimum u star in the sense that it minimizes endpoint variance okay and then in that paper in the whatever 1999 or 2000 paper what they found was that if they introduced this signal dependent noise that had some value to it then indeed if you had assumed a duration of p to make your movement and you generated those motor commands you star you got a velocity profile that looked like the real thing it had the shape and so forth of the real thing so that suggested that maybe one of the cost has been minimized is endpoint variance if you think about it that kind of makes sense because you don't want to just get to the target you want to get to it in such a way that you maximize the probability of getting there so you want to be as accurate as possible you want to not just have this mean of your distribution but also has small of a variance as possible so this is basically saying you generate a movement in such a way that you maximize the probability of success how do you maximize it by minimizing the variance of your movement all right but the problem with this formulation is as follows if I want to minimize variance why don't I just increase the amount of time that it takes to get there because what happens is that the shorter time you have the more vigorous motor commands you have to produce the stronger your input has to be so if you really want to get there by maximizing the probability of success why not just go there slowly so what happens is that if you increase p to p prime of course your velocities are going to get smaller right you have to follow the same distance but now if you look at the variability that you have at the end of your movement this cost it falls so if you look at duration and you plot the variance at time point p so as p increases the optimum variance is going to decline why is that because signal dependent noise because look to get there quickly you're going to have to generate a heck of a lot of you right when you generate you you are going to increase variance because variance depends on u squared it's going to penalize tremendously the very variability so if you want to get there sooner you have to generate a large motor command the greater the motor command the greater the variance right all right so here was the puzzle it's interesting that people take 40 milliseconds to make it you know 10 degrees so cat but if they took 15 mil if they took a little bit longer 50 45 milliseconds they will have even a lower variance why don't they choose 45 milliseconds that was the idea all right that's the puzzle that that I want to tell you about so the interesting thing about the way individuals make movements is that if you look at how a movement is made it turns out that if people are asked to make a so catech movement to a dot of space in space they do something like this but if you ask them to make a movement to something more interesting say a face here's what they do they make a saccade that's just a little bit faster this is a saccade made to a face this is saccade made to a dot and what this means is that potentially a face is something a little bit more interesting for me to look at than just a dot and the reason why I can say that is because if you show people a number of stimuli on the screen and you look to see where do they look at first faces are particularly attractive which is why when people advertise things they put faces there particularly faces of females so they attract attention what that says is that your brain values that stimulus a little bit higher than other stimuli but the critical point is that if you value something more you will move more vigorously toward it so that's the critical idea that you don't just move but you move because there's some reward there the greater the reward associated with it the more you're willing to spend energy so that this seems like such an obvious thing when you think about it basically if you know you have your girlfriend across the street or your boyfriend across the street and you're madly in love you're gonna move more rapidly toward them to say hello then if it's your mortal enemy you know it's just no question about this so reward influences vigor and something as simple as a saccade is indeed the case that people make more vigorous movements with their eyes if that stimulus that they are going to be looking at is more valuable to them so it's not just minimizing endpoint variance so endpoint variance is interesting but this duration isn't really fixed in a single individual reward alters the vigor but perhaps more interestingly across people some people just move with faster saccades and some people with the slower so what's going on there all right so if we believe that vigor which means the motor commands regenerate minimizes a cost then we need to have potentially a cost of time what's good about you know making a movement that ends in 40 milliseconds and what's better why why is that better than making a 45 millisecond movement so basically what I'm saying is that if variance drops like this why is it that I choose this particular duration not this one potentially because there's something that says you know things are getting worse there's some other cost that is that is increasing and my thought was that maybe this is cost of reward that it is better to get this reward sooner than it is later so suppose there are two kinds of costs you're making a movement because there is something good to be had so you want to get there with low variance but two it's good to get it sooner rather than later okay so what do we know about time and reward it turns out that economists psychologists for 50 years have been studying this question of time and reward now we're talking about milliseconds here but they have been studying it using experiments that go like this you come to the lab and you sit in front of these computer screens or cards and in the card it says which one would you prefer a thousand dollars in five years four hundred dollars today which one would you prefer four hundred dollars today or a thousand dollars in five years four hundred bucks today what if I give you three hundred dollars today versus a thousand dollars in five years you would pick three hundred bucks what if I give you a dollar today versus $1,000 in five years. So there is a point at which for you $1,000 in five years is equal to X today. So what psychologists and economists discovered is that if you look at the function, it looks like this. This is time and say I'm going to give you $1,000 in five years. This is equal to you to some amount today. And for most people this is on the order of $350. Okay? And they said that it's interesting because if you fit these functions to the way people make decisions about these hypothetical things about money that it looks like the value of some some amount discounted as a function of time t is equal to alpha 1 plus beta times t. This is called hyperbolic discounting. So it says that you're taking something that's valued $1,000 today and you're saying that if it comes to me in five years, it's going to be worth a lot less than it is today. And that less has a metric to it. It's a hyperbola that's discounting it. So this is interesting because you notice they're not exponentially discounting. Exponentially discounting of reward is something that one sees in certain forms of mathematical formulations of learning like reinforcement learning. If you look at theories that describe how machines should make decisions about reward, mathematics of those mechanisms is based on exponential discounting. And this is an empirical result that comes from psychology and economics that says people seem to discount money based on hyperbolas. And effectively what that means is that if you look at an exponential versus a hyperbolic, an exponential would fall to something lower as time goes on than a hyperbola. But there's something actually much more interesting about this that has to do with the concept of change in mind. Why is it that people change their mind? Let me show you what that means. So what I want to do now is compare for you a hyperbolic discount error like this function versus a function that discounts like an exponential. So let me give this following example to you. Suppose that on Friday morning you're looking at your calendar and you notice that you have a test on Monday. So Friday morning you're told you're gonna have a test on Monday. So it matters to you to pass your test. So you plan that, okay, I'm gonna study during this weekend. So passing your test is gonna have some value to you and you're gonna plan based on that. That morning your friend comes to you and say, you know on Saturday, Saturday night we have this plan to go out, gonna go dancing. And so that has a value to you as well and you compare these two things. Pass my test on Monday morning, go out dancing Saturday night. You decide I'm gonna study. You tell your friend no. Then Saturday night comes, your friend calls you up again. We're gonna go dancing. Are you coming? Okay, I'm coming. You change your mind. What happened? So if you think about it in terms of decision making with these kinds of discount functions, it goes like this. So here's Friday morning, here's Monday and here's Saturday. So you're looking down the road and you're saying, ah, passing my test. It's great. I would love to do it. It has some value to you today. And you also like, you know, this idea of going dancing and, you know, this dancing also has some value to you. Maybe something like this. This is also discounted. So Friday morning you look down the road and to you passing your test is more valuable than going dancing. But as time passes, this crosses. Now this change of mind cannot be explained with exponential discounting. Why? Because in an exponential discount, if the rates are the same, if all that changes is the value, all I've done here is change the value. In an exponential discount, these lines are gonna be basically parallel. So if this is gonna be more than this here, it's also gonna be more than this here. So an exponential discounter couldn't explain why a person changes their mind whereas a hyperbolic discounter can. So this is just some, you know, anecdotal evidence for this notion of a hyperbolic discount. So when we come to the problem of optimal control and generating movements, we see that minimizing variance by itself doesn't explain why there's a duration. So suppose that in addition to minimizing variance, we have this other cost that says there's a cost of time. And that cost rises the longer time it takes to produce that movement. And that cost is gonna have a particular shape to it. Suppose that it is a hyperbolic cost. Meaning that the longer it takes to do something, it's gonna cost you more. But it's not gonna, you know, be any arbitrary shape. Let's suppose that it belongs to this hyperbolic functions. So this function is gonna have a shape that looks like this. Let's see what should I erase. Oh gosh, that's terrible. So suppose that I have a cost that says minimize my variance at time point P, my end of my movement. But it's also gonna cost you some cost of time as a function of this distance P. And this cost of time is gonna be equal to how much reward is there alpha times 1 minus 1 over 1 plus beta times P. So this cost here is gonna rise. And if alpha is greater, which means that this thing is valuable to me, this cost is gonna rise faster. If this value is less to me, it's gonna rise slower. And beta tells me effectively how impulsive am I? How time discounts reward. So I have two things now in my cost function. I have how valuable the thing is for me. Is this really an important thing for me to get? In which case I want to get there sooner. Or is that I don't really care about it so much. I can move slow. And second, in principle, how does time discount reward? And that has to do with beta. Which is saying, if I am a sharp discounter, which means that if for me, a thousand dollars in five years is worth three dollars today. Sorry. It's worth almost nothing today. Something that falls very quickly. Then I'm an impulsive person. I can't wait. I gotta have it now. Maybe I'm on cocaine. And I cannot wait any longer. I have to have what is what I'm looking for today. So you see, the point is that the theory says that there's gonna be a cost of time. And that cost of time should have a particular shape. This is assumptions that comes into the theory. So now, if there is this cost of time, what we can do is just now ask, can we explain why movements have a particular duration? Not just why do they have the same shape, but why do they have a particular duration? And more importantly, we can say, is there a way to understand why there are differences between people and how they move? So that's the beginnings of this theory. And so when there are changes that occur to individuals because of drugs that they're taking, because of condition that they're in, diseases that affected them, that changes that cost of time in decision making, does it also cause changes in time in how they move? And now we have a sense of how to link control of movements across domains. So that person that talks fast, according to this theory, should have this cost function that would influence everything that they do, not just their talking, and how they walk, and how they whatever they do. And that's the basis of this idea. So we have two kinds of costs. We have a cost associated with generating success. You wanna get there, and then we have this cost of time. Okay. And so in the papers and so forth that were written in the last few years, we've tried to sort of solidify and test this theory. Now, let me end with a simple thing. Why should there be a hyperbolic cost of time anyway? Where does this come from? Why is it that people discount money hyperbolically, and why is it that when you look at movements, this hyperbolic cost of time seems to work particularly? Where does this come from? Why hyperbolic? So that has to do with the following idea. That if I ask you to compare some amount of reward, alpha 1, and I tell you that I'm gonna have to make you weight the amount delta, and I'm gonna say compare that to amount alpha 2 given to you at time delta plus t. So I say compare $350 today, $350 at delta delay versus $1,000 at some longer amount of time. So when I say that, okay, let's find the condition where these two things are equal, suppose that what you're doing is finding the rate of reward, which is reward by unit of time. Suppose what you're comparing is not reward, but the rate of reward. So if that's the case, then when I find the time t in which for you these two things are equal, what's happening is that you're saying the rate of reward is equal in these two conditions, which says that alpha 1 is equal to alpha 2 times delta plus delta plus t, which says alpha 1 is equal to alpha 2 times 1 plus delta minus 1 times t. This is hyperbolic discounting. So if the way you make decisions is based on rate of reward, what's the rate of reward? It's reward per unit of time. Then you will appear to me as an experimenter, as somebody who discounts reward hyperbolically. So when I ask you to do this, if what you were doing in your head was to compare rate of rewards, then you will appear to me to be a hyperbolic discounting. So this leads to the idea that rate of reward may be the fundamental unit of cost and that by our actions, perhaps what we're trying to do is maximize not reward, but rate of reward. So we don't want just to be, you know, getting something good. We just want to get it all the time. The more you get, the better. The rate is what matters. And so in the experiments that followed, we tried to dissociate between reward and rate of reward and you do that by having intertrial intervals. So you don't just look at one action. You look at the rate of reward, which means that you change how long people have to wait for things. And it turns out that as you change the intertrial interval, how long people have to wait, even though for every movement, they get the same reward. If the rate of reward changes, people will change the vigor of their movements. And the vigor, the speed by which they move depends on this rate of reward. Let me stop and ask you if you have any questions. Okay. All right. See you guys Monday.