 Good to be back and to see your Right faces We have a quiz or a midterm next week on Wednesday, right? So We will cover Everything up until the current lecture So the end of today's lecture we will be covering on the midterm and To give you guys some sense of the midterm I will have a David send out some previous tests so you can Play with it and try it out as before You'll have an opportunity to bring a sheet of paper With your notes on it. So you don't need to memorize anything. You can just write it out Okay, no books no books no books no internet just a sheet of paper the idea being that At least when I look at the courses that I took by making a sheet of paper you're summarizing for yourself things that you know and It's good to have a summary Okay Front and back font size of your choice Yeah, all right, so today I want to talk about the following chapters Let's see We are going to talk about 6.5 to 6.7 and I can see that this pen is not going to make it Let me see what I can do. It's better, but There we go. It's better. Okay, so The framework of state estimation that we've been talking about has been applied to the problem of learning in the sense that people And other animals have been exposed to perturbations scenarios where they perform an action and there's a Something that affects their behavior. So for example, they may be moving your mouse and when they move the mouse The normally the mouse goes here the cursor goes here, but now all of a sudden the cursor goes here so there's a perturbation that's added to their actions and then trial after trial they learn it and They're Individuals as they learn they exhibit some interesting properties in that learning and we're going to talk a little bit about it today And the the next couple of lectures But the notion of state estimation that we've been using has been applied to this framework in thinking about how the brain is Estimating this perturbation this perturbation that's occurring has a state and the brain is learning. What is this perturbation? so Some years ago back when I was a student like you guys the paradigm that I thought about as a way to explore this problem of learning was one where You perform an action you you pick something up and you move it and when you move it You misestimate the mass of that object So the idea is like you go to pick up a can of say coke and you think it's full But it's empty and so your arm goes like this So your brain has an estimate of some property of that object and when you move it you make an error And what is that error is an error between what you predicted what should happen? You hand should have moved like this, but it moves like this and therefore there's a Difference between what you predicted and what you observed Therefore you update your estimate of the parameter that you're trying to estimate say the mass of this object So we didn't want to give people objects to move around this kind of clumsy So what we thought about is that we make a robot that generates basically arbitrary dynamics So the arm that we built Was something that can be held you held the end of it and you moved it and as you moved it it generated Dynamics meaning forces as a function of states and when it did that what happened was that? It acted as a perturbation on the action that the individual was performing So you you know you hold this thing and you move it and you expected to move in Some way, but it moves in a different way and by the fact that there's a prediction error Here's where it should have gone. Here's where it went. I have a prediction I learned to estimate my brain learns to estimate the dynamics and trial after trial people get better now What was interesting about it was that? People as they learned it they Performed a number of things that was consistent across individuals So it really didn't matter if the person was young or old if they were you know Whatever differences there may be in that they seem to have regularity It was a consistent behavior One of the things was that for example when they learned that they didn't just learn something about the action that they did But they also generalize what they learned so they extrapolate it from the space in which they learned To places they hadn't been and that was interesting because you can ask them Why did they extrapolate the way they did? What did that say about the way they represented the task in their brain? Probably the most interesting aspect of this task was that it turned out that you didn't really need your conscious ability to Recollect that you've done the task to remember that you've done it so for example You know if you were to have an experience with this strange machine this robotic arm you come and you know You move it around and you get better at it you come back tomorrow. You remember that while I was in this room before So you know you remember having done the task before So that recollection is a kind of memory But it turns out that there are some individuals who don't have that ability. So for example amnesic individuals can Learn things and they experience things, but they may not remember that experience. So there's this famous patient named HM Henry He just died a few years ago. So Henry when he was a young person In his 20s or early 20s. He suffered from this is now in the 1950s he was about 20 some years old he suffered from epilepsy and he was a mill worker in Rhode Island and He he had a terrible time because occasionally, you know, he wouldn't be able to maintain his posture He would have seizures he would fall down and then you know after the seizure would end he would get up and even go About his normal business So he suffered from these seizures and these seizures weren't controllable with what they've always known back then so There was a neurologist in Montreal who was performing operations and he went through a his own doctor told him go try this and so he went up there and They to help control these seizures They removed two parts of his brain called left and right hippocampus part of the medial medial part of the temporal lobe So, you know, you have a temporal lobe the medial part of it the middle part of it It's called hippocampus and they removed that They removed that and it helped a lot it helped with his seizures his seizures more or less came under control But what happened to him was that he no longer remembered the events that took place in the recent past. So for example He would come and he would meet me. So now I'm up in Boston We've set up this experiment and we have this room where there's this robot sitting in there So Henry comes and Henry's now in his 70s or so and he comes and is in the wheelchair Um, not that he can't walk around. It's just easier to you know walk them from room to room because of his being on his wheelchair so he comes to the the robot room and He you know, I say hello. Thank you for coming and normally what happens is that people when they see this machine They don't touch it because it's kind of a weird-looking thing. It's aluminum You know looks kind of threatening. They just leave it alone So he gets up and I soak it sit down in this chair There's this chair here, you know, so he comes and he sits down and and what he does is that he's looking at this thing And I say we could go ahead hold it so he holds it. He's looking, you know It is hand he's moving it around and I said no, don't look at your head look up here And you know up here is a monitor and there's a cursor on the monitor and I said, okay now move your arm Right moves the arm right. You see the cursor move around so he learns okay So when I do this this cursor moves up and left and right and so okay So he's looking up at the monitor looking at the cursor and then you give him a target He gives a target appear so he moves his arm over there and the cursor moves to the target pretty good So he does this again and again and he you know, he does well now every time he does this What happens is that the computer gives him feedback about the speed by which he's moving So most people they start this task they move kind of slow to encourage him to move fast You give a beep, you know, not a very good move and beep move a little faster Eventually they move faster when they move the right speed the little animation appears on the screen the thing Explodes okay, so he sees this and he says You know and I was a kid I would go to my backyard and I had this Winchester gun and I would shoot birds and I'd like to shoot these kinds of birds this and that and so forth And he said it with great joy You know doing kind of kept quiet for a while Maybe a minute or two would pass and he would say oh, you know when I was a kid I would go to my backyard and I had this gun and I would shoot this and that bird And you know he told me about his backyard and it lived in Rhode Island You know he had this woods in his backyard and and so forth This experiment took about an hour and a half and we did four sessions over two days So he would come an hour and a half come back four hours later do it again And it comes back the next day do it again and so forth and on this this Explosions reminded him of him, you know going bird hunting and he would give it You know with the six with with this pleasure. He would describe it now after he finished doing the task And I'll tell you about the task in a minute He left he left the room and he came back a few hours later And you know this is now about after lunch about five hours later So and ask him, you know, how you doing and have you been here before? No, have you seen me before? No Okay, so he gets up now. I said can you sit in this chair? He sits at this chair and when he sits at the chair What he does is Very important. He holds the machine without me asking him He looks up at the monitor and he starts moving it around So the way I imagine what's happened is that he is part of his brain Knows that somehow this machine is something fun And that's a fun thing for him to do because he don't I think if he had shocked them with it He probably wouldn't touch it. So he didn't remember consciously having done the task before but his brain Recalled that here's how I hold that here's I move now what was even more interesting is that? When he's doing the task what happens is that you you have a move So you move your arm and the robot produces a force So what that happened what happens is that you generate a motor command you? This is this is your motor command. I add a Perturbation a force to it. Let's call that X and that becomes the consequence of your movement Y So what you learn to do is to estimate X so you form an X hat and by forming an X hat You generate motor commands That cancel this X hat So that when I add a perturbation to it This is removed So you learn to remove this perturbation that I've given you what that really means is that if I Push you to the right by force your brain learns to When I'm generating this movement to push a little bit to the left Okay, that compensates for that perturbation So the meaning of this is that if I were to not produce the force field on that trial So if if you expect the force to be to the right so therefore you push to the left But now in this trial, I don't give you the force field what happens to your hand is like this Right, so that's when you expect the can to be full but it's empty and so these are called after effects Alright, so he has learned to expect these forces and he's doing well at the first trial with them in the first Experimental set then he comes back a few hours later. He said I've never seen this before I don't know what this is about He sits in front of the robot on the very first movement he goes his arm goes like this He expects the force to be there so part of his brain knows What the forces of this machine is they know that he knows that this this machine somehow is a Fun thing to use because I want to hold it I want to look up here and he knows the kind of forces he needs to produce to produce the movement so This was one of a number of experiments that showed that Despite the fact that the brain may not remember from the cognitive standpoint from what's called? Declarative standpoint that I've done something before there's other parts of your brain that knows how to do that task and this Dissociation between kinds of memory is one of the hallmarks of neuroscience so there's different kinds of memories and different parts of our brain and you know it's just because you don't Think you know how to do something doesn't mean that your brain doesn't know how to do something okay, so in the world of Adaptation which is what this is about you learning to compensate for things There are a number of experiments that are interesting and tell us information about How this process of state estimation takes place and I and today what I want to show you is how we're going to use the Common filter to try to understand two sets of experiments that were interesting in terms of the kinds of information they provided the Experimenters the first in from the first set of data that I want to show you has to do To before I do it and then any questions about what a the story. I just told you Momento. Yeah. Yeah, I have yeah interesting movie. Yeah Yeah Yeah It's a it's fascinating, you know, I think one of the things you guys will see in this class is that that the concept of state And the concept of estimation is something that you know, I will show you your brain may have Multiple estimates of a single thing. So, you know, you have many parts in your brain each of them, you know provide Some potential Understanding of what the world is around you and what's really a miracle is the fact that we feel like one person despite the fact that There are all these different things in our brain that can have, you know knowledge of various kinds I may not say that I've ever seen this before but Part of my brain knows that it has seen it before and it knows how to interact with it Okay, so the data that I want to show you is the following It has to do with a very simple experiment that you can do Fairly easily doesn't really require any complicated Complicated the information. So the story goes like this. So typically what happens in in Laboratories that don't have, you know robots, which are a little bit more expensive and things to make are exploration of tasks where you move let's say a mouse and and What happens is that as you move the mouse there's a consequence associated with the cursor moves now to make it a little bit more Sophisticated suppose that your hand is covered by something and you move your hand and they instrument your hand so that they can measure Where your hand is located and then they point a little cursor on top of it So that it's supposed to represent your hand and what happens is that they give you a target So here's a target that they want you to go to and then you go there, you know, so you have your finger there That's that's you know you point to the target. That's fine So now what they do is that they they impose a perturbation and so by by perturbation what I mean is that here's to give you the target But as you move your finger there what they do is that they show your hand out here so what I'm going to call this is Y V which means visual You make an observation visually about the location of your hand. Of course your hand is actually here You can't see that though But you have a sensor in your body called proprioception which lets you know where your body is So, you know, even though I can't see my arm I know where it's here why because there's sensors in my arm that tell me what it is. I'm going to call that Y P Which means proprioception so I have two kinds of sensors here I have things that I can see with my eye I can things that I can feel with my arm with my Sensor proprioception and so what's happened is that you show me that my finger is here because what you've done is that you have perturbed something by an amount R and this R I'm going to call RV because it's a visual perturbation It has altered the visual representation of my finger from where it would be here to over here And I feel my finger over here, but I see it over here And of course what I have to do is with trial after trial I have to learn To alter this so that if this is the target What I should do is that so I see my hand there And of course for this to happen, so this is Y V it should be at this target my real hand would be over here I'm going to call this Y P. This is where I feel my hand This is where I see it now that the thing is that that this seems pretty easy to learn Well, okay, so if you want me to you know to perform your task I need to I guess move over here so that I see my finger over here But if you were now asked people to point to where they think their hand actually is So the hand is over here. I can't see it hands over there. I see it over here Okay, now Where is your finger? They point somewhere in between not at where the hand actually is and not where they see the hand So the felt position of the hand I'm going to call it H Each hat is here somewhere in between Where they feel it and where they see it Okay, so I want to show you what this means and how to how to describe it from the point of view of our our state estimation problem, so Suppose that what we have is the following model of All right, let me move it up here. So suppose you generate motor command. Let's call that you and You have your hand That is being sensed by these two sensors vision and proprioception and so we have the following So we have our hand that Is a state that's being sensed by two sensors yp and Yv now This hand moves because we send motor commands to it. Let's call that you and We know what these motor commands are because we generate them So you I can measure yv I can measure yp. These are my sensors. I can measure now There are things in this world that can perturb my sensors. So let's call these rp and rv So these are perturbations that can affect things that I feel and things that I can see There are also things that can affect the position on my hand. So are you is Gonna be Things that can affect position of mine. Let me give you a intuition about what this means. So say that you generate a motor command But there's something that adds forces to what you've done. So maybe you're holding a robot So if you generate a motor command the robot is also going to generate for motor commands It's gonna produce forces. That's going to be r u so that where your hand ends up isn't just dependent on the motor commands You generated you it also depends on the world out there and that's r u Now so that means your hand depends your position of your hand depends on you as well as our are you and you don't know What this r u is you have to estimate it the things that you see say you see a cursor right it depends on where your hand really is as well as Some perturbation that the experimenter may impose on it So then you know what you see may depend on where your hand actually is plus some some perturbation Right and similarly for what you may feel So in principle your sensors will give you something about where your hand is But they're influenced by these these unknowns these perturbations. Yeah Consequences yeah, so so I don't have a sensor that feels forces I just have position that I can feel and position that I can see so In principle though you do have these force sensors in your body You have these kinds of sensors called Golgi tendon organs that maybe they can Sense a force when it's pushing against you But what happens is that The reason why I didn't put it in there is because when people learn to move their arm in force fields After they learn it the force field is there of course. They've learned to compensate for it They they tell you you've turned down or turned off the forces So when they have completely learned it, they think it isn't there anymore. You're probably Yeah, very good point Okay, so this is a this is our generative model It says I generate a motor command you I observe y v and y p and you can do things to me through these these states so Let's see what what happens in this task where? You give me a target. I'm gonna call it y star This is my target And of course what I want to do is my my goal Is make it so that y v is equal to y star So what I observe in terms of where my finger is I want it to be at the target That's what the goal of the task is take your visual representation of your hand put it at the target now What I want to show you is that in this framework These sensory illusions can be explained in terms of the fact that there are many ways by which these perturbations can produce what I observe. There's not a unique perturbation to Observation relationship so in the real experiment with what's happened is that you set our v to be said say minus two centimeters that means that if you were to Move your hand to some location. I'm going to move it to the left by two centimeters and and you know that this is the truth rp is zero r u is zero. This is this is what you do But what I see is y p and y b and my estimate of What these perturbations are will be such to minimize the difference between what I predict and what I actually see And so I'm going to show you that you know, it's very easy to produce scenarios where I see my hand to be At the target I feel my hand at some other location and despite that I end up with these estimates of ours That are not the same as the perturbation and therefore I end up with a location about where I think my hand is That's different. We're actually so this illusion and where it comes of us So let me let me show you let me show you how to do it. So suppose that my my hand in Trial and the location of my hand in trial and is the motor command that I generate plus my this this perturbation that you give plus some noise associated with the motor command sigma u and Then of course what I see y p and y v. So y p in Trial and is going to be equal to h of n plus some noise well Plus r p of n Plus sigma p Y v of n Is going to be h of n plus r v of n plus sigma v and When so this is what I observe this is this is my out. This is my measurement equation This is what I see and how it depends on my hand position plus the The perturbations you gave plus some noise So on any particular trial what I want to do is to cancel what I believe to be the perturbations that you've given me so I'm going to write my h of n is going to be Where I'm supposed to go y star minus this perturbation that you've given me and so h is this this term The expected value of it is going to be u of n plus R u is going to be equal to y or v So the motor command that I generate on trial n is going to be where I'm supposed to go y star minus Rv minus r u So because I want to see my hand at the target what I will do is produce a command that gets me to the target while Canceling for these perturbations So let's set up a generative model and describe our our state estimate for for doing this and Then I guess before I do it. Let me give you intuition about what's going to happen. So RV the true RV is equal to minus 2 But what I can do is to say all right this I see my hand 2 centimeters to the left of the target So here's the target. I see my hand here This is what I see yv. This is where I feel it yp This can happen from the following scenario, so You could have given me a 2 centimeter visual perturbation or You could have given me a 1 centimeter visual perturbation plus a 1 centimeter Motor perturbation so it is possible that you have generated a perturbation to this that has made my hand move to the to that side and You have added this perturbation there. So let me write it down here Maybe RV hat is equal to minus one or U hat is equal to minus one But then how is it that I feel my hand over here because our p hat is Equal to plus one so in this case you see this Cancels this so I feel my hand over here, but I see my hand over here Let me go over it one more time. So you feel your hand here. That's why p You see your hand here. That's why v that could have come about With this scenario Why did that can also come about with the following scenario a scenario in which Our hat v plus our hat u is equal to minus two But our hat u plus our hat p is equal to zero These two cancel So that I feel my hand where it actually is these to add to give me the Visual feedback that you provided Perturbation any kind of it well in principle any sensor can be a Sensor that is affected by a perturbation So in this theory we assume that each sensor is affected by a perturbation as well as the action that you produce Yeah, yeah, so so right so that your sensor is biased Despite the fact that your hand here it feels it over here The way the way the way people would do it would be to put vibration on the This is the one way to cause these kinds of illusions would be to put a vibration on the arm if you vibrate the arm Then sensors become basically biased if you vibrate this muscle It will think like it's shorter than it actually is you feel like your arm is here now They didn't do right so you can't change in it, but they didn't do it here We're just saying that in principle the sensors that are measuring things about your body may be perturbed and In this experiment if you imagine that it is possible that there is this perturbation Disperturbation of this perturbation the fact that you see your hand here and you feel it here would this is what it means But that means that for this this thing has to be true That's and so this is just one this is just one solution that gives us this condition It could be others as well So let me write the generative model for this and we'll do state estimation So let's call our state variable X All the things that we don't know which are I'm going to start with I guess RV RP are you and hand location This is my state that I want to estimate I have Exit trial n plus 1 is equal to a Times x in trial n plus b times u of n plus some noise sigma Which has some variance q and then my observation y of n It's going to be c times x plus another noise Various q various r So what is this a? Let's look at this a matrix here this is going to be I'm going to put little a v little a p little a u and 0 0 0 0 0 this is going to be Hand position it's going to be 1 0 0 0 0 0 0 0 and then b is going to be 0 0 0 1 So let's see what I wrote. I have This x which is the matrix describe sorry the vector describing the state my hand position is going to be equal to a Times x which is going to be R u from this term here plus 1 times u so Where your hand position is is going to be equal to the motor command you generated plus the perturbation on to the So this is this equation Russian Yeah, this this equation is The the bottom row of this equation and every other equation is this R u and Plus 1 is equal to a r u of n Plus sigma u r v of n plus 1 is equal to a u a v Rv of n plus sigma v and then the same for r p Because the hand position only depends on the current you the motor command that you generate now Alters the state of the hand that's the only thing it depends on plus the perturbation now So what what I wrote here what I wrote here is as follows. I'm assuming that the perturbations are Not forever. They're going to have some decay properties to them. Whatever you're perturbing me with they're not going to be You know forever but at some point if you nothing else happens are going to go away. So this means this a here a u a V a p is going to be almost one, but not quite one There's going to be something 0.99 or something which means that that these perturbations that you're giving me I'm going to estimate them, but I'm assuming that from trial to trial You know they're going to be almost the same, but if nothing else happens they're going to decay away The other thing that I've written here is that the that hand this hand position here Is going to depend on the motor command you and the perturbation that you give it So just to be clear what what this really says is that this this perturbation depends on you you is here The generate generates the movement and then it causes it's affected by our you to give me my hand position Does that make sense? No, it's just hand position, which is a hidden variable. I don't know what it is Here it is look at it depends on you and it depends on our you you see it it depends on right, it depends on H depends on R u here it is R u and it depends on you Right, so remember R u is the third quantity here. That's what this is No, so I'm saying that hand position depends on our you and You do you see it? Okay, so that's the only one that depends on you all right So here's a state equation now. What about C? So C is going to be a two by four Let's see what are the things that we can observe so we can observe Y v and Y p. What does Y v depend on? Y v depend on H and R v so R v is my first term Zero zero one and then Y p is this term here depends on H and It depends on RP. So that's my That's my model. Mm-hmm. It's a vector Right because I have two things that I can see Okay So let's do state estimation on every trial I'm going to generate a command u I'm going to predict the sensory consequence as why and I'm going to learn from the difference between what I observe and from what I What I predicted so I begin with prior beliefs with R's and so I begin with you know, I usual I have x hat of N given n minus one. This is my prior belief So I generate a command u and so what is my command that I generate u of n is going to be equal to the target that I'm trying to get to y star minus What I'm trying to compensate for which are x hat v n given n minus one minus x hat U or I should call these are I guess So these are the these are the components of x are and you So I'm going to generate a command that gets me to the target by cancelling for the visual perturbation and cancelling for the The motor perturbation. All right, so if I do that What's my prediction about where where my hand is going to go then my prediction is that y hat of n What I'm going to see is going to be equal to the command that I generated. It's going to be y star That's going to be that line c times x so it's going to be C times x hat of n given n minus one that's where I expect to see my up my prediction So now how am I going to learn I'm going to have x hat of n given n is going to be equal to x hat of n given n minus one plus my common gain times the difference between what I see and what I predicted and So my common gain is going to be the usual relationship between my prior uncertainty Times c times The uncertainty of the observation which is c p of n n minus one c transpose plus R minus one. Let's see if that the Units work out so p is a four by four C is a two by four. So this is a transpose here and R is a two by two This works out and I have p of n given n is equal to I minus k of n Times c times p of n n minus one That's my posterior uncertainty after I make my estimate and now what's my prior uncertainty on the next trial? That's going to be x hat of n plus one given n is going to be equal to a times x hat of n given n Plus b times u of n Or u of n plus one I guess This is my prior belief is going to be on the next trial and my prior uncertainty on the next trial It's going to be that equation which is a p of n given n a Transpose plus q. I have my pen is running out All right, so let me go over this slowly so I have a Posterior estimate from my prior estimate and from my observation error. I have a Common gain that depends on my prior uncertainty and my measurement uncertainty I've formed a posterior uncertainty and then for the next trial I Send things a step forward by using those those equations So my prior answer my prior estimate the next trial is going to be my previous plus b times u and my posterior my my prior uncertainty on the next trial is going to be my previous uncertainty multiplied by the uncertainty associated with a and a transpose plus Plus q All right, so now why does this give you the data that I showed you? Why should this give you this this uncertainty when you simulate this? Here's what happens if you have Let's begin with Say that y star is Equal to one so this is the place you're supposed to go you're supposed to put your hand at one and and you know you This is where your hand actually is so here's here's here's y star is equal to one Your hand is there and now all of a sudden you give a perturbation so now all of a sudden the perturbation begins So down here. I have r This is r now at zero and this is r v r v now all of a sudden goes to minus two It goes to minus two There's minus one here's minus two so at this point This is RV it goes to minus two what happens to my estimate of things so my hand is going to slowly go to Plus three this is H The actual hand position My estimate of RV is going to go down here someplace Rv hat and then my estimate of r u is going to be the difference between these two It's going to be some hits here our hat u and now at p is going to go here And my estimate of where my hand actually is it's going to do this H hat so what has happened is that you gave this perturbation With this generative model that I have up there Depending on my model. I said well, you know this error that I'm seeing some part of it is could it could be because But also Part of it could be because of the perturbation to give it to other parts of my system or UNRP and in that scenario What happens is that I I have you know What I'm trying to do is minimize the difference between what I observe and what I predict and what I observe I'm a predict is the two sensory modalities vision and proprioception. I feel my hand Over here. I see my hand over here and this can be explained by this Division of the perturbations and if that's the case that I get my hand somewhere in between. Yeah That would be associated with the noise in each so remember when we were doing the Maximum likelihood and you had noise in your in your in your the way you feel things and noise in what you see things And the noise in what you feel is about three times the size of the noise in which is what you see so what you see You believe a lot more And you see in that case I put this RV to be a lot closer to RV hat Then then are you so you believe what you see in this case because the noise isn't what you see is pretty small But it isn't it isn't zero Credit assignments the fact that Yeah, exactly exactly so so so the uncertainty associated with what you see And what you feel is not something that is you know So big that makes it so that the only belief that can only be a visual perturbation Yeah, so the there is there is this distribution of the the credit to the various Sensors that could have failed that could have been biased and that makes it so that there is this potential So this is just a this is just a model that says here's one way by which to explain why there is this Illusion associated with where you think you're in so remember the basic data is as follows You learn to do this in order to see your hand go here But at the end of training your hand is really here you see the cursor here somehow you feel your hand is in between Why does that happen? this says because here's where you feel your hand to be because some of the perturbation that you are Learning to estimate has been assigned to things that really weren't perturbations at all did nobody actually perturbed you there, but You end up with this illusion All right any questions All right, so the last thing I want to show you is an experiment that is this is a bit similar to Backwards blocking that David talked to you about which is basically a scenario where an animal is learning under two contexts and there's this curious result that the specific way that they learn depends on the prior history of it and the way the way the backwards blocking worked was that there were two cues there was light and sound and If the animal learned something with light and sound it affected how it would then learn would just light alone and And the idea is that in this state estimation framework the past history of how you learned Will influence how you will learn now because of what has happened in the uncertainty matrix So if what you learned in the past said that there's this covariance between the parameters Then that alters how you're going to learn when only one of the parameters is present So the history of what you learn is reflected in that uncertainty matrix P because on certain matrix P Keeps track of what has happened in the past here and So if you happen to learn when most of the times the two cues were on together That means that there's going to be this negative covariance in your uncertainty matrix That's going to alter how you're going to learn in the future. So Here's the here's the basic experimental data that I want to talk to you about And then we'll do the math to show you that the framework to put it in so the experiment was like this They they did an experiment where they move the hand like this and As they move the hand there was this you know projector that covered the hand and there was a screen and on top of it There's a cursor that moved and then in another condition they move the wrist like this and First of all what they found was that so again, you know the moving the wrist They found that you know again There's this cursor that moves and and you know depending on direction of the wrist movement the cursor is moving So both conditions are very simple You know you move the hand the cursor moves to move the wrist and the cursor moves And I don't I don't remember exactly them You know I guess the plane maybe maybe I maybe we should think about it moving like this or moving like this So both in the vertical plane so The the curious thing was first of all people learned to do this a lot faster than this That's kind of odd, you know Why should you learn when it's with your wrist faster than you know with the whole arm the second thing they found was that When they learned with the arm and now they learned with the wrist It transferred so learning with the arm helped with the wrist But learning with the wrist didn't help with the arm So it was an asymmetric transfer This action helped learning with this but this action didn't help learning with this That was the curious result. There was three things here one wrist Association with this cursor movement somehow was learned faster than the arm To arm transfers to wrist three wrist doesn't transfer to arm What the heck's going on? So we'll set this up again using a state estimation framework, and we'll see why this comes out of the Uncertainty matrix, and it basically goes back to when we did the blocking experiment Which means that the history of how you learn influences the current ability to learn so In the past when we were thinking about blocking we had two kinds of cues We had light and we had dark. Oh, sorry. We had light and we had sound Right. So here we have two kinds of contexts. We have wrist and then we have arm But the important thing to realize is that when you move your arm you also move your wrist so the cute thing in this mathematical framework is to recognize that wrist is a part of the arm and When the arm moves the wrist moves with it, but when the wrist moves the arm doesn't have to move with it That's the idea So we're gonna have a context just like before when we had light and sound But now the context of moving the arm Includes the context of moving the wrist, but the context of moving the wrist does not include moving the arm Right because when I do this I don't move my arm, but when I do this I move my wrist That's the idea So by setting up the problem this way then you do get this funny thing where Learning the wrist is going to be faster than learning the arm Learning the arm transfers the wrist learning the wrist doesn't transfer to the arm. So let me show you how that works okay, so So I'm gonna have two contexts I'm gonna call this C my context and When I am in the arm context, I'm moving both my wrist and my arm my upper arm and this is this is my arm Context but when I'm just moving my wrist, I'm just moving my wrist That's what I mean by this context. Okay so You're gonna have this display where you're gonna show me this cursor and It's gonna be attached to my finger. It's called this E. I guess my finger called F So I'm gonna see a cursor on trial and and that's gonna depend on where my finger is and You're gonna add some perturbation to it called R and you know, that's my and then some noise Associated with the epsilon y Let's see Now this R is Gonna depend on I'm gonna my model says this R depends on C transpose times W and these W's are the things that I'm gonna try to estimate So I'm gonna estimate your R the perturbation that you're giving me based on which context I'm in and some weights associated with those with those With those contexts. So what this means is that you know what I'm seeing This R this equation it's gonna have a context in it depending on this this scenario So I have some target that you give me y star. That's my target That's where I'm supposed to take the cursor and I'm gonna generate you on trial and so that's equal to y star minus our hat so, you know Whatever I believe the perturbation to be I'm gonna try to cancel it so that I get to the target and And The key thing in this in this set in this set of equations that we have I'm also going to have my state state as state equation. I guess x of n plus one is equal to a X of n plus B u of n plus epsilon x and What is x? It's going to be the things that I don't know which is going to be w1 W2 so this is for The weight associated with each context so it's so I can get the so I can get the And then my finger position My estimate of R plus the finger so the state that I'm trying to estimate are the weights associated with this perturbation That you're giving me so that I can I can predict how much of a perturbation is that are going to be in each context Plus where my finger is this is the state that I'm trying to estimate I have some command u that I give and that changes the state from trial to trial And this is my observation equation. So this is my measurement equation and this is my state equation So the key idea in these simulations is that What is the uncertainty matrix p gonna look like? Well, if you have a scenario where you move your arm and that movement of the arm includes this This movement of the wrist then your uncertainty matrix p of for for this for this state State vector Let's forget about f for now So so this is going to be that the bottom row is going to be for f and the right column is going to be for f The uncertainty of where the finger is but most most important is this two by two matrix here That tells me my uncertainty about w1 and w2 and if I have a scenario where Movement of the arm causes movement of the wrist and what that happens when that happens is that basically these These off diagonal terms are going to be negative numbers These two terms here, of course, these two are going to be positive numbers Which are going to be my variance is associated with my estimate of the w1 and w2, but these two terms are going to be negative Which means that every time I've moved my arm. I also move my wrist, okay, so In a scenario where movement of the arm causes movement of the wrist But movement of the wrist does not cause movement of the arm. You have this Negative covariance on the off diagonal made off diagonal terms of the uncertainty matrix. So now let's consider Scenario where you do wrist training you take somebody and you have them Just move the wrist for them to produce a task. So in that case now C is equal to zero one So C is equal to zero one you know C is equal to zero one in these equations You know what C is it's equal to zero one so in this case when C is equal to zero one what's going to happen to them to the learning process, so you're going to estimate So here's some trial number and you're going to give a perturbation say, you know R It's going to be equal to plus 30 degrees So that's that's how the the the task begins and what happens is that these errors slowly go away and you know the The system learns to cancel this perturbation our hat gets better and better. So here's your our hat You learn it but the way you learn it is based on this w1 and w2 and the w1 and w2 look like this So w1 is associated with the wrist w2 is associated with Arm and what happens is that you learn that? for For the wrist there is this perturbation and so w1 increases and as w1 increases w2 decreases so the sum w1 plus 2 is Here at at zero So basically you learn there's a perturbation to the wrist and For that to happen because the negative covariance here The perturbation to the arm moves in the opposite direction now if you have a scenario where C is equal to 1 1 where it's the The arm is moving what happens is that your? These negative covariances make it so that you're going to learn slower I look like this Using the exact same initial condition the system will learn slower that perturbation and now Here's what happens w1 goes here w2 goes here the sum of those two is going to be equal to plus 30 so in this scenario if You were so w1 is how much you credit you assigned to the wrist w2 is how much credit you assigned To the arm what happens is that if you if you learn the arm condition you've also learned something about the wrist So therefore you're better at it. So all of a sudden you show transfer here You're no better at learning the arm because what you learn Put the move the estimate of w2 in the opposite direction. So assume if one assumes that This context matrix is as follows that in this condition It's a 1 1 in this condition It's a 0 1 now uncertainty matrix ends up having a negative covariance when you move the arm and in that condition You end up with scenario where when one moves up the other one moves down And when you do the arm Most of them move up and this gives you the three qualities that we had one in the wrist condition The system learns faster to in the arm condition It transfers the wrist and three in the wrist condition It doesn't transfer to the arm is asymmetric transfer all comes from this uncertainty matrix and the way the C plays a role in it. Okay So the it In this in this condition the wrist actually doesn't move with respect to the arm. It just is carried with it So it's like impossible to just do this right to not move the wrist, but just move the arm. It would be very strange Yes, it wasn't moving relative to the arm. Yeah. Yeah You know remember a few weeks ago you were saying how it's not an observable system These are all examples of that. There's no unique solution So it's just that when you change the C matrix you're changing the particular Subspace of parameter space that you're able to observe right right exactly and the way the way That observation takes place the estimation takes place is all based on the uncertainty matrix On certain matrix is telling you the weighting of how to distribute the error So when you see so in this case when you see your hand at one location or feel it in the other location The uncertainty matrix through the common gain tells you how to distribute that error to the various things you're trying to estimate And so it gives you the credit assignment and that credit assignment depends on The very critical thing which is your generative model So I so in these all these exercises I start with writing a set of equations that says this is the truth This is the way they the data that I'm observing is generated So the important thing of course to realize is that that's just a guess We don't really know how the learner is learning all we can say is that if this is the basis by which they Generated guesses then the learning that they did would be You know following these equations. So we start with a generative model now in a couple of lectures away What we're going to think about is about who gives us a generative model, you know How do you know this is the right model to begin with how do you know that a? How do you know the relationship between X and Y? How do you know that C? Right, so where does that come from? How many hidden states are there? You know, I'm assuming that X is the structure to it, right? Well, how do I know that? Where does that come from? So that's a kind of learning that's that's that you can think of it as there's a structural learning It's also is system identification there What we will we do is that we just make observations and we have to say okay What was the system that generated it and the system? Identification or structural learning is about not finding X, but finding a and B So of course if I knew a and B I can tell you what X is are because that's what we've been doing but The next problem is how do we find a's and B's? Okay, thank you very much. See you Monday You