 Thank you, Naira. Thank you for the invitation this morning. Thank you for everybody for coming. I appreciate it There are definitely other places that you could be So I appreciate you coming to hear me talk So by training I'm an engineer biomedical engineer, but most recently I've kind of found my way More into like looking at neuroscience to some degree so The talk today is going to be a maybe a little bit heavy on neuroscience kinds of things But the way I like to think about it is In a lot of cases all the work that we do in engineering really informs the neuroscience that I do So all the kind of analysis techniques that I want to be talking about today and the kind of experimental methods And even the kind of questions that I like to ask Basically come from an engineering kind of approach to neuroscience So I tend to like to work in the motor system more than anything because well, I think it's the best system to work in That's my own personal bias But and I always like to talk about the motor system from this perspective Right if we want to do something and I like to think about the upper limb more than anything as you can kind of tell from my Drawing here. We first have to decide what we want to do right? We have to select some tasks We're going to reach down pick up that pizza Transport it to our mouth, right? That's kind of one kind of a control problem that we might have to do Another kind of control problem would be Maybe if we want to pick up that that that can of soda or a bottle of water or something, right? There's a lot of computations that we have to make that go into kind of picking that up and Transporting it as the weight of the drink gets less and less we have to kind of adapt for that across time Right, so we're going to select that task Pass it to some kind of motor controller and then what I'm saying motor controller here I'm really talking about our brain the part of the brain that controls movement That's going to issue some kind of motor commands that are going to kind of go to this very highly Nonlinear thing that hangs off the side of our body. That's going to go ahead and make some kind of movement Okay, right? That's the simple kind of control chain that we're going to be talking about And it wouldn't be a reasonably good control system if we didn't have feedback in some way, right? We have lots of different ways to get feedback into our system I'm going to talk a little bit about vision and proprioception today Maybe just lightly not so much right and then we can kind of iterate through this control loop fix errors that we make Kind of move along and do things like that. So The question that I'm really interested in studying from yes, please No, that's fine. Yeah proprioception proprioception is The sense of if you stand there and you have the unlucky opportunity of you know having someone test your proprioception They're gonna ask you to close your eyes and try and touch your nose So it's the ability to know where your limbs are in space without seeing them So that's what proprioception is so So the kind of research question that I want to ask is to try and understand how the behavioral context Surrounding movement of the upper limb Change neural activity in the parts of the brain that control movement and I kind of want to unpack that for you a Little bit from this idea of behavioral context. What do I mean, right? There's a few different things that we could mean one could be what kind of control strategy do I use to do what? I want to do right it's one thing to reach out and pick something up That might be one kind of a control strategy something very ballistic Trajectory control or you can imagine right if you're driving some of down some of the roads right in fair Milwaukee After the winter there's lots of potholes and if you're holding on to that steering wheel Right your control strategy might be I need to hold this still even though I'm getting all these perturbations to keep my car going straight down the road So different kinds of control strategies are going to be important So that's one of the things that I'm kind of interested in talking about Something else that that's interesting to me is how does sensory feedback that either the presence of sensory feedback or the absence of sensory feedback play into This kind of control system and then the third being what I'll call Modification of the end effector right so and by end effector I'm talking about the arm and you can think of lots of different cases where our end effectors get modified right if we're Going to go to the gym right we're going to lift a lot of weights our muscles are going to get stronger That's going to change how we move and it's going to require that the brain changes how it's going to control behavior on the other side as we get older and maybe we end up with Parkinson's disease or some other kind of Disability that's going to change how the brain has to control this limb to kind of make it move I'm going to start for a second just talking a little bit just a kind of a cartoon example of giving you a flavor of how Context or changes is in sensory feedback can affect Processing of these motor areas so this is Something that I think if you're a graduate student in this kind of area We're all too familiar with it And this is basically just like the rawest of the raw data that we can get basically what we're doing here We're putting an electrode down in the motor cortex. This is the part of the brain that Explicitly controls how we move that's what it is and these are two neurons the the action potentials from two neurons That we're recording from that one electrode so these are two neurons and very very close proximity to one another and So these are all the different action potentials that we're recording You can kind of see them all stacked up on one another and this is when those action potentials occurred in time So think of this the y-axis doesn't really mean anything here other than the fact that we're just trying to spread out where those dots occur And then just looking at how they and they evolve or unfold in time And I take a picture of this obviously piece of paper because this is something I just always keep hanging up on the my basically board in front of my desk because it's one of these things where You'll do an experiment sometime. You make a hypothesis, right? You don't know if it's going to work This is one where I could see that my hypothesis was correct when I was looking at the rawest of the raw data So this was something that was kind of interesting to me So I'll draw your attention to this green neuron first and you can see this green neuron is going along spiking happily So each one of these dots that you see in this panel here is an action potential And then I kind of drew this line in here and I think what you can kind of appreciate is It's going along maybe at some kind of rate And then there's a kind of an increase in the density of the firing of this neuron in this time period here in this time period here I think we can appreciate this something for this yellow neuron. It's going along happily and then it basically turns off We don't see any firing for that neuron So these are two neurons in the same part of the brain that control movement that do two completely different things During this interesting time period. So this is kind of what I'm thinking about when I'm talking about context So what was going on in these different two two different time periods? So this is a little bit of whatever the experimental setup that I use Looks like as we use rhesus monkeys in this case and we're going to record the Neural activity from rhesus monkeys. So we kind of bring them in we set them in a chair And then they basically have their hand in a robotic exoskeleton So this is kind of a bottom view of the real experimental setup And they basically play video games with this robot, right? They put their arm in this robot and then they're just going to start making movements in this horizontal plane We project some kind of a game onto a screen that they can see Using a two-way mirror. So basically they can't see where their arm is at any point in time But they see a kind of a proxy for where their arm is you're going to see that in a second So I'm going to play this video here and give you an idea what's going on So this monkey is going to go ahead. He's good. So we're looking from the bottom up Here's this exoskeleton you can see there's a target and he's just basically going to move his hand and he's going to hit That target and do it over and over and over again and then we're going to play an interesting game with this monkey and we trained him basically to What you'll see so you see this line come up on the screen. He gets a little upset We kind of get him back in position and then he's just going to sit there not move Right and he's just going to watch what he just did He's going to watch that target move around and then there's a proxy for the cursor That's going to move around exactly how he just did it and he's going to do that over and over and over again And that's how we generated this kind of a task this this raster that I'm going to call it here So this time period actually this yellow time period or this yellow neuron so these time periods here is a time period where he was observing what was going on just by Feeling how his arm moved around okay, so that's what was happening in this time period here Whereas in these time periods here. He was using his visual Acuity to observe what was going on so what we see going on here, and I kind of Broke it down a little bit differently here is that we've got some neurons So this is just the firing rate of a neuron that's changing over This is on the order of 45 minutes across this whole thing as he's doing different tasks in a condition blue Where he's actually moving his arm around physically moving his arm around you can see this neuron likes blue conditions It's firing rate will go down it doesn't do much in the other conditions pops up a little bit in this blue condition here Pops up a little bit in this blue condition here There's another condition where this he's holding still just watching what's going on on the screen, and that's this yellow neuron It's not doing much and then every time you see a yellow condition it kind of pokes its head up So now these are neurons in the part of the brain that controls how we move We're not moving here, but that's there's still something going on context is changing how these neurons are behaving And then again, we've got this neuron that's proprioceptive related So he's not seeing anything on the screen his arm is just being moved around by these this robot that he's That he's got his arm in and you can see this neurons not doing much And then in these gray periods it picks up and picks up and picks up So we've got these neurons that are doing much different things and we can appreciate Over all the 80 some neurons that we recorded we see very Heterogeneous behavior in this part of the brain that controls movement. That's not only related to movement But rather visual feedback and kind of sensory feedback as well So these are kind of some of the ideas that I'm thinking about when I'm talking about context Kind of the rest of the talk I'm going to focus on a couple of the other segments here the first being the selection of control strategies and Modification of the end effector so I guess I'll talk about the goals of this talk a little bit Yes, please please Okay, and and so that seems to be appropriate here what you know, there's behavior what level of description of that behavior I mean how how high a level of description of the behavior Can you map that neural activity too? I guess that's my question So I think I think we'll answer that a little bit as we go along I mean it's like you can you can map it to very overt parameters of movement some neurons prefer moving to the left some More neurons prefer moving to the right some will prefer forces in one direction versus the other and all kinds of different variations in between very heterogeneous kinds of and this sort of Context laden Yep, that's of the description It's sort of akin to if I said that every supreme gray there yeah or shirt there Yes, yes indeed and and and those are the very those are the kind of questions that I'm interested in right because we find we find Ourself in in lots of different situations right and these areas are so densely connected and have so many nodes Right and sensory input or I guess inputs to these areas from so many different places We don't understand how right what how many other neurons one neuron is connected to So that what that neuron might do in one context might be completely what it does in another context Just because of all the different kinds of input. It's receiving from other things so what I'm going to talk about a little bit today is Going to be to try and relate some spatio temporal activity in the cortex To the initiation of movement so the process of going ahead and actually starting to move is a very interesting one And I'm going to talk about that a little bit from looking at local field potentials These are slow oscillations that we measure in the brain on the order of 3545 anywhere up to 100 Hertz or something like that as well as the kind of spiking activity the action potentials as well So we're gonna and what I'm going to show you is that if we interfere with some of the Spatial temporal patterns of activity in the cortex that we're going to be able to disrupt movement is movement initiation and then I'm going to kind of try and link the kind of Spatial temporal activity that we're going to see in the cortex here And I'm going to give you a little bit of an explanation Why I think we need to activate cortex in such a way to control this like really non-linear thing that hangs off the side of our body Okay, so maybe a little bit of background to start The part of the brain that we're going to record from here is called the primary mostly the primary motor cortex and this is kind of a Study from John Rothwell that he did back in the the mid 90s where they basically used transcranial magnetic stimulation in this case to Basically stimulate neurons in motor cortex and while they were stimulating those neurons in motor cortex they Either or they recorded the electromyographic activity either from the biceps muscle, right? Either from the biceps right the muscle on your hand or the hypothenar muscle on the side of the hand, right? And they would also stimulate the cervical spine to basically the part of the spinal cord that's going to control that muscle activity So what we're showing here is basically These lines up and down is where? Stimulation was delivered either to the motor cortex or the cervical spine here for the biceps muscle and the hypothenar muscle. So you can see Measuring the muscle activity either in the biceps or the hypothenar. We're seeing muscle activity Directly evoked by stimulation So if we stimulate this part of the brain that I'm recording from with electrical stimulation or magnetic stimulation We're going to evoke activity in the periphery, right? So if I took one of these things and gave you a little shock right around this part of your brain, right? Your arm is going to start just kind of moving because that's just what it does, right? That's how we're wired up That's what we do, right? So this is the part of the brain the kind of Output of the brain that's going to control movement and if we actually look at what neurons do in this part of the brain Trying to come back to this idea of description being able to describe neurons if we look at single neurons This is a study from Apostolus Georgiopolis Who was describing How the activity of one neuron changed when a monkey in this case with either making movements to the left to the right? Away from him or back kind of for aft left right kind of movement So what would be called center out movement? So basically when the monkey made the movement to the right this neuron was firing along happily This is where the movement happened It got very quiet and then it kind of started firing after the movement was over again Whereas if this monkey moved to the left it was firing along happily got to the place where the movement was happening Started to fire a lot more right and then when the movement was over quieted down So basically what he showed here is that this neuron Preferred movements kind of made to the left. This was a neuron that coded Whenever that monkey wanted to make a movement to the left. So these are ideas that have kind of Helped the the the field the study of motor cortex evolve across time Looking at their their ability or ability to kind of describe them This is some more recent work from March mark Churchland where instead of looking at single neurons and trying to describe their temporal dynamics He's looking at hundred a hundred or so neurons at the same time So right you've got this very highly high dimensional space Maybe seventy five eighty five ninety five dimensions and he's trying to boil down that neural activity down to a couple of dimensions Right so trying to represent the activity of a hundred neurons in just two dimensions And so for many different subjects in this case doing relatively similar tasks Basically basically what he can see is very orderly temporal dynamics So the way to kind of interpret these plots is where the dots start That's about time zero and then as these lines emanate from these dots that's time Unfolding and how this kind of neural activity is unfolding in this two-dimensional space as a function of time So you can see across all of these subjects that he examined we see some variation in the starting point Right where these kinds of high dimensional representations take place But they all kind of evolve across time with very very orderly dynamics where they're all kind of doing the same kind of thing So we can see there's very well-defined temporal dynamics in the motor cortex But the question becomes what about spatial structure, right? We've got a brain right that brain has to map onto our body in some kind of way and that's called the homunculus And so we have a very well-defined homunculus Both for motor cortex and sensory cortex such that if I came and I kind of did a little bit of say Electrical stimulation right at this part of the brain basically right at the top of your brain almost right in the middle right your leg might start You might muscles in your leg might move where if I come kind of way down Almost on the side of the brain by your ear and we would do some electrical stimulation there You might see some twitches kind of in your face things like that the part of the brain that I'm recording from is kind of up In this elbow forearm wrist kind of hand area of motor cortex. So if we would provide electrical stimulation there we would see Movement of muscles in the arm in the hand if we do some more detailed study. This is work from Peter Strix lab where they actually Look at the projections of these neurons onto muscles. So what muscle explicitly does a single neuron control we can see there's a very well-defined orderly structure kind of across the motor cortex such that we have neurons that are are Defined for controlling shoulder kind of movements up here. This is would be more Medial a little more lateral to that neurons that would control the elbow and then a little more lateral to that Neurons that would control the finger and so this is up on the surface of the brain This kind of area here is as we kind of start to come I should say down into this in this kind of sulcle area right one of these Infolds of the brain so we can see there's this very orderly kind of spatial pattern of neuron Organization in the brain as well. So we've got spatial temporal dynamics that kind of unfold across time So the question is is there anything that we can learn about how the brain functions from the spatial temporal dynamics? So the way we go and do that is we implant what's called a Utah electrode array And I think we're implanting these in different areas. So From an engineering perspective, these are about four millimeters by four millimeters With shank lengths on the order of a millimeter to a millimeter and a half So we take these lay them on the surface and actually gently press them into the surface of the brain So we're recording from inside the brain with these with these electrodes An idea of kind of where in the brain We're recording from you can take my word for it that we're in mortar cortex and the kind of activity that we're going to record Is going to look like this so these should look like action potentials that I showed you earlier You can see that we're recording many many many of them at the same time lots of different Spikes different information with with very high signal-to-noise ratios in this case These are signal-to-noise ratios on the order of oh 10 or 15 or something like that Which is rather remarkable for these kinds of Experiment so very high fidelity activity in this case a Little bit about the experimental paradigm. I'm going to talk about today talked about the the monkey Kind of set up already the task We're going to have the monkey do is the center out task the same one that George opera's talked about and it looks something like this This is kind of simplified the monkey's going to have a target It's going to be kind of in the center of his workspace and he's going to sit there and wait for about 500 milliseconds A target is going to appear in the periphery for a certain period of time It's going to flash when he's supposed to start to move and then he's going to make a movement to that target hold for a brief Period of time and then come back to the center and do that over and over and over again in a bunch of different directions So if we were to look at it schematically, it's going to look like this He's going to hold in the center a target is going to appear in the periphery after Between a thousand and fifteen hundred milliseconds That target is going to start to flash That's going to be his cue to move. He's going to have some reaction time His movement is going to begin He's going to make that movement to that target hold out there in the periphery hit the target hold there And then he's going to get some reward at the end of the target Apple sauce or juice or something like that something that he likes and what I'm really interested in here is looking at the neural activity During this instruction period before the movement starts and then why right around the time when movement is going to begin when When neural activity is going to take place and the first thing I want to talk about is local field potentials So these are these electrodes right placed in the cortex, right? And we can measure the spiking activity like I've shown that's very high frequency activity in the Thousands of Hertz usually but we can also measure these low frequency fluctuations So in here I'm using what's called the the Betel local field potential if you're familiar with EEG at all these are kind These kinds of bands of of slow oscillations and brain signals are named from EEG literature after Greek Greek letters But here we're thinking about 15 to 30 Hertz oscillations in these frequencies So you can see we might have some very kind of high frequency data here and if we band pass filter those Between this 50 or 15 and 30 Hertz oscillation, right? We're going to see a relatively Lower I should say a lower frequency oscillation and then if we just do a Hilbert transform on that Oscillation and then just look at the magnitude of that oscillation across time We can extract that envelope right and basically we're going to do that on a bunch of different trials in this case And then line up all those trials and we can what we're going to see this This waveform here that we're going to call the beta attenuation, right? So it's a it's a relatively well-known phenomenon that around the time that Movement is going to begin in the motor cortex this the power in this beta frequency band is going to drop Precipitously and we see that very nicely so like I said, we use the Hilbert transform to to estimate that beta amplitude and really what we wanted to try and Understand is if there was some spatial temporal relationship in this Beta attenuation across the array, right? So does it start here and then propagate over here? Does it start here and propagate over here? Is it random? Does it not matter? So these are the things that we were interested in here So what I'm showing here in this cartoon is for electrode 4. That's this electrode over here Electrode 44 this one in the middle and then electrode 87, right? So I'm showing the beta attenuation With respect to when movement begins on those four electrodes And I think what you can see here is it appears that the blue one starts first Then you get the green one taking place and then finally you get the red one after the after the fact, right? so we're seeing this kind of spatial temporal organization or Neurons over here doing something then neurons over here doing something and then neurons over here doing something, right? Yeah, please Stay tuned for two slides This is this is this is a question right so The way we're going to try and uncover that if there's actually some organization that we can that we can deduce there Is to just really fit a very simple linear model, right? We're going to take The row and column organization of an electrode where we recorded a signal and we're going to try and map that onto the time That this attenuation crossed this threshold, right? We set an arbitrary threshold whenever this time takes place We're going to log that time so we're going to basically try and predict the attenuation time based on the location Of the electrode on the array and if we do that across a bunch of monkeys We see something that's that's relatively predictable. We can see for this this animal here These electrodes start the earliest and then it kind of progresses across the array to that side In this second animal starting here progressing across the array to that side in this animal starting over here Progressing across that array to that side in this animal doing the same kind of thing So these are very generalizable phenomena all of the animals that we've looked at I show four here We've looked at six eight ten animals. We've done this in humans as well. You see similar kinds of things in humans We don't stick electrodes in humans brains. Those are usually people who are in the hospital having recordings done for other reasons But but they generalize there as well And what we found is quite interesting is that though they don't always propagate in exactly the same direction They always propagate along the exact same axis. Okay, so this is a very prominent landmark in the brain called the central sulcus It's right next to the motor cortex For all these animals and what you can see here is that all of these Propagation patterns are basically perpendicular to the central sulcus either propagating out or away So it seems like there's something about that axis into and out of the central sulcus That end up being important. I'm gonna talk about a little bit about how we Made sure what we weren't seeing is artifact in this case, right? So at least for this animal here we found that there was a very significant regression I think on the order of point seven the r squared for this 2d regression model that we fit in this case So we had a relatively good a regression fit, right? but what we did is to try and and Assess the statistical significance of that regression is basically to spatially scramble those electrodes, right? So instead of basically scramble the row and column mapping of that So we're basically shuffling this data set of data set around and we do that about a ten thousand times, right? and then basically estimate the distribution of the R squared coefficient for those shuffled fits, right? So if we move those electrodes around all over the place We should break this spatial pattern that we see and we in fact do that, right? So you see that there's almost no r squared for all those ten thousand shuffles that we did, right? So we have a very highly significant r squared in this case We do something similar where we basically then subs or break our data set up into trials that were in the first half Trials in the second half estimate that for both sets of trials, right and that's what we get in this This blue distribution here and we look at basically the propagation Direction for both of those halves and they're really quite similar in a lot of the in a lot of these cases But then we do the same procedure that we did in this other case is we take the first half And then we take the second half shuffle the second half and look at the similarity between the propagation directions And that's what this gray distribution is in here So we see a very highly significant Difference between these two different these two distributions saying to us that This really is an artifact this is a real spatial temporal Propagation that we're seeing across these these these electrodes in these field potentials But what about spiking activity, right? We can look at these these the fire the firing rates of individual neurons three different neurons happening to be on those electrodes that I showed you before For different directions, and I'm just kind of there's there's nothing interesting about the y-axis in this case other than the fact that We're looking at different movement directions And we can take those and kind of take means across the directions and estimate what the average firing rate for that neuron was For movements up so green movements up purple movements down and then use a Mutual in or basically an information theory to try and understand when this neuron Was most active with respect to movement onset, so we're using a measure of entropy in this case, right? So entropy the amount of randomness that's in a system so we would expect this these neurons to get less random the closer they were to Initiating a movement okay, and that's exactly what you see that entropy is kind of ticking along it Reaches a minimum where that sell becomes most informative about behavior, right? And then I'll go back to some baseline It's gonna hit some minimum and going to hit some minimum and what I'll ask you to under to to kind of remember is right? This neuron this electrode was furthest to the left this one was in the middle This one was furthest to the right and what you can see is this neuron is most informative earlier than this neuron And then as that one's more Informative earlier than that neuron, so we're not only seeing a progression of activity With respect to the field potentials, but we see it in the spiking activity as well, right where we see If we kind of it's a bit more noisy for spikes because that's just the inherent nature of spikes but we see that these spatio temporal fits work for the Spiking activity much the same way that they do for the Local field potentials, so these are spikes from one monkey the LFP from one monkey, right? And those are columns so we can see that the LFP in this monkey propagates this way the spikes propagate that way So if we would plot those two vectors on top of each other, that's what they look like similar for here similar for here So we see spiking activity and field potentials doing the same thing interestingly If we look during that instruction period that preparation period where the monkeys just holding still and not doing anything We don't see any kind of patterned activity. So this is something that's that's Unique to the process of actively initiating a movement and starting to move the arm so it's not just a Property of cortical neurons, so what we wanted to do then is really try and understand Well, is this really something that's important for initiating movement if we didn't have this kind of thing happening? Could we still initiate a movement? So the the experiment that we wanted to Take a look at there was to try and establish some kind of a causal link between this spatio temporal pattern and movement initiation And what our hypothesis was is that if we disrupted this progression we would Disrupt movement initiation in some way so basically what we did is we saw this beta attenuation Threshold and we came in and we decided we're going to electrically stimulate the cortex spatio temporally at a few different times either long before movement onset So this is long before movement onset around when movement onset happens or after movement onset So basically we gave trains of pulses electrical pulses on the order of There were about 10 micro amps or something like that delivered right into the brain 200 microsecond phases at about 250 Hertz and Importantly what we did is we tried to align the electrical stimulation with the pattern of progression of this neural activity So in this animal here, right his neural activity or his spatio temporal pattern beta attenuation Orientation I guess you could say was kind of up into the right So what we decided to do was apply three different patterns of stimulation one that started at the bottom left and Moved to the upper right so you could say with the pattern of beta attenuation One that started at the upper right and moved to the bottom left Against that pattern and then one that was completely random with respect to time Maybe it started where it started here and then went here and then you know here and then here and then kind of moved all Over okay, so I so our hypothesis was if anything, right? We should see a disruption in the initiation of movement because we're going to be applying applying electrical activity in the opposite direction that this This inherent activity of cortex is going to be Propagating along so what I wanted to show Briefly was this right? I told you earlier right if that you provide electrical stimulation To these parts of the brain something happens in the periphery right you get a twitch you get a muscle twitch or something like this We purposely Defined our stimuli such that they were below the threshold to elicit elicit those kinds of activity So what I'm showing you here is right? This is just normal movement Actually So no stim with against random, so right so this is no stimulation with the direction of stimulation Against the direction of attenuation and then that random so you can see that this is just looking at the velocity of movements across time Right, so we move velocity goes up and as we slow down velocity comes down and you can see that all of these lines Lie on top of one another so we weren't perturbing movement in any way with our stimulation Okay, I guess in the interest of time. I'm just going to explain this middle section So this is when we applied stimulation right around the time when beta attenuation that the kind of spatial temporal pattern takes place and if we look at The relative reaction time how long it took the monkey to initiate his movement we can see that we Significantly increase the reaction time when we stimulate against that direction of propagation So when we can kind of disrupt that spatial temporal activity in the cortex, we can delay at least We can't in completely inhibit but we can delay the progression of activity if we I guess I would hypothesize if we would Stimulate for longer periods of time. We would be able to kind of eliminate that activity. So this is basically Direct evidence that this spatial temporal progression of activity is necessary to Elicit or at least initiate movement. So that was the first part of the talk that I wanted to talk about I'm going to skip ahead for a second You just go so we'll start here. So That first block diagram I talked was talked about was a little bit simplified Right, we've got task selection. We've got the brain kind of doing its thing It turns out right that the brain is more complicated than just a block There's lots of different things that we can think about that go into Providing those kinds of motor commands that we need to move the arm We need to select an action plan the movement generate the commands these kinds of tasks are definitely are usually Attributed to different brain areas, and that's what I wanted to talk about briefly now is how can we then go ahead? and parse in some way shape or form these different actions that go into Creating a command and try and understand how that relates to behavior so if we think about action selection and movement planning those are usually thought about as Tasks that happen a little forward in the brain from the motor cortex usually it's something called the premotor cortex Where this kind of command generation is in this area of the brain that have been telling you about all the time Called the motor cortex and so what we did in this case I guess I'll say this so The general thought across time is that these premotor areas don't really represent movement as strongly as the motor cortex They're like higher up the level of Abstraction so they do a kind of higher level tasks where the motor cortex kind of deals with the nitty-gritty of what's going on with The arm and things like that so really this last part. I wanted to talk about Address a couple of questions the first is to try and understand if the Responses in premotor cortex Look like those in motor cortex have maybe a very low level representation or consistent with this old Hypothesized idea of having a very high level or Extrinsic representation so extrinsic am I moving up down left or right versus Intrinsic am I controlling a muscle am I controlling a force or something like this? And so can we see things in the population activity that look like this so we look at movements I guess this is what movements look like. We're having right these monkeys make these eight movements in eight different directions If we pick one to the left here I'm showing in gray here or in the dash line is y velocity in the Black line is the x velocity so you can see we get very nice velocity profiles for movements to the left the joint Torques at the shoulder and elbow change the joint velocities at the shoulder and elbow change as we would expect them to change to things that we expect to be Extrinsic we would expect to be represented uniformly in space, right? So velocity is an extrinsic variable. We're either moving left right up and down It doesn't change much whether we move right left up or down very very much a uniform distribution Things like joint velocity are very intrinsic kinds of representations. They end up being represented non uniformly in space So the question is is do we see neural activity that is uniform in space or non uniform in space in these areas? So in this case where we're recording from lots of different areas. I think this is almost 2000 different neural Recordings similar kinds of tasks that we that I talked about before center out task Interested in the instruction period in the movement period so these are kinds of Neural responses that that we can record So right we've got different directions of movement. This is during the instruction period This is when movement starts so we can see neurons maybe in primary motor cortex. They're sitting there quiet They do something during the instruction period that looks like it's slightly different for movement direction And then we get this very widely varying activity around the movement time That's definitely varying by the direction of movement. I Told you before that there's some neurons that prefer moving certain directions versus the other That's what I'm showing you here So this neuron during the movement period likes movements kind of up into the right Whereas during the reaction time in the instruction period like movements up into the left in pre-motor cortex In all three of those time periods this neuron does about the same thing and then we can see something Again different very heterogeneous again in in this pre-motor this other pre-motor area So I'm gonna come back to this slide We're not come back to this slide or talk about this slide briefly So this is taking those directional Representations for each of those neurons and then plotting them as a circular histogram, right? So the length of this line is directly proportional to how many neurons I found that liked movements in that direction Right the length of this line is how many neurons I Recorded from like movements up. Okay, so we're basically plotting those as circular histograms here for the primary motor area dorsal pre-motor cortex ventral pre-motor cortex, right? And I'm gonna come back to this Talk about intrinsic or extrinsic versus intrinsic kinds of representations again So if we kind of look at these shapes that we get from these distributions I'll submit to you that I think they all kind of look X or very intrinsic, right? They kind of have bimodal distributions They're not very uniform. They're not quite uniform in space. And that's exactly what we're showing with these yellow lines in this case those are the significant or the the directional specificity or the orientation of those distributions that we found were significant with a Raleigh test in this case And what I really thought was interesting thinking about right how these progressions change as a function of time is to track This yellow orientation or what the the angle of that orientation is as we start very early before movement takes place as Movement takes place and then after movement takes place So for these three areas primary motor cortex dorsal pre-motor cortex and PMV I'm just gonna focus on PM1 in this case the primary motor area you can see that this representation is very stable Let's say 170 degrees and it stays stable up until about 200 or 300 milliseconds before movement starts But right about the time that movement is initiated almost exactly looking like that beta attenuation thing that I showed you earlier right you see this very huge change in this Population level activity the orientation of this distribution Changes right from here all the way down to here right around the time that movement starts Which I thought was very interesting and kind of looks like those temporal dynamics I showed you earlier where they're starting in a place right and then they're all kind of emanating and rotating and changing as a function of time so I wanted to kind of ascribe some Relevance or some meaning to What was going on in that in with with that kind of temporal change in this Population hundreds of neurons the activity and so to do that I basically simulated a net neural network model that controlled the physical model of the arm So the physical model of the arm was just a two-link arm almost exactly the same thing that we're actually having the monkey to do it was fixed here at the shoulder it had an elbow and then it had an end point and it needed to Basically reach in eight directions the same task that we were having the animal to do and it did that through the use of six different muscle actuators that we coded in here, so there was basically Muscle actuators for the triceps muscles For the biceps muscles and then for some muscles that control the shoulder basically Different kinds of muscles and basically what we did in this network models We had a bunch of neurons in a neural network. We gave them some kind of task We gave them state feedback state feedback in that case. How far away am I from my target? Simulated that those those neurons ran them through this Biomechanics model that we had and they just had them reach many many many many many times This is the cost function that we used to try and Optimize our model so the cost was basically The difference between where we want it to be and where we are And then basically trying to minimize Neural activity muscle activity and then the weights that connect the different neurons in that case We look at the network model It was a two-layer neural network model that basically generated neural activities that got transformed into muscle activities Those muscle activities were transformed Basically into joint torques into velocities and then into positions And really what we were interested in trying to understand here is right you in this case you is basically the output of Transforming neural activities or actually the activities the neural drive to those six different muscles and transforming those actually into basically I Guess force relationships in that muscle or the the actual output of those muscles and we can take that output of those muscles and Transform them into torques at the joints Through this matrix basically H and basically the matrix H in this case is a non-linear matrix that represents the Physiological properties of the muscles so muscles are non-linear Their force output changes as a factor of how fast we're moving So there's a very non-linear relationship if you move faster too fast The force output will go down if you move too slow the force output goes down But up in the kind of there's an optimal range for how fast we move to generate the most force Forces all also related to the length of the muscle right when our muscle is relatively short We can pull harder than when it's very very long anyone ever done a pull-up before I've tried to do pull-ups all the time, and I'm very unsuccessful It's a lot easier to do a pull-up if you start in this position than if you start in that position, right? That's the force force length property of muscles So we did two different simulations here one where we just had a linear muscle model so basically activation is directly related to torque through this matrix M or We had a non-linear model that tried to represent the physical system where u is related to t through this non-linearity as well So we simulated all these neurons and we came up with these same distributions that I just showed you before for motor cortex So for this linear muscle model what's important to see right is this yellow line? Really doesn't move very much as a function of time as we move along right so the start of movement is here Here's before the start of movement moving along whereas when we use this non-linear, right? We try and Model the physiology of the muscles we can see that there is a temporal progression in that yellow line That kind of looks like this and we see something very similar to what we see in physiology So what this says to me? Like I'm actually gonna just conclude on this slide So what this says right is that there's something about the properties of muscles the properties of the end effector Right that are very important to the brain right in changing how the brain Functions to control that arm right so the brain has that very intimate knowledge of this arm and how this arm works Right and changes its neural activity to take Into account the properties of those muscles the force length property the force velocity property of muscles right and we saw Earlier right that we see this spatial temporal pattern of activation Across cortex so really what I think is going on and we're running experiments now to try and really kind of firm these Things up is this temporal activity that we see in these populations of neurons Comes about and this propagation of this neural activity across this array comes about to activate neurons at a very specific time In a very specific way so that we can take into account these force length or actually the non-linear The properties of the muscles that are are important to kind of help us to move along so with that I'm going to stop acknowledge all the the collaborators that I've had on this project The different funding Sources that we have and I'm happy to take any questions Not too terrible sure So The movement sciences it is the idea about what the unit of analysis is and quality and so my my You know when I look at these these neural patterns Have you ever thought about or do you guys of? Associate that with a notion of quality of the movement or and so you know one one idea of quality might be well um You've the goal of the movement The goal the end goal of the movement if that can be achieved with less energy Something then it's higher quality. Yep, and then so that seems like that And you know that could be mapped the some notion of quality. Yep mapped on to those patterns and then more relevant as The Really what I'm interested in is my guess is that quality and This goes back to the unit of analysis and an elderly person walking down an empty hallway Consider that versus the elderly person walking down the hallway surrounded by Rapidly randomly moving three-year-olds sure the quality of movement this I mean in the latter case You take sort of the individual in the crowd as the analysis and I would think unit of analysis and I would think that You'd get completely different Quality of movement sure ends of patterns and okay, so maybe let's stop there and Let me try and answer a couple of those questions and then we'll see where we go from there. So I think maybe what You can appreciate from even the first slide that I put up right that was basically this this idea of how neural activity changes across time Very heterogeneous lots of things going on Right hard to kind of think about these very high dimensional spaces from the aspect of quality of movement It turns out if you take those High dimension least high dimensional spaces and ascribe a dimension of interest right quality. Let's say Or Dimensions that are important to the neurons To get done whatever task wants to get done move from here to here in the shortest possible distance being as energy efficient as I can It turns out that the brain is very good at Minimizing variance along dimensions that are very important for the task that it's going to do well Letting variance in other dimensions that it doesn't care still be highly variable So it turns out we can't do a whole lot of that with these data here because these are very very very over trained Movements right these monkeys make thousands of these movements every day So they're doing things over and over and over again now what we can do Is play games with them See if I can find the slide Comes to mind it instead, you know, they they move from out To the center. Yep, and then from the center to the right. Yep, but if the task was to move from To the center, yeah, yeah, yeah, yeah, yeah higher quality of movement might be just to go straight indeed Indeed and these are things that that we see them do so I'll put this slide up for a second So this is a task that It's an adaptation task So basically we have them making these movements and then we change that we play a game with them We basically rotate their visual space with respect to where their arm moves. Okay, so Early on right they're making these pre-adaptation period They're making movements in eight directions then we switch them basically to a task where they're only gonna have to move north they only got to move straight and Basically well the game that we play here is So this is their hand and this is the visual cursor right so we dissociate the two such that their hand has to move This way for the cursor to go straight and for them to achieve their goal and we basically look at how the neural activity Changes while they learn this so that's basically What we're doing here? You get a very nice linear map there and it turns out that when we do that Oh, I don't have the right figure here You see very different kinds of neural activity I'll go here to this last one. It's not great, but it'll get the job done So this is kind of looking at neural activity in hide in a very high-dimensional space We're as they're kind of learning and as things are unfolding So you can see as as early in learning or when they're kind of learning this novel representation We're in one space and then it kind of progresses to another space as quality might improve So I think you could map quality onto an axis like this in some way shape or form So Well, I don't know how much time we have I'm happy to talk with you as long as yeah I ran across them because they have ballet tickets, okay But I'm more familiar with is the Eskall Walkman, which is in Israeli Okay, and so now Okay So my question is this So it seemed like you were you know when we were talking about Oh, did you get generalizability across monkeys? Yes So, okay, here's the way I put it. It seems like you could take These similarity maybe fuzzy similarity clusters of neural patterns associated with movement initiation And map that onto A movement pattern pattern as represented in one of the movement notation If you could do that, that'd be really cool because you'd be crossing the I'm not familiar with what you mean by movement notations so much Waterman it's like Yes, it's a three-dimensional space Okay, it's represented, okay, and it's just got a notation for like the arm comes up or if it's gonna I see here Yeah, yeah, you know they try to use it in choreography. Sure. Oh, okay Yeah, interesting. Yeah, I've never I've not I've not looked at It seems like it goes more kinematic kind of movement science kinds of things Well, I mean we're we're we're fixed in a very small repertoire of movements here. Yeah, so to speak But but there's different there's different things to definitely look at. Yeah, that's interesting Thanks, no problem sure indeed Probably because my questions are related to spasticity and the work of the total development Yep Yes So you're wanting to live with this research you're able to identify the patterns Yep So I see that you are you planning to use this for like stroke patients in terms of rehab or In terms of like other neurological disorders to kind of identify what could happen and can you like Especially I know a lot of neurological disorders. Yeah, correct. So what will so So this work is interesting I think from a couple different points of view a lot of what I Let's go let's grab this slide so One thing that I've been very interested in are these ideas of brain machine interfaces, so right if someone's paralyzed They can't control their arm anymore Can we use some kind of engineering technique? Measuring brain signals controlling some kind of end effector. So that's basically what this task is right So we're interested right in how these patterns unfold how these neurons behave in these different contexts So we could build better algorithms basically to transform the neural activity that we record Into a control algorithm that can could control a high-fifth a high degree of freedom robotic arm So right you could give someone back fidelity right basically a Luke Skywalker arm or something like that, right? So this task was something basically where we we have monkeys make movements or watch movements And then we try and build a relationship Through different techniques through from the neural activity to how he moved right basically a linear linear regression Is the easiest way to do it and then we basically put them in a different task where we say okay? Now use that representation to actually move something around so here. We basically Decouple the cursor from their arm and say this cursor is now moved by this representation So you hold your arm still put it by your side put it behind your back or something like that right and move this cursor around to hit targets So that's kind of one of the applications that we think about is trying to understand how this neural activity Unfolds as a function of time so we have a better Idea to how to create these algorithms one and context is an interesting thing for me From the idea that you can imagine if you had one of these algorithms or movement descriptions You might say it might work in one state, but when you go to another state it might not work so well So right how then do we change those? Algorithms across time to you know go from the task of I want to reach up and pick up my water bottle to I want to drive my car down the road and do that so that's kind of where we think about them There are people who've used these kinds of brain machine interface interfaces usually with EEG to kind of Facilitate stroke rehabilitation to try and reinforce patterns of activity that are related to movement Yeah Sorry in some like human brains as well With stroke like people who are typically there people who are in the hospital to have epileptic foci map so people have epilepsy a lot of times a treatment for Epilepsy that's resistant to medications is to actually take the part of the brain out. That's offending So, I mean, I know you can't put it in healthy brain Subjects, right assume. I'm maybe I'm making some assumption that a monkey brain is representative of oh, yes Oh, yes, that's that's there as close as we can. Yeah How does the brain like mapping from those subjects and human subjects relate to the normal they look they look almost exactly the same There's there's between oh wait They're almost exactly the same the kind of The neural activity that will record will try and record in humans so imagine Epilepsy usually happens in the temporal lobe So you'll have grids, you know kind of in this region here the part of the brain that we're interested in is here And it's usually not Affected by the epilepsy so it's kind of something where we'll ask the subject. Hey, you're having this surgery We're gonna put a big grid in here. Do you mind if we put a little one in here, too So we're gonna try and record Activity from healthy human brain or as healthy as it is it can be in those cases The kind of refinements that you see often track behavior, right so You know if you're gonna if you're gonna make a movement and all of a sudden I'm gonna change the world on you And I'm gonna rotate your visual feedback with respect to You're gonna see right I'm gonna move this way the cursor is gonna go this way And then so I got to learn to kind of do that to make it go straight You're gonna see the neural activity tracking what the arm is doing right and then basically then Like we were talking about variance before the variance kind of along the dimensions that are important gets big And then as you kind of hone in on okay I understand what that permutation is now and I can make that very straight ballistic movement that variants then starts to shrink back down So I think you can make Make an analogy to people who've had stroke usually what happens in a lot of cases if you look at their movements after stroke or As they're able to get better doing tasks like this after stroke they have lots of variance in dimensions that aren't important Even important dimensions and then they're able to kind of restrict that variance to kind of make better quality movements Even though they can't quite always get back to the best quality of movement Yep, no Has anybody done for the other parts the other region that take care of other movements and do It's right to say the same patterns or same same idea. So temporal structure seems to be conserved across species and across movements So if we take a big high-dimensional space and boil it down into some lower dimensional space we see similar kinds of temporal progression of neural activity across species across Movements now that doesn't mean that that space that we're looking at right is the same across all different movements So it seems like there might be right the orientation of a plane might change but the behavior within the plane tends to be conserved and and There's papers by you could look at papers by Mark Churchland They've done it in monkeys reaching monkeys locomoting I think he even shows data from a lamp ray swimming or something like that kind of rhythmic kinds of behaviors They all tend to be conserved the kind of temporal structure of movement tends to be conserved across across species All right, so upper and lower. All right So so right right upper and lower are two different kinds of things in a lot and well it depends on the the species of course but in humans anyway upper limb movement is a lot different than lower limb movement because a lot of lower limb movement tends to Be related to central pattern generators in the spinal cord kind of offloading some of that activity so And that that happens in basically almost every animal chewing is another interesting system that tends to be a very subcortical kind of movement Yeah, so upper and lower are very different humans and monkeys primates in general Have more well refined upper limb movements just because of tool use So in things that have to have dextrous movements of their hands and digits and fingers You see these very expanded representations and the kind of things that I've been talking about today Yep Yeah, I don't know I can't speak to shivering I've never studied shivering it's a very autonomic kind of behavior so It's kind of noise riding on top of things I think it might just kind of increase motor noise, so I don't know indeed