 This authored, or co-authored, more than 125 articles and abstracts related to today's topic. He is considered one of the outstanding researchers in the area of magnetic resonance imaging, and we are pleased to have him at this conference. We are looking forward to hearing you speak on the topic behavioral neurophysiology of the motor cortex, Dr. Georgopoulos. Good afternoon. Welcome to the afternoon session. First of all, I would like to thank the organizers of this conference for the invitation for me to participate in this neuroscience feast. It is a great honor and a privilege to be part of this occasion. Second, I would like to acknowledge that the data or the results that I will present that they have been obtained over a period of maybe 10, 15 years with closely working colleagues whom I will not acknowledge by name because there are too many and too precious to be somehow reduced to simply naming. But I'm a team person, I haven't done much all by myself. You can't do anything by entirely yourself in this field. And I would like again to acknowledge the unfailing support and help from all my collaborators. We might as well start then. Well, the conference, the theme of the conference is unlocking the brain. So I thought might as well show you a lock and a key. If there is something to unlock, we probably need a key to unlock it. And what the point I want to make though is that there are quite a large number of issues and problems in brain sciences that we still don't understand, of course. But I don't think there is any unique question that will have a unique answer. More or less, we're dealing with a lot of different locks. Therefore, a lot of different keys. And during the past few lectures, you heard some of these examples. Dr. Kandel had a beautiful molecular biological key to learning and memory. Dr. Hubert had a powerful microelectro key that I'm using as well. And Dr. Damacio had a sort of unfortunate experiments of nature that is lesions of the brain is one of the ways to look into brain function. So there's still around several grand theories of the brain. I don't feel qualified to develop one. So my talk will be rather focused, if you want, on specific issues and how we get some idea about what is going on in the motor cortex. The structure that controls movement rather than propose a generalized theory. Actually, a very nice summary of the different theories and problems and issues in the field. You heard yesterday from Dr. Churchland. Well, the first part of my talk will be relating to some recent experiments we have been doing with the human brain. Looking at the human brain using magnetic resonance imaging, the second part of my talk will be mostly dealing with what I'm doing for a day-to-day living, which is looking at the brain of monkeys and trying to understand how a small brain circuit works. Well, the human brain, you see a photograph. A photograph of it on the left with all the peculiarities of the sulcai and gyri and the difference. We appreciate the difference between the two hemispheres. And one you would say that what we call mine, as Dr. Hubel mentioned, is just the total sum of the immense combinatorial, almost interactions between the different elements. And that humbles us and makes us rather modest in what we're trying to achieve and understand in the long run. Now, there are a few different ways you can look at the brain. A venerable one is just to open the skull and look at it. So you see here a photograph from the work of Penfield, with the Penfield and Rasmussen. That work is actually was done and still been done for the purpose of removing epileptic foci in the cortex. And for these operations, you need to know where the seizure locus is. So one would open the brain and open the skull, look at the brain and try to localize the epileptic focius. Well, how do you do that? Again, by using a venerable technique of electrical stimulation and the simple way of putting a piece of paper with a number on the part of the cortex that gave you certain response. You see here again two examples from the work of Penfield and Rasmussen. And what was discovered with these techniques were two, I think, quite important things. One was, we learned quite a bit about the somatotopic organization of the motor cortex. And one reason I'm showing you these slides is to take advantage of that work and take from there on the more detailed investigation of one sub-area of the motor cortex. So if you stimulate electrically the surface of the cortex, you elicit movements of the contralateral, some contralateral part of the body, be it fingers or hand or face or leg. On the other hand, what was found by these workers by stimulating other parts of the cortex, especially the temporal lobe, that in some cases they could elicit very vivid memories from old childhood memories. And not that that particular observation led to any major discovery, but it was a very useful technique that had to be done anyway for medical purposes, but provided a worth of information on somehow the initial gross organization of some areas of the cerebral cortex. Now you understand that this, you cannot really do on a routine basis. Even if you volunteer, I think the existing ethics committees will not allow you to be paid volunteer for such an operation. It is still being done for epilepsy in a much more sophisticated manner. You might have heard of the so-called subdural grids. There are sheets of plastic with up to 30 or so electrodes so that you can implant subdurally that is underneath on the cortex, underneath the dura, in patients with epilepsy and you can record the activity, the electrical activity that you can get from these electrodes and trying to localize the focus of the seizures. So when an operation time comes, one can get a good localization. Modern techniques looking at the brain, actually this slide summarizes what I referred to a moment ago and that is that by electrical stimulation of the motor cortex, the somatotopic organization was confirmed that was previously inferred actually by observing how the seizures spread in grand mal draconian epilepsy. And it so happens that if you see here the central sulcus, there are, and you see words here, hand, face and so on and so forth. It means that electrical stimulation at these points will elicit movements of these body parts in the contralateral side. Our work that I will describe you in a moment and a little later deals with movements of the forearm and arm, this reaching movements in space and that is the function that I will be concentrating on. I wanted to simply to mention that if we did know about this somatotopic organization it would be very difficult to carry the work that I will describe you. Another way that you can look at that is shown here in the coronal section where you have the representation of different parts of the body. And Dr. Kandel mentioned the fact that the representation is actually proportional to the use that one makes of the parts of the body. So we have a big hand representation, a big face, but relatively small sort of leg and trunk. Well, modern techniques that by which one can look at the brain first of all in a structural fashion are essentially based on magnetic resonance imaging. You see on the left in the image of the cortex in an oblique slice, single slice of the right hemisphere. This is a 12 millimeter slice, superficial oblique slice of the cortex. You can take horizontal sections. This is a horizontal MRI with a pretty good gray, white matter differentiation and you see the different nuclei in the thalamus and the pitemmin and so on. Both of these slices were taken using a high field magnet that is in the University of Minnesota Center for Magnetic Resonance Research. And this work has been done in collaboration with our colleagues in that center. There are essentially three magnets in that center. One is a high field four Tesla magnet. This is the highest field that the FDA will allow for using humans. Give you an idea, four Tesla is about 80,000 the magnetic field of the earth and it's more than three times stronger than the 1.5 Tesla that is usually used for clinical work. In the center, there are also two high power, high field magnets that are used for animal experiments and in vitro work. Well the four Tesla magnet is one of four in the world and one of three in the United States and as far as I know the only one completely dedicated to brain research and it's pretty comfortable. It's 125 centimeters bore, that is you can lie there comfortably without feeling too claustrophobic and these are some other technical characteristics of it. Now one can use the MRI for two main reasons. One is structural one and you saw an example before. The other is which is very exciting is for functional imaging of the human brain and what we mean by functional imaging or dynamic imaging as Dr. Tamasio mentioned is trying to get an idea of changes in local metabolic and or homodynamic signals in areas of the brain that seem to involve to be involved in process specific tasks. The whole idea is the same that is used for pet imaging. That is the idea is that if you perform a task and you need a certain brain area to perform it then there is local use of the metabolism changes locally that elicits local changes in blood flow in other hemodynamic changes and that results in some change in the area that participates in the task and if you have the proper technique like the pet or the MRI you can identify which areas are responsible for which tasks. The advantage in general of the MRI is that you can do that in a single subject. You have very good special resolution for brain imaging. You have to be very careful not to move at all. It's very sensitive to motion artifacts so it has pros and cons but it is a technique that is evolving very fast and has a great promise. Now the particular technique that we're using for brain for MRI imaging is based on the fact that on the some outline here on the slide that is if you have a small voxel, a volume and you have a small vessel running through it then the hemoglobin can be in essentially two forms oxygenated or the oxygenated without oxygen and this is a paramagnetic substance in contrast to this which is a diamagnetic one and the relative ratio of the two of them will give you a signal that the magnet can detect and in fact the higher the field of the magnet the better the identification of this signal. This is a rather recent observation I think the whole field using hemoglobin is an entogenous contrast agent started 1990 or so by the work of Sergio Ogao at Bell Laboratories and has been quite a useful tool for the studies. So essentially what you do is the following we ask the subjects to perform a task and in this case the task was to make alternating movements between the thumb and the fingers. So before, during and after the performance of the task we take images, the images used to be like 10 seconds would take for an image, now it's much faster with modern techniques, you can take images 100 milliseconds or lower, although the biological signal that gives you the meaningful signal doesn't change until a second or two so the bottleneck if you want is in the biological signal not so much in the magnet itself but in any case you end up with a time course so these successive images, they were 10 seconds per image they were from our initial work and you can see here that during the pre-task period there is a baseline, this is arbitrary intensity signal intensity, essentially the ratio of the two forms of hemoglobin that I mentioned to you and then when the task is on one of the movements are being done in this case by both hands there is an abrupt increase in the signal that stays on and then at the task off you get back to the baseline. This is a trace from an old study somehow one reason I show it is that two reasons first of all the change in intensity here is wonderful is almost more than 100% and number two I am the subject here I have been the subject so it's been a motor physiologist I'm proud of my performance in the motor task and I want to share that with you. I usually the changes in intensity we get are about 10% or so we can detect down to two or 3% but it's usually of the order of magnitude of 10 to 30% that we see as changes. So what then we do is process the data and make a decision whether this is significantly different from the background, the pre-task period and we end up with maps of that sort that similar to those that Dr. Damasio showed you yesterday only with somehow better immediate image of the structure of the brain and here you see localized activation of the motor cortex that is in front of the central sulcus, the pre-central gyrus during movements of the contralateral fingers. Here there are two actually composite images one superimposed on the other one from an experiment in which the subject move the hand and you have the yellow activation here and another in which move the foot, the toes actually and you have the medial activation and if you move the upper arm and so on you have activation in between. So with this technique we're able somehow to it's not a matter of confirming but make sure that we can see similar somatotopic organization as one had seen, had shown before with the electrical stimulation technique. What we did then is my interest is more in sort of quantitative work or whatever that can be done was to measure the area of activation in the motor cortex in the pre-central gyrus under a couple of different conditions which are summarized here. What we wanted to know, there are two things first of all whether the left and the right motor cortices were symmetrically involved in the performance of contralateral or lipsilateral movements and number two whether there were any differences between left-handed and right-handed people. Now we all know the hemispheric asymmetries between the left and right hemisphere. The predominance of the left hemisphere in speech function and of the right supposedly in visual special tasks hardly much is known about any existing asymmetry in an area as supposedly simple as the motor cortex which is somehow the final pathway from the cerebral cortex to the spinal cord and brainstem structure regarding motor function. Now let me explain to you first the slide on the left side here. Let's start from the right hemisphere. So the data we had about, if I remember correctly the left-handed people I believe were nine and about the same number or actually there were I think six left-handed people and nine right-handed people. So we're looking here at the right hemisphere while movements were performed with a contralateral hand or the ipsilateral hand. Now the right hemisphere behaves much like we would predict as a null hypothesis if you want and that is that is involved mostly if not exclusively with performance of the contralateral movements with the contralateral hand. There is a small but significant activation with performance of ipsilateral movements you see here with the eye which was not exactly well-known but was not surprising since anatomically the corticospinal tract there is small percent that is uncrossed so you might expect to have disinvolvement. Now what was very surprising though is when we looked at the left hemisphere and there the picture was quite different in two respects. One one respect the relative activation on the motor cortex between contralateral and ipsilateral movements was quite different from the right hemisphere that is you had more involvement of the left hemisphere with ipsilateral movements relatively than the right hemisphere. So the ratio of the long bar over the shorter bar is smaller here than it is here on the right. And second even more intriguingly somehow this was even more pronounced in right-handed people. This exactly was counterintuitive we were expecting if anything the representation of the right hand the dominant hand in right-handed people to be quite prominently represented in the motor cortex in the left motor cortex. Instead we seem to have a reduction in this representation if you want and one can hypothesize it doesn't cost anything to put out hypothesis but it seems that maybe in the right-handed people the use of the hand is so well learned and so well practiced that almost as probably other motor skills it is not the burden of the day-to-day cortical motor cortical function and it is probably being delegated to sub-cortical structures. This is something to be further determined. Now what was also interesting is that this relationship that is the more predominant involvement of the left motor cortex in this effect seem to be or if you look on the side on the right now even quantitatively varying with the laterality quotient. So it's what it shows is the activation in all the individual subjects here the longer bar, the contralateral activation of the motor cortex versus the laterality quotient the anti-bork inventory laterality quotient in which 100 is complete right-handedness and minus 100 is complete left-handedness. You get this number by completing a questionnaire how you cut the bread and with which food you take a bowl and so on and so forth and what you see that there's a continuous reduction of the activation here of the involvement of the cortex with the contralateral with the right-hand function that is the left cortex is a function of handedness and we had even a ambidextrous subject that also seemed to be along the band of this relationship. Well, that was exciting somehow to us and because it was unexpected actually in retrospect it might have been expected there are results or observations in people with strokes and the capacities of use of their right of their good hand and there seem to be some hemispheric asymmetries there but these were usually attributed to to damage in association cortex not really to the primary motor cortex. What we're currently doing is running a series of these tasks of homozygotic twins that were very privileged again to have one very large inventory in Minnesota trying to see whether, I mean, twins that are left and right-handed in each pair and still have the same genetic makeup so we're trying to see if some patterns of activation could be related to genetic factors or are completely dominated by handedness. Now, imaging is very good it's the only way at the moment one can look at the human brain but you cannot get any information about the mechanisms that is how is it that information is being processed in a given area. So we're moving to another key in another lock perhaps and now this is more properly if you want how the cells interact in the motor cortex and I think I'll hopefully persuade you that we have found a small key that we can open a small lock but still quite interesting to us. Now, what we do and what we would like to know is all of these activation patterns while all of these are metabolic signals essentially or hemodynamic signals what is generating those signals in the vasculature somehow is how these cells work together and how intensely they interact they're making metabolites, synthesized proteins open channels, closing channels and so on and so forth and this is just a diagrammatic I wouldn't call it a cartoon exactly but a diagrammatic drawing of the cortex with the different cells, pyramidal cells with the dendrites going up and extending and the base of dendrites the afferent pathways and so on and so forth it's this beautiful complexity of interaction between these cortical elements that we'd like to know what is going on. Now the technique we use is exactly what Dr. Hubel described to you yesterday and actually Dr. Hubel he allowed me to use his slide that he almost showed, he just passed by very fast so my eye caught it and I asked if I can borrow it so what you see here is the microelectrode near one of the cells and it is the electrical activity in these cells that we pick up with this electrode we amplify, we look at the oscilloscope and that is our database now Dr. Hubel showed you a great video you heard the sound of these neural impulses are called and these are the spikes that's another word, neural spikes or impulse trains and so what do we do in our case unlike what Dr. Hubel described you use that we have an awake behaving animal of course you can study the mechanisms of voluntary movement in anesthetized preparation by definition so we have to have a behaving animal that is making movements and simultaneously we record the activity of cells in the motor cortex and other areas of the brain and we are trying to see how can we predict the movement that the animal makes how can we tell what the animal does or is planning to do or intends to do or remembers if you want based on the new signals that we get from these recordings now that means that you have to use an animal for that we can do it in humans and our animal has been exclusively the Rhesus macaque is a very extremely intelligent especially intelligent animal you see an outline of its brain that's the central sulcus here this yellow area is the motor cortex and the area that we are investigating is around here you see it much better here in the photograph from the brain of one of our monkeys in a small area of recording here in front of the central sulcus and that is the area that controls region movements controls movements of the contralateral hand in space so if you stimulate there you elicit movements of the arm if you record the activity of single cells during performance the cells are not firing are not active with movements of the fingers or the tongue or the leg but there are quite active with movements of the contralateral arm now one way we show this in a standard way of displaying this data and you'll see several of them in the rest of my talk I want to explain to you as the time that the spike takes to happen it's about a millisecond the time is small relative to the hundreds of milliseconds that the task is lasting so we tend to present them as point processes that is a single event and what you see here are six trials or so that are associated that were recorded from one of the same motor cortical cells during performance of the movement in the same direction and in the same amplitude in space so every time you see a small bar it means that T-cell fired an action potential and this longer bar the target changed from a current position to a new position on a plane the monkey was required to make a movement to the target a pointed movement so this is the reaction time from here to here all the trials as you said align to M to the onset of the movement and then the movement is done around this period here and it ends around here I don't have a sign to indicate that so what you see here is that this particular cell and by the way this is a histogram of the differences of the ongoing cell activity if you subtract this period where the animal holds without doing anything so when you follow that in this particular case shortly after the target shifted it was in a way the command to the animal to move you have an increased dramatic increase in cell activity within this is 100 milliseconds within about 120 milliseconds in this case so this happens during the reaction time that is while the movement is being planned there is no movement going out there yet and when the movement begins the activity of this cell actually subsides it retains a higher ongoing activity at the end of the movement so what I would like to point out is that about 85 or 90% of cells in the motor cortex will change activity very dramatically during the reaction time starting from as early as say 80 to 80 milliseconds after the onset of the stimulus and peaking at about 120 milliseconds after the onset of the stimulus so it's a very intense activation of the whole structure just preceding and planning the movement now the the task the task that we used and they let me seems that one slide may not have gone in place okay, fine just one slide but that's fine alright now the as this is a study of behavior we have to to control the behavior of the animal and as the study our interest is for its own movement and specifically on region movements in space we used two essentially two different ways to control the movement one was training the animal to make movements on a plane from a central point to a peripheral point so these dots indicate the lights that the light will come on the animal will move this handle there the animal will sit at A and then try to move when I said before that the target shifted I meant that the the center light went out and then one of the peripheral targets came on so the animal had to move to make a directed movement from center to number two in this case so if you do it in different directions you have movements that are done in different directions but different amplitudes excursion but the same amplitude on the right more generalized task in which the animal makes movements in three dimensional space from a central point pushing a button and then another light will come on the animal will move from one button to another so it will again make eight movements in different directions in space now what we wanted to know is what would be the activity in the motor cortex under those conditions you can see already similarity between these movement orientations in the orientation of the luminous bars that Dr. Hubels showed yesterday and our hope was back in the early 80s that what the case would be in the motor cortex is that we would find cells that would be sharply tuned to the direction of the movement so that if you want to move in a given direction you can then coherently and uniquely activate a subset of cells that would be very specific for that movement that would be very close to what we know for the visual cortex however what we found is what we found instead is that cells in the motor cortex are not specific for any given direction but are very broadly tuned with respect to the direction of the movement you see two examples here one from the three-dimensional task and one from the three-dimensional performance now these data on the left are all from one cell and the different lines indicate repetitions of the same movement in the direction indicated in the center so this cell increases activity very much with movements in these directions here then you have an orderly change for different movements and it's actually decreasing the activity quite a bit with movements in this direction if you take the average intensity of activation and you plot it against the direction of the movement you get a broad tuning curve it is really broad Dr. Hubel mentioned in the visual cortex you may have a plus or minus 20 degrees around the peak of tuning here is it covers the whole continuum from 0 to 360 degrees so it's very broad tuning the same interestingly the same general function was observed in the three-dimensional case where here you have movements made in different directions and still you have a broad tuning and what seems to be the important aspect here is this angle between the movement direction and what we call the cell's preferred direction see here which is the peak actually of this curve this is the direction of the movement of the cell would be highest now if you plot if you plot this function this cosine function in a polar form what you have is the preferred direction here and the length of the lines indicate the expected activity of this cell when movements are made at angle theta from away from the preferred direction so this is the polar form of the two-dimensional tuning curve on the right and if you look down at C this is the three-dimensional form of the same function which actually you can generate easily if you take this polar function keep constant this axis as an axis of rotation of the preferred direction rotate it in 3D space and then you get this volume where you have the preferred direction here oriented differently because you are dealing with a different cell where the origin of the axis is eccentric so that if you would make a movement starting from here in this direction the length of the line from here to the end of the solid would be the predicted rate of discharge so the function holds pretty well in both dimensions two and three dimensions and here is an example actually of another cell you see the broad tuning and the linear function that you get when you plot the activation of the cell against the cosine of this angle actually this is what this particular figure is now the next slide shows you actually that another I think very important finding from these studies and that is what is the distribution of these preferred directions in a large ensemble of motor cortical cells because you can say fine you have a broad tuning it is also possible though that these preferred directions will be very clustered in certain domains and that will be very important to know what you see on the left is another example of a three-dimensional tuning curve a little more elegant again the tuning volume in the preferred direction and on the right are the preferred directions unit vectors of about 575 cells that are recorded in the three-dimensional case and you see that they are distributed all around the directional continuum there isn't any particular clustering finding that has been confirmed by other investigators well I think because I had misplaced that slide I apologize to the projectionist I think if we can turn the slides on now and just in one moment I'll show you a short video I was inspired by the video that Dr. Hubert showed you so it's a very similar one you will hear the cell activity that is recorded in the same way that Dr. Hubert described however you're also going to see the monkey-making movements so you'll try to associate what you hear with what you see and as it is a video that you can look at from the top on the working surface you'll see the lights that are coming on and off and you will catch I think if you pay attention because we're talking very short times but you'll catch the activation of this particular cell quite early with the monkey's movements also on the top left of that video you see the electrical signals that we record that's where the sound comes from and this particular case for the experts the negativity is up so it's a negative it's a negative spike so when we're ready I think we can start the beta video I apologize for the little mix-up I had two blanks there to get to that we don't see anything that's the problem I'm sorry something is supposed to be shown as well if you can rewind it and show it as well well I hate to ask you if you have questions but if you do have I'll entertain somehow you know we depend to finish that video before I can proceed otherwise we're running ahead of our time well let me lead you in somehow to the next question here because you see these cells are broadly tuned and that means that if you as a lay person have that information still you cannot tell me what movement the monkey is making that is on the basis of the broad tuning care because of the symmetric part of it you cannot really tell me except for the very peak of the activity what is the movement that is being made and that does pose a conceptual somehow problem and that is of how we can infer uniquely the information about the direction of the movement if we just have these very broadly tuning curves so the idea that we developed to solve that problem was a relatively simple one and that was that it seemed that the whole information is still contained within the neuronal example that is it is not don't have to use one cell if you use more than one cell or a number of cells you can find perhaps a way that you can define uniquely the information from the neural data so you can predict successfully the direction of the upcoming movement so this is the video you see the monkey's hand moving so this is the spike task so this is the handle if you can put a sound louder if that's possible there is an ongoing activity there are others that about 30% of the cells wouldn't have any inatonic activity this just shows you in the same format that you saw a moment ago actually we can stop here thank you so we are left now in our summary I will see I went ahead of my somehow talk because of that before I go to the ensemble coding I want to mention just two properties of the cells in the ensemble the motor cortex that relate to their directionality one is that they tend to be clustering in columns much like the cells in the visual cortex they are segregated in columns according to their orientation tuning what you see here is an example of a penetration in the cortex that we reconstructed making small lesions and when we recorded the activity of different cells along the penetration in the exposed part of the cortex the preferred directions were very similar instead if one would make a penetration down the bank the anterior bank on the central sulcus where the columns change quite abruptly the unblocked changes of the preferred directions of the cells the other finding that I want to mention on the right is that it's not only cells seem to be arranged in columns but there seems to be a certain relationship between cells and their preferred directions and how potentially they are interconnected so what you see here is that what is graph shows is that cells have very similar preferred directions this is the angle between preferred directions so if that tends to be zero or in a small interval here then they tend to be synaptically connected interacting in excitatory function whereas if their preferred directions are opposite they tend to be inhibited and that is a continuum and this is again a relatively similar finding that has been an observation that has been also seen in the visual cortex that is that cells with similar orientation tuning their preferred direction here they tend to be interconnected now this the directional tuning then and this finding led us to just develop for the for this conference somehow a static part of it although we are using it as a rigorous way as a neural network modeling tool at what you see here how we visualize this network of cortical cells their direction is tuned so the the cones point in the preferred direction of the cell in three dimensions and the the lines indicate the connection strengths and if they are red they are strong and excitatory because the cells have similar preferred directions and you go through intermediate cases where they are light blue they are opposite they are inhibitory and in the yellow if they are orthogonal there are simply no interactions there I would not be dealing with these modeling studies that we have been doing but this is the this is the essential model that we use for these studies it's a massively interconnected directionally tuned network of simulated motor cortical neurons let me get on to the coding problem which will get as close to what I think would be a small key for a small problem and that is to understand how an ensemble of cells in the motor cortex could provide the unique information about the direction of the movement the left slide simply summarizes what we talked about the preferred direction of the cells the broad tuning that the preferred directions differ follows that the cell participates in the generation of movements in many directions and from this it follows that many cells will be active for the movement in a particular direction so you engage the whole ensemble the question is how can you get a unique information about the direction of the movement well the idea we had for that is simply it's very simple it is now almost trivial in 1983 we first proposed it it was not trivial at all but in any case it's very simple because the cells are directionally tuned we said why don't we regard the command say for the direction of the movement as an aggregate as an ensemble of vectors in which every line indicates the preferred direction of a cell where the cell makes a contribution in its preferred direction the length of the line indicates where the cell is and then the vector sum the simple vector sum of these weighted vectors we call the population vector we can call it a cortical pointer or something that indeed seems to point in the direction of the movement so if you have a movement in the up direction here and you look at an ensemble of a couple of hundred cells with this technique we get a unique signal a unique outcome that is telling you you can predict what the movement direction is the animal does make of course movements that are always straight lines so you have a variation in the trajectories and of course there is trial to trial variability in these data so one properly should compare the movement trajectories a family of trajectories with some confidence interval in the population vector and here this is the same ensemble of cells now we are looking at the movement in another direction so this ensemble changes not direction but changes the length of the lines are changing because the cells are now differently active and the vector sum points in the appropriate direction and you see the same ensemble coding for movements in a different direction now the big difference here from when we started we started with completely temporal spike trains the original data that you saw were simply points in time action potentials over time here we ended up with a measure that is special and which is actually isomorphic to the movement space and we obtained that through the interpretation of the tuning and the population processing idea fortunately this idea worked very nicely for the three dimensions where on the left you see the same ensemble coding for movements in different directions here the red line is the population vector and the yellow line is the direction of the movement so it's the same ensemble that again takes care of movements in different directions and again more properly one should construct a confidence interval in the population vector around here and the yellow which is the direction of the movement is contained within that confidence cone now we we move to the next step which actually is the most interesting in the last part of my talk and that is that can we use that signal now to look in temporal events can we as you saw before most of you know from the time you see a stimulus to the time you start the movement you have the reaction time the period of movement planning it's fine to say you have a measure that gives you unique information but is it useful for looking at the events in the brain as they evolve in time that's the main question what we did to answer that was to construct, calculate the population vector every 20 milliseconds here in the reaction time so here the movement is up T is the onset of the target and shortly after the onset of the target and about 180 milliseconds before the onset of the movement you see that the population vector increases in length, changes in length preceding the onset of the movement and pointing in the appropriate direction of the instantaneous velocities that are coming afterwards similarly if you have movement in another direction the population vector would point in the appropriate direction as well that to me was the most encouraging finding from our work because it opened all kinds of possibilities it opened the possibility that you can ask the animal to do complex tasks required to say withhold information, memorize information process information in different ways and that measure I hoped could enable us to get an insight into what is going on in the motor cortex while the animal is solving a problem and that is what the rest of my talk will be dealing with so our goal was to get some insights from neurophysiology into psychological processes and what we do in general for this in my laboratory is we start with a problem then routinely we study rigorously human subjects human behavior in cognitive psychological tasks on the basis of these results we construct hypotheses concerning the nature of the psychological process that underlies human performance then we train the monkeys to do as closely the same tasks as possible during performance we record the activity of these cells we will try then to relate the neural results to the hypothesis that we have built up about the psychological process see whether we have a match or mismatch there and then we move to another problem now the process is time consuming it may take time to define a problem and to do the human work but certainly it takes easily a year and a half to train one animal to do these tasks so that's why the publications that my introducer mentioned are quite low actually to quite a few other colleagues takes time now that is our strategy that we follow then go on the neural aspects we select the process this variable it is more of a logic in our case it's the direction of movement that we operate upon all of what I showed you before related to the understanding the neural coding of the direction of the movement so that we can use that knowledge to look what this variable is doing while the process is going on I mean the process of manipulating that variable in your mind therefore I somehow conjecture here that the neuron population vector will try to use it as a probe to decipher brain events underline a cognitive process and this is one of the modest keys you know for a very particular problem that I think can serve us well so the first if you can focus the right slide please so the first thing we did that is illustrated the results are illustrated on the left slide is we train the animals not to move so you saw the light and the animal is trained to withhold the movement until a go signal comes later during that period that is an instructed delay period there is activity in the motor cortex we knew that from other studies but what we were interested in finding is whether our interpretation the neuron population vector idea would produce an outcome that would tell us what is being the stimulus that is attended to or is being instructed so you see here three cases where the stimulus was pointing this direction or these different directions and indeed the population vector tends to follow the stimulus in different directions now we went one step further on the slide shown on the right here and still that is not very well focused but what we did here we saw the light for 300 milliseconds then the light was turned off and then after a period in this case after 750 milliseconds total the animal had to move in the direction of the light and it disappeared so during this period the animal had to keep in memory to remember where to go where the go signal came this is what we call the memorize delay period now during this period you see that the population vector first begins to grow and point in the direction of the potential target while the target is still on but after the target is going off at 300 milliseconds and in fact still points in the proper direction at this point the go signal comes and after a reaction time the vector increases further in length and the movement is emitted out here so this measure somehow became then quite useful in identifying the information that is being kept somehow in the motor cortex in a dynamic fashion both during an instructed delay period and during a memorized delay period in a said dynamic fashion because these are time bins this is every 10 milliseconds or so and this every 20 milliseconds so we have a continuum that we can read out that kind of information well the next task that actually we did very recently was entirely different and what we wanted here the animal is to do a memory scanning task that is to search in memory and identify a certain element and make the appropriate response so in this in this case the animal is shown targets in different places on a circle that indicate potentially different directions of movement and then one of these changes color so in this case is number three the animal is trained to move to the next one in the original sequence so it is a context record task it was actually first described by Sol Sternberg in the mid-60s and it requires identification of the test stimulus if this is the test stimulus you have to place it within the context of the sequence and pick up the next one in this particular case as your response so another name for that is scanning to locate is another name for that task now what one finds in this task is that the reaction time the time that it takes you to initiate a movement is a linear function of the number of elements that you have to deal with in your scanning and Sternberg in 69 made a semi-laborate scheme somehow I won't go in detail simply what you see here is that when you see the test stimulus you have to record the elements from the list to scan them, compare them with the test stimulus, make a decision what I want to point out here simply is that all of these takes time so the reason we believe or the high hypothesis I come to a high hypothesis how to explain the human results the reason that you have this increase in the reaction time is that it takes time for you to go from one element to another identify what is the element if it is the proper element and then you choose your response well we trained an animal to do this task and I'll show you a couple of results from this particular case now let me start with a slide on the left here which shows again spike activity impulse activity from a cell that we recorded under those conditions and in this in this case the numbers I'm sorry there are indicate directions 0 is to the right and 90 is up and everything goes counter clockwise every 45 degrees the stimulus was shown the target was shown for 400 milliseconds so it's an instructed delay task say in this case you have zero time here this period is 400 milliseconds and then at this point the ghost signal is given and the animal is moving in the appropriate direction which in this case is directly in the direction of the one and only target there's just one target here so this cell is directionally tuned as before but we are utilizing now the differential activity of the cell for a couple of different cases to illustrate the fact that after the ghost signal in the scanning case you first spend certain amount of time that's about 150 or so milliseconds for the direction of the second stimulus what this means here is at first the stimulus in this direction was shown 315 then at 45 degrees then this one and then this was the third one the signal was supposed to go to this one so what it does first looks if you want at the target at 45 degrees the first part of the activity here is practically a non-existence of activity it is similar to this and then after a short period of time of the order of 150 milliseconds you see the activity increasing and this is what you expect when you have this response here which is this pattern related to the upcoming movement in contrast if you look up here here the second target is 270 is this one so the initial part the ghost signal is given here the initial part when this burst comes the initial part relates to looking at this direction and then very abruptly it's completely inhibited so the second target was 270 and the response was supposed to be upward the point is that for a period of time short period of time you have the ongoing activity which is this one which is completely lack of response now if you look at this case in the population vector you can see that much more dramatically so here at this point in this arrow you have the ongoing activity downward and then very abrupt shift point in upward so you have almost a digital extremely fast shift in how you scan if you want from one direction to the other and this is one mode that we think is similar or at least is congruent with the Sternberg's idea that you spend definitely time target to target and that the switch from one to the other as you shift your processing is quite abrupt the final task though that I will show you is a very different one and that is I think the beauty of somehow this approach in the sense that depending on the task you can get a very different result that is the technique can be quite sensitive telling you what is the underlying process in this case the animal had to do two things first of all if the light was dim it was a force it was trained to go to the target if the target jumped to another location was bright the animal was trained to go 90 degrees counterclockwise from that target so here there is no scanning or I mean it could be it could be scanning for example but by its nature it's quite different task so if you could look at individual trials here in trial one the target is bright and it is here so you are supposed to go 90 degrees counterclockwise in trial two the light is bright you are going directly to it in this one it's bright and you go this way and so on and so forth so for every trial in every case you have to redefine you have a fresh to define on the basis of what the stimulus is and the rules of the game which is 90 degrees counterclockwise now how do you solve that problem what do the humans do actually let me skip that for the case of time the interest of time how do the humans solve this problem again you know you get a hunch you get some idea of how that is solved and the crucial finding was that the reaction time in human subjects increases as a linear function of the angle that you have to switch away from a given target so if you have to move away 20 degrees or 40 degrees and so on your reaction time will be higher and higher now there are not many ways you can explain that our hypothesis was that given a task that you have to move at an angle from a direction then in your reaction time what you do you shift your motor intention from initially being in the target direction to being in the movement direction and the time that takes to that increases here is taken because it takes time to rotate these images in your mind so in contrast to the memory scanning digital sort of process here in hypothesis anyway we are dealing with an analog process that is we hypothesize a continuous somehow shift in this internal image which if you as you can understand we identify with the population vector so the question is if you do, if you record from an animal and look at the population vector will it jump around or will it move in a continuous fashion and what we found was just a second so what you see here is the case where the light was bright to the right and the animal had to move 90 degrees counterclockwise to the left and as you move up in time this way we pick it up 90 milliseconds after the onset of the target where the population vector begins to lengthen and provide a signal so as you move up you see that the vector first points in the direction of the stimulus and gradually rotates counterclockwise to point in the direction of the movement you see it here also where the movement is in the same direction in the direct case in the direct task you have the population vector points in the direction of the movement but in the rotation task it points first upward and then gradually switches counterclockwise which you see it here more clearly to stabilize along the direction of the movement now the value of this is not it gives some unique again information about how this process is done but also in comparison to the memory scanning result where you had a very abrupt shift almost 180 degrees of the population vector it points to the fact that you can identify fundamentally different processes that the animal would use to solve a particular problem the other thing that is very interesting is that the slope here of the change in the population vector direction versus time is very similar to what you find in human subjects when you look at the slope of their function which boils down to about 400 degrees per second so this somehow completes the loop that I showed you for this particular case we started with the problem of transformation in direction space the human behavior showed us a linear increase of the reaction time versus the angle from this we inferred that we have a mental rotation of an internal image we trained the animal to do the same task we identified the population vector rotating so there was a good congruence between the two and then we move up to another problem ok now there is a 3 minute video to at least end in a positive note sorry for the previous trouble we had with the other video if you can show that please this is not neural data they are simulated data of what the database is experimental but you get an idea of what the dynamic somehow function is for this kind of analysis so this would be the straight movement task where you have the light and you have the population vector the cells increasing and the vector is the red one that points in the direction of the movement and then the movement is elicited the two things that you will see here one is the memory task and the memory scanning but the memorized delay task and the mental rotation task so this is the moving at an angle from a visual stimulus which is the mental rotation task so when the color is orange I don't know if this looks like an orange this is a direct task remember that in the task they were intermixed so from tile to tile the animal did not know which position to be whereas when the light here is green it indicates the movement directly so this rotation case now what we have been doing with the neural network just to entertain you we are not going to see it here is to exactly take it as an output from that network the population vector of the network and describe trajectories arbitrary trajectories or making ellipses or other aspects and look at the underlined constraints in such a network that are generated when it is forced it is trained to generate a dynamic trajectory out of thinking together a small continuous direct population vectors so here is the memorized task and see how it is the target appears it stays there for a very short time then it disappears and that's the memorized delay period so as the light appears there is a small increase in the population then during the memorized delay you have retaining the network retains that increment and then there is further increase actually during the memorized delay period there is further increase in this vector ok and that is something that we did not do but Andy Schwartz in Arizona has done and published now actually a recent paper in science not long ago where we visualized proposed that the continuous performance of the continuous trajectory would be done by the population vectors being strung tip to tail and changing continuously in time and that is you can stop the video please and the last two slides is pure aesthetic somehow on the left it's one way we visualize the rotation of the population vector from the stimulus direction to the movement direction and when my friend Richard Passingham professor of psychology in University of Oxford saw that paper he sent me a letter this slide has probably the first time I received a slide of that sort which is shown on the right actually and the letter was saying that he thanks me for the reprint and all of that and that I am sending a slide of a picture which is up on the wall in this department and that is the Department of Psychology University of Oxford in England the label says that it is by Alan Gross in Washington Street Gallery Cape May with your paper in mind I thought this picture should be named population vector making up their mind well thank you very much for your patience ladies and gentlemen we'll break for coffee out on the mall and there will be some responses to apostolous in the next hour but we're 10 to 3 and we need to be back here at 3.30 and you need to break so go for it