 So, this lecture I will be talking about the man machine interface. So, as of now, we have talked about the how the different signal, bio electrical signals are being transmitted, the action potential, the addition, summation, subtraction, inhibitory neural transmitter, excitatory neural transmitters, then different circuits in the form of a retinal circuits, cochlear circuits, the olfactory systems, the gustatory systems, the memory acquisition, the spinal cord signal, the stretch reflex arc and so on and so forth. We have talked about several such circuits and how the electrical impulses are being coordinated. So, one thing what I highlighted in one of the earlier classes, what really is neural code. So, say for example, I am seeing an apple and I can calling it as an apple or I am seeing an oranges, I am calling it an oranges. So, what essentially is happening is that my visual system is picking up those information, translating them into electrical signals and that electrical signal is coded in the brain as an apple or as an orange or as hot or cold or whatsoever, I mean depending on any kind of signal what you are getting. So, assume that we know all those signals and we could tune a robot with those signals, then a robot should not be able to execute that action. If there is a way we could interface those signals, so essentially that means is that suppose I am thinking something and I want to do this. If I know those signals and instead of me say for example, I am sitting here and so imagine like look at this. So, I want to you know lift this fine. Now, I am thinking that I have to lift this pen from here. So, this pen is sitting out here fine is could see this pen. I want to I am just thinking that I have to lift that pen. So, there is an electrical impulse which is going and suppose my arms are not functional. Now, I am thinking and I have this robot from my thinking will be transmitted to the robot and robot will exactly pick up the signal what I am thinking and I will do the job am I making sense. So, I want to pick up this pen. So, now my best way to pick up this pen is like this I pick up this pen. So, imagine that I this is a normal situation. So, what I did I have to do something. So, I needed that pen. So, there is a thought process I saw the pen I went there and pick up the pen and I start writing. Now, situation 2 my hands are not functional. I do not have my hands or some situation imagine my hands are not there. Instead there is a robot say for example, this mouse is the robot imagine this mouse is the robot. I am thinking and my thought process my brain waves are connected to this robot and then this robot will go like this. Just imagine it is not my hand suppose the robot is going robot will pick up the pen and move it is it making sense. So, this is essentially is brain machine interface where a brain can cross talk to the machine. So, the brain impulses could be configured in a machine and the machine can do that job. So, this class and this is the concluding lecture in terms of the admiral bioelectricity or from some of the insect bioelectricity I will be dealing later after the plans. In admiral city this is a one of the most beautiful success stories of last century where man has been able to translate the information of the brain to the machine. So, in other word the man has been successful if you read through this first slide what it says man has been successful in controlling the robot with mind and this is the story of a monkey called belly and I will be discussing this how that was feasible. So, before doing that let us see this slide where I was I repeatedly told you in several classes that the brain has different areas which are involved in different function. So, if you look at it you have the movement area out here you have the judgment area out here you have the sensation all the sensory inputs are coming out here. Then you have the hippocampus which is this seahorse like structure sitting out here which is involved in your memory acquisition and we have talked about what happened to those patients in whose case this hippocampus region is being is was removed. Then you have the reward area this is the area which ensures that if I perform well in the class I will get an award or you know if I do this I will the reward will be given to me this is one and so forth. Then you have the coordination area which ensures all the coordinated motion of your body all the different coordination visual coordination eye coordination hearing coordination likewise when so forth that you have the visual cortex which is in the back of the out here the complete visual fields and how to judge all those things. So, this whole structure what you see in front of you which is just part of it or a small fragment of it or small minuscule of the whole abilities of it work in coordination and they all work as I should say as a as a pool of cells bathed in a pool of electrolytic fluid where the electrical impulses are travelling in a certain pattern and decoding those pattern is the journey of the modern neural engineering or bioelectricity to understand those wonderful motion those fluxes and everything what ensures what we exactly see or what what we are observing the color perception the movement thought consciousness and everything. And this is a cartoon showing say for example, this is the reward area if we will go back so this is the reward area. So, this is what you are doing that you implanted electrode in the reward area. So, this is now we have slowly a trick into that experiment what I wanted to show you. So, what is this being done before I really get into the experiment. So, there these monkeys were trained in a way that if they do certain act they will get an reward say for example, if imagine this is a liver this pen what you see out here this pen this is a liver. So, I tell the monkey you know you hold this liver and you move it in one direction like this like this if you move it in this direction you will get some say juice and if you move it in the other direction you would not get a juice. So, or say for example, if it moves in this direction it will see a green light coming in front of it or if it moves in this direction it will it would not see that. So, this liver so I am training the monkey to move the liver in one direction. So, if it moves in this direction it will get a juice and if it moves in the other direction it would not get a juice or if it moves in this direction it will see another green light as well as a juice in the other direction it will see a red light it would not get a juice. So, if it has to get a juice or if it has to get a reward it has to move in this direction. So, these kind of training could be done when you are doing those kind of training if you look at this slide now. So, this reward area. So, your reward is very simple, if you move in a direction you will get a reward. So, the reward area gets activated, here the reward area is getting activated. So, you can pick up the signals from the reward area by putting an electrode there, what is the signal which is generated when you get a reward. So, now coming back into the circuit. So, this is the most primitive way or simplest way to put the circuit. So, this is the finger which is ensuring. So, look at my fingers. So, this is this is my finger which is ensuring that it moves in one direction. Sorry, this is my finger which is ensuring that this either moves like this or it moves like this. So, whenever I am doing so if I move like this. So, the signal now if you look at the slide the signal that the one in the pink color this signals are all travelling along this all the way to the brain. And in the brain in the reward area or hippocampus reward in all these areas there are a whole range of computation part of the signal goes all the way up into the sensation area and the movement area. So, this area the reward area sensation area movement area. Now, see the circuit sensation area movement area from here it moves to the movement area and the signal comes back. And now if you look at my finger now I can move my finger after this whole processing I can move my finger like this. So, this whole process this whole process of movement of the lever is involving if I had to get a reward if I move in this direction like this the lever is back now I put this and I bring the lever like this. So, in order to do that there are 3 different circuit now if you look at the slide. So, this is the level of complex circuit which is involved in it. So, there is a reward area there is a sensation area then is a movement area. And if I go back to this picture now and if I could collect all the signals what is happening. And if I could decipher these signals and then what I do once again that I translate those signal to a machine. So, this is what the next slide is about from theory to practice. In the early nineteen nine eighties Apostolos pre Georgeopolis of Johns Hopkins University recorded the electrical activity of the single motor cortex neuron in a macaque monkey. He found that the nerve cells typically reacted most strongly when a monkey moves its arm in a certain direction that is exactly what I was trying to tell you if it moves in a certain direction there is a very typically very high electrical activity. Now, if you read through read further yet when the arm moved at an angle away from a cells preferred direction the neurons activity did not cease it diminished in proportion to the cosine of that angle. The findings showed that motor neurons were broadly tuned to a range of motions and that the brain most likely relied on the collective activity of dispersed population of single neurons to generate a motor command. This is very important please read through this paragraph this is this is pretty much the basis of some of these successful experiment from here. So, this is what Georgeopolis experiment was being done. So, they were training. So, look at this the training this rat you know to move it in a certain direction and they were recording these are the electrodes which are implanted you see this. So, these are the electrodes in the f you see all these field potentials all these electrodes are being stored and depending on from which electrode it is coming this is the impulses and this is the raster plot these are the different raster plots which are getting generated out here. So, the overall field potential changes what are taking place during this whole process of its motion using the finger. So, essentially what the rat is doing is exactly the same thing what I am trying to do out here. So, I hold this and I move it in a once again I move it in a direction like I move it in a direction like this and now if you go back to the figure this is how it looks like. So, if somebody puts in my brain all those electrodes like this they should be able to trace the whole motor action which is taking place and if you go back to this picture you will see it is all in the sensation and the moment area where this whole processing is taking place which is essentially part of the motor cortex. Now, from here the experiment was designed that this was very interesting experiment which is done in Durham and. So, if you look at the American map. So, this is where North Carolina Durham Duke University and here you have Massachusetts the state of Massachusetts. So, this experiment was done. So, there was a monkey which was sitting here in North Carolina and this was controlling a robot which was sitting in Massachusetts and how this experiment was done this is what is all about the man machine interface one of the most successful stories of the last century belly. So, this was the monkey this is called an this is an owl monkey which was sitting in a special chair inside a sound proof chamber at our Duke University this is all taken from the discoverers website sound proof chamber at our Duke University laboratory which is essentially out here and North Carolina Chapel Hill or in at Durham sorry her right hand grasped a joystick as she watched a horizontal series of light on the display panel. So, imagine if you look at me now the way I am holding it. So, I am holding grasping a joystick as and I am seeing a lot of lights in front of me and this monkey knew. So, if you see the slide now this monkey knew that if a suddenly that if a if a light suddenly shown and she moved the joystick left or right to correspond to its position a dispenser would send a drop of fruit juice into her mouth. She loved to play this game and she was good at it. So, look at it how they have trained the monkey. So, basically what then this monkey knew that if suddenly if light suddenly shown and she moved the joystick left or right to correspond to its position. So, imagine now if I will look at me. So, here the light is moving. So, this knows if certain light is shown like that in a if it could really you know follow that light in a way then this monkey is going to get a get the fruit juice it is pretty much like that this monkey this belly was trained in a way that it could you know follow the light likewise and it will get a reward. Now, talking about the reward if you will go back here this is this is the area which will get activated and this activation is the function of these two situations. There is a sensation because it is looking at those lights moving the eyes like this is moving the eyes like this. So, there is a sensation there is a movement and there is a reward situation coming back to the. So, so this was what belly was trained with coming to. So, this is where belly is 600 mile reach it means 600 miles away. So, this is essentially wanted to tell you from North Carolina all the way to Massachusetts this is what they are trying to highlight. So, on the day belly first move multi jointed robot arm with her thoughts as I was telling in the beginning with your thought she wore a cap glued in her head. This is the cap which is glued in her head it is filled with all the electrodes out there beneath the cap each of the four plastic connectors fed an array of fine micro wires into her cortex. So, these are where the cortex if they are connected to the cortex as belly saw lights shine suddenly and decided to move a joystick left or right to correspond to them. The micro wires detected electrical signal produced by activated neurons in her cortex and relate the signals to a hard way box of electronics. So, this is belly in the laboratory room in Durham North Carolina here is the cap and here is the implanted micro electrode this is just underneath how the micro electrodes are implanted these are the micro electrodes. And this is the joystick which belly is holding and playing depending on the which light it is showing depending on the light it tries to correspond along with the light if the light is shown in the left it moves to the left which is on the right and moves to the right and it loves to play this game because it knows that if it does. So, it will get a reward get a reward for that the box and then you have the hard way box now I am coming to that hard way box of electronics. So, this before I proceed further please try to read through this scientific American exclusive online issue of May 2004 which has talked about this particular story of belly. The box collected filtered and amplified the signals the hard way box what we have talked about in the previous slide out here I am continuing from here electronics hard way box of electronics. So, the box collected filtered and amplified the signals and relate them to a server computer in a room next door the signals received by the box can be displayed as a raster plot. So, this is where the raster plot is sitting. So, this is the similar raster plot as I was showing you here this is the raster plot which is sitting out here which is basically showing essentially this raster plot is doing it is telling from which region of the cortex these signals are coming. So, it is basically a field potential plot it is learning this area is active this area is active or which electrode from which electrode the signals are passing through you can really draw the whole map looking at the complete raster plot coming back to that. So, each row of the raster plot represent the activity of a single neuron recorded over time and each color bar indicates that the neuron was firing at a given moment. So, basically you are getting a spatio temporal plot using a raster plot because spatio because at which from which neuron in the space this is coming temporal means with respect to the time you are getting it. So, it is a spatio temporal computation which a raster plot gives you at what time this is active. So, this neuron is active at this time this neuron is active at this time this neuron is active at this time and based on the plots you can say from neuron 1 to neuron 2 to neuron 3 to neuron 4 likewise this signal is moving. So, that is what the raster plot does raster plot gives you an overall spatio temporal changes in the signal at a given point of time depending on where all the electrodes are being placed. So, maximum overall electrodes the complex the raster plot become and more intense signal you started getting. So, coming back to the slide again. So, each row of raster plot represent the activity of single neuron recorded over time and each color bar indicates that the neuron was firing at a given moment. The computer in turn predicted the trajectory that bellies arm would take and converted that information this is very important and converted that commands for producing the same motion in a robotic arm this is the key out here. So, these signals the computer in turn predicted these signals are fed to the computer the computer in turn predicted the trajectory of bellies arm would take and converted that information into commands for producing the same motion in a robotic arm. Then the computer send commands to a computer that operated a robot arm in a room across the hall at the same time at the same time it send commands from our laboratory in dirham north Carolina to another robot in a laboratory 100 miles away in response both the robot arms which was the one robot which is sitting on the other side of this room the other robot which is sitting 600 miles away in Massachusetts both of them at the same time they respond. So, if you read through this in response both robot arm moves in synchrony with bellies moon limb. So, essentially now bellies thought process from the brain. So, belly is doing this act think of it. So, this is what belly is doing. So, you look at my arms now. So, so this is what belly is doing, but from the signal generated from the head of belly you could make the robot do that action for it. So, say for example, I want the robot should give you a glass of water I just think the robot will do that job because my brain will be conveying the message to the robot sitting in by your side to tell you give that glass of water to the robot think of it this is something amazing. If you really become intense and think over this whole situation you are telling the robot. So, essentially nurse has to think of in a medical situation a nurse has suppose there is a nurse and there are 20 patient who has to work. So, nurse will not go to the individual patient nurse will think and the patient will get the medicine. So, this is what man machine interface is all about that where you are thinking your process the robot is executing that task. So, this is something amazingly interesting and years and years and centuries to come mankind will be keep on trying you know to integrate it is thought processes with a robot because that is amazing I mean if you really could achieve that kind of feed then a lot of things what we can we can only thing we cannot really do could be achieved in no time. So, now going through this going through this whole layout of this different thing. So, this is where the computer on the left and the robotic arm this is the robotic arm which is sitting across the hall from belly and this is the other robot in laboratory in Cambridge Massachusetts where the second robot is sitting interface with the computer. So, the signal and this is where the belly is sitting the hardware box from me from the either net this will send to the computer from here the computer split the message or rather duplicate the message and split it in two parts and send one power send the copy of the message to Massachusetts and the send another copy of the message through the wires to the through the to the other room and the end result is both the robots where and this is what you are seeing that computer is translating translating the signal in terms of the three dimensional movement of what belly has done and this is belly raster plot. It was basically belly's raster plot which is generated from here from these from these wonderful electrodes this raster plot is generated this raster plot is sent to this hardware box where sent to the computer where it basically deconvillated the whole signal and send a copy of the signal to Massachusetts via internet whereas the other one was sent to the other room through where and both the robots at the same time while belly was drinking the juice or you know getting the juice belly's brain was making two robots to function in synchrony. So, if you think of this whole situation this was a landmark discovery of its time to. So, this is what if you have to really open up this problem. So, BMI stands for the brain machine interface. So, BMI multi multiple feedback loops being developed at the Duke university center of neural engineering a recess macaque is operating an artificial robotic manipulator that reaches and grasp different objects the manipulator is equipped with touch proximity and position sensors. The signals from the sensors are delivered to the control computer which processes them and converts to micro stimulation pulses delivered to the sensory areas of the brain of the of the monkey which I have already shown you to provide it with the feedback information which is the red loop you could see a series of micro stimulation pulses illustrated in the inset on the left. So, this is those micro stimulation pulses what you see on the left the neural activity is recorded in the multiple brain areas and translated to command to the actuators via the control computer and multiple decoding algorithms in the blue loop this is the blue loop which is involved in it arm position is monitored using an optical tracking system that tracks the position of several markers mounted on the arm or the green green loop. So, we hypothesize. So, this is the hypothesis is that continuous operation of this of this interface would lead to the incorporation of the external actuator into the representation of the body in the brain. So, basically you are decoding the neural code for that action if you know that neural code you can actually use that neural code to do some actuation using a robot. So, there is almost a imaging of the neural code which they are trying to do at this point. So, from here I will move on to the drab side of it now this part of the game all the different kind of electrodes which are involved in it this is very essential for you to understand those electrodes if you know this is how the those electrodes look like. And please make a note of the scale this is very important make a note of the scale these are the different configuration of the electrodes are very sharp electrodes which gets into the brain and implanted in the brain as of now I have always talked about electrode I have drawn, but this is physically how most of this electrode looks like. So, looking at the slide so, look at it I mean there was almost a array of 1 2 3 4 5 6 7 8 9 10 almost I think 10 by 10 array which is of electrodes sitting out there. So, then you have all the connectors shown by the arrow out here. So, these are this is a summary of the brain machine interface if you look at it. So, there is either you could have invasive technique or an in non-invasive techniques if the invasive techniques of the electrodes are implanted intracranially. So, basically you are pushing the electrode through the brain using surgical techniques you could have single recording site or you could have multiple recording site depending on how many electrodes you really can push into the brain. Then you have in the single electrode single recording site you will have a small samples whereas, you have the local field potential or the LFPs you could see that we could follow the code underneath the abbreviation the BMI the brain machine interface the e g is the electroencephalogram LFP is the local field potential M 1 is the primary motor cortex P p is the posterior parietal cortex. So, you have this LFPs where the brain machine interface based on decoding the LFPs suffer. So, there is a always there is another problem I will come to that then they have the large in on shambles and then you have the complete non-invasive process of e e g's electroencephalograms about which we have talked about the different kind of waves which are involved in it alpha wave gamma waves that is where we are decoding those informations from the brain using the surface electrode or the electroencephalograms. What I wanted to highlight here is this. So, from the beginning of this course I am repeatedly highlighting this part see this is a big the biggest central problem apart from any other problem is this cell electrode interface and if you read through this line very carefully BMI is based on decoding LFPs suffer less from bi compatibility issues. So, there is always a bi compatibility issue with the electrodes involved and this is very very important. So, the large on shambles large in on shambles hundreds and in future feasibly thousands of cell provide a stable signal to control multi degree freedom processes this approach instigates new computational solutions. So, if you look at this whole thing this whole brain machine interface this revolves around revolves around the fact that how good you are in terms of the electrodes as well as how good you are in terms of you know getting the electrode material which will ensure that there is maximum bi compatibility and long term signal harnessing ability these electrodes should have. So, coming to some of the vision of the future. So, these are some of the vision of the future of the man machine interface a brain machine interface might someday help a patient whose limbs have been paralyzed by a spine injury tiny arrays of micro wires implanted in the multiple motor cortex of the brain would be where to a neuronal chip in the skull as the person imagined her paralyzed arms moving in a particular way such a reaching reaching out food on a table the chip would convert the thoughts into train of radio frequencies. So, this is very important the chip would convert the thoughts. So, I am just thinking that I want to have a food into a train of radio frequencies signal and send them wirelessly to a small battery operated backpack computer hanging from the chair. So, this is where the backpack is handy sitting here and then this is the neuronal chip the computer would convert the signals into motor commands and dispatch them again wirelessly to a different chip implanted in the person's arms. So, this chip is implanted out here. So, basically what is happening in this situation the signal from here cannot travel because of injury in the spinal cord injury side when neuronal commands die. So, in this situation this person is thinking and this thinking process from here through this is transferred to the to the arm now the second chip would stimulate the nerves needed to move the arm muscle in the desired fashion alternatively the backpack computer could control the wheelchair motor and staring directly as the person envisioned where he or she wanted the chair to roll or the computer could send signals to the robotic arms if a natural arms are missing or to a robot arm mounted on a chair. So, this is done by Patrick Wolfe in the university has built a prototype neuronal chip and a backpack as envisioned here. So, this is basically those kind of chips what we are talking about this is some of the futuristic ideas. This is another futuristic idea where fully implantable multi channel recording devices are being put and there are touch and position sensors sitting there mechanical actuators with both power and accuracy and wireless link by which a person can really do certain things which he or she cannot do because of certain injury. So, coming back and summarizing this whole process. So, here you have the machine and here the brain and the percept and the intent and the commands and actions and the stimulus. So, this is the physical interface and this is the coding interface and this is the neuronal interface and the organization of the brain machine interface in the output BMI a neural interface detects the neural coded intent which is processed and decoded into a moment command. The command drives a physical device a computer or a body part of paralyzed limb. So, that the intent becomes an action for input a stimulus is detected by a physical device coded into the appropriate signal and then delivered by its interface to the user to elicit a percept such as touch or vision. The use of these inputs and outputs is determined by the individual through the voluntary interplay between percept and the desired action. So, essentially what this means is you are thinking something those signals are being picked up and we put those signals to a robot and the robot execute the task. This is essentially all about the brain machine interface and I have given you sufficient number of references please go through them and this is where we will close on the human or the animal by electricity and it is a long journey of mankind in future. Those of you get inspired please pursue these kind of fields because these are something very amazing areas where there are lot lot lot more to be done. We are nowhere even close to anything these are discrete one or two examples here and there, but will this be a reality someday the future or the success everything lies on the young brains who will take up these kind of challenging problems and will make a difference for the society. Thank you.