 Well, today is going to be a nice chapter, chapter 16 of my book. It's on the web. Once again, I would appreciate receiving any feedback. I've not really been inundated. I've sort of received three typos. I assume in this 250-page book there's more things that are wrong than just three typos. I mean, I could, of course, be perfect and write perfect prose, but the experience shows it's unlikely to have happened. Now, in particular today, this thing you're going to talk, you'll hear today, is going to be critical and important for your homework, the one that's, I guess, you next Monday, I mean Monday in 10 days from now, because it'll be about binocular rivalry, which is one big part of what we're talking about today. So you should refer to today's lecture notes and also this is one additional review paper by Logotitas and Blake that's on the website that you'll need. You'll also need stereo color glasses that are here. So pick up one. Please don't take more. We don't have a lot. And can you please return them once you're done? They are because those are the only ones we have once you're done with the homework. So the D is a red-green. Now, they're not perfect. The best way to do binocular rivalry is to get two separate images, totally complete images, one on the left and one on the right eye, but unless you have mirror setup at home, then it's just to have green and red images and green and red glasses. But of course, those things are not perfect. The color spectral resolution is not very good, so some images will leak across to certain extent to both eyes. But it's enough to, I mean, you should be able to see the basic phenomenon. There you go. Okay, so we'll talk today about what I call perceptual stimuli. There are no commonly accepted words, so I just decided to call these perceptual stimuli. And perceptual stimuli put to lie the basic notion that most of us have about the relationship between the outer world and the inner world. The idea that we have is there's an outer world out there. There's some physical stimuli, some event that happens. The photon gets generated or a sound wave or a single molecule of odorant floats around. And your sensory apparatus picks that up and translates that into an unambiguous percept. And this is one-to-one objective relationship. There's one stimulus, there's one percept, there's one percept, there's one... So every stimulus has one percept that you can uniquely assign it to, and for every percept you can assign a unique stimulus. That's sort of what most of us, whether we think about it or not, certainly if we don't think about it, that's what we assume implicitly. Now, you've seen already many illusions where that's not the case. Here we're just going to study them as a whole because they give rise to some of the best experiments that track the footprints of consciousness as sort of consciousness steps around in the brain. And so in all these cases, all these perceptual stimuli, we can dissociate, we can manipulate the relationship between what's out there and what's in your head, between physical, objective reality and subjective percept. Phenomenal content, as philosophers would say. So these are different illusions. So some of these illusions are binocular ivories. I mentioned they're part of the homework by stable percepts. I'll show you, I mean, we'll show you an example of these. Here you have the case where you have a sustained input. You have a constant input very often, yet you can see two different things. So here you have the case where you have one physical input that gives rise to two different percepts. And when you see one and when you see other, it depends on all sorts of factors that we'll study. These factors depend, for example, what did you see before? Flash oppression is another case where there's one input, but you see two different percepts. Most new blindness, I'll show you that again, we saw them in the very first lecture. Again, you have, well, not a constant input, but you always have the same sort of input there. You have the stationary yellow spots, and then you have the moving clouds. Sometimes you see one thing, sometimes you see the other. So all these stimuli, all these illusions, if you want, these perceptual illusions are characterized by a nonunique, by a very often absence of a one-to-one relationship. That this one-to-one relationship that we think holds true for most things in life, certainly in this case, does not hold. Now this allows you to dissociate the input into the system from the percept, and so now you can study systematically in the brain, you can study where the neurons, where the part, where the neural mechanism that respond to the input, the physical stimulus, and where the neural mechanism that correspond to or that manipulate the subjective percepts. Normally these are very, very, you can't associate them, right? And then normally when you've got a physical input that always gives rise to the same percept all the time, then it's very difficult to understand where as you move from the periphery in the photoreceptor to your high-level neurons that generate your consciousness, where are the neurons that are more responsible for coding the input and where are the neurons that code actually for the percept? In these class of stimuli you'll see we can begin to do that because you can dissociate the input from the, the physical input from the phenomenal percept. The classical one is called a NECA cube, although it's not the oldest one, this is called a NECA cube and E-C-K-E-R because the person who first described in the literature was a Swiss mineralogist who was studying minerals and he looked at this crystal and then he saw, he could see it and he talked about it, probably people see, you know, the Greeks assume, seen this already, but it was first noticed by him and so it's called the NECA cube. You should all be able to see this cube in one of two different orientations, right? Now, of course a very interesting question is what triggers the transition? You know, if you just look at it, suddenly this one for me is very, is very labile. It sort of flips back and forth. For me it's very easy to get the transition. Sometimes you associate it with eye movements but not always, I don't know what feelings you have. Can you sort of without trying to move your eyes? Of course it's difficult because you might make small eye movements that you might not notice. You have the feeling that you can flip between the two without moving your eyes. It's a controversial question going, there's somebody who wrote a large monograph on a Levelle, the Dutch guy, on to what extent this is an evolutionary control and probably that's very difficult because you've got a separate evolutionary control that you might exert through small eye movements for instance through blinks or something like that or very small eye movements that might induce changes from, in the absence of any movements can you trigger change and that's very, very difficult to get at scientifically. Okay, then here, artists have made use of these for a long time. I think some of the earliest one was in a Roman villa in the city that was destroyed and what was it, 72 after Christ next to Pompeii that was destroyed due to the volcano. Some of these have been seen already there. But essentially most of you should see, for example this, you can see as a vase or you can see a silhouette, this is a nose, eyebrow, in fact this is the silhouette of Francis Crick. His eyebrow is forehead, right, you can see the head. Here this is a picture from Dali, well it's from a painting, but then in a black and white. So here it's a scale one. If you really look at the detail, you can see the nanny, I think it's supposed to be his nanny and the little boy here and the painting this is supposed to be ship, this is supposed to be here. If you just blur your eyes, what do you see? It's very obvious. You can see a face, right? So this is not quite the same because here you actually have to sort of blur your eyes. I find it different. What's the difference about the nanny? You first saw the face there. Now I can see it in both. Well okay, so this one is called the Schrödinger's staircase. You can see it in one of two orientations. Although there's probably a bias seeing it as a normal staircase going up, because that's what we expect to, but you can also see it more difficult. You can also see it flip the other way and ceiling. Does anybody see it any other way? Yeah, sort of it's inverted, right? It's like this or it's like this? It's more difficult. I think for two reasons, A, because staircases that we've seen normally, we don't see staircase coming down from the ceiling so there's a powerful bias at work. We always seem to assume that the light source comes from above and so that bias is the thing. This one took me a long, long time to see. There's an old woman and a young woman and you can see it? It took me a long, long, many, many minutes staring at it to see it and now I can sort of flip back and forth. This is the same, this is an old man and a young man. I don't really see the young man in here. This is a modern version because it was, you know, for gender equality reasons. This is a good one. And here let's see, these are ambiguous because you're supposed to see them either as sort of oriented in this way, for example this cloud. This has to do with what psychologists call grouping. Grouping principles. These have been studied a lot by psychologists in the 10s and 20s. They studied these sort of things. You can see it grouped in this direction or you can see it grouped in this direction along here. You can see this grouped in this direction and sometimes less weak you can see grouped in this direction. Here I find this stronger. Well, these things you can either sort of see as flock of geese moving horizontally in this direction where these point in the direction or you can also see them as moving downward here. Yeah, that's exactly. So the point here is in all of these cases nothing changed over the last two minutes on the input, right? Yet you can see different things. Now, interesting is the question can you perceive two of these percept at once? Can you at once, and of course exactly what does it mean at once? You have to define some sort of scale etc. Can you see for example the silhouette of the two people and the vase at once? Or can you see the face here and you see the old woman and the young lady at once? Very small. Okay, so the question is so in this case I find it difficult to answer because in the Neckar cube I find it easier and I mean just for the record whatever you perceive is I'm not going to argue with your perception if the claim is that you cannot see them at once that really if you can see any this form or this form you never see them at once and you never see a superposition in between. You don't see this and then sort of a state that's halfway sort of in between. You only see this form and this form and you can never see this at the same time both. Okay, that's a very good point. Yes, sometimes you can get incomplete exactly where you see sort of this is in one orientation that's in the different direction. Yes, so there essentially you don't see this a single object, you see this two separate object. Yes, now I don't have a picture well I have it actually somewhere in my laptop but not in this lecture. For example, the one with all the monks that go up this tower and they all go upwards you know they only go up which of course we know can't be and so again that the solution breaks if you can look locally at it but I mean globally if you would put it together into smooth into continuous percept it would be this impartial percept of a staircase always going up. Yes, so that's a two observation but the claim is that if you have a single at any given point so if you just look at here or just look at this, you cannot seem in two orientation at once. Now, okay this is the same well it's a similar version but it's a dynamic, it's a dynamic by stable percept so these are all called by stable percepts and you know there are dozens of books, it's a huge website illusionswork.com where there are hundreds of these illusions you can buy lots of books in art shops etc that have this is a dynamic one did I show this already? Lights so this takes a longer one, this has a longer transition what do you see? which direction is it rotating? Okay, so A of course you can focus on any one part but then you don't really get the perfect this is called structure for motion so you can extract out of you know just two dimensional on the screen but you can extract motions called structure for motion we'll talk about next year in vision class but the way it's constructed it's perfectly ambiguous so you can either see it move or roll away from you and the point is you can only see it in one this I think unless you see sort of one thing here and one thing there if you just look at the whole thing you can only see it in one direction and this at least for me has a long transition period I can see this move for many many seconds in one direction this doesn't flip very fast it seems much slow at least for me I don't know how it is for you than the NECAQ for instance this is much more okay so you have the same feeling okay and then just to remind you this is the newest member of this family this is the most compelling one because it's not the same it's not by stable because you always see the you always see the clouds but sometimes you can see the dots and sometimes not and I find this the most compelling because if you learn to avoid eye movements sometimes these dots disappear particularly the upper one there's an asymmetry between the upper part of the visual field and the lower part they're gone like for many many seconds it helps to have a fixation point do you all get that okay so now some of these are obviously more accessible to scientific manipulation than other ones so all the by stable ones the NECAQ etc it's unclear really what to vary people have done some studies no animal studies I know they're very difficult to manipulate in a parametric way so it means they're much less accessible sort of to psychophysical and electrophysiological exploration the paradigm that's so far been the most successful is binocular rivalry so you can experience this yourself what you should do it's actually shown here so you should all do this right now so you should all take a piece of paper because it's very striking it's not the same but it sort of it gets at it like this so with one I put it in front of one eye and then I mean rest the this piece of this tube inside the knock of your hand here like this and put it in your right eye if you do this with your left hand and look with both eyes and you'll see a huge hole in your hand yeah so otherwise you have to do okay so so I'm giving you the instructions for right hand most of you I am right like most of you are right right eye dominant so if you put it into your right eye and hold it like this with your left eye okay then there's going to be a big hole in your left hand now of course if you are dominant in the other eye then you have to reverse it it works in fact it works even better in the non-dominant one whoa there's a big huge hole particularly it works extremely well against a bright dark background against a dark background it works much less well because then the other the hand dominates but against a bright background it's a huge hole in your head hand not your head you all see that it's quite compelling isn't it a bigger one yeah but in Maya it's certainly strong on the other one okay so this is sort of it relates to binocular ivory this phenomena so binocular ivory is the following the best way to do it is to do it with mirrors but as I told you in your homework you'll do it with red green anaglyph glasses the principle is the same so essentially you put one image into one eye and a different image into the other eye and you separate them best with a divider and using mirrors but you can also do it with these glasses now of course in real life the images you see in your left and your right eye are never exactly identical right you can you can just check that this image and this image is sure similar you know the angles are slightly different and you use that difference between the left and the right image to get depth that's called binocular disparity and you use it most of the time to get a measure of distance certainly within a couple of meters but now when you have images that are quite different that usually you don't get in a normal circumstance except if for example you do this where now your left eye sees the hand while the right eye sees something totally different under those conditions then what does the brain do so in other words if you put two images into the two eyes in corresponding locations well in principle you could imagine that the brain does some sort of superposition you get some sort of amalgamation you get the left image superimposed on the right image and sometimes you can see that actually but in general certainly if you do your experiment correctly what you see is that the brain doesn't like that there are these two input now and they rival for perceptual dominance they compete against each other and your brain resolves it and it does it independently of you you can sort of be the observer and you can observe yourself seeing different things the competition is resolved in favor of one object for 3, 4, 5, 6 seconds and then you get a transition and then you see the other object so for example if I put horizontal gradings into my right eye and vertical gradings into my left eye I don't, I mean for very brief time I might see a checkerboard sort of superposition of the two but typically what you'll see is for one or two it'll take some time to set in and then you'll just see the horizontal gradings and then you might see this for 3, 4, 5 seconds and then something happens you move your eyes or something changes in your brain and you see a transition period where you might see what's called piece me rivalry where up here it might still be horizontal but down here it's already vertical sometimes you can see sort of a wave moving a wave that moves across your visual field that in front of the wave it's the old percept, the horizontal and behind the wave it's a new percept, the vertical and then after this is over you only see the vertical ones and then it flips again 3, 4 seconds sometime later it flips back and goes to this and this is indefinite, it goes from one to the other with these transition periods and then we ask you to plot this, in fact we'll ask you to plot the distribution times of those because the bottom line is that the individual duration of this is stochastic, it's a stochastic variable so once it might be 3 seconds the next time it might be 5 seconds and they're also independent of each other so it's an independent stochastic process but there's a great deal of regularity in this stochastic process now the great thing about this like all the other perceptual stimuli you can use this now to track the footprints of consciousness because you can ask ok so in your the right eye only ever sees the horizontal stimuli ok so clearly in the left eye only sees the vertical so clearly in the right eye because I know the direct connection between the eyes and optically they're isolated and the right eye I'm only going to see ever neurons that respond to this great thing ok so I can ask now well if I go to higher stages like LGN or V1 at what stage will I see neurons that begin to respond to the to the stimulus that I actually don't see ok because at the high stages wherever my NCC is I'm going to only see neurons that fire or at least the simplest assumption is that when your NCC is I'm only going to see neurons that say if I currently see this well only neurons that correspond to this will be on and the neurons you know and the NCC does not have access to the neurons that represent this because I don't see that right so I can I can ask a question where are the neuron populations that switch with my percept right because those are potential candidates for being the ones that actually give rise to this percept that not only correlate with the conscious percept but that give rise to the conscious percept this has been enormous productive research program still ongoing very much ongoing it started in fact here 12 13-14 years ago was John Orman who was the first person who tried to do this in a monkey the I mean the psychophysics goes back to at least Helmholtz but the idea of doing it monkey and then recording is a relative new one and the person has really done more than anybody else to advance this is Nikos Logotitas and David Leopold who are currently interviewing and considering make an offer here to do to come here to Caltech okay so let me tell you about this studies this is a sort of classic studies now although they are only sort of less than 10 years old so they recorded this is the monkey brain primary visual cortex here the front of the brain and remember the the ventral pathway that originates in V1 then moves down here to the infrared temporal cortex and so this correspond in humans would be the fusiform gyrus and here you along here you find neurons that respond ever more selective to things like faces and colors and things like that and they record in the most forward part of this part of the brain anterior-medal temporal sulcus the most forward part of the infrared temporal cortex because they know them we know from experience from lots of other studies here you find neurons that are very selective to individual things like they might fire a face or they might fire to a specific face or they fire to basketball or they fire to rather weird things that is difficult to predict ahead of time so this is the let me tell you about the X-bomb the X-bombs are a little bit tricky partly they're tricky for a conceptual reason they're tricky because if it's true that you see if I just show you this if you see, assuming you're not lying and I'm assuming the monkeys are not lying you all see what I'm seeing a laser pointer in other words I know what you're going to tell me ahead of time now in binocular rivalry that's not the case because there are two things let's see in the simple experiments the horizontal and vertical and I don't know what you're actually seeing I know your input is this and this but whether you're seeing this or this I have no way of knowing from the outside it happens in the privacy of your own head and only you know however, as I mentioned there are phenomenal laws for example I know that if I do this many times there's some statistical distribution I know that they're statistically independent I know what happens if I make the image in one eye brighter I know how these distributions will shift I can cheat sometimes and instead of putting these images on I put actually identical images on left and right if I do this carefully you won't know I mean if I do this carefully even in humans you won't know because if I now put identical in left and right I know exactly how you're supposed to answer because then you can only report a horizontal because the horizontal is the only thing that's present so in a monkey of course the situation is much more difficult because the monkey can't directly tell me that he doesn't talk so I have to train the monkey but the monkeys are just like humans they are very very clever and very adapt and they're lazy fundamentally and they'll use any and all tricks they can to get by to get their own shoes or fruit juice every physiologist can tell you stories how they got fooled by the monkey how they thought the monkey did some experiment but actually the monkey for example every time this one stimmers on somehow there was a mechanics and the mechanics generated a noise and so it turns out the monkey only responded to the noise because it was much easier or things like that so you have to be very very careful but the point is you can train monkeys just as you can do typical subjects in these experiments and then not too surprising to biologists the monkeys behaved very similar to the and to the undergraduate subjects in other words they have the same distribution and they have it's more or less the same visual system as I've been in pain to stress the visual system of humans is similar to the visual system of monkeys not identical but very similar and now they train them to do the following they train them, they put them in a chair and the monkey could push one of two buttons one of two levers so for example they train them every time when it saw the sunburst pattern to press the right lever and every time it only saw the butterfly to push the left lever and when it saw both for example in optical super position of both where you know where logatitas just put them both together or when you know you cut out half of one and half of the other and put them together the monkey was trained not to push anything so the monkey was trained only to respond only saw the sunburst pattern to respond pull the lever and only when it saw the butterfly ok so that's like a human telling you you know yes I see the butterfly I see the sunburst pattern so now here you have sort of what is this, this is 25 seconds here so this is one cell, this is a different cell when it's gray the monkey is actually experienced by natural rivalry so here in his left eye is the sunburst pattern the right eye is the butterfly super position of both no sorry, here it's just it's just the butterfly here's the super position of the butterfly and the sunburst pattern so here again these are these trick these catch trials that logatitas puts in to make sure the monkey isn't cheating because here you know the monkey has to pull the lever for the butterfly because only the butterfly is present here the monkey has been trained not to push anything because it's a super position of both you know what the monkey is going to do because you truly cannot look inside his head so the upshot of this particular cell is that every time let me see there's a strong correlation between so this is one particular trial this is average here and this is the behavior of the monkey so right was sorry he can see right is when the monkey sees the butterfly so here every time there's a strong correlation between the strength of the cell firing and the fact that he pushes the right button the right lever indicating he sees the butterfly so here strong cell firing the cell and the monkey tells you I'm seeing the butterfly here the cell stops firing more or less not totally but more or less and the monkey tells you I'm seeing the sunburst pattern here it starts up again and here of course because there's only the butterfly present the monkey not unexpectedly pulls the lever but again there's this tight relationship between the single cell and the behavior of the single cell and the behavior of the animal and you can make this mathematically accurate you can do for some information-threatening calculus you can put into this standard information-threatening machinery and predict with what probability am I going to predict that the monkey is going to say left lever or left lever here's another case a different cell also in infotemporal cortex here's just the sunburst pattern here's the superposition of sunburst pattern and young orangutan face here's true binocular ivory here again it's just you know there's only the orangutan face up or only the sunburst pattern face and here the relationship is between I guess the face yeah it's between the face and the okay so here every time the monkey pulls the right lever which again is the face left lever is always the sunburst pattern the monkey the cell responded so there's a correlation so here the cell doesn't fire very strongly there's always some background firing going on remember this is half a minute there's some background firing going on and the monkey tells you I'm seeing the sunburst pattern there's a very strong firing going on and then the monkey tells you it's seeing the face and here again the cell fires not completely but intermittently and the cell and the monkey tells you it's seeing that face now you can see it's not perfect this is not a machine these are neurons and nobody's claiming that the individual neurons that exactly mimic behavior so you can see these facts here for some reason nobody knows sort of stop firing and here of course it fires a little bit all the time but nonetheless there's a very very strong relationship from over 90% of the cells you can read out with very high fidelity the behavior of the animal you can predict the behavior of the animal by looking at these cells here of course these cells are very far away from motor cells because you could say trivially well if I record from a motor neuron that controls my right the monkey's right arm if I record from another neuron that controls the left arm then of course trivially I can predict the behavior of the animal but these are very far away in the purely visual part of the brain you can do control experiment to show that there's nothing to do with the sort of the motor behavior by itself so here the claim is really you're tapping into percept you're tapping into the monkey's percept some of these so 90% of the cells correlate so of course these cells are likely to do a plurality of different jobs you know, rivalry is a complicated phenomenon for example there's always every time you're in a rivalry you see one percept there's always some time later you see the other percepts so some neurons have to be involved in doing that if you look at the different neurons you get a catalog, for example here they fire quite regularly some neurons fire at 5-6 hertz cedarism, you can see it very dominant some respond very sluggish some respond very sustained there is zoo of different responses and right now we have no idea what the function of all these neurons is here's another case, this is a related phenomena, closely related I'll be not identical, called flash suppression so in flash suppression you do the following you put one image into one eye let's see the laser pointer is into my right eye and you look at it for a second and then you put a second image let's see this elastic band this elastic band into my left eye ok so this is on you flash this on my right eye for a second I see it, then you flash on this into my left eye if you do things properly this new input will perceptually suppress the old one so this will just be gone from sight you can see it here you put this left eye, nothing in the left eye in the right eye is the face of the young orangutan you flash on a 0 millisecond this one and the monkey will always see this one this is very reliable advantage over by not a rivalry experimentally it's much more controllable in a rivalry you've got this internal thing that triggers this transition it's internal to your head, it's very difficult to exactly control it here it's always controlled by the external stimuli whenever you flash on this other stimulus, this new stimulus for short time half a second or a second will perceptually suppress this very reliable to look at the behavior of this cell again is close to the infotemporal cortex superior temporal sulcus this is a neuron that happens to like for whatever reason this picture of this young orangutan it flies strongly to it in a sustained manner you flash on this new image and then 70 milliseconds later boom, the cell goes to 0 we're not talking about 10% modulation here the cell just totally goes to 0 shuts off although its preferred stimulus is still present and here's the opposite you do exactly the opposite here is that stimulus that over here suppressed it but by itself the neuron doesn't respond to that stimulus then you add this new one this new stimulus suppresses that one and the monkey sees this and the cell 60, 70, 80 milliseconds follows very reliable so the point is that this physical setup and this physical setup is exactly the same in both cases in the left input IE of this and the right IE of this yet here the cell doesn't fire at all totally zero here the cell flies very strongly and of course here the monkey doesn't have any visual sensation of the orangutan while here the monkey sees the orangutan at least that's what we believe from its behavior nobody asked the monkey directly so this is really a very, very nice example and again the majority of cells in this part of the brain respond like this they follow the percept and they do not fire to the perceptual suppressed stimuli yeah this is one of the same cell it's not the same cell as the cell before I mean it's not the same this is one cell, this is the one cell and that's the third cell this is actually rivalry yeah it's the same cell oh yeah otherwise it so this is really to my mind some of the best evidence that the best sort of scientific evidence we have right now for neurons that could be quite close to the neural correlate of consciousness there are probably many other cells here involved in switching and expressing a memory and all sorts of other things so it's probably likely to be a subset of these neurons now of course all of this is correlation all of this is correlation and ultimately you want in neuroscience we have to to go away from being a mere observational science to to something when we can talk about causation there are many sciences when we can't do that like in astrophysics we can't go from correlation to causation particularly we can't cause stars to disappear to appear or supernova to happen at least not yet but there at least we have very quantitative theories that can sort of predict with high degree of accuracy the evolution of stars and where they are along the Hertz-Pong-Grassel diagram and all of those things now of course we are very very very far away from that in neurobiology so at least what we want if we can't do this prediction at least we want to be able to go to causation now that's not impossible for instance there's a type of operation that people do since Oswald not Spengler but something like this in Germany did this first and then in this country most popular is Penfield actually in Canada in Montreal so what you do doing certain types of brain operations you go and and take out a part of a brain because it gives rise to epileptic seizure because of a tumor and then very often because you want to be sure you're not taking out let's say language cortex you're not taking out motor cortex what you do you stimulate the brain and Penfield did this and developed this in art form and did this for a couple of thousand patients literally and we know already from the Bill Newsom experiments of course for example if you have this balance situation remember in Newsom you had this cloud of dots that could move in one of the other direction and you inject current and you have columns in columns in MT and you have only code for motion in one direction and next to it you have new ones that only code for motion in the other direction so now if the same exist in infotemporal cortex and there's some evidence for that let's say there's clusters for faces and there's some good evidence for that so let's say over here you have a whole bunch of new ones that code for faces and then somewhere else there's a whole bunch of new ones that code let's say for more abstract forms like these sunburst patterns what you could imagine that you have a situation like this actually it's totally symmetric or in rivalry where you can see either one or the other and now if I inject current into the patch of neurons that all code for faces maybe I can get enough of these neurons for example if I can get enough of this neuron in its bodies I might be able to systematically shift the percept of the animal and that would be a first decisive step to go in a way to moving towards causation not just this correlates but actually it doesn't only correlate it's actually if I inject current I can write in a percept so it's a bit like matrix so matrix reloaded in fact I'm on this joint grant with a DARPA grant with people at MIT Jim DeCala where we're trying to do something like that in a much simpler case of optic recognition in monkey IT but the idea is that we not only read out but we write in, this is all read out but in principle you should be able to write in and as I mentioned you from clinical experiments we know we can do that in humans under pathological conditions these are epileptic patients and when you stimulate parts of the brain you can stimulate as I mentioned before you can stimulate these phosphines if you do it in V1 and people are blind you can get these sort of flashes of light so the question is in a monkey this might be a pathological experiment that you can do although it really works only if you have clustering if you don't have clustering, if the neurons are arranged randomly then it's not going to work so that's one step for example towards causation because you can either do viral experiments but they're not, right now we don't have the knowledge that we're supposed to do this in monkeys because ultimately for example if you identify there's a subset of neurons here that are involved you know something about where they sit let's say they all sit in layer 6 and then you can genetically characterize them by getting a protein that's only expressed in them then you can transiently inactivate them using sort of like a second or third generation molecular knockout then you can also interfere with the system very delicately, very deliberately very transiently and all those experiments are going to happen in the future so Logotitas and his people there's a whole number of studies by him they did this sort of a rivalry flash depression in different cortical areas they tried it in V1, V4, empty and then what I just showed you was down here in the Infratemple Cortex and here this shows you the percentage of neurons that follow the percept so by the time you get to this high level part of the brain the vast majority of neurons follow the percept and interestingly there are no neurons that represent the suppressed stimuli so if you believe in Freud you can say there's no evidence for Freudian remember the unconscious and all of that well there's no evidence for Freudian unconscious in this part of the brain because you could say you can put a Freudian spin on this you can say well there are two stimuli and you suppress one actively and you can say well where in the brain is a suppressed stimulus present it has to be present because after a while the other stimulus sort of comes back again there has to be some memory in the system so somewhere you have to have there has to be a representation for the suppressed stimulus and if you want you can think of it like the Freudian unconscious well it ain't here because here the neurons don't respond to the stimulus and suppress in V1 there's now recent experiments you can show very beautiful very small fraction of neurons much less than in this old diagram it's a very small fraction of neurons that is weakly influenced by the percept the vast majority of neurons in V1 fire to the whether or not the stimulus is perceptually dominant so in other words whether or not the monkey saw let's say the sunburst pattern or saw the orangutan face it didn't matter at all in IT in infotemporal cortex the neurons kept on firing more or less for some neurons a small modulation so the remarkable thing is you can have literally a million neurons firing away in V1 yet you don't perceive it that's very interesting because it tells you that not any cortical activity gives rise to consciousness because you could say well maybe one explanation is there's all sorts of subcortical stuff going on but when there are lots of neurons that fire in my brain in cortex that's got to be consciously represented well in this case we know it's not the case there are many neurons that find V1 but they might fire to the perceptual or they do fire to the perceptual stimulus to the suppressed stimulus and that's of course what Francis Crick and I postulated when we talked about not in V1 so I like this of course among others because it very strongly supports our conjecture that you're not conscious in V1 that V1 is necessary for most normal forms of seeing but that's not where consciousness is generated and then you go between V1, V2 and these high level areas and you see some intermediate change so here you have like 25 or 30 percent of the neurons that follow the percept and interestingly in both areas you've got quite a number of neurons that only fire when their favorite stimulus is suppressed so let's see in MT they did it they did the study with using moving gratings when you have two moving gratings and you see one on the other and there they find many neurons only so let's say a neuron fires to a stimulus moving upward well if the animal sees a stimulus moving upward the neuron won't fire if the animal doesn't see the stimulus moving upward but it suppresses it then the neuron fires so the neurons that nobody knows why that strongly respond in an anti-correlated manner no yeah we don't know anything about the cell tide so this is just a cautionary note so these are some of the neurons that project from that part of the brain that Locotetus was recalling from infotemporal cortex here that project to front of the brain so again I mean it's certainly quick and I think that these neurons are going to be key involved in generating consciousness since one of the function of consciousness to generate is to provide the summary of your current percept and send it off to the planning centers and then many of these are just one of the catalogs of all the different cell types these are distinct cell types as defined by Natin you get sit in this part of the brain and as I mentioned the different cell types physiologically these neurons behave in a very different way some find this very chopped this oscillatory manner some find maintain some of them in this much more sustained as a transient manner some are delayed some are not so as we begin to dissect the circuits we realize that all these auxiliary neurons are involved in the memory are involved in the encoding of the percept some neurons are going to be involved in the suppression of the other percept it's going to be a very complicated system and again we have to use these perturbation approaches for example we inactivate some of these cells either conventionally by just going inside the brain dumping some neurotransmitters some pharmacological poison or something that turns off these neurons or more cleverly we want to do it genetically where we can target those neurons and sort of switch them genetically affecting them animal with a virus or something or we could do this in a mouse and use it in a transgenic mouse all sorts of different possibilities yes how do we what how do we get oh yeah so there what you do essentially blindly well all you know it's a part of the brain so then you essentially you lower your electrode and you listen to it till you have a neuron sort of that response for example you show different phases or something till you have a neuron when you show a phase that sort of nicely responds okay so there are techniques for doing that so you can for example plot the action potential how often does it fire and if it's a single neuron if it fired one spike it's not likely it's extremely unlikely to fire for the next two or three or four seconds that's for example one way you can also look at the actually the shape of the individual action potential and from that you can make some determination whether it's generated by one neuron or by several neurons the trouble is you do this in a totally blind manner right now so you're lowering your electrode very often you don't even know are you in deep layers or in superficial layers sometimes you know and you have no idea what neuron you're recording from I mean it's like doing molecular biology and saying well I don't know there are some neurons there's some proteins there I don't know what they are they're just proteins you know and I'm just manipulating these proteins that's the state the scale it's a scandal the current state of electrophysiology in these animals now in for example in you know if you record from well I guess in your case in from Locust right there you do know to certain extent at least in some part when you're recording from you have a much better idea of what neuron you're recording from but here you gotta remember they're per millimeter so this is roughly two millimeters from here to here so you know in in let's see you know this chunk here they are on the order of 100,000 neurons now so it's a big problem sometimes on some days you know I just figured we will never understand it but we will so let me finish off telling you a similar study that we do in humans this is now our own work that we do in humans where we use these sorts of stimuli so these are just the stimuli we use colorful images so this is done in collaboration okay the work was done by Gabriel Keiman who got the closer price for this of the best PHD at Caltech last year and it was done at the clinic in downtown LA here with a MD-PhD person neurosurgeon Isaac Fried and we are recording there's a very very rare opportunity that in fact now up to three different groups at Caltech are taking advantage of as Erin Schumann's lab and also Richard Anderson they're doing it with hunting we're doing it with UCLA we're essentially you have patients who have epileptic seizures and they can't control the epileptic seizure anymore with pharmacological intervention so the drugs are inefficient or they become inefficient they adapt to them and then one of the options is actually a very successful option is brain surgery where you go inside the brain and you take out surgically you cut out or coagulate the part of the brain that gives rise to the epileptic seizure and it's brain surgery so you don't want to do this at home but it's rather successful in the sense that many many of these patients go most of the patients have significant improvement and some patients a significant number of people have epileptic seizure at all anymore in most patients it's significantly improved a small fraction of the patient doesn't make a difference and that's probably when you actually didn't get the right foci oops well I mean I don't want to make fun of it it's because the trouble is that's not why I'm coming to it because very often the brain looks normal so in some patients you can with the EEG based on the symptoms you know if you observe these patients you know okay so epileptic seizure is a very diverse phenomena there's very different etiology different origins, different genetics different type you know depending where the seizure is you get very different symptoms you know some people have aura some have don't so all of that if you're a doctor you see these patients can tell you what part of the brain is left or right or is it frontal or temple it originates and then many people and then if you use those sort of symptoms together with X-ray or MR and EEG most of these patients today you can locate where the seizure is and in the MR the brain that location looks a little bit abnormal and then you can just directly target and take that out you take out for example very often the hippocampus sort of you know take out a little piece of brain tissue like this and you remove it just remove it all together and you close the brain up and the patient goes home and it's relatively normal it's amazing if you talk with neurosurgeon what sort of with what sort of nonchalance what sort of relationship they have to brain move it down and you know this is somebody's brain they just have a very different relationship anyhow now in some subset of patients that's not the case so unfortunately from the outside even with the EEG you don't know whether the seizure originates on the left or the right brain and if you look at the MRI it looks relatively normal so then what you do you implant you literally implant electrodes into the patient's head so now you put up to 12 of these macropoops so here you can see one in the hippocampus very often it's where seizure originates for some reason this is part of the brain that's very much prone to seizure in fact HM no we haven't talked about that yet and so they developed this procedure it's now done at many many universities around the world where you monitor you put in so called the different electrodes and so if you talk to doctors one for example is the doctors do they put in just grid electrodes they just sort of shove them under the skull at the top of the brain then these are more invasive but sometimes you need them if you really want a pin point where you actually go inside the brain and so these are like 1.2 mm thick in diameter and you slowly push them in and then you put 10, 8, 10, 12 of them in you seal them up and then the patient lives now on the ward for 3 days, 4 days, 5 days a week until and he's monitored 24-7 until the patient has a couple of seizures and then because you monitor the brain you can now sort of pin point you can essentially do triangulation and you can now recover where the seed originates and then you de-plant the electrodes and then a couple of days later you go in another operation you take out that part of the brain and that's considered invasive surgery but it's quite successful and the mortality today the mortality in any of these brain operation is probably less maybe 10th of a percent or something like that but the mortality of other things that bleeding is the biggest problem that's probably roughly a percent and now of course when you're normal you don't want to have this done but when you are a person who has 3, 4, 5 seizures a day you can't drive, you can't really hold down a job then this makes a huge difference now what they did at UCLA they hollowed out this volume here and they added then these micro wires platinum iridium micro wires so now essentially you can do what you do in a conscious person you can record from their brain you can record individual nerve cells so Gabriel showed this here this is a cell in this is the the electrical signal and this is the visual stimulus I mean it looks no different from a you can see when this was done this was done in the election campaign it looks no different than what it would look from an animal so this just shows one cell this is time scale from 0 to 1 second the stimulus, those images were added at 0 and were removed at 1 so they're exactly 1 second on and here we show the time 1 second before and 1 second after the horizontal dashed line is the average firing rate so this neuron on average 3.5 spikes per second and these are different categories of input we showed and this neuron responds to either face drawings like Batman you saw or Clinton or something or famous people like for example in the clip we showed pictures of Bush and of Al Gore now this picture just like the American electorate this neuron just like the American electorate was unable to decide statistically whether it preferred Gore or Bush although let me know it did fire a few more spikes to Gore than to Bush but not statistically more because it is a different question remember that cell I showed in the very first lecture the Clinton cell that comes from here because some cells actually a very rare number of cells do respond to one picture much much stronger than any other we have like 12 of those neurons now so the Clinton picture I showed you responds to 3 different pictures of Clinton a line drawing of Clinton a photograph a potential portrait of him and Clinton with Chelsea and somebody else and Clinton physically looks very different one in color one in black and white yet the neuron still responds to it so some of those neurons respond very very specifically some or many more or more broadly like these cells seem to respond to any famous person or to a face drawing of a famous person they don't respond to unknown people so here we have unknown actors displaying various emotions like happiness and sadness and anger it doesn't respond to them anyhow so now we can do a gavel comment did this first PhD we can now do what Locotita has done in a monkey but the two differences A of course it's a monkey versus a human and you know you never know what goes on in a monkey and B in the monkey you have to train for many many many months for many many years to do this experiment and so you always have to worry what's the effect of training or these patients you tell them for two minutes just like I told you and they experience a phenomena okay so here here's a cell that happens it's one of the cells I think I also showed in the first class that likes curly so out of the 40 49 or 48 different images we showed it it only really responds strongly to curly statistically at the 1% level so here you showed in the right eye curly left eye the patient sees nothing the neuron responds this is sort of the average response and the perceptions curly is nothing else now we flash on the grating so this is flash oppression so perceptually this suppresses this and what you'll see and what the patient told us he saw is a grating and the neuron with some delay that's like a 100mm delay stops responding here the opposite occurs left eye the grating the neuron doesn't fire we leave the grating on we add curly curly perceptually suppresses the grating the person reports curly and the neuron fires very strongly so let's just go to this here so we show this for two different types of cells here all the cells that are selected for categories like the neuron I showed you first that fires to you know bush or to go as far as we can tell to any well known person well here are the the smaller number of cells that respond very selectively like just to curly we have one to Michael Jordan we have another one to Paul McCarty the beetle or the Clinton etc the results are strong but they're similar here so this average over all these neurons so let's say let's just do it for the Clinton cell and call the effect of stimulus Clinton and ineffective all the other stimuli that the cell doesn't respond to first of all we let me see so here is nothing then the effect of stimulus comes on nothing in the other eye and the neuron strongly fires to the effect of stimulus and the person says I see the effect of stimulus and then we flash on the other stimulus the other stimulus is perceived and the ineffective defying rate of the ineffective stimulus goes down and here the opposite is the case you don't put the effective stimulus on you put some other non-effective stimulus on let's say picture of somebody else and the neuron doesn't respond to it the person sees that picture though then you add the effective picture the Bill Clinton picture person reports you see Bill Clinton and the cell fires very very strongly so again you can do information theoretic analysis to show there's a very nice correlation that you can in most cases you can read out the behavior of the cell the behavior of the person what the person sees what the person tells you he's seeing from the behavior of the single neuron and you can also ask the question to what extent so in this period here only one image is present this is the first stimulus when you only have a monocular stimulus on so you only have one image on well in the second period you have both stimuli on so you can ask is there a difference between this period when let's say you have Clinton on and somebody else but you're seeing Clinton and this period when you're only seeing Clinton can you tell so perceptually it's the same case in both cases you see only Clinton but here you have nothing in the other eye and here you have something in the other eye so you can ask the question you're only can you distinguish this distribution from this and the short answers you can they have the same delay they have the same duration and they have the same amplitude okay so let's finish here so so so here we only have a small number of neurons because these clinical you know we are in a very strong constraint here we only have very little time to record because the patients are only in the hospital for very little time and of course most of the time the clinical concern you know they want to sleep they want to eat you know the relatives are there so we have very little time compared to monkey so we only have very few cells but the cells respond very nicely they're very similar to the monkey recording two-thirds of the cell follow the percept and no cell ever follows the perceptually suppressed stimuli of course this is in a higher area before the cells at logotitos recording from all in part of the brain it's the sort of the last purely visual stage of the brain it's the ventral pathway, infratemple cortex the last purely visual stage we are recording in this part of the brain called middle temporal lobe we'll hear it next week it's critically involved in memory it gets output from the visual but it also gets output from lots of other sensory areas it's a multimodal area that responds to vision as you can see to visual input but it also responds to other things that's key involved in transferring things from short to long-term memory we'll talk about it in the movie the movie we're going to see in class that also involves the mentor that also involves injury to the part of the brain, middle temporal lobe just to finish this class we are introducing these perceptual stimuli perceptual stimuli allow you to dissociate the input from the percept so you can separately manipulate the input or you can manipulate the percept and there's a very active area doing this in humans and monkeys also a lot of fmi and what the the electrophysiology seems to show very clearly that in the early part of the brain in LGN, I didn't mention that at all there's no effect whatsoever of perception in LGN and in V1 based on the single cell electrophysiology there is only very weak effect now there's an interesting discrepancy here the number of papers that appeared in the human literature using fmi using functional magnetic resonance imaging and there they do see a much stronger modulation in the early area V1 then in the electrophysiological so in the electrophysiological literature as I mentioned you have millions of neurons that fire away in V1 and without having any influence on the percept or the percept doesn't have any influence on the neurons activity in primary visual cortex there's a discrepancy in the functional literature it seems they find a different result that has to be resolved but that in the in the higher stages it is very clear in the higher stages infratemple cortex and medial temporal lobe the neurons a significant fraction of neurons follow the percept and there seem to be no neurons that respond to the suppress stimuli and a subset of these neurons are probably going to be critically involved not only correlating the percept but generating the percept and so a big push now is to try to understand where the neurons are actively involved in generating and trying to find out techniques that can zoom into them and then manipulate their behavioral state so we can make the transition from correlation to causation okay next week we'll talk about zombies