 attention. So today I didn't finish last Wednesday's lecture just to remind you when we talk about attention, attention on selection. By the way, yesterday I had this great experience. I gave a talk at the senior curriculum, a senior college, downtown Pasadena. There's a senior college for people who are retired and you know the office scientific classes and so I gave a talk there. I was very full with like 100 people there and the gorilla in our midst worked beautiful. Like 10 people saw it and 10 people saw it and you know everybody else didn't and it really worked very well. Of course it's a you know it's a somewhat more in general more you know attention does get impaired with age. But even here there were a number of you who didn't see that. I remember the gorilla. So I mean just to summarize there are two forms of attention selection very crudely speaking and there you can think of them as bottom-up attention and top-down attention on selection. These are ways how our brain voluntary or involuntarily automatically selects relevant information from the mass of information out there and presents them to you I mean to the conscious you. So the bottom-up one exists all the operates all the time it's present all the time it's automatic you can't really you can willfully ignore it but you have to do it willfully it comes at a price. It's usually rapidly it doesn't depend on the particular task am I just looking around am I looking for my friend am I looking for an L among teeth silence is always present and does not guarantee access to conscious awareness. Then there's top-down attention which is much much more powerful you can attend to essentially anything but it requires voluntary effort as we all know from trying to concentrate when we work you know if you do it for a long time it's actually you know it's actually somewhat exhausting. It can be spatially prescribed but can be like a search out of attention that's a metaphor that's being used often. It can be directed to location space as a search light it can be focused at an entire object or it can be focused at specific attributes like you can attend to everything red in the visual scene then sort of as it were a red becomes salient everywhere in the in the in the scene. It takes longer it's sustained but it also takes longer to set in. It depends on the task it is another visual volition control and it seems to guarantee access to consciousness in other words what you voluntarily attend to you seem to be your conscious of. I don't know a case where you have the dissociation maybe it exists I just don't know it. Now you can ask the question what can you see outside of focal attention. So for some people as I mentioned for some people that's a nonsensical question to ask because by definition people say whatever you're conscious of you have to tend to sort of axiomatic but at least you can say okay so let's let's get away from that sterile debate because this of course revolves around what exactly do I define attention and if I just do it by words or by you know if I confuse it if I say everything that you're conscious of you attend that seems to conflate two separate terms and so ultimately we need a neuronal definition which we don't have. So let's use an operational one. Let's say we'll do this dual task paradigm and I'll tell you how we do it. We'll get you to attend to one location in the part of the visual field and we get you to really pay attention there and we can tell when you're not attending because then your performance will draw and then we can ask what can you see outside your focus of attention. Now already your introspection seems to suggest to you I think correctly that you know I can really attend to the tip of my finger here, can really really focus on that you know the detailed structure of my fingernail and it's clearly it's not that I have tunnel vision right you know I could imagine that if I really attend everything else sort of fades that's not the case. There are some there's some deficits some neurological patients where they have something like that it's called the balance syndrome we'll talk about a bit later. Some balance syndrome essentially one of the symptoms is there are a number of different symptoms. One of the symptoms is that you're really only aware of things at your focus of attention when you attend to and everything else you're literally not really aware of so that's sort of a bit more like tunnel vision. But certainly normal I mean neurological intact individuals don't have this. Okay now the dual task paradigm is sort of I'm talking about here something that was perfected by Achim Braun who's a professor now at Plymouth he did this here at Caltech. You essentially ask people to do a demanding task and I'll show it to you in a second. Task at the center of the display so when you're looking at you're looking at the center of the display and I very briefly let's see 100 or 150 milliseconds it's a difficult task I flash up five letters and the letters can be either all the same five L's or five T's or they can be four T's in one L or four L in one T then they're different. So like often in psychology it's a two alternative choice task you have to say it's a more different. Those two answers you can guess if you don't know you should guess. So random performance is chance is 50% and it's titrated with a mask we'll talk about that in a second. So you can so if you have less time for example if you quickly look away you totally unable to do this task you really have to dedicate you all your resources and so tightly regulated that even if I give you 10 milliseconds less it's already a very difficult task. 10 milliseconds less there's a measurable decrement in performance. Now at the same time I flash up something in the periphery. Okay in focusing what I can flash up is a picture. I can focus on flash up a letter in L versus a T or I can flash up a bar and and I can ask you is the bar let's say horizontal or is it vertical? And so now there this is obviously displayed. It displays always at the center you have these five letters and then in the surround in the periphery not always at the same location to avoid eye movements but sort of randomly on a circle I flash up this this other test that I want to test. For example for sake of argument let's say it's an L versus T. So the typical display will look like this that at the center you have sort of let's see they can be rotated. So this would be here the answer would be different because it falls in one teeth and then somewhere on a circle you know 40 I think it was 40 degrees eccentricity let's say I put a single L or a single T and now I'm asking you to do three different things. Well I'm asking in three different configuration sorry. In configuration one I'll say forget about the surround just constantly on the central task and I'm going to just just tell me the central task performance. So here I plot the percent performance of the center task so this is central performance. 50% is chance and 100% is perfect. So let's see and I try train things with a mass so that you do let's see on average 85% correct. Okay second trial and intermix these trials. Second trials I say well forget about the center I want you to look at the center I really want you to look at the center and I can track that with eye movements with you know eye scanner and I ask you okay just do this round T versus L. Now this looks of course sounds trivial but A this T can be any orientation so it can look like this or you know even upside down plus it could be anywhere here and it's very briefly presented so it's not an easy task but by itself you can perfectly well do it. 50% is chance 100% is perfect so arrange things that you do it you know arrange things with a mask I showed only very briefly so you do this again do on average 85% correct. Now in the third condition I ask you to do both things at once I can ask you okay now tell me so now it gets a little bit more difficult and you have to train subjects because now they have to pull two you know they have to push two two sets of buttons one for the central and the other one for the peripheral performance that's more difficult and you have to train people to do that they change their brain and now you can ask the question well if both things are truly independent let's say if this requires attention if this one doesn't require attention well then the performance on average should be here right so here's a performance with one the peripheral by itself here's a performance central test by itself I ask you to do both then you know your your your performance will be over there and if it's not for example let's say this one requires attention well then it depends exactly on what I tell you and I usually the the subjects are usually instructed to if they have a doubt to first do this task and only then do this so typically when there's a performance you um when there's attention interference you get some of the performance along this diagonal okay so that's the that's the setup so I'll think uh sorry so let's see okay here the task is is an animal or not in the scene this was done actually in the lab by Fei-Fei Li and Rufian van Rulen so you can see it's not altogether easy did any of you see that okay so but remember you're supposed and I mean of course you're sitting at very different distances but what you're supposed to do you're supposed to look at the center tell me whether there's else whether they're all else or all T's at same or whether they're four else in one T or 14 one L that's different and then also say that image that you so you see brief image which is then asked that image does it contain an animal or not let's just do that again and of course I can control the you know timing very well on this you know my mac and the lcd do you have a feeling that you can do both uh yeah although it's pretty big from where you sit um okay I mean it doesn't really matter I mean this is just illustrative right the timing is totally off with the lcd and the computer display so okay so let me just tell you some of the results so for instance if I if I do this with L versus T in the periphery you cannot do this it's really quite remarkable so in other words when you're my intentions are gauged here there's an L on T here somewhere yeah I can clearly see it I well I can see something there I know there's something there right on this empty background but I can't really tell whether what I mean what what the letter is whether it's L versus T which is really very remarkable on the other hand this you can do this surprise this is to to everybody surprise this task okay I'm not gonna show I mean I'll just draw the data out on the on the on the so you can do simple things like you can say whether it has a particular color or if there's a big if if there's an orientation the periphery that's different from neighboring orientation but L versus T you can't do what you can do you can say whether a scene contains an animal or no animal or whether the scene contains a means of transportation a car or bus or a truck or plane or no means of transportation now if for any of you who've ever done computer vision this is rather surprising should be rather surprising why well because doing a letter even arbitrary rotated L this arbitrary rotated T computation sounds rather simple and essentially you have to look for the alignment it's alignment like this is it like this this is a T and this is an L or for example something else you cannot do you're unable to do this task if this is green and this is red or telling this from from this disc in other words in the surround somewhere I flash it you know you do the task here in the surround I flash this task that's either this disc that's either green red or red green and you cannot tell which one is which I mean this is really computational this is trivial right anybody can write a computer program but distinguish this from this yet your brain can't can do that at the same time your brain can't take one of these images and decide whether there's an animal present in the image and the animal can be a lion can be an ant can be a flock of bird or can be a school of fish yes no no no no so the the pictures of course are much larger but the the animals often are small the animals inside you know sometimes they occupy you sometimes the animal is a small portion but the relevant thing is it's always the you normalize it always such that by itself the performances but the task itself is always roughly the same so you just see the size and everything l versus t so by itself you do 85 percent and the the the animal non-animal so by itself is 85 percent well what's right so so so that the performance always the same it's just remarkable that some things you can do independent of the central task and other things you cannot do and what's more surprising that by itself is not so remarkable what's surprising and what we just don't understand one of the many things we don't understand is why should something that seems very simple like red green green red seems you can't do without attention but something to us is complicated as animal non-animal right now there's no algorithm there's no machine algorithm that can tell an arbitrary picture whether that picture contains any animal or not there's no algorithm like that around I mean people will make it but right now they just don't exist so so that's we we don't really understand it but there it is so so there the the important point is that certain things we can do and certain things we can do and we have to look for an explanation for that and the explanation will have to involve competition among receptor field competition among neurons I'll talk a little bit about it next week what what uh what Leila who's who's your TA what Leila showed just recently together with uh with Patrick working here is that you can also do even more demanding tasks you can tell if I take a face I remove all the facial facial hair and then I ask you is it a girl or boy male female gender discrimination and again that if you look at the faces they look very they look you know you really have to look close I mean to me it looks you really have to look closely to tell you know because they wear no makeup no glasses no hair whether so you just have to look at the facial structure to tell whether it's a woman and a man yet again you can do that without attention probably what it says is that those things that are I mean one way to read this is to say well those things that are ecological relevance those things are really important for at least when we evolve like seeing whether there's an animal there obviously gender discrimination is crucial for us those things are um you can somehow do it without attention but those things like L versus T is a rather unnatural task you have to do this very fine face discrimination that requires attention of course that's not a mechanistic that's not a neuronal explanation that's a high level functional explanation yes don't look sorry covering this doesn't really help a lot it's an excellent question I know we discussed this I don't think that's an excellent point because um yeah I know it hasn't been done yet I think we're supposed to do that but no we haven't done it yet no that's an excellent question we have both the same so people have done other experiments like that where they have the same task in the center in the surround and sometimes you do get interference and sometimes not so I'm not I don't think it'll change it'll change significantly because we know we we know um that's why it comes back we talked about it but we haven't done it but we're really sure because the question comes up often I don't think it'll make a difference but it's it it should be done as control yeah I mean that's the point so that's why I don't think it'll make a difference whether you have a car let's see an image where a picture was an animal at the center and the same task in the surround I think can still be done but you know that's the nature of science we should do it maybe we're going to be in for a big surprise and we haven't done that for faces either yet right it would be nice control good question any other question okay so this is one case I wanted to discuss and I'm I'm someone confused about this topic and I'm just I think it just reflects well my confusion and also the lack of detailed neuroscientific understanding of what attention is and what conscious is so as I said many people most psychologists uh would identify when you attend to something you are conscious of and when you're conscious of of course you have to attend to it but certainly if you use this operational definition of top-down attention the one I showed you in this dual task you can show that you can attend to certain things outside and other things you cannot you cannot detect so and the fact that we don't perceive the world as a tunnel at least tells me that this even the absence of focal or the near absence of focal attention you can still see things outside on the other hand those experiments in inattentional blindness and those gorilla in the in our midst experiment seems to tell us that we have no expectation whatsoever I mean this is a compelling thing about this gorilla in the midst story if we have no expectation forever if we have no expectation whatsoever and if we really attend at least some of us are blind to very large long-lasting signals in the image I mean the gorilla you know took like five or ten seconds to walk across and and so it's not just a very brief input yet we totally miss it um so I'm I I don't know right now what to say about the relationship among among consciousness and attention in principle I would say they're different but maybe with it is you know you could argue without this expectation at least you don't see things um now there's um there's another instance of a phenomenal gist perception this is another instance where you can argue that you don't have attention really yet you still see it or you can withdraw attention and you can still see the gist or even inattentional blindness so when when it's totally unexpected when I ask you to do this remember this experiment where you have to judge is the horizontal or the vertical arm longer and I totally unexpectedly flash up a single picture of a scene very quickly you're very good at getting what's called the gist so the gist is a high level description right the gist of this is it's an office scene you know it's a lecture hall with people that's gist knowing who they who it is or how many women and how many men and you know the color of your clothing all over that's not gist anymore it's not tell precisely I mean it's difficult to really define it precisely but just it's a high level semantic description of a scene like mountain scene with people you know river with dog you know that sort of gist so to demonstrate this faith he made this this was one frame now again I don't really control it some principle it's I don't know one frame what's this well I don't know that 30 what's the clock of the lcd yeah but the lcd isn't 60 hertz or is it whatever so it's a bit unclear what the exact timing he is did you see it no yes was it Elvis what was it yeah yes yeah that's exactly what it is we can look at it again yeah so so here and you can do it people have done experiments on this you can show it very very briefly and you still see something in fact somehow a movie makers have played around with it would there's a famous episode in one of the Woody Allen films where he kicks in one of his very early films where he kicks a can then he has this very brief flashback and so movie makers experimented already a long time ago at their frame rate which usually is 24 hertz how many frames do you need in order to induce a particular feeling in in in your audience and they discovered that you actually can be very very brief people have made these have made these movies where they show you for example images at 10 hertz or they show you different you know like MTV except even faster than MTV so you will have let's see 100 images and each image is only on there for a 15th of a second so you have 15 images per per second and these are let's say line drawing so each image by itself is a single line drawing of a tree you know a bicycle a baby a truck whatever and they come you know 100 of these at you at this 15 hertz rate you can perfectly well see each one of them it's quite remarkable you might not remember them of course for the most but you won't but that's different we'll talk about that later but you can certainly see them so so so a that tells us our digital system is very fast which is very surprising if we look at computer algorithms because computer algorithms are very the today existing machine vision algorithm are very slow and very fast machines or the performance is is is still not nearly as good as all performance and of course they operate on machines that you know it's now gigahertz well you know the the sort of the clock speed if you want to use that metaphor it's not a very good one but you know the switching speeds of neurons is on the order of a few milliseconds you know one two three four five six milliseconds action potential is a millisecond across the synaptic transmission takes half a millisecond or something like that yet we can perceive things in 150 milliseconds so you know you don't have a lot of you don't have a lot of time to do processing so there's an interesting difference huge discrepancy in performance between us and and and machines and here the here the claim is that you can do this just perception without really requiring attention only minimal attention resources so a d d you have to explain what are the neuronal representation that support this and be that you can at least be conscious of something and there this is phenomena that people really don't study it all in the lab and i could call i call it like spaced out phenomena and we do i mean humans do this all the time you say i don't know it's a technical term for that you know you know you can focus on when you drive i love you know i go climbing you know i drive so i drive a lot of red rocks and you're cemetery you're on the road there you know you're dreaming and sort of there isn't too many too much other traffic on the road you can sort of you know you can do all sorts of things in your head while you're doing that yet and so you this is what i call by spaced out or when you just sort of war ambling through the world thinking of something else but we do that all the time and you you don't really you tend to something that in your head right you imagine some conversation or replace some conversation yet you still it's not that you have no idea what's out there right clearly you know you can drive you can make meaningful decisions whether to cross the road or not and we'll talk about that separate that sort of instance of these zombie systems but i claim you still always have access to the to the gist you know roughly where you are you know you know if you drive with an over path coming up there's a truck ahead of you you know you have to pull over all of that still going on and so i think sort of this spaced out phenomenon relies partly on on on gist perception it has not been studied really in the lab as far as i know okay so the conclusion focal attention necessary necessary for conscious vision the classical answer is yes this is almost by definition uh certainly with focal attention tied down somewhere in the image you can still see things certain things um for gist or familian isolate so the claim is or my claim is either for gist over things that are very isolated or very familiar so if there's a scene and there's nothing in that scene but sort of that car you know single cars and again you don't need you know a single object a single face or anything again i don't see any need why you why you need attention now this debate um between the relationship between consciousness and and attention which which i right now would probably say well they they often co-occur very often live they occur together but they actually separate mechanisms or separate processes this debate cannot really be resolved right now we really need to go to the underlying neural mechanisms to understand both consciousness and attention and then sort of to be able to understand under what conditions are they very similar you know is it just pipeline first you need to attend it then once you know you attend like in a pipeline architecture you're conscious of or do they saw the other differ i mean what's the exact relationship between the two psychological method by themselves that in general does not powerful enough to be able to answer this okay the last point i wanted to make about um um attention is a is a i wanted to allude to a problem that's very often discussed when attention and then consciousness comes up and it's a problem that's inherent certain types of architecture of computational architectures it's called the binding problem binding it comes out of the brains architecture um and the problem is the following and it exists in two different versions so in one version is let's say you attend to my face okay let's say only my face is present forget everything else in the world well we know um that let's see you there is a higher resolution image of my of my head if you're looking let's say you're looking at my nose of my face on v1 in your primary visual cortex if i move as i do constantly that's going to activate neurons in mt you have i have certain color you know color hair and color choose of my skin that's going to activate some other neurons my voice will activate neurons in in vernicus area and the speech areas you know something who i am or you know i what i say makes sense to you so that sort of activates other parts of your brain yet if you're looking at me so the problem is a single complex percept like like my face talking moving will evoke all sorts of activities in very different parts of your brain yet your percept is not of that nature your percept is a unitary percept right in general unless maybe you're schizophrenic or something you you're looking at me the voice comes out of my mouth everything is put together right the shoe isn't over here when i move it's not like sort of the the color that gets dragged but the shoe is on my face the voice comes from a mouse and my head moves it's not motion isn't attached to something else it all fits together and how can this how can this occur in this architecture when you have these widely distributed parallel networks this is known as the this is known as the binding problem now it gets even more complicated when you have two objects or multiple objects so let's say you have a scene okay let me let me just put that in your shorter memory there that in the binding problem has been looked at both from an point of view of ai or computer vision or neural networks as well as from a from a computational perspective as well as also been looked at from a perceptual perspective and this is again and trismann the psychologist who sort of pioneered some of these feature integration theory in visual search paradigm so the her claim is that okay so she uses this language that i don't like at all she uses high level psychological language she talks about object files and things like that i don't like that because i don't know where i i think it's a very bad metaphor and it's misleading metaphor and i don't know really what objects files aren't where they exist in the brain but the but but but but the problem is but she says you the here the claim is in order to correctly attribute different so the idea is you have different feature maps the feature maps of color for orientation for for texture for emotion etc in order to combine them you need attention so that attention solves the binding problem attention with some of select out the relevant objects in the color map in the more in the motion new among the motion neurons among the depth neurons the texture neurons will all bind them together that if you have very simple as very simple features or objects like just a single bar that has essentially only orientation and location you don't need attention in particular if you have neurons that are already that are coded already sort of genetically to represent those things and we know for example a simple cell in v1 will represent the location of a of an elongated object with a certain orientation so this cell sort of in hardware solves already the the binding problem for location and for orientation but if you if you want to do things that you that you didn't you know that your brain didn't wire up a particular neuron like if i show you some of those search tasks where you have to combine color and orientation for instance well then you i mean then you need selective attention and so then now let me come back to the second more difficult version of the of the binding problem so okay these are two different colors so let's see what happens if you have um if you can have an l or t and the l can be either brown and the t okay so here now i have four i have you know two two dimensions letter which can be lt and i have two colors so now it's the brain so now let's see there's a scene that has this okay now in a topographic network if i if i map this now into network that's sort of topographic right so here i have sort of maps one map for for color and one map let's see for for letters as it were well then over you know in this color map i'll get activity here corresponding to this and i'll get activity here corresponding to that in the letter map i get it for here and and you know this activity can activate at the black neurons i mean this is all very abstract and this activity over here activates brown neurons and so then i can match the color activity here with the left activity i can say oh it's a it's a black l and it's a brown t however if i if they are maps in the brain like they are in these high level visual areas like for example in the object recognition in the object area it sort of at the end point of the of the dorsal and object perception the what pathway there's very little there's some but there's very little receptive field organization left there's very little topography left it's early on in the brain but you know as we mentioned the receptive field becomes larger and larger and topography sort of slowly disappears it's not abruptly but it slowly disappears and it's very little of it it's present in these high level visual areas then i have the problem how do i how do i keep either i wire up individual neurons that you know a neuron that will only respond to a black t and i have another neuron or set of neurons that only responds to brown t and a third one for red for what is it a black t and a fourth one for black l or i haven't i have because otherwise i have this problem right if a neuron is active how do i know uh how do i know it was a black a black l versus knowing it's a black t right if i separate somewhere represent color so all i say no it's black and you know uh brown and l and t how do i know this goes with this and this goes with this rather than the black goes with the t and the brown goes with the l so this is sort of the more difficult version of the of the um the more difficult version of the binding problem when you have two or more objects how do i keep the property separate how do i sign that one set of properties to one object another set of property to another object and she and she had these ideas because sometimes she discovers people make so-called conjunctive errors that if i flash it seems very quickly you know these remember she did these experiments where you flashed the l's and t's and horizontal and vertical or various letters or various colors sometimes people would make errors but if she analyzed the pattern of the errors there were more than expected more than by expected by chance these sorry illusionary conjunction there were these illusionary conjunction errors so here for example the claim is that when i flash you lots of arrays with l's and t's that sometimes you would say oh there was i saw a brown at brown t or i saw a black l but the idea is that people combined this i think this is true but i think it occurs i mean as far as i can tell from the literature and certainly it it's not very common in life i mean yes there's sort of occasion these episodes you know when you have a you know this this sort of the story here the the girl wearing the red pants and green t and and you know maybe you could say oh she actually you know had a had a green pants and red t-shirt but i i mean i so these these illusion conjunction certainly exist i don't think they're all that common but they could reflect the fact that if the system is severely limited if i show you these images only for very briefly then you don't have enough computational you know you don't have enough time for the networks to settle down into their steady state as it were and signal correctly things and sometimes they can make these illusionary errors um yes and this is essentially it's due the computational version of the problem as i just talked about it in these maps essentially due to this german physicist christoph von er marzburg who's a professor at usc uh half half half the time half the time is a professor in germany and he talked about long time ago already in 1981 in the very well-known paper he talked that the brain somehow has to solve this binding problem the binding problem is a big problem in these non-topographic networks um when you want to represent multiple attributes and you don't do it without hard wiring you know for every possible percept you have a group of neurons and this problem doesn't occur but the argument is that's computational very expensive you need to have lots and lots of neurons for every possible combination of every possible object and the brain surely can't do that and therefore it has to solve this problem in some other way and then the claim was that these oscillations that i briefly mentioned early on these are synchronized oscillations that the brain could do this by using temporal information that yes you have different maps that code for color and you have another map that codes for letters orientation but the things that go together they fire together so essentially you have the group over here that codes for l and the thing over here that codes for brown they sort of fire together they they're synchronized that discharges and that this fact that they fire together rather than the other neurons sort of fire independently that it's a critical code the critical piece of information that tells the posts in optic networks the neurons that look at the the network that looks at that saying okay these objects go together because they fire together and therefore i'm going to perceptually combine them into this unitary percept now the binding problem is well known it's still very controversial whether in fact it really exists in the sense that you could argue well you have really two mechanisms you have a very sloppy mechanism because you have a very sloppy mechanisms for those things that i really haven't seen before and i code them in again in in neurons but i don't code them very precisely and if for all those things for all those objects all my friends all my enemies i don't think i have any but for all you know for everything i've seen and grow up and see every day the font of my computer my dogs my you know my dress my you know the people that are around me all those actually are code with individual with groups of neurons and if you think about it and if you do sort of crude back of the envelope estimation there aren't really that many discrete objects you know or different type of people of faces etc in my environment you know call it 10 to the 3 call it 10 to the 4 you know if you code it's if you code each one of them by a couple of dozen neurons and you remember the 100,000 100,000 neurons per cubic millimeter of cortex and you know you have on the odd of 1200 square centimeter cortical tissue these numbers are not really that big so conceptually the problem is well known as this binding problem that's why i think it's important to talk about it to what extent it's a real problem for the brain remains controversial certainly there are cases like these illusionary conjunctions when you incorrectly combine things not randomly but in this interesting way where you combined or you know attributes of different objects and and and swap them out certainly under some condition it is but that also suggests the fact that these illusionary conjunction are not that often you know it's not that often that you compute you know that you can confuse let's say you know when somebody's walking the dog that you can confuse the color of the dog with the color of the person right you know or the car you know when there's a person walking in the car next nearby how often you confuse the color of the car with the color of the person that those things don't really occur very often in normal life suggests that maybe the problem it's less of a problem we think it is okay that was finished off are there questions about binding problem or tension you know it's all clear because i can be looking at something and be a million miles away no i mean i don't see how you know i mean i can tell you know once i have access to your body i can then maybe read off you know skin conduct and things like that but just by looking at you know no i mean that's why it's also called covert shift you know i mentioned there are over shifts of attention i'm moving then covert shift of attention and those you can see from the outside that it's much more difficult to study them that's why you need to use these indirect techniques when you really force you to pay attention there otherwise you don't do the task and you need to use these indirect techniques to make sure you're actually attending no because you can look at me and be a million miles away and of course we all know that happens but not in this lecture okay okay so i'm going to talk about related problem which is that time in consciousness so there's a huge and deep literature going back to the middle of the 19th century at least if not earlier that that ask questions about the evolution of conscious perception or the evolution of of perception particularly in the visual domain again so there's an early experiment this was done i think in 1895 in paris at least it's published in j physical the paris where they did the following experiment so they asked they had a single light incandescent light and there was a standard there was a standard reference light that was on all the time and there was a little light that was on for various for various time and the the subject was asked to compare the impression of that of the transient light for you know when it was very briefly on what it was long on compared always to the the constant reference light and then you get a curve like this so in other words the idea is that for short times you know it's it's and not bright compare it's less bright than this reference light then it's in fact more brighter than the reference light until for a long time sort of it's identical to the reference light obviously if if there's the the flash light is on as long as the reference light you're going to get the same answer now often this is often be misinterpreted so this would suggest if you just look at this and it has often been interpreted in this way which is erroneous as i'll tell you in a second that actually that the perception at short timescale changes continuously in other words here the interpretation is that if i look at a light first it's very weak then it's brighter brighter and then sort of it settles down to its steady state that is not the case this as far as i know and i would really love to know otherwise for short times perception appears to be all or none and what happens here is that what you have to read this in a different sense you have to read this in a sense that if i flash a light this long if it's on for whatever this time is x then you see it as for this amount of time if i flash it on for this time you see it that bright if i flash it on for this amount of time it's quite bright and if i flash it on this amount of time it's uh it's sort of as bright as the reference light this is the this does not imply that the light becomes sort of changes in time as far as we can tell i think it's a very important point because it tells you something about the underlying neural mechanisms the the dynamics of the underlying neural networks as far as we can tell certainly for short time things don't uh you don't see change in the individual percept they don't blend into one another there's another experiment by effron where he did um he flashed on for 20 milliseconds a red light so if you look at in in time so 20 milliseconds 40 milliseconds so here there was a red light and then there was a green light uh now what you see is sort of uh some sort of yellowish which probably has slightly greenish hinge you don't you don't ever see on these conditions for the short times you don't see a red light but then go turns into green light now if i do it for long times then of course you do see it if i leave it on for a second red light and then the second green light then you will see some sort of some sort of transition but for short times the point is that the way we perceive that things are that there's some sort of there's some sort of mixing that the that you know you get some sort of you know the mix of red and green which is some sort of yellow but you don't get but but the percept is constant so that this stimulus gives rise to constant percept not to an evolving percept and i think that's terribly important um because it does suggest it does this sort of model for the for now and this is pure speculation but this does suggest this sort of model for the origin of the neural qualitative consciousness let's say for these simple stimuli that you have to go about threshold and then once you're about that you have some neural mechanism that explicitly codes for the phenomenal visibility otherwise for the fact that i see it i have these sensations i cannot only push a button and indicate how long or how bright it is but i actually see it and that that seeing has to have a neural physiological correlate and that correlates let's say some sort of process that goes above threshold probably that requires feedback i'll tell you why in a second and the question is to what extent do any further perturbations of let's say this is a let's say this is a group of neurons that fires so well what is it that you actually perceive what is it that the the the corresponds to the quality to the to the sensation of seeing the stimulus is it just the fact that you're about threshold for certain times or is it actually the time the detailed time cost of this it's a question we just don't know right now now there's likely to be hysteresis in a sense these are physical very complicated systems so it's very unclear so it's probably not the case that this threshold and this threshold are the same there might be feedback for some excitatory feedback that sort of where you get hysteresis like phenomena like in ferromagnetism in in in physics now this also this is an idea by by samozeki who has some evidence to back it up this also suggests that there might be for different aspects of consciousness who says i mean it's a question that really has never been investigated until the last decade or so who says that all of our conscious perception have to occur at the same time who says that the conscious perception for red has to occur at the same time as a conscious perception for the brightness or for the location of the stimuli right when i look at a color stimuli i see everything at once it see the color comes on i see the color i can tell you it's red it's very bright and it's at you know it's just a little bit to the left of my computer but who's to say that if i look at the detail time scale that that they all occur at the same time in fact might well be that if i look at the underlying neuronal processes that they occur different times right we talked about the fact that the different neuronal subcomponents have different propagation velocities remember we talked about the mark on the power cellar pathway the mark no cellar pathway originates in a retinal goes to primal visual cortex that seems to encode rapidly moving objects the power cellar pathway seems to encode objects with high spatial precision and seems to encode color well they have different propagation velocities and so therefore they might reach you know do whatever they do go about threshold or whatever at different times and there is some evidence it's it remains controversial by Simer Zeke it remains somewhat controversial but there is evidence that it is the case if you actually look at the detailed microstructure of perceptual phenomena that there are actually discrepancies on the order of 30 40 up to 80 milliseconds between different aspects of of different attributes of objects like color and motion so he coined this very nice term micro consciousness and so his idea is well you have a micro consciousness for color and you have micro consciousness for orientation and for for texture and for stereo so if each attribute has its own has its own consciousness that's really just i mean just looking at it in terms of ncc you have different ncc each conscious aspect that you see each attribute has its own ncc you have an ncc for color and for motion and they don't necessarily in fact it's probably very unlikely that they all go above threshold at the same time and they all go below threshold at the same time then of course you can ask the question well if that's really the case that seems like a pretty messy system why don't i see that why don't i see these things if i move rapidly you know imagine let's say last summer i had gorgeous red hair i got my hair dyed i'm going to do it again this summer i had my hair dyed this bright orange and um i have a picture i can show you but i'm not gonna do that um and so you know so imagine i rapidly move my my in my hair now imagine there's actually a difference as zacky claims of 80 milliseconds between color and motion well shouldn't that then imply that actually if i move very quickly that you see the motion but the color actually you see the color only some time later right you know like in some of those televideo conferencing right if you move very quickly you know it takes time to to shift yet yet we don't really we've never seen that i mean certainly in your own life this never really happens but now so it's an excellent question and um one answer could be well if it is if the delays are small you would only notice them if there's an explicit process in your head that calls for those things and that's really a very important take-home message no matter what the neural correlate of conscience is that what you can only be conscious of those things if you have an an explicit representation somewhere in the head that codes for them if there's no such explicit representation you're not going to be conscious of it and because there is no homunculus we have to get rid of the homunculus there is no homunculus so it's not like you know it's not like of course if i'm gone i look down at my brain and i can see oh there is the more the the ncc for color and there's the ncc for motion and they seem to be delayed for 80 milliseconds okay but if there is no such if there is no such entities uh no such neural network that explicitly does that computation then i'll miss it unless it you know unless it occurs at such big time scales where i can notice where i can notice it so that would be one explanation why these differences exist they might actually not be um you wouldn't notice them unless you did you know fancy scientific experiments similar things to the blind spot right we don't you know at the location of the retina there's no information but we don't have an explicit representation for that in fact our brain goes to great goes to great pains to compensate for that lack of of representation at that at that location and you know it tries to fill in um yeah for short times yes oh yeah for short times well wait see that again you mean when it's so bright it goes through your eyelids i don't well okay so an after image yeah an after image it becomes it of course it fades gradually but that takes that that takes uh place over long time scales i mean in many seconds when i talk about short times i'm talking about tens of milliseconds very short times you know those short of times you don't see something that changes uh for example if i do this experiment with 20 milliseconds you don't see a red light followed by green light you just see a single light of um of um of somewhat yellow this raises the question that people are testing is called chromatic color fusion so let's say i take a grating and i modulate it red green red green red green do it faster and faster and faster at some point it's called a flicker fusion frequency for color it's like 12 hertz i'm unable i'm unable to follow the individual periods i don't see it as so if i do it slowly let's see half a second red half a second green half a second red i can clearly see red green if i do it at 10 hertz or 12 hertz i don't see the individual colors anymore i just see i see it uh yellow it's me it out yellow and how yellow this depends on you know the exact setting of the red and green yeah i just mentioned that that's this experiment here there's a there's a another question i can ask about the brain can i tell for example this from this this stimulus from this stimulus green red and people have done a lot of these temple uh simultaneously or or temple order judgments uh now i in this case i can because what happens if i so randomly i give you either this stimulus or this stimulus and i ask you which comes first now here what you do you translate this temple order judgment into color judgment because this you're going to see a little bit more greenish and this is going to look a little bit more reddish the time runs from left to right here as usual and so this the red has a little bit more impact on your perception than the green because the green has already decayed a little bit here therefore this looks a little bit more reddish and this looks a little bit more greenish so on at this condition i can i can i can tell the objects the objects apart uh i don't know effin would have done that then you then okay so then you have to so in other words you're asking let me see if i make this a little bit shorter and this a little bit shorter right so they actually look the same well then okay so then the claim is my strong claim is you would not be able to tell it apart but when they do temple order judgments so there are two ways you can do temple order you can either use a more general mechanisms and what you have to use here so here if you equilibrate the it's a good idea if you equilibrate the color you're not going to see the difference you will see the difference once it's more than 50 or a 60 or 80 milliseconds in other words if you have 100 milliseconds red 100 milliseconds green that you can tell from 100 milliseconds green 100 milliseconds red but unless you for example if i give you two tones they've done this a lot with tones now um of course if i give you if i give you two tones and they have very short time interval between them then i hear them inside my head and i can hear more you know the right ear leads you know this leads by one millisecond over the left ear then i hear the tone inside my head i hear it sort of on the right side it's a phase shift and when this one leads a little bit i hear it over here so that's one way i can tell i can tell very short times using using using auditory cues or folks i mean if i just flash two lights now ask you which light comes first if i put them next to each other and i flash a light here five milliseconds then this light here five milliseconds if i put them close to each other you can tell because that translates into motion and if this is first and this you see them moving like this or the other way if i avoid that if i put them for some great distance then you're not able to do that anymore it's another experiment here that was done in the 60s where they had sort of um i guess this must have been done on our oscilloscope because they didn't have you know x y bitmap monitors at the time where they had a ball sort of you know approach one ball approach you know one ball which is stationary and then the after a while the second ball moved and they asked what do people see and if the time between you know it hits the first ball and the second ball moves if it's less than so here you see the first this curve here if it's less than sort of 80 milliseconds or something certainly if it's less than 70 milliseconds you people perceive it as causality that the first ball push you know like billiard balls it you know consists of impulse to the other ball and the ball moves if it's longer than you know 100 or 140 milliseconds you just see it as two totally separate events and if it's somewhere in between you see it like it hits it and then it stores energy and then it begins to move so there are a lot of experiments like this temple uh simultaneity apparent simultaneity that leads you to to the conclusion is yes there are all sorts of specialized processes in the brain that allow us that allow me or animals to tell time and i can do particularly in the auditory domain for so for example we can do auditory localization this is a mark work of marconisci here but also can do this exceedingly well to the resolution of one degree and i do it by comparing temporal cues so i can certainly do that and the the claim has been i can do that ultimately my hardware has a distinguished spikes at the at the microsecond level but but that's a very special purpose process and in general when i don't have such all folks i can use color here shoo when i don't have this general when i don't have the specific process i'm thrown back in a general process and this general process has a crude resolution of 50 to 100 millisecond and the claim of one set of people is and their theories of that that goes back to the 19th century it's never really been proven satisfactory is that sort of you have these that that things that occur was in 50, 60, 80, 100 milliseconds this interval might be a very great deal on the modality on the type of stimulus whether you attend etc but that if things come within an interval you're unable to perceive you're unable to see them as separate you will see things as simultaneously if they occur within one interval and one conclusion there has been which is remains very controversial that at some level the brain sort of is this i don't want to say clark system because it's not a computer clark but the brain is some sort of system where you don't have a continuous evolution of percepts but were percepts sort of in discrete chunks and you see something for short times and then you see it differently so it would be a little bit like a movie where each individual clip isn't fixed time like in a movie but actually can be variable there are lots of theories like that around that remains very controversial partly because it's very difficult to test that at at the physiological level i don't know i'm sure they are you know if you have a motion lesion you'll probably have a lesion in your ability to see fine motion i've never heard of a patient who has a generalized lesion that and i wouldn't expect that because you know each of your different modalities you know this process is a very different one that you use for doing motion which is a different one that you would use here right and then of course we have separate mechanism for doing temporal duration that people are now beginning to study and there's some interesting illusions there right so there's probably a whole bunch of separate mechanisms that they occur a different time scale you know if you close here i say you know tell me when a second has passed or tell me when a minute has passed and people can do that reasonable well and they make sometimes they make mistakes they make you know there's a certain pattern of mistakes so there are other mechanisms that sort of you know clock-like mechanisms that people use that regulates the perception of temporal passage but once they they model here like so often apologies that these are all separate discrete processes while you might think well okay it's all in a computer it all would done you have a central clock and everything is always referred to the central clock it's very different in the brain where you have very separate totally separate mechanisms that do all of these things independently sometimes you know in cooperation sometimes you can have discrepancy between the two this is let me i'm trying to remember the experiment here this is an experiment also by affron so affron was this guy it was he still is i think at the in northern california he does a lot of experiments relating to timing and here he was arguing that there's a minimal perceptual duration sorry minimal perceptual moment this is it's minimal perceptual moment moment not motion sorry that in he did that by by by manipulating two lights the on and off that's a very tricky experiment he manipulated the on and off set of light beams and says and again argue that that perceptually there is a minimal moment so this well what this plot says is the relationship between the physical duration that the stimulus is on let's say on the retina and the perceptual duration how long you perceive it on and the aim is is that even a very brief stimulus even a very brief stimulus that's an impulse response you know think of it in terms of physics you have a physics you have a system you kick it you put in a delta function you get an you know you get an impulse response function out well this is obviously a very nonlinear system and you have one of the properties you have this highly this this nonlinearity in the sense that even very brief stimuli give rise to a to a minimal duration and for this time really you know all these all these different duration will give to stimuli that you cannot tell apart directly now you might be able to tell it indirectly by by brightness for instance so the claim is that if you just see how long the duration of the stimulus is that whether it's this long or this long or this long it always gives rise to the same perceptual duration now of course if i ask people to return the first first choice did i present the stimulus whether this long or this long where people can do because of a a phenomena called Bloch's law essentially if i leave on the stimulus twice as long you've got twice as much signal energy right if you just look at the total number of photons there's twice as many photons and there and for very short times there's a direct linear trade-off between duration and energy so you know i can make a stimulus twice as bright and half as long and you have the same you have the same percept as i've do it twice as you know twice as long for half the brightness so so again one has to be aware it's a very complicated system with different processes and you really have to think hard what system are you querying when you're when you're asking observers to do something but here based on these psychophysical experiments the claim is that even for very short times there's always a minimal duration and then there's some sort of scaling probably not entirely linear it will probably say it's probably going to set it will have some non-linear saturated non-linearities in it so again this tell this i think is very interesting because it suggests something about the underlying dynamics underlying neural network dynamics that have to underlie these ncc's yeah so here the stimulus time is 130 milliseconds and it was 220 milliseconds perceived time but that might really depend totally on the stimulus that really might depend on the stimulus if you use a brighter stimulus it might be shorter i mean i suspect he didn't do it for different he did it for this one stimulus but i suspect it will depend on the stimuli you use yes well okay so no yes it's a very good question that's why these experiments are tricky so you use two different stimuli then ask at what point do they interfere with each other and at what point you see one stimulus changing into another stimulus so you cannot do it directly because again there's no referee you know there's no absolute reference signal of course so you all you know it's a the the trouble is you have this very complicated system you only can measure the output what i say or the buttons i push so of course there could be lots of transformation that go on between the the ncc let's see in my visual brain and the representation by the time i go to motor cortex or by the time i speak in bogus area right they couldn't go there'll be all sorts of further transformation that i don't really control because i you know i can't open up the box so ultimately so that's the one of the limitation of all these methods that if i just look at the output it's very difficult to infer directly what's what's inside the system but can i i'll give you the single most i think important evidence why conscience has involved feedback and this is a phenomenal backward masking so this is the same image you just saw it shows the street scene for one frame whatever the frame is here and then followed by what's called a mask in this case a mask was okay so what you'll see these stimulus and this is a very common term in psychophysics you see an image okay you see some people here and then i leave that on for some time the is i this part of your homework incidentally and then i flash on what's called a mask and the mask sort of can be anything that sort of destroys the idea is that i want to put on something on on the on on the retina that destroys active that destroys the image relevant activity in the retina in in the rest of the brain and so i want to use something that has lots of structures but it's random structures and so the result is there this is now here the stimulus was on for one frame then the mask the next is the image is on for two frames and then the mask that you can you can see a difference right and now it's on for one frame with no mask but that should have been the most visible of all right so we so we can just do that once once again this is called backward masking in the homework there's also forward masking so backward masking is you know time goes like this first are the stimulus and at the mask i can also put the stimulus before that's called forward masking it's not as efficient let's just do that again so here you have the single image and a mask difficulty after a while by the way this is also learning after a while it's certainly easier to see right two frames and no mask now if you imagine if the image is very bright of course you're going to see an after image and then the image de facto although i removed it within 20 milliseconds if it's very bright it might stay burned into your phosphor you know like if i do flash photography right you see the pattern burned into your eyes for many seconds as Irma was just asking so that's one function that's why you want to use the mask because the mask allows you to precisely control how long the stimulus the relevant stimulus actually present on the retina so if you mask it then the stimulus will only be present for this amount of time you know between the onset of the stimulus and the onset of the mask this time this is called ISI interstimulus interval so you can precisely control how long it was present now what does this imply think about it what does imply in terms of the architecture if you have an image and then up to up to 80 to 100 milliseconds later this image can still be rendered invisible uh... by uh... by a mask we can still interfere with the perception of the mask it seems to me that this really implies you have to have some feedback pathway in order to be conscious because if you just have a feed for just think you have a totally pipeline architecture the total feed forward architecture right so you have this series of boxes you know retina alzien v1 v4 it you know motor cortex spinal cord you push that you know you push your fingers and so you always have these two images racing first you have the activity corresponding to the image and then 80 milliseconds later the activity corresponding to the mask or if it's just a feed forward architectures then there shouldn't be any you know there shouldn't be any interference so the fact that up to 80 or 100 milliseconds after the image is removed from the retina this i can put on the second image that can interfere or erase it really suggests to me very strongly it's got to be feedback that there has to be that conscious must involve a feedback so the activity he has to reside for certain times the feedback that happens then some magic happen as in the famous yorker cartoon and then you get bingo you get conscious sensation is this better this better yeah but yes but in because in this case this the image was not the mask what is not part of a psychophysical experiment i just didn't want to answer your question