 Is it good, bad? I don't know, but it's something. People have done some research. Marty Banks at Berkeley, for example, has considered making head-mounted displays that have multiple depths. There are some lenses, for example, that have two different indices of refraction based on a current that you apply through it, based on, say, putting an electric potential across it. And so based on that and a series of lenses like that, you could get a discrete number. If you put three lenses like that together, you could get eight different focal lengths. And then you could track how the eyes are converging to estimate what depth the eyes are trying to focus on and then change the focal length of the optical system based on that and do it all quickly enough to have low latency. So it's possible, but very difficult and probably a long time before we'll have engineered solutions that fix this. So it's something we have to deal with. So if you want to put things very, very close up in virtual reality, I would say be careful. Use that sparingly. Don't make it so that there are things very close up in front of your face. If you put a sign that someone should read, some kind of menu or something embedded in the world, but I would put it at least a meter or two away. Does that make sense? All right, questions? I'm going to now start to talk about what happens to the information or the signals that are absorbed by the photoreceptors. What happens after the photoreceptors? In order to do that, I'm going to show a number of pictures for that. So let's take a look at the eye again very carefully. So again, it's a standard kind of picture as we've seen and all along the retina, we want to zoom in and get a picture like this. So let me zoom in a little bit more in fact. So here are the rods and the cones all the way in the back. Notice that the light is coming in this way and there are some collections of cells here. So there are a bunch of neurons that are part of your eye and they're doing image processing work if you'd like to consider that before it ever gets transmitted back to your visual cortex. So there's a lot of work going on before the signals even leave the eye which I find fascinating. So some basic kinds of filters let's say are doing their work beforehand. There's also something peculiar. Let me zoom in just a little bit more so you see the same kind of thing here. The light is coming in, the photoreceptors are in the back and then there are a bunch of neurons in the way here doing some kind of work, different types of cells. And I will talk a little bit about what these are doing in just a moment. But the interesting thing is that over here on the left this is roughly how our eyes look and on the right are how the eyes of cephalopods look which includes octopus, squid, cuttlefish. And in the right case you see that the photoreceptors are exactly facing the light and in the human case although the shape of the eye here is the same as the shape for cephalopods, it's inverted. So this case is like the way the human eye is and if you decide to do it like that if the photoreceptors are facing let's say the wrong way then you have to get all this wiring it's like a bunch of cables looks just like electrical engineering you have to route the wires somewhere. So they're all routed across the front and then there has to be a hole in the field of photoreceptors so that all the cabling can go run back to your visual cortex eventually. And so because of this there's a hole in the retina and this shows up in this picture here where the optic nerve which sends all the information back to your other parts of the brain needs to occur. So there's the blind spot we talked about that in photoreceptor densities correct right remember where there was a blind spot very easy experiment to find the blind spot this is one of these amazing things that happens with the human brain. So if I just put an X on a piece of paper and also draw a circular blob about the size of a small coin I put them about 12 centimeters apart and then I try this experiment where I close my let's see if I do it right I close my right eye I use my left eye to stare at the X and then I change the depth for me right when I'm here the dot that I've drawn completely disappears and I perceive blue lines to be covering it. It's a very powerful phenomenon so it just vanishes so I was able to put this dot exactly in the place where the optic nerve is and there's this blind spot in the array of photoreceptors that we have it's important to focus on the X because if I rotate the eye to look at this then it just points to fovea so I'm not gonna let the eye cheat like that you have to look at the X and you get it just right make sure you pick the proper eye otherwise you have to do it this way so it's over to the side a little bit right the fovea is straight ahead at least so hooray for cephalopods we have it the wrong way it appears so because of that we have a blind spot but the brain is quick to fix that for us most of the time yeah why do you think we have a one like that instead of like an octopus what's the purpose of having it the inverted way? I don't know I've seen a little bit of speculation in theory but I think it's an example of a local optimum so in other words evolution got stuck there let's say along some path so it doesn't seem to cause great harm I think having this tiny blind spot I don't think there's a purpose as far as I know so I would not assume that evolution is gonna reach a global optimum every time I believe that based on the conditions it's going to settle in various local optima and maybe that line will become extinct when it's eaten by a better design at some point so if that's the case but it seems like having this blind spot was not enough to cause us to get eaten so it'd be very difficult for a predator to figure out our blind spot and exploit that I suppose but I don't know it's very, very speculative but there seems to be no purpose all right, so we have this and we have all of these structures in between until eventually the light hits the photoreceptors in the back and so a lot of people study the details of this retinal circuitry I'm not going to drag you through the details of that but we go from the photoreceptors the rods and the cones and eventually end up at the ganglion cells where the axons or the output part of the ganglion cells are connected they form the optic nerve and connect to what's called the LGN in the brain which is kind of a central switching station and eventually signals are sent onward to the visual cortex and so I want to talk about a little bit of what's going on inside here but not drag you through lots of details people make very complex electrochemical circuit models of what happens from the rods and cones into these cells such as if I go back the first step are these bipolar cells that's one of the main kinds of them so what exactly are these doing and people form very complex circuit models of this which reminds me of reverse engineering right as I said so imagine if you're electrical engineers you build circuits you analyze them here you're given some kind of circuit that might as well have been designed by aliens and you have to put sensors in try to figure out what it's doing during its normal operation so you can speculate what these pieces are doing and there's still a lot that we don't understand in fact we probably the parts we don't understand are probably greater than the parts that we do understand one thing is very clear it seems there are around a hundred million photoreceptors and there are only around a million ganglion cells so some kind of compression is happening right that's one thing that's very clear so people think that there's some kind of funneling that's occurring or some kind of filtering that's occurring each one of the intermediate cells has a receptive field it gets information from multiple photoreceptors in a neighborhood around it and it could be mixtures of rods and cones they do some kind of detection let's say and then they pass that on further so that by the time information goes into a ganglion cell through its dendrites and the cell does some kind of processing and then it sends information along the optical nerve it may be taking into account on average about a hundred photoreceptors right or more could be quite a bit more because they're not necessarily single photoreceptor is not necessarily sending its outputs in one direction only right so if I go back to these different types here there's three different kinds of cells in the bipolar cell layer I should just say a little bit about it so there are often on bipolar cells the the role of these is to detect a change in photoreceptor activity so the on cell will be fired if the photoreceptor changes from being hardly receiving any photons to suddenly receiving a lot of photons and then the off bipolar cell detects in the other direction like it's turning off right so the change is what's being noticed there there are horizontal cells as you see in the picture here as well the horizontal cells connect information output from multiple bipolar cells together and they're looking at lateral information spatial lateral information such as uh... there's been a change here but not here that would be called a lateral inhibition there's some kind of place where there's a change in the change across space as a change over time but but but it doesn't occur spatially uh... so that's interesting they're also at amacrine cells and uh... people are not really sure what those are good for those are still a lot of speculation so very difficult to figure out what's going on inside of here so when we get to the ganglion cells there are three different kinds uh... called midget parasol and and small by and by stratified cells are and uh... the midget cells are there's seventy to eighty percent of the ganglion cells are called a midget cells and these are mainly used for photopic photopic vision uh... parasol are used for both uh... photopic and scotopic and by stratified cells are mainly for photopic as well and they have different functions of the midget cells also using this opponent see kind of idea which is looking for changes inside of a spatial region let me give you some examples of that so but by the time you get through the ganglion cells they're mainly generating something that one might call a neural image which is on the right the original image that we might see uh... which may pass into the cornea they look like the left by the time it's communicated through the optic nerve it may only be like something like edge detectors and things like that right so the kind of primitives that you're familiar with in computer vision so for example is one of the most common types of ganglion cells called on-center off-surround or off-center on-surround where they will look for distinct patterns like this so how this might look in terms of photoreceptor arrays there's groups of photoreceptors there's a receptive field that'll be firing so for example uh... here they may be very simple patterns that correspond to it is a photoreceptor on red photoreceptor on green uh... if i have a very complicated pattern together let's say this uh... look at this uh... this case there may be some greens uh... surrounding some reds and uh... they have the ability to detect uh... changes spatially with regard to those colors so here's a very uh... simple example of these on-center off-center cells so it may be the case that this shows the firing of the output or uh... axon of a neuron it may be the case that and by the way the firing of them ends up being a pulse train and uh... the higher the frequency of pulses the more active the the neuron seems to be so it's it's getting excited let's say when there's a bunch of blips or pulses and when there's very few pulses and it's not very excited anymore so that's how the signals are communicated uh... so this particular type of cell will get most excited let's say in the second case here so this is what a ganglion cell is doing uh... believe it's example of midget cell and it has a uh... a very heavy firing occurring here because it's detected activity in some small region but surrounding that there's no activity right and this dashed line out here corresponds to its overall field of view so everything inside of this disk it can detect information with regard to and there's no stimulus let's say are low stimulus in the uh... low stimulation in the outer ring part but high stimulation in the interior part but then as the stimulation region gets larger it goes back to not firing again so it only likes case where it only gets excited about this case where uh... there's a small region firing but not too small because then it goes down again and uh... there's an outer ring where there's there's there's low stimulation so that's very interesting so it's detecting very specific kinds of patterns after the information goes through the optic nerve it goes into this area uh... that's part of the thalamus which is very very low level uh... with regard to brain functions i should point this out that uh... one thing that's central to these visual pathways is hierarchical processing like i'll just write that to make sure we get that hierarchical processing right the raw data comes into the photoreceptors it goes through these intermediate cells uh... that we talked about the bipolar cells the amocrine cells and the horizontal cells doing low level processing then it gets up to the ganglion cells that have a larger receptive field for each one of them and then it goes through the optic nerve and by the time it gets to the visual cortex notice there's a criss crossing performed as well as is seen in the picture there by the time it gets to the visual cortex when you think about the number of photoreceptors involved in firing the neurons in the visual cortex it becomes a very large number and the overall spatial field ends up being very large for what's being considered in the visual cortex uh... this is another picture of this as well people have done experiments where you show particular stimulation to the eye and then you can do what's called a single unit recording you can measure how neurons are firing based on the visual stimulus and you can get an idea of how high level the detectors are let's say the neurons that are doing the detection in the visual cortex for example you can take a stimulus such as a rectangular bar this has been done on on uh... primates many animals as well very large variety of kinds of studies this is a very simple one where you just take a bar and you uh... start rotating it and you can find a neuron that fires only in a certain direction right so for example maybe i have my notebook here and i start turning it around and maybe the neurons will be firing when it's right side up there be one neuron that likes it when it's right side up and ready for reading right or the bottle the same way right so this orientation is quite important other orientations make us a little nervous when we know this is water so there's reasons for having orientation detectors these are some kind of plots where people have done a study for uh... cells that uh... you know some some detect orientation some do not so for example uh... this is in polar coordinates this cell does not seem to respond very much to particular orientations this one does but it has a symmetry with regard to upside down and right side up so it doesn't distinguish between those this one has a particular direction that it prefers right so so there are different cells for different kinds of cases in the visual cortex eventually we could get to higher and higher level parts of the visual cortex it gets harder and harder to measure these but we have regions in our uh... in our visual cortex that correspond to higher and higher level kinds of concepts faces houses places things like that and one of the big ways to to divide up information or processing capabilities inside of the visual cortex is to divide it between uh... where and what right so some parts of the visual cortex are doing classification what am i looking at and parts are also trying to determine where exactly is that in the scene right it may also be then combining with information about your place i mentioned place cells in the very first lecture so there may be combinations and connections between all of these things that come together to give you a coherent view of the world right any questions about that so that's my uh... little bit of neuroscience going through this hierarchical processing some not giving all the details but it gives you some kind of view of what's going on