 So, there is one more part of the bird's eye view I will get through part of this this time and then I will probably finish it next time. So, the final part remember the first part was hardware second part is software the third part of this which will be a big part of the class is the sensation and perception part. So, this is the biological the neuroscience the perceptual psychology part sensation and perception. So, how do the senses work and how are they interpreted by the brain sensation and perception. So, there are a lot of relevant VR questions here. How do we perceive depth and how do we know how far away something is? I think most engineering students would answer by stereo right stereo vision right we have seen you have seen you have done computer vision projects before perhaps where you reconstruct depth from stereo images in a camera. It turns out that a very very large overwhelming amount of depth information comes from monocular vision just from a single eye from the brain's expectation of where everything falls together in the physical world. So, you get a lot of depth information from that they are called monocular depth cues that is something I want to cover. So, in engineering people tend to over emphasize stereo when when the truth is monocular depth cues provide most of what you need and for many applications it may provide everything that you need in which case it simplifies your engineering design considerably. What causes nausea or fatigue? What are what are the root causes of these? What has it do with sensation and perception? The information coming from these sensors you have engineered alternate stimuli now to present to these senses which confuses your brain. Sometimes your brain does not mind sometimes your brain may respond by causing you to want to vomit right. So, what is going on there right how do we understand these things and try to avoid these problems? How do we perceive the source of an audio signal? Where is the sound coming from right? I have noticed on the campus here there are these incredibly loud birds called the Asian Coal right not sure what your local names are for them, but very loud I notice when I step outside I can identify which tree they are in very very closely which is quite interesting when you think about it. We do not have a full imaging system for our ears how are we able to do that to localize the audio source to pinpoint where something is coming from? How much resolution is enough on a display right? How do you answer a question like that? Well you need to take the human body part the human vision or the the organisms vision part into account and this whole idea of presence I gave you a little bit of an idea, but we can get more into that as well. The thing that is very confusing is that the processes that are going on for us all the time in our daily lives we are using perception constantly our brains are doing a lot of work to build a coherent view of the world around us and you know what it is effortless right? It appears effortless to us it is not something you spend a lot of time on you understand where you are going and what you are doing. But if you really stop and think about it carefully you are doing very very complex tasks ones that are very very hard you have around 10 billion neurons in your brain doing a lot of work a huge fraction of those are devoted to vision interpretation and you know the best computers we have the best machine learning algorithms they are getting better these days, but they still cannot compete with the human brain and it is quite impressive and all this work seems to be happening effortlessly for us right? So, we are not even trained to be aware of it. If you do virtual reality you need to be trained to be aware of these things because you are interfering with their operation. So, if you want to be a good VR engineer you have to understand some of the reverse engineering of the human brain and the human vision system and the other senses in order to be a good engineer of virtual reality systems. So, that is why these things come together and why we are motivated to study these. I have already talked about different senses I want to go into a little bit of classification of senses. So, I just want to talk about for each one of our senses what is the sense called, what is the fundamental stimulus, what are the receptors, what is the familiar sensory structure or sense organ and I might not think of might not hurt to think about some kind of engineering equivalent. Again this is just a bird's eye view so, I just want to quickly go through these. So, for vision that is the name of the sense vision, human vision let us say. What is the stimulus, electromagnetic energy so, just think about electromagnetic energy or photons if you like right. So, electromagnetic energy in what is called the visible spectrum why do we call that because we are we consider ourselves to be important we have selected that part of the spectrum because that is what our eyes respond to right call the rest infrared ultraviolet outside of that all right, but it is all electromagnetic energy. On your retina of your eyes you have photoreceptors just like you have pixels on a display that is the output pixels the photoreceptors are like input pixels. So, these are the parts where you sense. Sensory structure well it is a human eye. The engineering equivalent I guess would be a camera right and the way we design cameras very similar to the way our eyes are designed except all these extra movements and things that our eyes do is very complex and interesting. Let us go for another sense so, that was one of them vision the next one is hearing. So, if we go to hearing sense is called hearing what is the stimulus yeah so, air pressure air pressure vibrations or air pressure waves let us say right. So, it can be considered as a kind of mechanical vibration good that you use the word mechanical because the receptors are called mechanoreceptors right. So, we can get into a little bit of the details when I cover audio although I am mainly going to be covering vision of course, the sensory structure is the ear and I guess the engineering equivalent would be a microphone perhaps right we put microphones which are very close and designed to speakers. Let us see if we go through touch right so, the sense of touch in this case the stimulus is tissue deformation right. So, the surface of your body is deforming in some kind of way. We have mechanoreceptors for that we also have thermal receptors. So, part of touch could also be a change in temperature right you feel that something is burning hot or a cold piece of ice on your skin right so, it can also be thermal. So, there is thermal receptors and mechanoreceptors sensory structure skin all over muscular structures and all throughout your skin you have the sensory structure. So, that one is the most distributed across your body right. So, we could break that into further and further finer categories if you like. In the engineering world we have a pressure sensors and force sensors and thermometers right. So, we make all kinds of things the physics is the same either way so, there is usually an engineering equivalent for the things in our bodies. One extremely important one for us and this has a lot to do with simulator sickness is our sense of balance. It turns out that we have inertial measurement units inside of our ears and it is not part of the hearing system it is part of our balance system. So, the sensory structure organ is called the vestibular organs which are mainly located in your inner ears there is a pair of them and the stimulus is accelerations could be due to gravity could be due to linear motion. So, that is the stimulus that we get our accelerations and there are again mechanoreceptors for receiving that information. So, it is one of the most mysterious organs we rely on it every day, but most of you probably have not thought too much about how it works or what exactly it is trying to accomplish other than from its name it is obviously helping us to balance. And finally, I will put these both together taste taste or smell so, in this case the senses taste or smell the stimulus is chemical composition right. So, we have some chemical composition and it turns out in this case we have chemoreceptors right. So, some kind of chemical receptors if there is a change in the pH for example, in acidity you may be able to measure that and send signals to your brain. Sensory structure well these would be in our mouths and nose right and use surface of our tongues inside of your nose. So, you have places where you have these chemoreceptors that can measure these chemical information chemically based information as it propagates through the environment. So, among all of these senses the one I am going to cover the most in this course is vision number 2 is the hearing part and I am going to leave off the rest right, but they all can be considered very important. You can easily spend an entire course on the touch part many fascinating issues there balance is going to be important though too I better put that in here we are going to cover some of that because that becomes extremely important for making comfortable experiences. Let me mention one small thing here and then I will finish up for today so, which is as these senses come together if you are taking control of one of the senses by providing alternative stimulation and then you have neglected the other sense your brain may not become happy about integrating those two sensors now because it is going to start detecting mismatches and that is mismatches might happen at a subconscious level and you may become uncomfortable or sick from that and these are the kind of difficulties that happen. So, the most important for us is vision plus balance when you mix those two together balance are vestibular you have a problem. One of them is called Vection again this is just a high level view what happens in this case is this is the illusion of self-motion have you ever been sitting on a train and the train is stationary and out the window another train starts moving and you get the sensation that you are going backwards, but you are not moving that is an example of Vection. So, your eyes have convinced you that you are moving and accelerating, but your vestibular organ knows that you are not and it is disturbing right it is really unusual. So, in virtual reality you are doing that over and over again every time you grab onto the controller and move your character you are convincing your brain that you are moving, but your vestibular organ your balance organ knows that you are not. So, that is a problem it is called Vection what do we do about it not completely sure I have some suggestions, but a lot of people working on how to overcome this. Another interesting thing I mentioned it is called the vestibulo I mentioned it once already ocular reflex or VOR. This affects our perception of stationarity. So, if I put something in front of my face like this piece of chalk and I go back and forth like this my eyes are counter rotating with respect to the rotation of my head. Perfect one to one correspondence it is a very tight feedback loop that takes less than 10 milliseconds and bypasses my higher level brain functions it is a very tight control system that is been evolved to do this and it affects my perception of stationarity it is very hard to not do it you can try to not do it and just have the chalk look like it is going back and forth instead very very difficult you can put your finger in front go ahead and try this put your finger in front and go like this it will not hurt. So, you know right. So, you are doing this all the time this affects your perception of stationarity you have to get that exactly right in virtual reality. If you have optical distortions or you have latency other things your brain may learn to compensate for that and then when you take the headset off the real world will be messed up for you. Those of you wearing spectacles you will notice that you may notice that when you take them off the world might look like it kind of swims back and forth. Your brain actually has learned a different gain constant for your spectacles or glasses when you wear those due to the magnification of your lenses the counter rotation rate is different than the rotation rate, but your brain learns that it is no problem. So, if you have a bad virtual reality headset your brain might learn to compensate for its flaws, but it is not going to tell you that it is doing that you will just think it looks right because your brain is adapting to it and then after 20 minutes you might have a headache or feel sick or something you know and your brain is trying to compensate for these things pushing the limits. It is better to engineer it correctly and make it comfortable, but these things tend to be invisible to people. So, this is a very important intersection or combination between two different senses is there any which equivalent yeah that is a very good. So, you could provide vestibular stimulation maybe you provide some kind of vibration that distracts the sense enough you could do electrical stimulation of the vestibular organ, but it tends to be painful I don't think there is an easy source, but that is a very natural thing of course just make all the senses become together in harmony and that would be a solution right it is just very difficult to engineer that. Low frequency vibrations and rumbles might help there has been some research on that, but it is not I don't think it is completely conclusive. So, a very good question alright I have a little bit more left on this I don't know if I could either take 10 minutes or wait until tomorrow I don't know what is best should I finish it alright I will just finish it up so that we hit it in one shot. So, a few more things on the bird's eye view and then we get it we get a clean shot into the rest. If you read the first chapter of the book by Mather on sensation and perception you will get all of this and much more detail and I think it will be very helpful for your perspective as you go through this class. So, I just want to introduce a few small concepts here in the remaining 5 to 10 minutes. The subject of psychophysics alright so this is the scientific study of connecting physical stimuli to perceptual phenomena perceptual phenomena. So, your brain reacts and comes up with some conclusion for example, what does it take what do I have to present to your eyes so that you declare that something is red instead of blue or instead of orange alright. What does it take for you to declare that something is bright or dark or loud or sour what does it take for you to declare that something is sour if it is taste right. So, these are really interesting questions right so one of these is clearly a reaction of the human, but it starts with physics that is why it has physics in it has psycho in it you put these 2 together and you design scientific studies for that. So, there is usually a kind of plot in a lot of these works to look something like this you have a probability of detection and then you may have let us say a stimulus parameter here and the plots could look you know varied depending on the experiment, but perhaps it looks something like this the probability should go from say 0 to 1, but maybe for some particular experiment there is like some crossover point where everything should match perfectly here. So, that is some kind of parameter for example, maybe you are changing acidity in some kind of taste experiment so that at some point it becomes it gives you a sour taste let us say right. So, you have a whole bunch of subjects and you get them to start trying you give them these chemical taste tests as you vary your parameter. So, down here nobody would say that it is sour over here when you turn the parameter to some extreme everybody will say it is sour and then there is some kind of in between region where you know around in the in between region some people will say it is sour and some will not right. So, that is the kind of experiments people do in psychophysics they vary they turn knobs and they try it out on people and they see what happens. They may also be measuring response times to stimuli. Sometimes you need 3 or 4 subjects and that is enough sometimes you may need 100 subjects or more sometimes you may have to consider demographics of your subjects could be age gender various things might matter some other kinds of studies it might not matter at all it may have to do with just a common wiring that is in all of our bodies and it does not really matter therefore, you need 3 or 4 subjects and you are done. So, it just depends on a lot of things like this and it is a very scientific rigorously defined subject and there is all kinds of interesting results from that. One of the most interesting general patterns that seems to come up over and over again is called Stevens power law. So, let me go into that Stevens power law this is from around I think 1961 let us say M is the magnitude of the stimulus is the perceived magnitude perceived magnitude this is a pattern that will show up over and over again and it definitely shows up throughout virtual reality. So, there is a kind of mapping looks like this P is equal to some constant C M magnitude to some power x called the exponent. So, C is a constant depends on the units depends on some kind of curve fitting. So, people have done many many experiments with human subjects in many different settings and they end up with plots like if I put M on one axis and P on the other. So, I increase the magnitude of the stimulus and then I measure the perceived magnitude by the person based on what they tell us. There are some cases where it will go like this in which case the exponent is greater than 1 there are some cases where it appears to be perfectly linear and there are some cases where it goes the other way. It is very easy to find examples of this if you look up Stevens power law on the internet you will find lots of examples. So, it goes off in this direction in the case of x less than 1. And so, as an example if you take a look at loudness you ask somebody is this twice as loud or not right you can keep doing experiments like this give them a bunch of signals at let us say a single pure tone at 3000 hertz and then you start varying its amplitude and you try to measure their perception of how much the amplitude is going up right. If you do this the exponent will be 0.67 about 0.67. So, it looks like this. So, it starts getting louder and louder, but it kind of tends to saturate you know it does not really is not perceived as being that much louder interesting right. If you ask someone to estimate how long a line appears to be on a screen it ends up being linear they are pretty good at it that estimate as long as there is no other strange interference or optical illusions they end up estimating it to be twice as much. So, that is interesting if it is really twice longer if you electrically shock somebody then it looks like this a little bit more shock feels a lot more painful. So, so these things all depend that is Stevens power law. Another thing that tends to happen over and over again and this I mentioned already with the vestibular ocular reflex is plasticity and adaptability. So, over time you tend to adapt. So, if there are flaws in your virtual reality system you adapt if something sounds really loud after a long time listening to it it will not sound so loud anymore right. So, all kinds of things happen through adaptations that is just another very very powerful principle which I need to emphasize is adaptation ok. So, one last thing I want to say is I want to I want to present some optical illusions to you just to help emphasize to you how much processing is going on that you might be barely aware of. So, you might have seen some illusions before there are visual illusions there are audio illusions as well they are all kinds of interesting things that happen I am just showing visual illusions because that is the main emphasis of this class. So, let me put some up here you seen this one before. So, I seems how many legs are there I do not know. So, you can present things that look reasonably plausible as long as you do not stare too closely. This one is great which one of these line segments is longer right turns out there the exact same length right. So, the upper line segment here the lower line segment here same length does not look like it. It looks like the lower one is shorter correct. We can play the game like this which one of these line segments is longer right. These are perfectly vertically aligned, but this monocular depth cues that I talked about that gives you the sense that the upper bar here is far away gives you a feeling that it must be longer, but it is not it is the same length right. So, your brain is doing all this work for you and people who work in vision science like their job is to be a scientist and study all day long human vision. They love finding new examples like this that reveal something that is going on when you are doing the reverse engineering of our vision. This one is pretty good. So, the gray square here labeled A and the gray square labeled B are the exact same intensity. Does not it look like A is darker than B? They are exactly the same. You can look up you can grab this image and look exactly at the pixels if you want they are the exact same RGB values. They are exactly the same. It is just a shadow effect that is causing this gray square illusion. So, as your brain tries to integrate the whole picture together it is going to tell you that B must be lighter, but it is not the exact same shade as the A. That one is crazy. Does it look like the wheels are rotating? Well they are not. Do you see motion in that? So your eyes have very very tiny motions called micro-sacods and they are always doing that to refresh your photoreceptors. This illusion appears to confuse that and it gets especially bad when you when you get up closer to it. Maybe I can leave this at the end of the lecture and you can get up closer to it especially bad at the periphery. And finally I will end with this. This is called the haunted swing. This was made over 100 years ago in a festival. A bunch of people get into a swing and then the entire room around them is actually rotating. But they do not know that. So, the people who designed this they took the furniture and they nailed it to the floors. And then they actually spin the room so that people perceive it as they are moving here and sitting upside down, but they are actually not. They just perceive it as that. So, this generates vexion. And guess what? The people got very sick from this. So, this is exactly the kind of thing that is going on. This looks like virtual reality to me in a lot of ways, right? They trick them with exactly these kind of motions, you know, it gradually starts rocking back and forth and then you are convinced the swing is going completely around 360 degrees, but really you are motionless, right? So, again another kind of optical illusion. So, that completes the overview. That is the bird's eye view. I covered the hardware part, the software part and a sensation and perception part and gave you motivation for the for this class. So, the next lecture I will get into the basics of geometric modeling and some of the transformations that we apply. These are the beginning steps of making alternate world generators. So, you can look at chapter 6 of the graphics book by Shirley at all if you want to get started on some of the basics of that if you do not have that math background and things. Thanks.