 Ok ready for the next part um we finished last time with an abstract view where I showed you how when you can take the physical world and in the physical world place a display which is connected to an alternate world generator that renders to the display and then that provides the appropriate stimuli for whichever sense you are taking control of and then this in some sense fools your brain into believing that you are in some alternate world right. So I drew that and we had a very good question about by display do I mean visual display and I said no I mean um not necessarily visual display but including that and it could be an audio display could be a um a tactile display could be many one of many different kinds of displays. So I want to talk about um displays and rendering and remember this is just a bird's eye view I am just giving you a high level overview in the rest of this course in the next two weeks we are going to go into more details on these things I just wanted you to be able to see everything in one shot. So that is what this lecture will be the rest of the bird's eye view. So let us think about um the audio part and hearing right so audio and hearing. So the display is um most commonly called a speaker right so we have a speaker right letting out some sound right. So that is perfectly fine when we are trying to use a word that is common for all the senses we will call it a display and rendering produces the appropriate signal that then gets converted into sound pressure waves by the speaker which is the display in this case right. So rendering um produces the appropriate I do not know whether it is called the audio signal or the sound pressure waves depending on where you are at exactly but ultimately um produces the signal let us say to convert to air pressure waves I should say ok I am going to erase this announcement for a minute again it is very important to connect on a piazza for the class forum. So follow this link and register for that. I want to think about two different kinds of systems here for audio and then we will talk about the visual part. So compare um two audio systems that you should be familiar with. So in one of them suppose you are sitting in a movie theater right so I will draw some wonderful art here um so you are sitting in a movie theater and you have your ears you can turn your head however you like when you generally do not leave your seat let us say and then all around there are speakers of some kind which are audio displays um let me put a bunch around here and um in fact if I put it like this I could put labels on these I guess like there is a front left and front right and front center put a couple of subwoofers here some middle left and right speakers some rear left and right speakers and then you have what is called 7.2 surround sound right so one particular uh technology standard by putting enough speakers around. Let us compare that to wearing earphones may also call them headphones but I am using the term ear to make it very clear that we are just talking about the audio part because this is kind of like a headphone too when you wear a head mounted display for the vision part. So you have earphones so now you are still there with the um of your ears but you have some kind of I will exaggerate the size of these these will look like earphones from the 1970s and or it may be a very expensive pair let us say and you have this connected together somehow. So now when you turn your head the earphones follow your ears directly and when you are here and you turn your head the location of these uh displays is changing relative to your head right did you ever think about that difference right it is happening all the time for you right something fundamentally different is going on between these two and people do not seem to make um a big deal out of it right um also pay attention to the fact that here in 7.2 surround when you are at the theater the display is far away here the display is very close right. So now I want to think about virtual reality systems for the visual part we can argue about whether or not this is virtual reality for audio but it is just a different sense. So let me think about it for the um for the visual part I will use reuse this space here. So I told you about two different kinds of virtual reality systems in the last lecture one of them was a cave type system and the other one is a head mounted display I think from this drawing over here it should be pretty obvious which would be which if I want to make a perfect correspondence. So let us suppose again I am here I guess I am not wearing earphones in this case I am just have my eyes looking around and I guess I am not emphasizing the audio part and I have some kind of projection systems here that are projecting onto screens all around and I guess I do not have to draw all of these but there is at least three of them you could put four or six you could have projectors all around you could be inside of a cube if you want. So this would be a cave system again this should in terms of the way it is working on your body should be very similar to 7.2 surround correct right and again the displays could be very far away now there is something more that could go on here which is do I want to change the view point based on where my head is right. So it could be that as I move around I want to change what is shown on these screens to be dependent on where your head is and where you are looking that you would need for stereo or just simply to change the perspective so that when things are very close it moves so that you know you walk up to something and it moves appropriately. So you may need to do that interestingly enough I guess you could do the same thing in the theater right you could have the audio custom tune just for you so if you decide to walk around in the theater the audio changes that goes into the displays right has anyone ever done that for you in the theater no right but it could so it is very interesting to compare these two. The other one we are making a perfect comparison here would be a what is called a head mounted display I will just write HMD head mounted display. Another word for that is iPhones right turns out Jaren Lanier used that term for his product with these goggles back in the 1980s so it is a different kind of iPhone than the one you know today right. So it makes perfect sense if we compare these two you have earphones here iPhones here they are both different kinds of headphones and I guess you could call both of them a head mounted display right even if you put on these earphones here you could call that a head mounted display where the display corresponds to speakers right all right. So I guess in this case I will draw my head again and of course when you wear a head mounted display you have the usual problem of there is a brick covering your face right so you look a little bit ridiculous to everyone else because you appear to be wearing some kind of blindfold right so that is how it looks in the head mounted display case but you know this looks slightly ridiculous but we have become accustomed to that. You cannot hear what is going on around you here you cannot see what is going on around you if you are wearing a virtual reality headset that is not designed for augmented reality if it were designed for augmented reality then it might look just like a pair of glasses right and then it has extra information appearing on it all right. So what is interesting is to make this work do you need to track your head to figure out which way you are looking to figure out what to put on the display. If you choose not to then whatever you see is going to look like it is fixed to your head right. So if you want to be sitting in a virtual movie theater watching a virtual movie it will look like the entire movie theater is glued to your head right it moves effortlessly of course because the physics is different there but that is how you will perceive it because if everything is moved to your everything moves exactly with your head. Interestingly that is exactly what is happening with headphones as well right if you turn your head while wearing sorry earphones if you turn your head while wearing earphones you should perceive it as everything moving exactly with your ears. Why does anybody care about this right we have been using earphones for a very long time nobody worries about head tracking here yet they are now now the head tracking quite easy to do but here everyone became obsessed with head tracking it became required to make it work anybody know why strange isn't it we are just comparing these two senses nobody cared about head tracking on this side on the audio part. Why is it seem to be necessary on the visual side I am sorry the direction but it is the same in both direction let us think about this I guess so I mean ultimately the direction that the sound or the visual signal is coming from is important I think in the case of audio especially when you are wearing headphones listening to let us say recorded music where does it sound like the music is coming from sort of like inside of your head is what most people say I talked to some audio engineers and it sounds it sounds in most cases when you put on headphones listening to music close your eyes imagine where the band is at playing the music for example it sounds like it is inside your head and we seem perfectly comfortable with that. So when you are rotating your head it does not seem to make much of a difference it is still inside of your head and we have become accustomed to that the vision sense is much more detailed right we have an entire picture of what is going on around us it is not simply signal processing of a single signal at with higher frequency going into our ear canals it is you know through the power of optics other things will get into you see an entire image and there is so much more detail there that you notice if you turn your head and everything is moving with you versus it looks stationary just like it does in the real world that makes a very big difference. So I think a lot of it is just the level of fidelity with regard to the visual sense in comparison to the audio sense and the fact that in audio we have grown accustomed to just hearing the sound source inside of our head. If you present left and right speakers you will perceive it as being interpolated in between right the sound. So it just ends up interpolating right in between and it feels like the sound is in your head now we have just grown accustomed to that there are tricks you can do to make it sound like it is coming from the outside and then you need to do head tracking and some other tricks things like head related transfer functions I hope to have one lecture where we cover that way I expect to and so we can get into some of those details. So you can take some of these kinds of concepts that came up in virtual rally for vision and apply them back in audio and now thanks to the latest head tracking technology that is easily available it is it is quite reasonable to do this. So even just on the earphone side you can improve this technology considerably. So that is very exciting I think. All right let us also think about I mentioned this already but think about the distance of the display from the sense organ right. So here the display is far away those of you sitting here in the audience you can see the screen does not have anything interesting on it now but you can see the screen and it is far away from you right. If you wear a head mounted display say iPhones then the display is very very close in fact it is so close that we need to use some lenses to put so that your eyes can still nevertheless focus on the screen that is very close. So we are kind of helping your eyes do not have enough optical power to focus on something very very close even if you are a small child. So the lenses provide some help in that and will make the screen appear like it is infinitely far away and filling your field of view completely. So it ends up being presented very very close. Well what happens I guess if the if the if the display is far away then it needs more power right needs to be larger needs more power. If I want to you know think about the size of a movie screen at the theater I guess they use projection technology so that they do not have to make a gigantic screen right out of say OLEDs or LEDs or LCDs or something like that. So they do not have to do that thanks to projection technology but it has to be enormous right. It has to be very loud in the theater because the audio rendering the speakers are far away. If you have earphones and iPhones then it does not have to be as loud and it does not have to be as large it can be very small. So there is some engineering advantages but to make it work you may have to do more work on the head tracking parts right the head tracking may become necessary. So there is some more engineering work to do but as the technology improves it can lead to greater portability and lower cost and generally the amount of signal or power that you need to generate of course is going to be less the closer it is to your actual sense organ correct. This may cause you to want to go further why even bother with putting some display very close to your face why not beam information directly onto your retina right the light information directly on your retina. So there are retinal displays that do exactly that you are probably wondering why not go one step further why not we just cut out your eyeball all together and just directly provide stimulation to your optic nerve right. If we can match it exactly that ought to work right. Again there is going to be a lot of to make that work there is going to be a lot of engineering challenges to match it perfectly we do not really understand exactly what goes over the optical nerve but if we did and we could reproduce exactly that signal that would be like having an HDMI connector directly to the visual part of your brain you just feed in the signal and everything is fine. Certainly it is going to require head tracking to make that work and very low latency and lots of other things and unfortunately it would be sort of a one-way trip right we could maybe wire you like that but then it is hard to get you back to normal again once you are done so you will have to be a permanent cyborg at that point. So it is interesting to think about right where exactly is the place where you want to interrupt the stimulus and present the display. So try to get closer to make it more portable and lower energy is needed but the closer you get the harder it gets engineering wise to achieve the more tricks you may have to do. Alright I want to say a little bit more about hardware and in my bird's eye view I have three components. I am going to talk a little bit about hardware which I have already been doing and then I want to talk a little bit about software and then a little bit about perceptual psychology sensation and perception and those will be the three components of my bird's eye view. So just continuing on the hardware I just want to say a few things it will go into much more detail on these things later.