 Welcome back. I am continuing onward with audio for VR. So, we introduced that last lecture. I talked about the human hearing system and gave you some of the basic biology of how our hearing works. I also talked about propagation of sound waves and made some comparisons to light and vision along the way. We talked about auditory localization and I finished up with that. I talked about monoral cues and binaural cues and now I want to talk about the rendering part. So, you have sound sound rendering just like we had rendering for the visual part which we call computer graphics there is an audio part that has many similarities to the visual part. So, I will describe four steps and just give some comments about this as I go along and it will be significantly more brief than it was for the visual case. So, four steps. So, one we have modeling. So, for the visual case and even before we covered vision versus audio or anything we talked about geometric models right. So, we will again have geometric models and have stationary or movable models. You could have movable models you could have a moving object that generates sound or maybe sound will be bouncing off of a moving object. So, walls, obstacles, moving bodies perhaps you have a bird that is flying in VR and generating sounds. When we were modeling objects for computer graphics for rendering in that case we were concerned about the material properties in some cases right. We wanted to know how the light reflects off of the surface of our objects. So, what do we want in this case we want to know how the sound reflects off or absorbs into the objects or maybe transmits through the objects or the diffraction that occurs. So, how does this happen? So, we have acoustic material properties and just like we had light sources we have sound sources right. You could have a point sound source or you could have some kind of parallel wave source maybe an entire vibrating virtual wall that makes parallel waves. So, we could generate those directly or in the case of a point sound source I guess we could make the same statement as we did for light that we never end up with truly parallel waves right. So, we have these propagating spherical wave fronts right. So, we could have point source or a parallel wave source like a vibrating plate we can talk about the loudness of the source just like we talk about the brightness of a light source alright. So, these are the modeling components that we have right just as we had for the visual case. Then we need to think about propagation through this virtual world that we are constructing. So, using the acoustic material properties we want to figure out how is the sound going to propagate and again we have all the things from before I will just write them here reflection diffraction could be 2s there refraction or transmission right. If we are moving right if the avatar is moving through VR we may also get the Doppler effect right or if some object is moving towards us and making a sound. So, that is something that we did not worry about that in the light case right. The red shift that occurs right in in light is useful for tracking the motion of stars in the sky for example right, but it does not end up being enough of a shift to be significant for VR, but here it ends up being important. So, the Doppler effect and overall amount of attenuation. So, waves get weaker as they propagate through the air at a more significant rate than light waves . So, attenuation. So, these are all important types of propagation or factors that are involved in propagation and if we divide up computational approaches we want to think about doing a computational approach. Remember we are making an alternate world generator right that is what we talked about very early on in the course we are making a kind of simulator to propagate this audio that is being constructed in the virtual world. So, there are 2 big alternatives I would say at the highest level we could do it what I would call in a numerical approach versus a combinatorial approach. In the combinatorial case we will use concepts like visibility or ray tracing is that look familiar right. So, the same kinds of concepts that we talked about and considered for a graphical rendering for visual displays the same things can be used for audio rendering. So, we can talk about propagation of sound and we can look at rays of sound as they propagate and reverberate in an environment. So, that is one possibility and you know this is fast and approximate in some kind of way and the question is going to be is it good enough. When we do this in computer graphics remember we looked at different shading models and we talked about how they were not physically realistic, but they seem good enough because they look fine on a screen. Sometimes they look fine in VR sometimes they do not and we have these difficulties to deal with right. People making computational simplifications by doing visibility based methods keeping them very simple not considering all of the propagations simplifying the materials as much as possible making things like the Blinfong shading model which is not physically accurate, but is very very fast computationally. So, the same kinds of choices will exist on the audio side, but it may be even more difficult to tell whether or not you have done it right right. You may hear the sounds, but can you correctly localize them in the same way as you would in the real world? Very difficult to determine that you have to do more difficult kinds of experiments with human subjects determine whether or not you have gotten it right. Whereas, in the case of visual maybe you can just look at it and say yeah looks fine right enough people look at it seems to be fine looks um physically realistic enough and people are happy with it. So, so you can go down this path, but again it becomes difficult to verify that these approximations are correct. You can go the numerical way which is write out partial differential equations you can calculate what is called the solution to the Helmholtz wave equation which I will not write here I will not make this a class on partial differential equations, but um in other words you are doing numerical PDE calculations um using things like a finite element methods and so forth. So, you end up in the regime of partial differential equations you end up with um lattice or grid representations over the entire 3 dimensional space and you are doing some kind of numerical wave front calculations. So, the numerical computations propagate through the space you can imagine that this is very complex. This is equivalent in terms of difficulty to the full global illumination problem that we had for light right. If you want to do very very real physically realistic rendering then you again want to resort back to numerical computations and maybe even do all of the PDE calculations and people do that in computer graphics sometimes to make very highly photorealistic renderings. So, you have the same kinds of choices for audio you can go this direction or you can try to find some kind of combinatorial quick visibility ray tracing kinds of hacks let us say that are hopefully good enough. So, these choices um exist as well this one being very very expensive, but very physically accurate this one hopefully being good enough. I think there is a lot of research and a lot of development to be done in this side here to make um um fast and efficient methods just as was done in computer graphics across the 1970s and 80s when people had very limited hardware and they still nevertheless generated um very nice looking pictures and then eventually videos synthetically. So, we are in the same kind of place here with audio it is not known how much of these models and the acoustic material properties need to be fully specified how much of this information is going to be critical for a good audio experience you will hear something, but will it be reproduced in in the same kind of way as it would in the physical world giving the abilities to localize the source for example as you can in the real world and that could be very important for the type of experience that you are building. As I said in the beginning it depends on the task right what is the experience you are trying to make. So, if you are trying to make a um a horror experience right or maybe some kind of scary experience perhaps where the sounds are coming from in some kind of not very well lit environment might be a an important aspect of that right and so you might want to capture that very well. If you are just making an experience where you are just talking with your friends maybe some amount of localization is important, but maybe not a large amount right. So, depends on the application, but if you want to get these things right you may have to go further down this path unless we develop unless the community develops some very good um very fast methods here for generating synthetic sounds and propagating them in a way that is convincing questions about that. So, that is one step is modeling another step is propagation I want to talk about two more steps. So, three rendering, but one of the things that we have to take into account for rendering and rendering is going to be to determine what output needs to be given to the audio display which we may call speakers right. So, what needs to be presented there? Well one thing we have to do is we have to use the head position and orientation to determine the appropriate um let us say air pressure signal at the right and left ears just like we have to give the right visual information to your virtual right and left eyes in VR you have to figure out where the wave should be hitting your virtual ears in in um right and left ears in virtual reality. So, that means if you are moving your head around this has to be tracked and you have to adjust the sound that goes into your ears based on that. So, what goes into the audio display or the speaker should be dependent on your head position and orientation right. And if you grab on to a controller and move your character that will also change your position orientation in the virtual world even though it is not changing in the physical world the audio should be adjusted accordingly as well if you want to convince your brain that you are in fact moving which may even further contribute to this vestibular mismatch or maybe it helps that problem maybe it overwhelms when you put audio and visual together helps overwhelm the vestibular signal I do not know which one will happen you have to do experiments to see