 Hello, welcome back. In the last lecture we finished with the audio component of virtual reality and then I gave a little bit of an overview about interfaces for VR. And so I gave you 5 categories which I will go into detail on in this lecture and the next one. First one was locomotion, second one is manipulation, third one is system control, fourth one is social or social interaction and I made a fifth category which I called other which corresponded to specialized kinds of interfaces and methods for one particular kind of application. Whereas, the first 4 categories correspond to fundamentals these may show up in category 5 as you develop particular systems, but you may also find very unique issues. For example, in category 5 maybe you want to develop software or write a textbook, maybe you want to play some kind of a hand to hand combat game with your friends with some kind of actual interaction at a physical level. So, so different interfaces would arise obviously for problems such as that they are very specialized and yet some of the general principles may still apply. So, let us go into locomotion. So, locomotion which I will call traveling in this alternate world that we are making could be virtuals could be some kind of capture of the real world and in order to understand locomotion I think we should make 2 different extremes and reason about them. So, one of them I will call all physical and the other extreme I will call all virtual and then we will have this kind of spectrum between them. Let us go to the all physical case first. So, in the all physical case suppose we are in a cave like environment and we can walk around and there is no other way to induce a motion beyond the actual physical walking that you are doing and the moving of your head to change the view point and that is all or if you put on a head mount to display suppose you are in an entire and an entire cleared room in the physical world and you can walk around you can look at things you can get down on the floor and lie down and look up to anything you want all of the physical motions are being captured and the view point that you observe everything from is being tracked and transformed into the alternate world. So, that is there is no mismatch of any kind right. So, that is a very nice extreme that is the most comfortable case assuming there is no latencies or other kinds of tracking artifacts and display artifacts this is the most comfortable case. So, let us say walk around an open space if you would like to walk around in an enormous open space and you will need a lot of physical space for this. So, it is not always very practical. Let us go over to the other extreme and look at the all virtual traveling. So, if you are playing for example, a first person shooter game or some other kind of game that is first person perspective on a screen then all of the motions you see are virtual right. You grab on to your controller and you move your character around through some kind of virtual world, but there is no tracking of your actual motion of your head right. So, the entire thing is virtual. So, all looking I mean in a sense that when you change your look at you have to use a controller for that too right you grab on to a controller and you may control looking up or down or left and right using the controller. So, all looking or generation of look at and walking is done by the controller. So, it does not correspond to tracking. These two extremes make sense. So, this one generally is the most comfortable in virtual reality. The all virtual case may be comfortable on a small screen and if you sit in front of a large screen some people do complain of discomfort in first person shooter game sitting in front of a very large screen very close to it. And through adaptation people tend to overcome that large majority of people. However, if you put on a virtual reality headset and you feel completely immersed and all of the motions are done by controller. So, that you are not even having head tracking this would be the most uncomfortable case. So, what most people are doing if you want to visit a very large world in the alternate world right a very large alternate world. Then you are in the middle here somewhere where you have on a head mounted display and you are seated in the physical world right. Let me let me grab a chair just to illustrate a bit. So, if I am in the seated case right I may grab on to a controller, but as I move my head around and imagine I am wearing a head mounted display I look around and all of the motions that have to do with the look at based on my human body have been matched using a tracker right. Now, if the tracker does orientation only then it can only handle about this much looking around and then I need to have position tracking as well so that I can do motions like this. But notice that I am seated in a chair and so what happens in virtual reality? Well first of all one issue is that are you seated in a chair in virtual reality or are you standing in virtual reality? Even that mismatch of height causes some confusion and discomfort for people. The next question is if I want to move myself around in virtual reality well I could go forwards and backwards let us say. So, I grab on to a controller and I move forwards and backwards and so I see the motion of that. There is an interesting question about what happens when I am turning. So, I could grab on to a controller and press some buttons and then in the virtual world my head turns right and we have talked about vestibular mismatch in comparison to vision and I will remind you of that a little more later. But one easier alternative is if I am in a rotating chair I could actually just rotate my body like this and then push forward and then I can rotate back some other direction and then push forward right. If I did that then I am using one more degree of freedom from the physical world I am actually doing the yaw correctly. But the mismatch then corresponds only to the additional components the pitch and the roll and of course the translation or positional changes. So, this by rotating in the chair it is more comfortable for the for the user because it involves less of this rotational vexion may eliminate the rotational vexion all together if the tracking and everything else works correctly. So, that is interesting however, not everyone is going to be sitting in a rotating chair and if you rotate too much if you have a cable system then you become tangled up in the cables. So, these are the interesting kinds of trade offs that we have when thinking about locomotion. So, over here closer to all physical is you could rotate with a swivel chair or office chair and over here you rotate this is just a yaw component. So, I should put this yaw with controller or buttons on the keyboard, but the keyboard is not at all ideal. So, rotating in the swivel chair that is closer to all physical therefore, I claim that it is more comfortable. However, as I said it is more complicated to implement in the physical world. Somewhere over here for walking around you might want to make a treadmill perhaps it is a one dimensional treadmill perhaps it is some kind of omnidirectional treadmill you could make a very expensive and complicated mechanism for walking in any direction horizontally. You could be walking inside of a giant spherical ball and feel very much like a hamster right. So, lots of things that could be done here perhaps you have a bicycle. So, you can feel like you are riding around in a virtual city. So, these are not too bad there is some motion going on that corresponds to what you would be doing in the real world if you were in fact, moving, but your head is still staying roughly in the same place. But it is causing I would say a lot of vestibular signals to be generated and a lot of physical body motions that are familiar in the real world. So, I would suspect that it becomes more comfortable because of that. But you know this is also more expensive of a setup more difficult to get working and so it is a bit more of a challenge, but it is further down to the side of all physical which is generally more comfortable when implemented correctly. There is one beautiful trick which it which you can do if you have a large enough physical space to walk around in and assuming we do not have cables that limit us. So, you could be using a purely mobile headset and it is called redirected walking. This is due to Rozaque, Cone and Witten from University of North Carolina 2001. The idea is that you provide visual stimuli to trick the user into walking in circles even though they think they are walking straight. So, it turns out that as we integrate our own internal gyro which is the vestibular organ as we integrate the data from that it is possible to fool your brain. So, that you think that you are going straight over a long distance, but you are actually curving. Now if you make the curvature amount too much so that in VR it looks like I am walking straight, but maybe I am walking in only a 3 meter radius you will figure that out it turns out. So, there is an interesting question of what is the minimum radius. So, just to keep this clear in the real world the character may be tricked into walking in circles all the while they are trapped inside of a large box. So, you may have a bounded space here we always have a bounded space in the real physical world. So, they are trapped inside of here, but as far as their thinking goes if they imagine themselves they think they are just walking perfectly straight the whole time that seem all right. So, you just keep introducing artificial yaw rotations to guide them. There are interesting challenges as far as you know you cannot ensure that they will not hit the boundary right. They may reject the visual stimuli and keep trying to fight against it maybe they are just bad luck and they may end up hitting the boundary anyway and you will still have to do something. But generally speaking it is possible to guide people into circles like this over long periods of time and experiments have been done showing that. And if you want to make an enormous space for this, but make it let us say as small as you can get away with and still have this effect work reliably. Researchers have said that the radius should be greater than about 22 meters. So, if you have a large space. So, if you were to walk around outside say in an enormous cricket field right then you would have enough space and then you could walk around in an entire virtual city right. So, you can walk around and explore an entire virtual Chennai let us say until you are just you collapse with exhaustion and you will be maybe convinced in your brain that you have been walking all over the place and exploring right. And you can have a map with you and it can show you where you are going. And you do all of this in a very comfortable way while in fact you are just walking around in some physical space and you have everything tracked. Of course, you need to have your tracking system extended to the entire space right to make this work. If you are outdoors you may be able to get some positional tracking information from GPS we will not be very accurate you will have to figure out what to do about that. Questions about this? So, I want to talk a little bit about what this means in terms of the transformations that we apply because we have gone through the chain of transformations before. And so, we have to add a locomotion component to that how to apply controller based locomotion. When you may recall the chain of transformations T viewport T canonical view T I which we broke into two components for left and right eyes in the stereo case, but I am not going to worry about that today and T rigid body right. So, these are the transforms that we did before worrying about rendering just trying to make the geometric models end up in the right place. So, they end up being in the right place on the screen corresponding to the view point of the eye. So, if we have locomotion we are going to be artificially changing the look at of the eye. So, recall consider or recall how T I is formed from the look at which I covered a few lectures ago where we had these various direction vectors eventually let us see we had a position which I called E the eye location and we had some coordinate system I believe I labeled as x hat y hat and z hat. So, we have these components that corresponded to which direction you are looking which way is up and the location of the eye. So, in the case of all physical when we are at that extreme then T I is generated entirely from tracking right. So, the all physical case T I obtained by tracking in the all virtual case I mean virtual being just a locomotion part. So, if the locomotion part is entirely virtual then T I is entirely determined by the controller operation right. So, you may be press buttons move knobs on the controller. However, whatever kind of controller you might have you do some kind of interaction with it and then from that T I is transformed. Let me give you a simple example that would be common for a head mounted display imagine this is a top down view of the world right. So, the alternate world that we are generating. So, this is the world frame feel like showing how things look in the world frame and then inside of here I like to imagine that we are riding on a kind of virtual cart right a virtual platform and this platform is being transformed from the physical world while I am sitting in my chair into the virtual world and then when I get to use the controller to move I am just controlling this platform that is how I like to imagine it. So, here is a platform it is like a robot or a cart and. So, this one will have its own body frame right. So, imagine this is a vehicle frame just an example of a body frame will call this kind of virtual cart a vehicle and then somewhere with respect to this vehicle's frame I have my eye location and direction of looking. So, I want to have everything referenced with respect to this vehicle frame in terms of where my eye position orientation is rather than with respect to the world and then I will also consider how to place the vehicle into the world and I have these two transformations now that come together to replace what was once just T I. So, I will do it like this I will say I have I have two transformations now I have T vehicle which places the vehicle in the desired position and orientation you only need those components just yaw rotation and xz right it is the horizontal plane xz coordinates for placing the vehicle into the world assuming that the vehicle is driving on level ground if it is driving on a terrain then you still only need two coordinates if it is flying through space right if you have some kind of virtual spaceship then you need more degrees of freedom for that, but nevertheless the math at this level looks the same. So, we have this we have T vehicle that we apply. So, we can apply this directly after the rigid body transform. So, we put that here and then we have the transformation that corresponds to the tracking part. So, which part is being tracked that transformation would be applied next to place the virtual I in the appropriate location with respect to the vehicle. And now you might remember that we formed T I by taking the inverse of the rigid body transform that places the I right remember that we had to we had to do the inverse because it corresponds to how the world looks with respect to the I rather than how how it would be in the world if we were to look at the I and see where it is placed right. So, it is actually from the perspective of the I. So, it is inverted. So, T I is actually um equal to the inverse of this and you might remember that when we did the inverse I will just take the inverse here you might remember that when we take the inverse then we swap the order. So, T so we just get these two transforms now that are replacing T I questions about that. Now, there are some interesting issues that come into play here um one of them I will show I will illustrate again with this chair it is good to have a prop. So, if I initialize my system and I grab on to the controller and then I decide to rotate. So, I decide to rotate my cart right or vehicle the question is what should the center of rotation be for that? That is a very interesting question because I might not even be aware that I am on some kind of virtual vehicle um what should the center of rotation be? So, one possibility is to make the center of rotation be wherever your cyclopean eye is. So, then when you grab on to the controller and you do a rotate you rotate about the point between your pupils. Um now what if I am leaning like this all of a sudden? Would it make more sense to rotate about the center of the chair? So, that I feel like I am going like this when I grab on to the controller or would it still make sense to rotate about this point? What would be more comfortable? Or should I just continue to rotate about whatever point I was rotating about with respect to the world whenever I started the system right in the physical world? Um should I just keep rotating about that point? So, there are several choices for the center of rotation and um well it is a great source of debate. I think um I found that um various people um will have varying opinions about what is the best and most comfortable way. So, um I am trying to I think I think I think I think my favorite was um having it be the center of the chair. So, when you are leaning it feels like you are rotating like this, but um I cannot even recall anymore. So, it is um which one was my favorite? And remember if you have a um a vehicle moving in the plane you get um XZ and yaw coordinates if you have a vehicle that you perceive to be flying through space then this T vehicle transformation may include a full 6 degrees of freedom right. It can have the entire yaw pitch roll which you may want to represent as a quaternion. I would recommend that and then the X, Y and Z uh translational or positional components would be there as well. I want to remind you of the vestibular mismatch problem. We have already talked about it. Let me just be brief here. It becomes very important in the case of locomotion. In fact, locomotion is the primary reason I think why um people are getting um some simulator sickness and virtual reality. This is the primary motivation. People want to locomote move around explore larger worlds than the one that they are sitting in and um even if the headsets are functioning perfectly this vestibular conflict that you get is what is causing the trouble right. So, remember as you as you go forward you get optical flow presented to your eyes that goes outward. When you go backwards it goes inward and we talked about the other cases when you are rotating or moving sideways. So, problem is uh vestibular mismatch or conflict and you may recall that I mentioned the case where if you instantaneously start moving that is actually more comfortable right. If you if you instantaneously go from being stopped to being at a full normal walking speed that is actually more comfortable than gradually ramping up um because when you gradually ramp up you may have for example, 2 or 3 seconds of vestibular mismatch because your eyes are perceiving an acceleration, but your vestibular organ knows that you do not have that right. Then we said that just doing a quick impulse is better is more comfortable. You can do the same thing for orientation um. So, you could you can make it so that every time you tap a button on the controller for example, you may rotate by 5 or 10 degrees rather than doing a gradual circular motion. And this also seems to help with um this vestibular um mismatch. Now, if every time you tap the controller you rotate 90 degrees you may have a lot of confusion about which way you are facing. You get a kind of disorientation from that um. If you every time you tap the controller you only move 2 degrees then you end up having to can tap the controller many many times really fast. In which case you will experience vexion again you will have the rotational vexion. So, um so there is a kind of balance you have to strike between having them be very small motions and being very large motions they each have an unfortunate side effect. And whether or not some number in between ends up working really well for everyone I think it remains to be seen. So, in the 10 degree range 10 to 15 degrees perhaps that is a good number to try and every time you hit the button you rotate 10 degrees or so. And that is still not going to be as comfortable as sitting in the chair and rotating your body in the physical world if you have a rotating chair and do not get tangled up in the cables. So, you understand that in the case of circular motion even at a constant angular velocity there is vestibular conflict right. Does that make sense right you may remember that let me just write it quickly um so rotation or circular virtual motion. So, suppose this is the radius vector um in the virtual world or alternate world I am moving along in a circle at constant angular velocity omega and from basic uh physics the linear velocity is equal to omega cross r the omega component is along the z axis if you write it as a 3 dimensional vector it has one um outward component yes maybe it is not the z axis here maybe the y axis because of the axes that we have chosen to use I am in physics and in most of engineering outward be the z axis. So, um so it is the outward axis to keep avoiding confusion. So, this would give you a linear velocity vector v that seems appropriate because as this if you are on a merry go round spinning around this is the instantaneous direction you are moving you are about to be hurled in that direction right um except for um the forces of that are due to accelerations which um acceleration just using basic formulas from physics is omega cross v which is then equal to omega cross omega cross r which gives you a component like this of acceleration right. So, inward so there is an acceleration component and um if you present visual stimulation that convinces your brain that you are in fact rotating in a circular arc like this at constant angular velocity then you are perceiving an acceleration again with your eyes nonzero and um your vestibular organ knows that you are not not rotating in this way and not does not perceive this acceleration. So, you have a mismatch. So, in the straight walking case if you are walking at constant velocity there is no mismatch right there are no accelerations going on. But in the case of turning just be very careful this is a very uncomfortable case and um I think it ends up being one ends up being effectively the worst case um these kind of rotations. So, we need to do something better um to handle this difficulty and I will say one final thing um even if you are moving along at constant velocity I still do not think it is completely comfortable. So, you start looking around in different directions you you see optical flow the system may be working perfectly or nearly perfectly I still sometimes experience discomfort and I am not completely sure why I think it is left for researchers to figure that out perhaps you will figure out why why that is happening um I do not think there is any case in the real world where we are moving along truly at constant velocity with no additional forces right even if you are on a train or in a car wherever you might be doing um coasting on along on a bicycle there are always more vibrations more motions going on more stimulation to the vestibular organ along different directions and to have perfectly smooth straight motions with no other stimulation going on is I think odd it is a bit unusual and I find that uncomfortable in virtual reality myself so even at constant speed so just something to pay attention to however we know that when there is a mismatch it leads to discomfort for large fraction of people if there is no mismatch may still cause some discomfort for some people and we are trying to figure out why all right um so in response to this um what you should be thinking about in the design of a virtual reality system is the following so is continuous self-motion important or critical right so for the application that you are building the experience that you are making is it important that you drag your virtual self through all the gradual motions that we would be doing in the real world right what if I can just point somewhere where I would like to go perhaps on some kind of map that pops up right or even if I'm in a big room I can just point and click and say I want to be there now and then all of a sudden we're teleported there if you could do that in the real world you might choose that as an as an option right you become lazy very quickly of course but it's nothing wrong with giving people superhuman powers in a in a virtual reality or alternate world experience um there are interesting questions about how real does it feel when you're able to do that my guess is that with a very little effort you'll be training your body to convince yourself that you have these superhuman powers like being able to teleport to another location easily so that when you come back to the real world you'll feel sad and frustrated that you don't have those powers anymore so that's what I think will happen but but which ones of these abilities become very natural and which ones will just stay unnatural it's very hard to say very hard to speculate about that um so just something to pay attention to you can do uh what I call teleporting just point and click where you'd like to go um if you have a virtual concert you're at a virtual you're at a um sporting event maybe you're in a virtual movie theater could be watching a live sporting event perhaps and there may be cameras located at different places inside of an actual stadium do you want to try to do all of the intermediate motions or would you rather just have a seating chart pop up and then you just click where you want to sit very often when you buy tickets for a concert you have a seating chart anyway so you might as well just have it said as soon as you click and buy the ticket you just appear there right and you're in the right place so you may decide um select a seat from a chart or map right um perhaps you want to navigate in a city and you have panoramas that have been captured like in the case of google street view you may have a bunch of panoramas of a city you don't even have intermediate video that will transition you from place to place people may eventually make all these things and connect everything together but while you don't have intermediate video why not just pop up a little map and you can see where the different places are you can stand and you just click where you want to go make sense so so even if you don't have the data you may it may it may guide you towards a more comfortable experience because standing and looking at a panorama is very comfortable because it does not have this kind of vestibular mismatch there's one interesting general topic called wayfinding and this goes back to what I mentioned in the very first lecture um when when we as humans or um other organisms um explore physical spaces we we end up developing place cells um in in our brains part of the neural structure we start allocating neurons towards particular places and they fire the neurons fire whenever we um arrive back in those places again so there's an interesting question of if you start doing these kinds of tricks of teleporting maybe using a map to get around can you still do wayfinding if you if you were to explore a virtual city suppose we picked um Washington DC and you want to um go exploring that um maybe you've never been there before and you start looking at panoramas you have some kind of map and you start moving around then I take you to the real Washington DC will you know where you're going will you have learned that or will it be a completely different experience and it doesn't really match so so that's something I wonder right is how how far can you go and generally if you're going to explore a large space like a very large building or a big city in virtual reality um after doing some of these kinds of tricks to reduce simulator sickness can you find your way easily or do you tend to get lost right now some people tend to get lost in the real world very easily right and um in the virtual world maybe they're less likely to get lost maybe they're more likely to get lost at least it's interesting to ask the question of under what conditions are is the performance comparable between the real world and this kind of virtual world of exploration using some specialized techniques like teleporting using maps and and such right so our abilities to use a map to navigate in a real city vary from person to person right questions about this