 I am going to go on to the next topic. So, locomotion was the first of the of the interfaces topic manipulation is the next one. So, I will divide this into a few parts selection. If I say manipulation can sort of add to that manipulation of objects objects. So, selection what object am I interested in manipulating um what are some of the factors in that um. So, factors that become important um distance right. So, how far away is the object from me and um what direction is it is it convenience do I have to reach way back and I cannot rotate easily perhaps um is it extremely far away. So, I need very high precision in whatever I do or do I have to locomote myself all the way over there to go to the object right. So, what are the what are the interesting um what are the challenges here for that related to distance will be size. Of course, if we can go back and talk about perception again depth perception and size perception as you are trying to manipulate something ah clutter. So, that is also complicated in the real world right. If you have ever played one of those um games where you put some money in and there is a claw that comes down and you try to grab a prize right there is intentionally a lot of clutter around and a lot of bad prizes and things that are difficult to grasp. So, that you have to move and and try to pick things up. So, clutter is going to be make it difficult and generally um occlusion perhaps you would like to grab something, but there are other things in the way. Maybe they are just partially in the way and. So, selections one big thing one big part of manipulation after selecting an object um you may determine that you wanted to grab it the next thing is um placing the object back or at least doing some kind of positioning of the object. I guess we could get very complicated and talk about grabbing a bunch of objects at the same time and doing some kind of putting together with your hands and go down we could go down that path I am going to try to keep away from that it is a bit more complex than I want to cover today um. Now, let us just talk about positioning. So, I would like to place the object back. So, we can have a distance and I will put distance and um and direction of the initial and goal configurations here I will just call them placements right. So, how far am I moving the object and where do I have to bring it with respect to myself and one thing that becomes very important is the positioning precision requirement right. So, when I place the object do I have to get everything down and down to the level of sub millimeter accuracy in the virtual world assuming we have perfect one to one correspondence um. Does the object have a basin of attraction right maybe that I just leave the object in some location and the software that you write knows that it needs to snap into a particular location and you are always happy that it snaps into the right place right and there is no confusion with that. You have seen these kind of techniques being used with um just simple line drawing programs right to make the vertices match if you are drawing a nice polygon as a as a closed polygonal curve you may have some setting so that it just snaps the ends of the polygon together perfectly right. So, similar kind of thing will happen here with positioning um is there a basin of attraction that will just bring the object to the right place is that reasonable to do for your application. If so, you should not be burdening the user with performing very precise placements unless that is the purpose of it it could be a game where you try to um see who can position objects the best with the most accuracy. Adding to positioning we could have merged category 2 and 3 here, but I will just emphasize separately rotation. So, the amount of and precision of the required rotation right. So, how accurate do I have to get also um how many degrees of freedom. So, rotational manipulation ends up being one of the more difficult and if you can limit it to being only let us say yaw rotation that is required by the user then the now would be better right. Um when we perform manipulation let us think about um different types of input device categories. So, input device categories I will divide them into two I will say there are metric where the motions are tracked through space and we should think about the degrees of freedom for these. So, it could be the case that you have a um maybe just a simple mouse on a desktop and. So, you have um um 2 degrees of freedom perhaps 3 if you have a more sophisticated mouse and it can handle the rotations as well. Um and um you could also have an air mouse that moves through space right. Maybe it has a 3 degrees of freedom of rotation that are possible may even have 6 degrees of freedom. You could have hand tracking software that is tracking the motions of all of your fingers is it reliable it may or may not be depending on what additional information it is able to use. Um the current state of the art if it is purely based on computer vision techniques um you may not get enough accuracy for the kind of applications that you care about. You may be able to do simple gesturing, but not very precise placement of your fingers. Um maybe you can put on gloves to track your motion as I mentioned gloves before it goes all the way back to um Jaren Lanier and the beginnings of the virtual reality industry in the 1980s. People had head mounted displays and gloves to capture the motions of your hands. So, that you could do um manipulation in a in a in a metric way where you see all of the motions. Um just turning on to a simple knob and turning it back and forth grabbing on and turning a knob back and forth 1 degree of freedom would correspond to metric. 2 I will say the other one is a switch. So, motions induced by pressing buttons or pushing and so forth right. So, a joystick or buttons right where the precise position of them does not matter so much you are just pressing and going. And very often if you have a button to press then you may have some assumed speed of motion associated with that right. So, if I um um um if I want to press a button and have my hand move in some way then at what rate is it moving while I am pressing the button could get very frustrating to control. One general thing that is very interesting and appropriate um important to pay attention to here um note that um both of these are essentially all of these um input devices that we consider have what I call scaling issues over space and or time. And so, we see this already with a computer mouse right there is a speed setting. So, you move your mouse a tiny bit and the pointer on the screen move some amount and people have different preferences for that. But um I just want to point out that um you do not need to have perfect 1 to 1 correspondence. Now if you are going to do some kind of um hand tracking of your motion through space in this metric way then you might want to get that working exactly the same, but you do not necessarily have to. And if you have some small controller that you grab on to or maybe you grab on to an air mouse and you move it you move it around a small amount it may be ok for your virtual hand to be swinging in a large way. However if you have to do very high precision motions if you if that is required by the manipulation then you may become frustrated if um the scale is off by too much just like you may be frustrated with a mouse. If it is set for a very high speed you may move a tiny bit, but then you cannot even click on the buttons anymore. And if you set the mouse for a very slow speed you might get frustrated having to move the mouse and move it again and move it again just to get it across the screen. So, the same kinds of issues show up here. And um one thing that is very interesting here to note in addition to this note is that there is um no vestibular mismatch as long as you perceive this motion as being a motion of your limbs let us say um or some other motion of an object that is just in the world with you as long as you do not perceive it as motion of yourself from your own viewpoint then you can do all kinds of interesting scalings and rescalings. And um you will not experience simulator sickness from that. Now you may initially go well this does not seem like my body, but who knows what you can learn and start to associate as being part of your body right. I do not know about you, but I am riding around on a bicycle I almost feel like it is part of my body you know it feels so natural to go flowing along on the bicycle. So, I what kinds of extensions of yourself will you accept very naturally. This is a very interesting question in general which um kind of ties these together which is um what are the learnable motor programs. So, in the case of a mouse you move your hand back and forth on a horizontal surface some small amount and it feels like you have a virtual finger that is 90 degrees rotated. So, it is vertical now and it is moving around on a screen over much greater distances. It might seem intuitive to you that you would like to just have in a virtual world a finger that you are holding in the air and pointing around. And this will be cool for maybe a couple of minutes, but then it becomes very tiring. It is a problem called gorilla arms where you get tired you feel like a gorilla with heavy arms and after a while you do not want to reach around in space anymore. You would like a very simple interface where you can lazily do very small motions very efficiently and with great accuracy without having to actually use your physical muscles if they are not required for the task. So, unless you have signed up for an exercise program right you might not be required to do all these extra motions just to select menus to other simple things like that right. Why put the user through all of that? In fact, you may make the user feel more powerful if you can identify some very simple learnable motor programs. That involves a lot of trial and error and a lot of creativity. I think this is a great place for people to be exploring in this field. Questions about that? Let me give a very couple of simple ideas and a general recommendation. So, here is a very simple selection idea. Why do not I just make a virtual laser pointer? So, in the virtual world I just point at an object that is how I do selection right. So, I have some kind of controller that is letting me do that. Maybe it is an air mouse in the physical world and I just move it around and I see doing ray casting right. So, you use ray casting just as we talked about for computer graphics of graphical rendering. So, you have a virtual laser pointer and it is as easy as that. So, why would I want to in the virtual world walk virtually 20 meters grab something and come back I can just point at it and click and then have it appear into my hands. If you could do that in the real world you probably would prefer it in most scenarios right. So, could get you in a lot of trouble too I guess. Let us see. So, that is one way to do selection that does not correspond exactly to what would happen in the real world. A positioning idea you may be able to have a virtual hand right and maybe it just does simple grabbing, but why not make it so that it feels like your arm can just extend indefinitely. So, you could have one controller button that just makes your arm longer. So, while you are making something like a laser pointer you might as well have a rendered hand that can just go out you figure out where you are at and then you do your manipulation and perhaps when your arm is grabbing something it can even give you a close up view still looks three dimensional and you can do the manipulation and then retract your hand back why not right. So, you have this long rubbery arm that can extend out. So, I will say extensible arm with virtual hand I guess the arm is virtual too. So, now one general principle is to reduce the degrees of freedom as much as possible in the task. So, think about your application think about what you want the user to be doing a beautiful example that is Minecraft right when you are manipulating objects in Minecraft it is extremely simplified right you grab objects they go into your inventory you quickly throw them back out again you do not worry about placing things in a very precise way using all 6 degrees of freedom. So, it is a wonderful interface great example of a good interface it caught on very quickly to to millions and millions of especially children all around the world. So, you know there is no reason to drag the user through very precise placements if it is not necessary and no reason to drag them through extra degrees of freedom because whatever degrees of freedom you have them try to execute as part of the manipulation you are going to have to have a controller for that and they may have to learn a motor program for that or it might have to correspond exactly to the physical world and then you have a very difficult tracking demands. So, why not simplify it so that it does not have a lot of degrees of freedom make sense right one final issue for manipulation general missing part that I have not mentioned here that I conclude this topic with it is haptic feedback is not a nice when I grab an object that I actually feel it in my hands depends on the task depends on what you are trying to do. If you are trying to do a medical simulation or training or tele-surgery something like that probably a good idea to give some force feedback to the user some haptic feedback to the user. So, perhaps you can just give a general rumble in some other applications right just shake the controller a bit maybe you actually need to present some pressure or force to the user. But you know a simple rumble might let them know that they grabbed an object and that may be very satisfying what associations can the brain learn they are good enough. If you need to know precisely exactly how much force is being applied right like in the case of a Professor Mani Vanan's lab there he and his students are working on training people for performing CPR right. So, you need to know exactly how much pressure or force to apply and so it is important to give that feedback and measure the forces as well as part of the training. So, it seems like there it is critical if you are just building some virtual world perhaps in a computer aided design context then these pressures and forces my guess is they would not matter very much. So, you have to really take the task into consideration. How do you implement these kinds of things? Well there are all kinds of haptic devices and things we could have some kind of gloves that you wear maybe there is a manipulator. In other words what I mean is a robot arm and hand and it could be some other kind of robot, but basically there is a robot that is pushing back at you and is programmed to be synchronized as well as possible to compensate or interact with your emotions in the real world and provide some opposing motions and forces. And finally, very nice idea is to make 3D printed objects right. So, I could just print an object and maybe try to use some similar materials to whatever it is I am trying to manipulate in the virtual in the virtual world. So, that matches the real world as much as possible for some kind of training applications. If it is augmented reality you can even print a very simple object and then augment it with additional surface structure and information if you line everything up. So, this may be another interesting and useful thing that you can do.