 So, evaluation VR systems. So, in other words how do we know whether you are making progress right you you made one experience ah I have improved it I have made something better are you sure right. So, how do we know um remember that um when we talked about um we talked about neurobiology in the very beginning lecture and perceptual psychology as well um we have ah where VR was applied in a lab to study how organisms are affected by that we talked about um VR applied to ah insects such as a cockroach we talked about it applied to rodents as well. So, imagine applying it to a mouse um all the way up to perceptual psychology which corresponds to applying virtual reality to humans right. So, when we do that there is very clear separation of the scientist and the laboratory subject. However, if you are an engineer or developer of virtual reality systems that in this case you may commonly be moving back and forth between these two without even being very much aware of it right. So, just be aware of it every time you try your experience you become the laboratory subject and then you go back and try to be the scientist again right um and it becomes even worse because you know the scientist might at least at least in principle it is supposed to be dispassionately trying to determine whether some hypothesis is true or false. So, they really would just like to know what is the truth. The developer would like to be done with their task have everything work well and either it is part of a business or a class project whatever the case may be or just making a good experience um. So, they have a vested interest in having the hypothesis come out. So, that they have made something good right. So, um so that is another kind of problem. So, they are already even as a scientist if they were they are quite a bit biased um for the outcome to be in their favor right alright. So, what options do you have I will mention two options one ah train yourself to be a critic I will say a good critic or maybe a bad critic in the sense of train yourself to be very negative and hostile towards your own designs difficult to do maybe um two design an experiment and recruit test subjects right. So, recruit people to try it which one should you choose I suggest choosing both of them right. So, obviously, the first one is the the laziest one right. So, um train yourself and then you can evaluate what you are doing with very quick frequency and without doing a lot of extra work. Number two may require a significant amount of extra effort and work and. So, you end up doing this depending on how much um your life depends on it right. So, so what exactly you are doing are you building an entire company around this you better do a lot of experimentation and testing are you building your entire career basing it on this then do a lot of testing. If it is just something for fun to share with your classmates then fine let a few of your classmates try it and um and get some feedback right and that may be enough to make you happy. So, choose both let me give some advice under these two cases. So, what if we choose number one? So, for yourself let me give some key points one thing is recognize your own bias recognize your own bias. So, one of the things I will say about that is um you are going to know more about what you are trying to do what is the intention of your experience than anybody else right. So, if you know what you are trying to do it may be the case that you have some kind of motion occurring and you know that the object is moving not you, but if someone tries it and they do not know either way they may think that they are the one that is moving and then they get sick and then you complain why are you getting sick and then and then um and then you say well um you know it look like I was moving and then you and then the creator might say well well duh of course, you are not moving the object was moving did you get it you know. So, um so, it will actually affect the outcome of sickness here in the simulator sickness based on what interpretation your brain is giving. So, you have to be very careful about that um the other thing which I already mentioned is you might be very proud of what you have made or lazy right. So, you you you have done you want to be done with it you do not want a bunch of feedback that says do it all over again right or so, you would like to think that you are done right. So, that is your bias right you you are either proud or lazy you do not want to redo it or you are happy of what you have made you are biased by that and um only you understand what it is supposed to do. What else um adaptation this is from perceptual psychology. So, you might perceptually adapt. So, you may have made some bad choices there may be some errors in your system you have learned to adapt to those they do not bother you anymore. Maybe you have made something that is horrible from a from a vestibular mismatch perspective, but you have made so many horrible experiences before that they do not bother you anymore right. So, you you adapt to it um the more experience you have in general with virtual reality the harder it may become to um to detect the flaws. So, you may need fresh subjects. One way to fight against that of against the adaptation is to put yourself through what I would call perceptual training. It is like training a detective to look for clues at a crime scene right look for the clues try to try to find the things that you would ordinarily miss and look for them and then as you develop experiences make sure you are finding them again. So, try to um try to see the flaws that maybe were invisible before. I am going to give some examples of this that I myself experienced. So, one observing latency. So, tracking and display latency ok. So, suppose you have some head tracking and maybe something has gotten messed up right. So, so it is hard to tell you would like to verify that um here is what I like to do. I like to look at a vertical line about 2 meters away in VR. So, it is comfortable it may be a door frame or or side of a window and then I go like this. So, I move my head back and forth at about 2 hertz. If I shake too much at a higher speed then the headset might start to move with respect to my head and I do not want to do that. If I move too slowly then I might not perceive latency anymore. So, I picked a kind of intermediate speed about like this. If I move at too far of an angle then I might also have problems because of optical distortions things like that. So, if I just move like this small angle at about 2 hertz then um I can adjust with some keys let us say the amount of prediction that is going on in the tracking to try to compensate. Ideally it should all be working fine maybe there should be effectively no um latency, but you can then move the keys and you can show yourself what does it look like when there is a 40 milliseconds of latency 100 milliseconds of latency. What does it look like if I overshoot I try to predict and end up arriving the images end up arriving 50 milliseconds ahead of where they should be right. So, you train yourself to look for that and then um try to zero in and get it to match as closely as possible as it does to the real world and again you can go to the real world and look back and forth at something try to match it you are using the vestibular ocular reflex then do it in VR and see if it looks right. Um when I started working um on VR in 2012 I could only see latency down to about 40 milliseconds when I first started playing with these systems. After about a year and a half and and perceptually training myself I could detect latency down to about 2 milliseconds. So, so I think that is very close to the limits of human perception that is very important. So, all of a sudden I have trained myself to be a verifier of whether or not the latency has been handled correctly. So, you should do that rather than training yourself to just adapt to the latency and have your brain convinced that that is fine right and then you may have some discomfort or or um um maybe fatigue or maybe simulator sickness at the end because you have not trained yourself to look for it. Um another one um observing optical distortion. Remember that we have lens aberrations with a high field of view lens. So, what you need to do to see optical distortions in any kind of problems with those is you have to be looking through the sides of the lenses. Now, you could just go like this move your eyes to the side try to look around, but why not use your vestibular ocular reflex. So, what you do is you look at an object in VR you fixate on it and then move your head back and forth a lot. And in fact, better than picking an object go to a place where there is a lot of rectilinear structure where there are vertical and horizontal lines maybe in a nice room as I look around on this classroom look at the blackboard I see um I see lots of um structure vertical and horizontal lines. So, do that and then move your head back and forth so that you are looking out of the sides of the lenses. If there are optical distortion issues you will see a significant amount of warping or swaying in these vertical lines. So, so look for that. If you go do that try that in the lab you will find that even if you compensate for it in software you still cannot compensate for all of it because your pupils when you use your vestibular ocular reflex like this are translating away from the center of the lenses. So, you are moving across different parts of the lenses. Um another one observing stereo mismatch. So, I am supposed to be showing perfect stereo rendering for the right and left eyes um what if I got it wrong. So, what if for example, I I have I have everything match perfectly except I have decided to draw a butterfly, but the butterfly only shows up for the left eye. You think you would notice? You might not your brain will still fuse everything together in one coherent picture. So, the way to do it how would you do it? Well, close your right eye and look through the left and then close the left eye and look through the right and see if you get different pictures very simple, but people usually do not do that right. So, that is so that is one one thing you can do close close one eye and then close the other and compare the images that you get right. So, very simple. So, there are many many procedures like this you may need to be clever about them as you suspect different kinds of flaws in your system. Questions about that? Alright. So, now that is for yourself. What about um designing an experiment? Now, there are plenty of books and courses just entirely on this and we have to design experiments when there is human perception involved. So, if that is the case um I highly recommend you read about and use methodology from perceptual psychology. You should believe this from the things we have talked about in this in these lectures for this course especially the area of psychophysics. Let me just give some highlights you definitely need to have a clear hypothesis right be very very clear about what it is you are trying to establish not just always my system pretty good you know I just try to come up with very very clear um hypothesis about what what you are trying to reason about you have two methods of locomotion for example, method A method B which one is more comfortable from a simulator sickness perspective which one is is um more effective in terms of um way finding you know are you actually building a kind of map of the environment while you navigate around or not or as it become too confusing and disorienting. So, formulate the hypothesis very clearly um do you need um a few subjects or many maybe hundreds of subjects? You do not always need a lot of subjects for some experiments in psychophysics it may be just that three or four subjects is enough. It may be based on a simple phenomenon maybe the response time of your vestibular ocular reflex and you do not need to have hundreds of subjects to determine that it may be um very very similar very very close range of a small amount of variation let us say from human to human you just need to measure it. So, you need a few humans to make sure that you have gotten the measure accurately you have done things correctly um. So, perhaps you just enough to um to to believe the data um and not be overwhelmed by one or two outliers right um versus some kinds of phenomena especially when we talk about simulator sickness you may need to study it over a very large number of people um related to this is our demographics important who is going to be using your experience and does it matter you know men and women may have different um um different experiences and maybe different results based on gender maybe different results based on age maybe different results based on the amount of experience in terms of sitting in front of giant displays right. So, so um if you have a group of people who have been playing first person shooter games for the last five years very regularly they may have a different response to the kind of stimuli that cause simulator sickness they may be quite immune to it in a lot of cases um can you take automatic measurements can you take automatic measurements or do you have to ask them how they feel right did you like that experience did it make you sick maybe you can measure the galvanic skin response which has to do with the conductivity across their skin which I measure whether or not they are sweating which may then give an indication of whether or not they are starting to get sick. So, there are different things you could measure or at least it may have some measurement of their emotion while they while they use the system there are also other measurements you may be able to take that may give um um indication of um of simulator sickness maybe how they move their head how their balancing is affected um you may be able to measure that if you do not have to tell them to give you a response in some kind of questionnaire you might have an advantage if the automatic measurements um can directly correlate with the the hypothesis that you are trying to resolve. So, automatic measurements or do you have to ask them questions you have to make a questionnaire um do they know what you are trying to measure which do you think is better should they know the hypothesis no way right whatever they know about they may be trying to help you right or they may be the kind of evil subject they may try to harm your experiment right they may say ah I know what they are trying to measure I am going to keep answering the opposite you know so um so it is best that they do not know um part of that is you should provide this is very common in psychophysics and perceptual psychology provide a stimuli in random order like for example, when we talked about perception of equal loudness do you want to start off soft and just gradually increasing it may be better to just present them with sounds at random levels and then have them indicate whether it was um louder or softer right if if you are trying to determine that um could maybe ask them if it is equivalent, but then I guess there is the difficulty of how much time has gone by since they have heard the base stimulus. So, um so if you have the opportunity it is better to present the stimuli in the experiment in some kind of random order without them having any idea of what you are trying to measure um another one is um adaptation adaptation is that an issue or fatigue right. So, it may be the case that as they are doing the experiment they adapt or maybe you have had the same subjects coming back day after day well you might need some fresh subjects or maybe they become fatigued right if they become fatigued then they will have different responses near the latter part of your experiment good thing use random order at least right if you if you save the best for last you know then um then that ordering and their increase of fatigue will affect the outcome. So, these give you some general ideas of um the kinds of things to pay attention to in designing your experiments it is not at all easy and it is especially difficult for engineers, but there are people who have studied this they do this all the time um they write books on it you can get advice you can talk to perceptual psychologists at your university um and um and get um ideas on what you should be doing here. My questions about this all right have one one final reference that I think is very helpful for general research scientific questions whether it is engineering or um perceptual psychology aspects and it is the handbook virtual environments um make sure you get the second edition which is from this year 2015 it is an edited series by Hale and Stanny um it is a very thick book an enormous collection of of survey papers that survey many many different aspects of virtual reality or virtual environments. So, um so I find it an excellent source for all of the current up to date scientific literature at least up to probably 2014 and then the and then the book was made, but this is a great source you can find many references in there and then search for references that reference those references that are that are newer after that and you can find a huge access to the the literature on that um. So, that is all for the course let me just remind you of what we went through we started off I start off giving you a definition of virtual reality that um emphasize that its engineering plus some organism that is being subjected to artificial stimuli. So, remember that the organism is a key part of that. So, and in the case of um most of what we talked about in the class um you are the organism that is participating in it. So, make sure that um you take that into account and whatever you evaluate. I gave you a bird's eye view where we talked about the hardware the software and the software mainly involved in alternate world generator which today are mainly um its most popular to use game engines, but who knows what the next generation is going to produce. There should be virtual reality engines that make it very easy for developers to develop experiences. Game engines are not ideally suited for that right now, but they're trying to adapt them and I gave an overview of perceptual psychology. Then we went into geometric transforms and models right. So, how to model these um alternate worlds and transform them so that the right information appears on the screen. Then we talked about light and optics. We got into the physics of it and then once you understood optics and light we talked about lenses and then optical systems that are engineered to make for example, a head mounted display, but then we got into the human eye as an optical system. And then we went into the physiology of the human eye. We went into photoreceptors which are like the input pixels. We talked about the optical system of putting a display in front of your face and going from pixels to photoreceptors. And then we went into human perception and physiology. We went from photoreceptors to um talking about the retina, the neural structure that's part of the um the eye which goes from the um the photoreceptors to the ganglion cells. The information is communicated along the optic nerve arriving into the visual cortex. So, then we got up to the levels of perception. We talked about hierarchical processing of the brain and the whole visual system. And then we had perception of um motion, right? Um part of that is is complicated by the fact that our eyes move. So, we covered um the motion of our eyes. We covered perception of motion. We covered perception of depth. And I emphasize that there are many other kinds of perception like perception of scale, perception of color. So, now that you understand that template you can study other aspects of perception. Um we then described systems of tracking because we have to track the motion of um the human head and perhaps more of the human body in order to make a perfect kind of correspondence between the physical world and the virtual world. Then we got into rendering for for the visual case. Much of that was standard computer graphics kinds of concepts, but there are many particular aspects to be concerned with when we get to virtual reality. For example, aliasing becomes much worse because of the changing viewpoint that we have and because of the stereo perspective that we have. Um so this becomes a serious and difficult issue. Then we talked about audio for virtual reality and we covered many of the same kinds of concepts that we did for visual. Um there are rendering issues. There's human physiology issues. There are perception issues. We covered these for audio. We could have done the same for haptics and touch or for um for smell and taste. Whatever other senses you want to bring into the virtual world. We talked about we could talk about these as well in the same kind of style. Um the vestibular organ was also mentioned because it's very important, especially in the case of vestibular mismatch. It also is what helps power the vestibular ocular reflex, which is very important. And then finally we ended by getting into uh interfaces. Some of these higher level aspects going across locomotion manipulation system control and social interaction. And then I concluded by giving you um an overview of how to evaluate VR systems, whether it's yourself, especially emphasizing perceptual training and then the design of experiments involving human subjects. And that brings us to the end. Thank you very much.