 So, that is all I want to say about system control I want to get to the the final of these four categories remember I had an other category, but I am not going to go into ah specifics on that I am just trying to cover the generic categories . So, interfaces for social interaction. So, many years ago the idea of virtual communities was developed ah this can be considered as a part of sociology right how do we how do we interact and socialize build societies together ah. The term virtual communities came out of a book by by um Howard Reingold um it is a long history of virtual communities one of them is called PLATO um which is from the 1970s I will say 60s and 70s um which came from the University of Illinois and um was a system for doing um for education. And so, in in that time there were forums appearing. So, it was networked computers brought together and people were interacting in the context of classes um now a forum like like the um the forum we have for the class the online forum seems very natural this is developing a kind of community for learning right. So, you do not have to have a completely immersive virtual reality experience to develop a community of people who have some kind of common goals or interests. So, I just wanted to point that out. So, in addition there are chat rooms been around for a long time email lists perhaps not as interactive, but we tend to use email lists all the time with threads going and discussions um news net multi user dungeons been around for many years. So, this idea of virtual communities has developed for quite a while and it is quite an independent thread to virtual reality right. You do not need virtual reality to build um virtual communities and have some kind of social interaction right. Human beings will do this in many other formats or many other kinds of media. So, we have it is already been established that they do not have to be physically present in the same physical space to form a community correct. So, a special case of that is through some kind of virtual through some kind of virtual reality. We have seen this idea all over in science fiction. Let me just kind of um let me list a few books that have inspired a lot of um people in this space. Usually they involve some kind of dystopian future society of people connected together through virtual reality um. So, you can look these up if you have not seen them before. Neuromancer um from the 80s snow crash from the 90s and ready player one from a few years ago. If you want to read about um all of the things that might happen to us one day if we um immerse ourselves in virtual reality almost all the time or all the time. Um coming back to the real world MMORPGs is worth looking up if you are not familiar with that massively multiplayer online role playing games right any of you involved in those right. So, you have heard of this before right. So, it just depends um on the audience um and these these these online games they involve some kind of teamwork and interaction perhaps with a bunch of strangers right all around the world. So, it is it is fascinating um I would say this corresponds to some kind of virtual worlds with social interaction occurring. And um second life this platform um started around 2003 grew to have many um users tens of thousands were connected together at the same time building some kind of virtual worlds for interaction. They are real people who are interacting through avatars. So, they have cartoon like representations of themselves. They are wearing clothes. They go to the shopping mall and buy other uh buy outfits. They they they socialize in virtual pubs um they started buying real estate. There became a currency called linden dollars that was actually exchangeable for real currencies like you could exchange it for US dollars. There is an exchange rate for that. This is one of the early examples of a kind of independent currency. You see bitcoin uh today very often. So, this was a the currency that evolved here was a precursor to that. All you need is some kind of social interaction community and people will start exchanging money that becomes effectively real money at least as real as the money we use that are backed by governments. Eventually it got into a lot of trouble in a lot of ways because um there is no government inside of here really. And so it became hard to enforce rules and laws and theft and other kinds of things became a problem. People would start to trick others out of money and it's actually changeable for real money. So, it's a fascinating what happened inside of here. So, the creators of Second Life um are were big fans of virtual reality. They just did the best they could using the technology of the time which means do it all on a screen. Don't worry about the problems of head tracking and immersion and simulator sickness. They worried about the social interaction part. And so um the people the creators of Second Life are continuing to be involved in the virtual reality community and want to see it brought to the next level where you feel like you are completely inhabiting these worlds. Alright. So, virtual reality is increasing the amount of immersion this feeling of being present together. Is that important for social interaction or not? I don't know. We started let's say a long time ago with telephones for for talking to each other. And then we carry around these things we call smart phones. How often are we phoning each other versus um sending text messages? Right. So, which what's more immersive and what what gives you more of a feeling of presence when you are talking to someone on the phone or when you are text messaging them. So, I would think the phone feels a little more immersive. In fact, I will go the other direction and I will add video to that. Wow. Now, I have a video conference going with my friend. How often do you do that versus just sending a text message? Right. So, it is very interesting right. So, you feel like you get pretty good interaction with text. So, I don't think it is always very clear um you know it is not necessarily the case that more immersion more feeling of presence somehow improves the interaction right. You would like to just roll out of bed and start sending text messages and not really worry about how you look for example, right maybe that is an issue. Um. So, something to pay attention to. So, um on the subject of face to face interaction which again might not be necessary at all um gosh maybe we put on our virtual reality headsets and then we just read text messages from each other right. Why would you do that? I don't know, but if you are spending all of your time inside of virtual reality you might still be texting with your friends then right. And in second life all of the interaction was done by texting. And I ask people who are um who are in that um very active in that community. There is also an open source version that I encourage you to try called open sim or open simulator. So, if you are all especially if you are on Linux platforms you may find that very interesting and hopefully people are running servers on that. So, um I asked them why don't you just transmit the voice and they care a lot about the anonymity of it. They like to be someone else you know they become their avatar. And so, the texting part let them communicate with each other and they would walk around in these virtual worlds and only when they get close enough within what would be normal hearing distance then you can see the texts of each other. So, the the texting replace that. If you wanted to do an audio track version you may need something like a voice disguiser then right. Because they are already running around with avatars that disguise their appearance. So, you would need to disguise the voice to have it be comparable right. So, that that could work you could make a real time voice warbler or some kind of um distortion um transform to the to the voice, but a texting works just fine there. So, when we get to face to face face to face interaction we have on the one hand avatars which is some graphical representation of oneself. If you want to look like a monkey it is up to you you can be anything you like you could look like a a piece of fruit it does not matter really, but um so you just you decide to have some kind of other representation probably something that animates itself not a piece of fruit, but you know it is trying to give you extremes versus real faces. In other words yourself captured as well as you possibly can capture it maybe just capture with a video camera and transmit it maybe you do something more sophisticated to provide 3 dimensional information. And for the audio part maybe you have text versus the real audio right. So, that is the visual part in the first line and the hearing part in the second line. So, on this side you are approaching realism you are trying to match the physical world experience on this side you are approaching fantasy right you have not a chance to escape the real world, yet still have some kind of interesting social interaction. So, these are both interesting and useful extremes think about your task think about the experience you are trying to make. So, regarding achieving realism um how might I do that right. So, maybe I just placed in the environment like in this classroom today I just put an omnidirectional camera all right. So, using multiple cameras that are synchronized using lenses optical techniques for a wide field of view however you want to put it together build a coherent panoramic video and then transmit that and someone remotely can look around and feel like they are here. That should be very realistic it would be better to capture all of the information in stereo or capture the entire light field somehow. So, they can move their head around get parallax get everything working just right that would be an extreme level of realism and you know maybe not too difficult if you just want to do a monoscopic panorama and transmit the live video you could replace your face to face on a screen video conferencing with um a full live panorama right. So, that is not too bad um one problem is that if each of you is wearing a head mount to display then in the panorama you will be wearing a head mount to display. So, then I cannot see your eyes right. So, we have a great conversation, but every time we meet we have black bricks on our faces right. So, it is kind of unfortunate, but that is one of the outcomes right. So, what do you want to do about that? You could put little cameras inside. So, I could at least try to capture the part of your face that is being obscured by the headset, but I still have to reconstruct the rim part that I cannot see anyway you know it is a mess no matter how you do it. So, what exactly are you going to reconstruct? Another interesting problem is that um what if I would like to extract the person out of the real world and put them into a virtual world. Maybe I would like to move if you and I are interacting maybe I would like to move us to some virtual world or perhaps we would like to be standing in Paris talking together right make it appear like that. Now, you have to do a lot of difficult work to extract some representation of our bodies and move us to that location right. The moving part is not too hard once you have a good representation of the entire person. So, um it is a lot of work and it is going to be very very challenging to get that accurate without having access to your own private motion capture studio or you can put markers all over the body and try to completely and perfectly extract a representation of the person. So, that is a very difficult problem. So, it was very easy if we all stay in the same place and we just capture it with omni directional video because very challenging if we want to extract just the body maybe put a blue screen behind and extract the body and move it somewhere else. Does that make sense? And always take into account the problems of delays if we want to move ourselves to another location on the earth for example or some other virtual world there may be problems with synchronization. And if if you see my lips moving it might not be synchronized perfectly with the audio. So, another issue maybe it is better to delay them both just so that they are synchronized all right. So, that is some some comments about achieving realism. Let us think about with avatars with avatars we can achieve the impossible right. So, you can achieve some things that are not possible in the real world. So, this is some kind of transformed social interaction right beyond what even normally seems possible. For example, you can change you can change your gender and see what happens as you interact with people right. You can change your race whatever that means you can change your species right. It could be some kind of fictitious alien or creature of some kind you could be running around like a virtual Godzilla and see how people interact right. Do your friends respect you more when you when you when you run around like that. You could change just very simple characteristics like maybe you just change your height right. Change your eye color your weight and just see how people interact. It is very interesting because this gives you an opportunity to maybe experience empathy right to put yourself in someone else's place right. Very often when people complain of racial issues, gender bias, things like this why don't you try it yourself and see how people treat you right. So, you can put yourself in their perspective. So, working with an avatar gives you an excellent an unusual opportunity to experience what it is like from someone else's perspective rather than just form an opinion quickly without really having experienced it yourself. So, that means that empathy can be taught. For example, perhaps in police training videos you may be able to have the police enter certain situations, but they get to be different characters in the scenario and everyone can look at it and learn evaluate it is everyone being treated fairly you know they they may get a feeling of what it is like to be the one being arrested right and why are they being arrested perhaps they feel like they are being discriminated against right. There is a lot of interesting opportunities there. Along the same line of achieving the impossible you may be able to look at all people simultaneously. This comes from a paper by Jeremy Balanson at all from Stanford in 2004. Jeremy Balanson has done a lot of research on social interaction and virtual reality. So, this is just one example is a lot of interesting papers from his lab. So, for example, in today's classroom there are a bunch of students I will just draw you as circles if you do not mind. So, there are a bunch of students sitting in the classroom and there is one lecturer me and I look in some direction at a given time. So, I can only look in one place. So, this is the real world and there has been a lot of studies that show that the learning outcome is enhanced by live presence right. So, I am doing a live performance today and that increases your learning capabilities or your learning abilities. And so, if I am recorded then I believe it goes down quite a bit right. Part of it I think is just maybe a little bit of the fear that I might look around and see if one of you is not really paying attention maybe you look like you are sleeping doing something else I can look around at you and watch while I am giving this lecture. But if you, but if I do not pay attention to you for a long period of time then you might start to fade away right or what if you have the fear that I might call on you and ask you a question right you know it is safe with a recording. Well, in virtual reality I could make a transformed classroom where people are all sitting there and I could be looking at all of you at the same time because when you are wearing the virtual reality headset I am looking at you right. So, I can be looking at all of you at the same time and give you this feeling and maybe I am not paying attention at all maybe I am recorded maybe I am not maybe once in a while a real person appears and is in fact looking at you and is going to ask you a question. So, you will get a little bit of that fear that maybe someone is paying attention to you, but maybe 50 percent of the time or less there is no one there at all and no one even looking at you it is just completely a recording right. So, there is all kind of interesting possibilities to help you stay more focused and pay attention me out of this fear that a human being might care and that human being might assign a grade to you at the end of the course right. So, things like that. So, I think that is a great example of how you can achieve the impossible something that is not even possible in the real world, but it still leverages your real world experiences and expectations questions about that. So, suppose we have an avatar and I still want to bring some realism back. So, I am kind of going the other way now I started off with realism over to avatars I want to go back and I should say before I go back the avatar representation could be very very simple how much do we need right do we just need some simple lines and some very very simple expressions that indicate our emotions maybe that is enough right. We know about emoticons and the power of those in texting is there going to be something similar like avatar visual virtual reality emoticons that just give us an idea of the emotion of our of our friends or people we are interacting with or just an emoticon or something like that it gives us an idea of whether or not they are paying attention right. So, how much do we need. So, that is one extreme we get a very simple representations or you could try bringing realism into the avatar. So, if that is the case well maybe I can do some eye tracking right. So, maybe every time I blink my eyes in the real world my avatar blinks as well maybe I can wink I can close one eye right. So, I can do some simple gesturing if you like to call that with my eyes you could also figure out where people are looking if you track the eyes right. So, wouldn't that be kind of creepy you have this cartoon like representation, but the eyes of the cartoon like representation do correspond to exactly where the physical real person is looking. So, it is not nice you could bring that information back. You could go even further and do face modeling I should go back for the eye tracking I mentioned foveated rendering before and said that you had there were very high performance demands for that for simple social interaction the eye tracking demands are much less you know maybe just generally you know where is the person looking some latency maybe is not too much of a problem. So, I just want to point out this is much more feasible at let us say a low cost with with minimal kind of engineering requirements, but if you want to do something like foveated rendering as we have talked about before then the performance demands are higher. So, this is called as basic eye tracking face modeling. So, what if I would like to do modeling and animating and tracking of my face. So, while I am making different gestures here there is a avatar face that I have that is moving along with me capturing as many of my motions as possible. So, you may make a model of the kinematics and dynamics the muscles of the face try to match everything up as well as possible. If you would like to see some work on that I suggest looking at the work by Paul Debevic who is at the University of Southern California he has done a lot of facial modeling and animating and tracking for the motion picture industry. And so, you get very realistic looking faces now one problem you have when you go down this path is what is called the uncanny valley. So, whether it is in motion pictures where there are synthetically generated actors and facial expressions it is also the same in robotics for humanoid robots. As you start to approach human realism people do not like it and then finally, there is some threshold that is crossed when you come out of the valley and then it is considered acceptable. It has to be really really good and very close to reality for people to accept it. Otherwise there is this uncanny region where I do not know it feels like maybe a dead person talking or something it just does not seem right something is just not quite captured correctly. And so, that is what you risk when you go down the path of very realistic face modeling is gee it seems like my friend, but they look more like a kind of dead puppet or something that is that is moving along and it is very creepy and uncomfortable. So, that is something to be aware of and of course, you can go down the path of hand and full body tracking. How important is that? Well, I can get gestures if we are just sitting in chairs talking it might not be very important if we are walking around together maybe it is important. If we are playing a sport together if you want to play a virtual tennis then maybe that would be very nice to have everything captured. For playing virtual tennis that would be difficult it would require a lot of space if you want to move around completely and play just like in the real world and you have the haptic force feedback interaction part with the tennis racket. So, that would be difficult. Nevertheless you can do this one warning I want to give here though is that poor tracking poor tracking is worse than none at all right. If I have a tracking algorithm I am moving my arm around imagine you are visualizing your friend they are moving all of a sudden it looks like their elbow has been broken and their arms popped in the other direction oh then it pops back again. So, very disturbing very uncomfortable annoying. So, a lot of tracking systems even if you see a great demo you may watch a video online looks great. If you go try it yourself you will see a lot of flaws they usually show you the best cases not the worst cases and usually they are not even showing you the typical cases. So, it is a lot of interest in tracking more and more of the body, but it is not very reliable and accurate it is better in many cases to not do it at all. Think about what you are trying to do for your task determine what you need how accurate does not need to be is it really required to match the physical world or not. For basic social interaction probably not you get a lot of social interaction as I said just from text messaging. So, what exactly do you need here think about it very carefully. Questions on that? If you would like some more reading that is related to these topics one thing I suggest is I mentioned this I believe last time 3D user interfaces although it is not particular to head mounted displays is more general for virtual reality augmented reality and a few other examples in that general area. Book called 3D user interfaces by Bowman et al I believe it is 2005. So, lot of the technology examples in that book are significantly dated, but it does give you a lot of the general principles and I also strongly suggest that you look at the Oculus best practices guide which you can download online for free. This was written by a coming together of experienced video game developers and perceptual psychologists. So, it is science and game development and experience merged together and it provides a lot of recommendations. Some of these things about like where to place menus gives you guides for that very simple recommendations like never turn off the head tracking. So, as an example one of these apps that you might use in the lab that allows you to experience panoramas that were captured by for Google Street View. If it is loading another panorama it tends to turn off the head tracking that is the most nauseating part of that experience it turns out because the head tracking all of a sudden freezes while it loads something else you are in trouble. So, do not starve the tracking thread you have to keep the tracking going. So, examples of recommendations like a very practical advice and it is based on a lot of the scientific aspects that I have covered in this class from a perspective of perceptual psychology and human physiology all right. One final comment on interfaces. So, I said this before any interface from the real world of the physical world can be simulated in VR. So, that means that any VR interface can be simulated in VR right well no it does not mean that does it, but I will add it as an extension it does this is would be in a generalization of this. So, if I start simulating interfaces in VR from the physical world I can then simulate VR interfaces I guess that in VR I can put on a VR headset and I guess keep doing that indefinitely right. So, it is a lot of weird things you can do I will give you something bizarre to think about. So, you can have more and more levels I suppose. So, take anything in the physical world simulated in VR, but then even putting on a VR headset can be simulated. So, very strange. Questions about interfaces? I have one final part which is advice on evaluation of virtual reality systems. So, let me go to that part. Yes. Sir putting on a VR set in VR that use more simulate a sickness by any degree. Who is the person is like he knows that you get simulate a sickness by use virtual reality. Yeah I do not know what happens I guess once you put on the simulated VR headset in VR if it is assuming there is no additional latency in head tracking because it is not real anyway then I suppose your brain is convinced that there was just one level of virtual reality right. So, reminds me of the movie Inception where you have all these different levels of dreams and you keep going further and further down. So, except there is no time scaling distortion. So, I assume it is just we will have the same problems again because it really ultimately is bit the problems of simulator sickness are only based on the physical layer which is what are the visual stimuli that are being presented to your to your eyes.