 I was just I was just practicing in my meeting simulator. It's pretty cool. I can show it to you later if you behave So You're probably now wondering who's this girl. What does she know what we are and what does she have a funny accent? so my name is Naomi Fragera and Yamasté And I am from Spain Master degree in computer science. I also have two more degrees and in hard work and software from Spain and Ireland and I really like testing. I really like new technologies and I really like the art I'm right now a senior software developer in tests for netis games. This is a Chinese company I don't know if you would have heard of it and It's pretty cool. By the way, and I also work in these other companies and different roles testing and developing and We're in different hats like some people will will call it so I hope everybody knows in here was virtual reality and Just to give you a little definition. It's all about creating virtual worlds that are immersive and This thing here is called a sort of demo place and you probably can tell why is this It's the first VR register device This device was created in 1968 so you can probably tell and How old we are is many people think that we are is a new technology, but it's not at all So and my goal for this talk is first of all for all us to learn a little bit about virtual reality and also to Get some conversation starting for the networking bits about what we are and how can we use it? how can we test on it and Also, hopefully to keep you all awake because it's not the best time for listening to anybody and First of all, I just want to talk a little bit about what we are the devices and then we'll go into what's important for us as testers and also We'll get to see what the different type of testings are and how can we apply them into VR and Also, how can we get started with the R and a little bit about the future. So right now? There's two main type of devices, which is phone devices and the headsets So the phone devices is something like this. It's also the cardboard and There's two main types as well between the phones ones are the ones that you have a clicker inside And the other is the one that you don't have a clicker inside So you have to come up with a way of clicking into objects if you need that the headsets are usually Not only the device, but also handsets and the hands can do different You can have different buttons in the hands so you can do different actions and And the headset itself will be usually more powerful than the phone ones Then I also call all the devices we will talk about that in future testing So what else is happening and within virtual reality? So what's important in virtual reality for us as testers? so first of all You're gonna have a lot of objects around you so the visibility of the objects that are important for the users and How can you reach those objects? It's very important in VR It's something that we have to take into account because yeah, the application looks great but the user cannot reach the object and Then there's another things about the scales the colors and sound all of this is super important in VR because you can create Different experiences so for example if you have a lot of objects are way bigger than you you can feel like you're small and Also, you can change the size of those objects to get the feeling that you are shrinking or growing Colors lighting shadows can also change the experience so it can make the same room Feel like some safe space or some dangerous scary area The sound is really really important in VR. I can't stress that hard enough because you get 360 degrees experience so maybe something is happening and it's behind you How can the user tell usually is done by either some kind of dialogue or sound that is happening behind so you will turn your head Motion sickness and it's basically what happened to you in the boats when you're sailing and you feel like a bit nauseous this issue is Unique from VR because what the user see and what the user feel is not the same thing the user may be static and It can be seen like you're moving so this is very important that is well tested in VR and usually we have We have for for this type of type of things not to happen Some solutions that are available online that you can look for how to avoid motion sickness and Google You find them like for example being in a static cabin and Having the objects making sure they're not blurry having a high frame rate and also This threshold for motion sickness is not the same for everybody So if you have someone in your team that is particularly sensitive to this You should ask this person to do the testing on the sickness if there's movement involved in your application In my case, that's my mom Poor woman has a lot of patience and she always does the emotion testing for me Then again the quality of the materials and the lights are very very important to create different Experiences and to make the world really immersive. So the more the higher quality the materials are the higher Function the lights and the shadows Work in your application the more immersive it's going to be So I'm going to go through the different type of testing now and I hope I'm not forgetting anything sorry, I have my chit-chits here and So let's let's go through them. So the design details I call this feature testing and this is like any other app. So it really depends for VR What device you're using so it's not the same to test for a phone application and for a PC application Exactly same happen in VR only that we have so many different devices So if you're doing mobile tests, you can only do one click right? So it would be easier to test than if you tested a headset in which you have two hands in which you can have menus and different buttons and so obviously the hardware is really really important and also Hardworks in VR are still slightly a bit little bit expensive So most developers will just develop for one platform and say that it won't work in the others Because it's very expensive to actually get that you're handsome one of other device and tested I'm actually very lucky in my work that we do have many devices and I can do the testing them and And there's three main things that you have to test for functional Testing in VR, which is a code accession. So you need this back-end API's Assert the components. So basically the objects that are in the screens So what the size should have what the colors what visibility what is it visible invisible? and also Assert the interaction with those objects. So If two objects are together should they disappear should they bounce should they collapse? So the object interactions are very very important for testing as well. So, how can we do those testing? so in unity Well, just a little break here I'm gonna talk a lot about unity because it's what I have more experience on it's not the only platform for doing VR We can use Unreal. There's also a web VR. So there's also things for the web But I'm just talking about unity because it's the one that I have more experience on So again Unity does have testing tools there are for unit test and for integration test and And this testing tools the only issue that I see on them is there are a little bit Written from the developers perspective. So you can do only like small little tests So in order to do things like end-to-end testing and we will need other sort of tools Luckily, this is a field that is very new. So many people are working on this but there's also a lot of room for you guys to work on that and Me myself I am working in this right now This without making too much advertisement This is a project I score a test project. It's done by netis is open source and We have this is a cross-site Cross-platform automation IDE and We have several platform one of them is unity and right now. I'm working and making this available for VR as well so I'm gonna leave some links as well at the end of the talk for you to get and and Feel free of going there checking it out So in terms of performance testing and As I say before the quality of the objects the number of objects the lighting is all great to get an investing experience, but it all Damage the performance right the more of these you have the more shadows and more lights The the worst the performance is gonna work and it's very important for performance testing that you test in the right device It happens to me many times when I've done a VR applications that they work perfectly We're on the computer and the minute I pass them to the phone. They stop working altogether. They just don't run so again unity has this tool which is called the profiler and With this tool you can see things like the CPU usage. I don't know if you can read it so you can see the CPU usage the GPU usage. There's a memory memory utilization as well and Whereas there's a Audio and physics so you can test many things on performance with this tool Also apart from this there's many other things that you should take into account in performance Just any other app. So the memory utilization and the battery utilization the network connectivity and impact crash reports So in general For testing this you can use any other performance tool and it should work similar. It's not the same Also for performance Don't think that your application is going to be just in your computer There are many applications nowadays that are being created or already created for Multi-user Are you awake? So well, so for example, it's an application that is called VR chat And this basically you can select which avatar you want to have and then you can go there and chat with other people That are dressed up like other avatars. There's a little bit like the movie so Again, don't think that VR performance is only in your computer So accessibility testing is very very very important in in virtual reality Because there are some applications that are being built just for this. So there are therapeutic Experiences for people that have issues and And there are so many things that can be different in VR So even just for the length of your arm, it's not gonna be the same as so the person Right. So when you reach out for an object Maybe you can't reach it even though in the test space it was fine, right? So you have to really take into account your users the mobility on the users give options For example, if you cannot use your hands, how can you click things? So one of the Solutions that we have for that right right now is if you look at an object for long enough And then that means it's clicked if you look and go back and look again that may be a double click So all of these are different options that you may have for your users Also in terms of audio Maybe the person cannot hear well So you may you have to make sure that there are subtitles and that the subtitles can be read Don't use contracts like green and red because that could be bad for someone that is color-blinded And and make sure that everything in yours thing is visible for as many users as possible and also The opposite right so if people cannot see well, they should be able to hear well, right? So are the source are the sounds coming from the right sources? Can the person tell what the sound is coming from and you know check that object and In terms of mobility again Is is your application made for someone that is going to be walking around? Do they need to duck do they need to be fast? Do they need to balance or kneel all of these make the testing one application from another way We different and as usual this lands in the tester territory because most of the developers Will just want the application to be as much immersive as possible and wouldn't think of the what ifs As we do so again many sources online if you want to know more about how can you? Have accessibility testing. There's many many as much much information So very quickly on this to Security testing and localization testing. They're not so far from any other applications and in security testing and You have to take into account that these applications are usually connected There's gonna be some virtual property and you have to be careful that there's no virtual robbery There's good cryptocurrency is going on. So just be careful on those for localization again because we are adding dialogues and We are adding subtitles These have to be working in all the languages that we support. So localization is also important Why do I put them together? Well, because there's another thing which is the digital currency as I say before But not only digital maybe you're using your country's currency And maybe you imagine that you set up a VR shop and people can buy things If you have set up the price in Indian rupees and someone is buying with dollars That transaction have to be fine, but you know in this case like I think dollars higher, right? So it should be fine for for you if the other person and The currency gets confused and is paying on dollar what costs on rubbish, right? But what if what for the other person, right? They're maybe paying way too much So all of those have to be very careful kind of land in between security and localization I've seen in other apps happening. So It's again, just think about what can make the user uncomfortable or unhappy and that's that's the way you find So there's one test that is very particular of we are and I call it we are safety testing And and there's two type in in between this. So first is the real world dangers So basically you're gonna be in an immersive world and you're going to be inside a physical world, right? So you can hit your hand on objects around you. It has happened to me many times by the way Maybe someone is speaking to you and you can hear that person. So What if again as testers is the question we ask what if there's an alarm? There's a fire. There's a robbery in your house There's a carbon diocese Issue happening you can't hear it because you're in your immersive world. It's super dangerous You could actually cause death with that So there must be a way for the person to disconnect and to hear this as I say before the developers are going to want to get the immersive work and we are the ones that have to say wait, what if and The second thing that happens in virtual reality and as was on many apps is the virtual assaults so this is basically when you have The possibility of getting really close someone else. I know it's virtual, but it doesn't feel virtual So luckily it hasn't happened to me yet But I have read a lot of stories about this and I've seen it in some places as well of people that are Maybe dressed as a female and then someone else is getting too close and pretending they're touching them so the Distance of which users can be together is very important as well and we are and we should check that it's going to be comfortable for the other person and It happens a lot as well in VR chats when someone gets recognized as a famous YouTube for example That people just straight away going to them and pretend and play around So this make the person feel really uncomfortable and maybe they don't want to use our app after So it's something very particular of VR that we should be testing and Just another thing It's not really how to test in VR. It's more about how to use VR for teaching testing So right now I do have an app which is like a virtual museum. It's very small You go around and you learn a little bit about what the different type of testings exist are But you can use VR for for teaching testing There's a lot of applications that are for teaching things That is either dangerous or very expensive to use in the real world So we just use a VR in order to learn them and One particular case is very interesting is Submarines so apparently if you stay in the submarine for a very long time You can guess because of the pressure us you can get many issues for example you can get cross-eyed Just because of the submarine so they were thinking in applications Like VR to do the testing for them so they can practice before being in the actual submarine And I listened to this in a podcast not long ago and I was thinking okay great But is it safe to be in VR for a very long time? Because the way VR work is basically you have two images a little bit a split from each other Really close to your eyes So obviously being in VR for a very long time is also not very healthy So we should have some reminder for the user some First reminder of the be careful have enough space around you And then maybe if they're using it for a very long time remind them as well to take breaks So if you want to get into VR, I'm just gonna tell you what I did to get into VR So my first experience with VR was in a employee hackathon with Microsoft and Just one of the interns I was part of the organization and one of the inter brought their occulus drift was I think the second version still didn't have the handset and So that was my first my first experience. I thought well This is really cool, but it's very expensive people are not gonna like you have to be really either geeky or Really like games in order to buy one of these right? But then for the long they added the handset and that really changed things prices are also getting a slightly Leave it a little bit cheaper Like this I think was me. I don't know and rupees how much you will be back. Maybe Less for sure less than 10 euros. Just this advice, but they're getting very very expensive There's courses online on how to build them if you want to be one is not hard And then you can use it on your phone Everybody has a phone nowadays that is smart phone. They will have the year scope, which is what is used for that so That is now something that many users can can use Other events I went to I also attend a hackathon that was organized by NASA in which we did AR and VR and In this hackathon by the way, we won the popular vote And I got a little me more feeling about how that works I also attend another event, which she was called tech tech for good in Dublin And that's basically they were using VR in order to test different Experiences and make people aware of things just by basically living those experiences through VR so that was also very interesting and Watch online playthroughs again. I'm a bit geeky on that sense I like watching all others play but when you're watching people using the application or something similar to the application that you're Developing developing or testing That's really make you understand what they need what they want what they appreciate in this application So it's also I think a good thing to do You can get into a course I have been coursing the VR developing nano degree for a while I find it really interesting and learn many things performance all about performance. I learned it on on that course You don't need to be developer to course this you can just Want to know more about VR in they will teach you VR They will teach you how to create videos So it's it's a very nice course. So I would recommend but I'm sure then there will be more not only that one And also play around just Don't watch some VR application in your phone if you have Android. There's the Google VR They have many applications there There's also a lot of applications online that you can download you can get maybe unity or maybe on web VR Try to learn a little bit about that play around try to create an application. I just created the meeting Application basically I asked the organizers for pictures of this and now if you look at the Application if you want after the meeting you can come over and I can show it to you You may think oh, it's not exactly the same but consider that this was done with just pictures, right from pictures I got a VR application. So how interesting that you can do it. It wasn't hard It was it didn't take me so long and I got an application work working So, how does the future of VR look like? This is a picture of this sonar Hong Kong which is a electronic festival. I won this year In which they they show what Kai is some some other scenarios, right? So they did have some all-factory PR So basically you were wearing your headset and someone was spraying some things around you to get you the feeling that you're in In a different maybe in a forest and then they switch and then you get the feeling that you're in a different room So that was very interesting as well Obviously, it's hard to take to the user because how did you take the actual mess? But it's something that can happen. We are very Developing faster world so it can it can be it can be something and they also had this other thing It was for snow. I think in which The platform was moving with the girl. So they had this thing so you can hold it, but it's also like kind of Moving it with the wind So the our application is not just going to stop on the handset. They're going to be platforms they're going to be and other materials like this Like this big thing that she's calling right so a Again any hardware device is different The many things that are coming up and I think it's going to be very fast First thing that is going to happen probably is the wireless There's already some HTC and some Oculus that are absolutely wireless. So one of the issues with the VR right now I will call it again from the VR safety is that The wires you can tangle with the wire you can fell off because of them. So making them wireless is a really big advance These devices also can localize where are you in the room? So you can actually walk around without the needing of a platform The only problem with that is that you have to have a really big room or being outside and even so it's still dangerous because you can walking into the wall for example and There are many examples of that on the internet if you like dark humor you probably can search for them and Of people fell falling off or doing crazy things because of wearing the headsets and That's all. I've been very very fast. I can come over and Review other things, but do you have any questions? Don't be shy Very nice presentation. Do you think it's at all possible to automate any process of VR? Yes, definitely. Yeah, so one of the things that I face when I was working on this Air test project that I mentioned before is that there's one thing in particular different from VR than other application Which is that you don't really click on an object you look at the object and then you click anywhere So that's that's something different, but it's absolutely doable. So Again, we are working on that in this project, but I know other people are also working on this It's absolutely doable. Yeah, but I was like wondering about the accuracy of something so immersive because generally I work with an application, it's really static and We use selenium and other automation frameworks exclusively because we know what's gonna happen But for something as immersive as and particularly Unpredictable as VR Isn't it a great challenge to address the automation problem in testing VR apps? So definitely if there is a challenge on addressing automation, but it's not as as big as you would think Yeah, so it's it's kind of basically the same thing. So you have yes, you have So you consider the experience as an interface rather than yeah, yeah, okay So you have a 260 degrees, but you can still you know look around you can click objects you can Verify the object integration. It's exactly the same as any other application. Yeah, exactly the same. Thank you So follow up on that. Do you use unity to simulate all this behavior? Yeah. Yeah, we use in unity for this Okay because on on this particular project I was talking about we do have our unity Solution so you can work and gains and automate gains already with this I'm just adding the VR functionality. It's pretty interesting. I'm really really happy to be working there. Yeah Do you have any more questions? Yeah, so there are many applications online that you can So again, just in case someone didn't hear him. He's asking if there's examples of applications There are many examples. You can download the VR A cardboard application for Google in your Android and in there you see some applications already You can also Google for VR web and in VR web is I think also Google has something to do with it And you have also many many of different applications that you can connect your phone to and test Also, if you want to go a bit more deep and you want to buy a device And you have the device at home You can also have the Applications for sampling the steam there are many many many applications and games created there If you want to buy a device my first advice will be check if your computer is Powerful enough You can do that by basically going to the website of the device that you're interested in and they usually have Some download Xe that you can use in your computer to check and verify that is Powerful enough. It happened to me that I try I wanted to really want to buy an HTC But my computer was not powerful enough, but it was powerful enough for mixed reality. So I got a Samsung device So really you need to be careful with that Don't buy just a device and then you have to buy a new laptop after unless you run to buy a new If we don't have more questions, I probably have skipped some things I can go over again We have time Okay, so and again just if there's some more questions I'll go over this a little bit. So this is my blog my Twitter account my LinkedIn my github in which I will upload the meeting simulation This is the project. I'm working on with NetEase That's the Twitter again. They're link their website and the github account and Just want to leave you with a thought so I don't know if you know about it there's something that is called the simulation hypothesis and That's basically what if we're already living in a simulated world. Thank you so much