 Hi, I'm Ada Rose Cannon. I'm here to talk to you today about WebXR, both what it can do today and what's planned for the future. First, a little about me and who I work for. I'm a developer advocate for the web browser Samsung Internet. Samsung Internet is a really good, highly customizable Android web browser. We have a focus on privacy and security. We also have built-in tracking blocking and extensions for ad blocking. As a developer advocate, I try to help developers with the latest features in the web platform to build websites which make the most of everything the web browser can offer. If you have any questions about Samsung Internet or WebXR, please reach out to me, either through email or social media. As well as being a developer advocate, I'm also co-chair of the W3C Immersive Web Working Group and the Immersive Web Community Group. These two groups work together to develop web standards for working with immersive hardware, such as VR headsets. The core part of this effort is the WebXR Device API, or WebXR for short, WebXR at its core. An API designed to access the sensors and displays of immersive hardware, such as VR headsets and AR headsets. By knowing what the user is looking at, we can render a scene using WebGL from the user's point of view. We can then send this view to be displayed on the headset so that when they move their head, they see what they would be seeing if they were inside the virtual scene. This is how you get the virtual reality effect. The main difficult part in building these APIs is supporting the wide range of immersive devices with their wide range of capabilities. The first type of headset we supported were immersive VR headsets, like the HTC Vive and the Oculus Rift. Some types of immersive VR headsets are tethered to computers and are very powerful, but usually require a direct wired connection. These usually both have very high quality tracking and very good graphical fidelity, but there are other VR devices. The old Gear VR, which is no longer supported, was a very popular headset, but it was driven just by a phone. Modern standalone headsets, like the Oculus Quest, are similar in power because they are also driven by a phone chipset, but they have the great tracking, which you would typically find in the more high-end headsets because of this great tracking giving very good immersive experience and the very good user experience on their ease of setting up. These have proven to be a very popular type of headset nowadays. A bit later than immersive VR headsets came immersive augmented reality headsets. This is also known as mixed reality. These headsets, like the HoloLens and the Magic Leap, show the potential of immersive augmented reality. These work by overlaying virtual content onto your physical surroundings. It can scan your environment. Objects are placed on the floor, walls and tables. It's a very incredible experience if you ever get to have a go at it. These headsets are normally very expensive though, but currently they're more popular in industry than with consumers. Immersive technology isn't just about VR headsets anymore though. In the beginning, a popular method of engaging users in WebVR was to give users a preview of the virtual reality experience on their handset so it kind of looks like a window into the virtual world. This is something which needs to be supported today through WebXR. A few years ago, Pokemon Go took the world by storm and it's still pretty popular. It was a huge commercial success and it showed how you can combine and held augmented reality with a really engaging user experience to build a fantastic product. This is another kind of experience the WebXR device API should find support. These examples really show that immersive content in the Web isn't just head-mounted anymore. I mentioned just now the API called WebVR. WebVR just targeted VR headsets, which is in the top left quarter of this chart. But because of the wide range of devices that now need to be supported, WebVR wasn't going to work long into the future. A new API needed to be developed to handle augmented reality and handset-based experiences and that's where WebXR comes in. And even though WebVR was deprecated at the end of last year, I hope this is the beginning of a long future of virtual reality on the Web. So this is what virtual reality on the Web looks like today. In this example, the user navigates to a page, they press the enter VR button and then they are placed in an immersive scene where they can use their controllers to interact with the world. This video was recorded in Firefox Reality, the Oculus Quest. But because it uses WebXR, it'll work in any VR headset if the browser for it supports WebXR. You can try it out here if you want. If you have a VR headset, try using it in that. If you have a cardboard headset, try opening it up in your phone and trying it just on your phone and then in the cardboard headset. This demonstration though, only shows off the very core module of WebXR and there is so much more to WebXR than what is shown here. And that is what I aim to speak to you about today. The first feature I'd like to point out is a really important one because when I recorded that demo, I was using the Oculus Quest headset. The controllers look like this, but if you remember, that's not what the controllers look like in my hand in that demo. I was holding those gray sticks and ideally I want to see the real hardware I'm holding in my hand in the virtual scene. The WebXR device API has two things which work together to accomplish this. The first is the WebXR gamepad API. This tells you what buttons are being pressed on the gamepad. Normally in WebXR, the only events you get are the select event, which is usually the device trigger, and the squeeze event, which is from the button usually triggered by your other three fingers. The gamepad API gives you a gamepad API representation of the controller. Because it's a gamepad API, it's polling-based, it's not event-based, but it behaves exactly the same as you would expect from the JavaScript API. The only difference is that you can't access the XR controllers from the gamepad API itself. These are exposed on the XR input object. The second part is a library which is maintained by the immersive web groups. This library is a complex project with three main parts. It has a large suite of 3D models or as many VR controllers as we can get. These models are usually provided by the manufacturer and they're ready to use on the web. They're fully rigged for animation. You also have JSON files, which describe how the various controller buttons from the hardware match to their index in the gamepad API so that when you push the button and it is revealed to the gamepad API, you know the corresponding part of the 3D model it represents. You also have a library to help handle this connection. Hardware vendors can update the library to add their new hardware whenever a new hardware is released. This enables virtual reality experiences to continue to work well into the future. The end result is that while holding the real hardware you have in your hand, you can see the model and the buttons on the model will move as you press them on the actual hardware. This functionality is even built into some of the popular 3D libraries so they make it pretty simple to use. Once we've built this functionality, this is where we considered WebXR VR complete and ready to replace WebVR. That's only half of the XR story. The other half is augmented reality. Augmented reality is brought to WebXR by the AR module. It primarily handles making the screen transparent. Either by having the cameras on the outside of the headset, play their image behind your rendered scene. On phones it will use the rear camera to display the world onto the screen. Some AR headsets already have a transparent screen and this enables it to work on those as well. Many VR headsets have externally facing cameras too but they only use them for position tracking and don't let them use them for AR. But this is missing a key feature of augmented reality. Although you can render any content you want onto the user's environment, you don't know where in the environment you should render it because you can't tell where their tables and walls and other objects you might want to place objects on are. So we need to work out where it is and to do this we use the Hit Testing API. The Hit Testing API allows you to cast a ray out into the world and get the real-life position relative to the augmented reality scene. In this video I'm placing a dinosaur in my living room. At the beginning, each frame, I work out the new position from the center of the screen and I replace the dinosaur there. Once I am done and I tap the screen I stop using the Hit Test API and just leave the dinosaur wherever it last was. It then stays in place and I can interact with it by combining the ability to scan the real world and display virtual content over the real world. We have the cornerstones of augmented reality and can build many experiences. You can try this out in Chrome today and it will be coming to Samsung Internet very soon. So now we have the core features for virtual reality and augmented reality. How do we actually build these? Well, firstly because WebXR is WebGL based you need to make a WebGL scene. You can't use any HTML or CSS with WebXR at least not yet. The trouble with working with WebGL is that it's extremely verbose and very low level. The code for just rendering a single cube is hundreds of lines long. So it's best to work with a library, of which there are a few. There are some of the libraries I've come across in my day-to-day work. BGS is probably the most well-known. It's maintained by the community and it's a really good VD engine. Then there's A-Frame. A-Frame built on top of 3JS by providing a web component wrapper for it. BabylonJS is another really good 3D engine. It's maintained by Microsoft and has many features for doing high-fidelity graphics. React 360 is kind of like A-Frame in that it's a wrapper around 3JS. But it's a React wrapper rather than a HTML one. And finally PlayCanverse, which is another really good 3D engine. I'm going to go through all of these, but I'm going to show you A-Frame first because it's the one I probably use most in my day-to-day. A-Frame is a web component wrapper around 3JS. It allows you to create a 3D scene declaratively using HTML. So you write this. This is a full HTML document. Head-to-header tags and body tags and everything. And you put in elements, like a box, a sphere, and a cylinder. And it renders it with WebGL to the scene. And since it just uses normal web component, you can control them with the same JavaScript you would use on a normal HTML document. It's also extremely extensible. Since it's built on 3JS, if you're handy with JavaScript and OK with 3JS, you can start writing your own components to take it even further. And it has a large and active community. So the finding help isn't too bad. Out of the box, it's ready to use virtual reality and augmented reality whenever the browser supports it. This is really incredible. It's a great way to get started building VR and AR if it's your first time doing any kind of graphics development. But since it's a JavaScript conference, you'd probably prefer something more imperative, such as 3JS, rather than something declarative like A-Frame. So I'm going to go through how you set up WebXR in 3JS. But this is assuming you already have some kind of VR scene set up and ready to go in 3JS. Fortunately, 3JS has really good documentation. Loads of examples and a large and active community. Your findings on boilerplate code to start from shouldn't be too bad. Once you've got your scene set up, you need to import the VR button module. This module lets you get a button and a month. You can add to your document. This is the button the user needs to start virtual reality. This is because a user interaction is required to start the WebXR device API. Next, you need to set XR enable. This is because some laptops have multiple graphics card. The VR headset will be driven by the more powerful one. But if your scene is rendered on the lower powered graphics card, then it won't be able to be displayed on the headset. So you need to make sure it's rendered on the high powered one. You also need to use the built-in animation. This may seem strange because if you use 3JS, or if you've done any kind of animation in the web before, you may be used to the request animation frame API. This is what this is using under the hood. WebXR provides two different request animation frames. There's the one for your normal monitor, the one you'd use as normal. But there's also the one for the headset, because the headset will have its own frame time. And often the headset will run a lot faster than the frame timings of your monitor. VR headsets can run not just at 60 frames per second, but sometimes 90 or even 140 frames per second. This gives you very tight budgets for rendering your scene. So be wary. Making sure stuff can be rendered efficiently. But now we've covered what you can do today. Let's take a look at what's coming around the corner. Firstly, I'd like to show you is the DOM overlay API. Because right now you can't use HTML for anything in WebXR. It's purely a WebGL API. The DOM overlay API will let you pick a HTML element, and you can take it into the WebXR scene with you. This lets you make controls and interfaces using just HTML and CSS, which can be interacted with in the usual fashion, like JavaScript, add event listeners. You can set classes and everything. The DOM overlay API is very similar to the existing JavaScript full-screen API. The DOM overlay API is similar to the existing JavaScript full-screen API. The user agent will take care of where to place it. So if you're doing automatic reality on a phone, it'll probably just replace it full-screen on the phone on top of your scene. If you're in an augmented reality headset, it'll probably place it headlocked around the user. And it'll do other things in VR as well. But using HTML like this is really good. Because it can be rendered by the user agent in a way that looks much clearer. So especially if you're doing stuff with text, it'll get a much better result than if you were just rendering text to a bitmap texture and then displaying it on a 3D model. You also gain access to the accessibility features of HTML. So it should work fine with a screen reader and be very usable. On top of that, you have all the 2D layout power of CSS. So you can build really fancy interfaces, the kind of thing that will be very difficult to implement from scratch in WebGL. Lighting estimation. Lighting estimation enables virtual objects to approximate the lighting from the surrounding scene. When virtual objects are lit in a similar way to real objects from the user's environment, it feels like they're a much more natural part of it. This enables a much more engaging or augmented reality experience. A feature that has a similar effect, and which is very subtle, but gives a huge improvement is anchors. And I haven't got a good illustration of anchors because the effect is that subtle. And it looks to solve an issue. So as AR scenes run, the underlying platform and the coordinate system might need updating. But if you do this, what happens to the objects you've already placed in the scene? Do you have to change the scale or slightly move it? All of the objects are going to get slightly shifted and are going to drift slightly. But if you placed something on a table ten minutes ago and you come back to it, you might find it's moved by a few centimetres. And this just doesn't feel that realistic. But by placing an anchor, items can be attached to that instead. So when the environment is updated, the underlying system can make sure that anchor stays in the same place it would have done. So any objects connected to that anchor will also maintain their position. This greatly increases the visual fidelity of the scene in a very subconscious way. Everything just feels more real, more anchored in reality. Because as you move your head, stuff doesn't slowly drift or move as you look around it. It's a much better experience. This is a feature that many of the higher end augmented reality platforms have built in, and it just feels really good. The next feature is layers. Layers are a method of efficiently displaying images and videos on simple shapes in the scene. Kind of simple shapes are a rectangle. The inside of a cylinder or the inside of a sphere. The rendering of these layers is handled by the user agent, instead of by using WebGL, which provides two significant benefits. It's far more efficient because you don't have to copy the textures from the video element into WebGL just to render them to the headset. The user agent can just render them directly. It also gives you clearer rendering because they can be rendered a lot closer to the time when the actual pixels themselves light up on the screen. It gives you a much crisper experience because they're not slowly moving as the user's eyes and head move. Normally reading text in VR is something that's not recommended and usually a very poor experience. But if you do it using this technique, it's actually not too bad. The text really does stay in place and is very comfortable. The last feature I want to highlight is hands. So some AR and VR systems have the capability of tracking your hands instead of using controllers. This is a really naturalistic way to engage with virtual objects. You can poke stuff or pick it up or inch to grab it. It's a really nice way to interact with a scene and really makes you feel as if you're part of it and is a feature of which I am very much looking forward to. If you want to find out more, please check out these resources. I maintain a website called immersiveweb.dev where I try and keep up-to-date information about useful tools and techniques, some articles and some instructions on how to get started with different systems. If you want to see what's being worked on right now in the immersive web working group, check out our GitHub. Everything we do is done in the open. You can read our old minutes. You can look at the standards as they're being written. Check out the PRs. And if you want to see what's coming and some of the more features which are on our radar, check out our charter in which we outline some of the stuff which we intend to look at over the next couple of years. And finally, if your company is a W3C member and you want to get involved in building some of these standards, ask your AC rep about participating in the immersive web working group. Thank you so much for listening. I've been Ada Rose Cannon and I really hope you have fun in the immersive web site. Thank you so much.