 My name is Thomas and today I want to talk to you about the explorations that we've been doing with the Zoom team over the past few months and some of the specific advanced APIs that we've been exploring together. As you've all probably seen, Zoom has become a stable in many homes and so it's critical that we're able to provide a good experience directly through the browser. Zoom does have a web version today but compared to its native client, it's missing some features and sometimes misses the mark in performance. We wanted to change this and so even before COVID hit, we started working with the Zoom team to understand exactly what changes they would need and what new things in Chrome they would want to create a truly great experience. Now, Zoom is of course a video conferencing application. You can't talk about video conferencing on the web without talking about WebRTC. WebRTC is a really great full stack solution that provides a complete package for achieving video conferencing on the web. WebRTC was built and standardized about 10 years ago and now ships in all major browsers. This makes it the best choice if you want a complete solution with broad support across browsers. WebRTC's strength of being that complete solution can however also be a challenge for someone like Zoom who have their own custom protocols and their own architecture. Zoom would rather want a set of more simple low-level APIs that they can then build their own architecture and system on top of themselves. And the three specific ones that we've been exploring and that I want to talk to you a little bit about today are WebAssembly SIMD, WebTransport and WebCodex. I'll mention from the start that all of these are fairly cutting edge and most of them are in active development. So while they're all in a place where you can start to play with them, these aren't shipping APIs just yet. But hopefully this presentation will cover some of the early parts of it and by the time you're watching this it might be in the future and you'll be able to actually use these directly and the law already shipped. So first of all I want to talk about WebAssembly SIMD and how it can provide really highly performance code. Most of you have probably heard about WebAssembly already but as a recap WebAssembly is a new low-level binary format for the web that is compiled from other languages and offers maximized performance. This means that you can take something like C++ or Rust and then compile it into WebAssembly before shipping it to the client. WebAssembly has been out for a while and has been shipping in all major browsers for a while but we're continuing to expand it with functionalities such as SIMD which stands for single instruction multiple data. To explain SIMD let's look at this incredibly simple loop that just adds two arrays together. Without SIMD the CPU would go through this loop and add their different elements together one by one requiring four full steps but with SIMD the CPU is able to vectorize these elements and then take just a single CPU operation to add them. The best part is that because compilers are so smart they can automatically detect these optimizations and do them for you. In Inscriptin you just need to pass the dash m simd 128 argument to emcc and for Rust you can pass dash c target feature equals plus SIMD 128. This will cause the compilers to automatically find and use SIMD where possible. Sometimes you also want to have more explicit control and this is where you'll want to use SIMD intrinsics which let you use the SIMD instructions directly. This is more detail than I can cover here but if you're interested I highly encourage you to go and check out these links. SIMD can be used for a huge variety of things including highly performant ML models such as this hand tracking, a real-life invisibility cloak, and real-time automated background removal. And this last use case is just one of the things that Zoom is excited about using SIMD for. They have an awesome feature where you're able to automatically remove the background so that people in conferences can't see all the random stuff that you have in your background and then replace it with fun videos or animations. If you're interested in diving into WebAssembly and SIMD here are some of the links that should help you get started. WebAssembly SIMD is doing an origin trial in Chrome M84 which will start rolling out to users in July 14th. If you aren't familiar with origin trials it's basically a mechanism for you to test out features with production users while we may still be making some changes to the API. You can read more about those origin trials at this link as well. So the next API I want to get into is WebTransport which is a next-generation networking API for client-to-server communication. Let's look at the definition of WebTransport. WebTransport provides bidirectional transport through both unreliable datagrams and reliable streams-based mechanisms. That's a mouthful but let's see if we can't break it down and understand it a bit better. First, bidirectional means that it enables easy two-way communication. With something like HTTP the connection has to be initiated by the client and you have to send all of the requests at once and then wait for a response. With WebTransport you don't have these limitations and so you can enable a much more interactive session. Looking at the two different mechanisms unreliable datagrams are one of the mechanisms for sending data through WebTransport. These datagrams are similar to UDP datagrams in that they are packets of information that gets sent but with no guarantees about delivery or ordering. Reliable streams in contrast are similar to TCP streams and provide reliable and ordered data communication. So now that we have an understanding of the definition of WebTransport let's understand what you might actually use it for. Firstly, WebTransport will be the only mechanism to do unreliable data communication without leveraging WebRTC and this is exactly why Zoom is interested in looking into WebTransport because it'll allow them to simplify their deployment and put it a little more in line with the other platforms that they support. It's important to note though that WebTransport won't be just a pure UDP sockets API since it does have some requirements around encryption and congestion control. It does offer an alternative to WebSockets and to understand exactly how it compares to WebSockets and WebRTC let's look at this chart. So to understand the differences let's dig into each of these pieces. First, WebTransport and WebRTC offer both reliable and unreliable while WebSockets only offer reliable delivery. WebTransport is an in development API while both WebSockets and WebRTC are widely available. While WebRTC provides a fairly high level complete solution to the problem of video conferencing, WebTransport and WebSockets are both much lower level APIs that doesn't solve everything for you but gives you more of that basic access. WebTransport also enables multiple cancelable streams whereas WebSockets can only do a single stream and WebRTC can also do multiple streams but they aren't cancelable. So here is a quick example setup for how you can actually use WebTransport. In this part of the code we really just set up our new quick transport which is specific subtype of WebTransport and create that object passing in the URL that we want to connect to. Then we just set up some simple logging function and await the transport being ready for us to use. Then we can simply grab the writer from the sendDatagrams function of our transport object which we can then use to send data at any point. Remember that this data that we send does not have any guarantees of delivery or the order that it will be delivered in. Next let's look at how you can actually read data from the server. Here you see a simple example where we get the reader from the getReader function and then in a classic while true loop we just read things from that reader and then detect them when we're done and console log out the actual values that we're able to read. WebTransport is still very much in development but we do have a blog post already published about how you can use it and you can find that and more information at these various links. So now it's time for us to jump into our last and exciting API the WebCodex API which aims to offer direct codec access on the web. But first let's back up and remind ourselves what exactly a codec is. A codec is a device or computer program which encodes and decodes a digital stream or signal. While many of us have not worked directly with codecs we've all seen common examples like mp3, vp9, h264 and many others. Codecs are actually used in tons of places throughout Chrome such as the audio and video tags, web audio, webRTC and the media recorder API. However in all these places where it's used you can't really configure and get pure access to just the codec part. For example web audio allows for decoding a media file but needs to work on the complete file and doesn't support a streaming-based approach. Media recorder has some controls but they're very high level and you can't really configure it to support extremely low latency use cases. As mentioned previously webRTC does give you a lot of this control but it needs you to bring the whole package of webRTC along and without doing that it's hard to get access to just the encoding and decoding parts that you want. As a result of this lack of configuration some apps have started compiling these codecs to JavaScript and WebAssembly. Some of you may remember that this is how the awesome application Squoosh lets you resize and re-encode images. This approach is really cool and workable today but has some specific drawbacks. Specifically it increases your bundle size lowers the performance causes slower startup time and reduces the power efficiency. Really what you want is to avoid shipping these codecs all together and just get the direct access that you need through the codecs that are already shipping as part of the browser. And that's exactly what the goal of WebCodecs is and in their own words the goal of WebCodecs is to provide web apps with efficient access to built-in both software and hardware media encoders and decoders for encoding and decoding media. WebCodecs main advantage is that it lets you get the direct access that you need to again build your own systems on top of the basic codec access. This completely unlocks some use cases like video editing since you really need that frame by frame access and faster than real-time encoding and decoding to do this properly something that's currently completely impossible on the web platform except for maybe shipping codecs with WebAssembly. Additionally many existing things that are possible to stay on the web but only if you use WebRTC things like cloud gaming live streaming and video conferencing will get more flexibility about how they can interact with these codecs. Zoom for example is looking into using this API in conjunction with the web transport API. They're hoping that they'll be able to take encoded video frames and send them up to the server using web transport at the same time that they'll be fetching down encoded frames and then decoding them to show to the client providing a really smooth integrated experience. Next let's look at some of the simple examples for how you can use the decoder part of this. Here in this canvas setup part we're really just grabbing a canvas's context and then making this very simple function to paint a video frame to that canvas from converting it to an image bib map. Now when you want to set up the decoder part you just call this new video decoder element and you set up the output function that we defined previously as well as just constant logging out in the errors. Then you configure it with the codec that you want to use and then you have this incredibly simple function that you just pass in your encoded chunk and call the decode function from your video decoder and then it does the rest of the work for you. Web codecs is still extremely new but for those of you who are curious you can go and check out the explainer to see what the team is currently working on. We will also be doing a web.dev post about web codecs so if you're seeing this in the future be sure to go and check out that and that brings us to our overview of these three new exciting APIs that we've been exploring with Zoom. You hopefully get a better understanding of some of these new and advanced APIs and hopefully an understanding of how they will be bringing all of us closer together in the future. Thank you so much for your time and I hope you enjoy the rest of the sessions.