 One third of the time spent in Chrome is spent with media. So that's listening to music and videos and cat videos. Give or take, that is about 40,000 years worth in a single day. For context, the Neanderthals were still walking around 40,000 years ago, so this kind of number is mind-boggling. But honestly, I think it misses the mark for why it's important. So quick story. I was a painfully shy kid growing up in a small town in Florida. And then there was this new peer-to-peer music sharing service. And for the first time, I had access to thoughts and ideas that I wouldn't have been able to experience in my little town. And so my world view just opened up in a way thanks to web media. So I felt confident and rebellious. And my parents hated it. This was a CD my best friend burned with some of our favorite crappy downloaded punk songs, with us being punk, obviously. The music I found back then on the web helped me find my voice. And I feel really lucky to get to work in that space today, especially now that it doesn't take 12 hours to download a single song. Creating movies and music, that's really hard. But we think that putting it on the web should be easy. And that's what our team tries to do. We build tools to make it easier for developers and content creators to connect with Beardo punks or whoever your demographic is. So today I want to talk about three ways to do that. Number one, by the end of this talk, you'll learn how you can increase watch time and close a gap between native apps and the web using two of our newest APIs. Number two, you'll learn what went into the redesign of Chrome's native media controls. And then my engineering partner is going to share how you can get them for your application using Chakra. And number three, you learn about a user-centric framework that underpins all of our work. You'll see how we used it to connect people to content no matter the device or platform. So let's start there. Users take in media in one of three ways. Immersively, totally focused on the content. Transantly, just kind of in and out, keeping an eye on things and ambiently. Ambient music experiences is like when you're working and you've got some music going in the background, the music is just kind of there. So let's talk about this one first. We know that users listen to a lot of music in the background. In Chrome, 60% of audio is played in the background, which is there if you couldn't spot it. It's not too bad if you like our typical user only has a handful of tabs, but if you're in this room, I'm willing to bet that it looks a bit more like that. We all know that trying to find music in a sea of tabs is annoying, but I want to nerd out for a second on why. Of course, it's hard to find tiny icons, but it really starts with brains. Human brains can't do two things at the same time, especially if they're similar. So for example, you can't read and listen to someone speak because they're both trying to use the verbal processes of your brain. And so this is why it can be extra frustrating when there's sound coming from some mystery tab. Your brain is trying to do two similar functions, reading and then suddenly hearing an ad or lyrics. Good for us humans, though, we don't have to do two things at the same time because our brains can switch back and forth quickly. The catch is it increases cognitive load, and that's bad for users because rapid tasks switching increases cognitive load, and that makes people up to 40% slower than if they were focused on a single task. So for example, in ambient situations, when I'm not paying a ton of attention to the music playing in the background, the moment I start writing this email, the lyrics from the song suddenly become really distracting because they're both verbal, especially if I need to look up something in another tab, because then my brain is switching between reading, writing, and audibly comprehending lyrics. So designing for ambient media is all about controlling content in ways that don't compete for the same brain processes. In my case as a designer at Chrome, it's a pretty safe assumption to think that people are going to be reading stuff on the web. So to pause the song, the keyboard is a better place since using the hardware keys doesn't require verbally comprehending an interface. And that might sound obvious since native apps have that functionality for a while, but until recently, media keys haven't been available in the browser. We launched keyboard support for media keys in March for play, pause, and stop. And since then, more than half of their usage has been from media being controlled in the background. We've gotten support from partners like Spotify. They've implemented the API to bridge the feature gap between their PWA and their OS native apps. The functionality in and of itself might seem tiny. It's wins like these that help us close the feature gap between what's possible on the web and what's possible with apps. So designing for ambient media is all about controlling content in ways that don't compete for the same kind of brain processes. This helps users stay focused on their primary task and doesn't generate extra cognitive load. In your own work, you might think about adding alternate input types into high-frequency kind of actions. So another example I love is how you could just turn your phone upside down to silence it. It's using the gyroscope and light sensors to make that happen. So it's a different kind of input. So next, I want to talk about creating for transient media experiences. Developers and PMs, I think you're going to like this one. Transient media goes back and forth between low and high focus. The content itself could be long-form. So imagine you're watching a soccer game, and it's just sort of in the periphery until the announcer yells, and then it suddenly has your full attention. This is exactly what we created the Picture in Picture API for. It gives any site the ability to pop videos or music into its own Chrome window that always stays on top. To go back to the story about brains for a second, picture-in-picture isn't so much about watching video at exactly the same time as doing something else, because we know that's not possible. Rather, we're making video easier to access so that users can stay oriented in their primary task and just keep an eye out. Nest has implemented it. And so while I'm working, I like to keep an eye out on my dog. This is my dog, Sweet Potato. Since adding picture-in-picture support, Nest times watch time is up almost 4%, which could very well be attributed to me just watching Sweet Potato. What I can't take credit for, though, is 10 cent. They've implemented picture-in-picture with huge results. They've seen a 16% increase in watch time and a 16% improvement in completion rates. For those of you in the media space, you know what kind of revenue correlates with bumps in watch time of that magnitude, and it's significant. Spotify did something else that was really cool. They made the picture-in-picture window into a music player. So rather than playing video, it uses album artwork to create a small media player. Personally, I think that's been really cool about this project is to be able to get to work more closely with partner teams. As a designer, I've thought about my role as the user's advocate in the past. And so since picture-in-picture is an API rather than something baked into the browser, I had to think about the need of content providers as well. So we started with a really basic MVP, just two controls, close and pause. We worked closely with our partners to understand how we could expand the feature set. So far, we've added previous to next, mute, and back to tab. It's still early in the process, but it's been very cool to be able to deliver meaningful results with just a really basic MVP. To recap, in transient use cases, you'll remember that media should be easy to control, even if somebody is not paying full attention and that the content can be longer short. Across all platforms, one way to make that happen is by making the UI absurdly easy to comprehend. Like so overtly simple that you think it's probably overkill and that's when you know you've got the right amount. Picture-in-picture is an example of how we apply that principle to desktop. So I want to talk about how it worked for us on Android. Android was a really important platform to get right because YouTube's mobile site uses our controls. So you might be able to see here, in our previous generation, our controls were having a little bit of trouble. They're a little difficult to understand at a glance. There's a lot of icons and different colors which makes my eye bounce around. The controls also stay on screen while the video is playing, and so since they're always visible, they can be a little bit distracting. And lastly, the scrubber isn't very easy to interact with because it's so small. This makes seeking really difficult. One user went so far as to say, scrubbing is annoying. It's just tiny. I might as well not touch it. Ouch. So we redesigned them from the bottom up. We spent nearly two years conducting research and drawing on expertise from YouTube and iterating and then failing and then iterating some more until we were able to launch what you'll see next. The result is a finished product that we believe helps elevate the standard for what Media on the Web could be. After that, my engineering partner is gonna share how you can get these controls for your application using Chakra. But first, let's deep dive on what's working and why. On mobile, we were able to make the play button four and a half times larger by booming it to the center. This increased the tap target and really increased usability, especially on smaller devices. This also made room for a larger seek bar. So now, instead of just taking up the 144 DP that it used to occupy, it now spans the entire video width. This makes it a lot more friendly for fat fingers and all fingers. This change also allowed us to add double tap to seek functionality. This is YouTube's signature gesture and it's had remarkable success. It's used twice as much as the scrubber and so being able to include it has been a really huge win. Volume is also a little bit different on mobile. The most common way to control volume on mobile is to use the rocker buttons on the side of the phone. So we removed the volume slider, reduced it to a simple mute toggle. So now, the controls lay on top of the video instead of beneath and so they're bigger and they're more conspicuous. I used a dark scrim to create a high amount of contrast between the video and the controls and so this removed the eye-bounced problem that we were seeing before. So as a result, the interface is more scannable. And lastly, since all that UI lays on top, it's important that it hides when it's not in use. The controls are there when you're interacting with them and then gone when they're playing. The result is a mobile player that's optimized for transient playback. It's scannable, it's ephemeral and it's easy to seek. As a result, it's easier to comprehend at a glance. I'll say one of the really great things about working and creating on the web is that you make something once and it works everywhere. And then one of the tough things about creating for the web is that you make something once and it really has to work everywhere. It's one of the things that makes this a really fascinating problem space. As a designer, I love constraints like these. It's like those cooking competition shows where the test of a true chef is if you can create something amazing out of soggy broccoli and a tube sock. The space is riddled with constraints like that. In particular, we have a shared code base for all of our platforms. So this means the same UI has to work on tiny phones, like the ones that are popular in emerging markets and desktop and giant TVs. They have to work with any input type. So touch on mobile and then of course on a mouse or even a stylus on desktop. And then it's got to work on any and all OS. Just imagine having to put the same UI on Android as you have on Mac. They have totally different navigation paradigms and so this is really challenging. So because of that, we decided to make a couple of selective modifications. These were crucial because larger devices are more typical of the third type of media experience I want to talk about today, which is the immersive kind. Immersive media experiences absorb you in content. This is the bowl of popcorn with the lights out kind of experience. You'll remember on mobile, the space was constrained and large tap targets are better of course. And we moved the play button into the video frame. In immersive cases like those on desktop though, it was important not to obstruct the video. So we moved it to the bottom. Another optimization, the scrubber head goes away when you're not using it because it's not necessary to create a larger affordance when you're using the mouse. You've got more precision. And then more behavioral nuances. On mobile, the expectation is that double tapping would rewind or fast forward. On desktop though, the expectation for a double click is that it goes to full screen. This is one example out of probably a dozen of these very nuanced differences that we needed to make sure we incorporated. Going back to volume, the mental model volume is also different on desktop and in immersive cases generally. On mobile, users tend to think of volume as sort of one and the same as what's happening in the media stream. But in immersive cases, they're thought of as two separate volumes. Users kind of intuitively understand that the volume on the media stream is different from the volume on their device. So in other words, if the sound isn't working, they'll look at the screen, see where the slider is. If it's still not working, press the button on the keyboard. So with that in mind, we included a full volume slider in addition to mute. You're also way more likely to have lots of options in immersive video, things like higher quality, different playback speeds. So this area in particular was an exercise in future proofing just to make sure that this would grow with us. We created a collapsible paginated menu so all the controls didn't have to fit on screen like they did before. I could grow to accommodate new features that we knew we wanted to add picture in picture and have room for others that we were still thinking about too. It turned out to be a really big win for the Shaka team who you'll hear more from soon. It's tempting to make everything want to fit really perfectly as you're starting to map out your information architecture. But a helpful exercise that I found is to sort of double the number of controls that you want to include and see exactly where the pattern breaks down. If it starts to break down, it might not work. And so that was something that we played with here as well. So we launched what you've seen so far in the last year and the reception has been really positive. Versions of Android and Chrome OS have adopted our controls as their own native controls. And it wasn't necessarily a smooth ride to get there though. So these are just a few of the dozens of explorations that I tried before we landed on what we have today. One constraint I wrestled with in particular is iconography. Like if you asked an alien what this is, they would just say shapes. But if you ask any human on the planet who has ever interacted with recorded video or media, they'll say play and pause and stop. There's some things you can't innovate and these controls are one of them. So I tried to be all clever. I thought there was one place we could differentiate and I was using the scrum color. We were testing it and I thought it was looking pretty good. And just as I was starting to feel real proud of myself for changing the color of a thing, I got this request from my counterpart on the Daydream team. He was translating the controls so that they would work in a headset. And he thought that we should make the scrum color dark because in Daydream, the white color was uncomfortably bright. And it was like being in the screen, like inside the screen. Instead of, after experiencing the scrum in that way and I couldn't unsee the problem, which was probably from the retina damage. So anyway, today we've talked about how immersive experiences can be required deep focus over a long duration. And you can't do that if the UI is making you hurt. Even in 2D, after having that experience, the white scrum made me feel like I just flipped the lights on in a movie theater. And you know how the story ends though. We changed it and everything was perfect. The new video player on Chrome 6.7 sucks. Thank you for that insight, Mark Taiwan. He gave me permission to say that. All our experiments are public on Canary, our test channel. It was great because we could get actual user feedback with ideas we were playing with really quickly. So what was Mark Taiwan talking about? He had three critiques and they were spot on. He said, the play pause button in the center of the player distracts from the actual video and places it too far away from the other video controls. He's talking about desktop, which we just talked about as an issue, right? His second critique was, the volume control is now an on off toggle without the ability to adjust the volume at all. Remember the stuff we were just talking about about how users think about volume on mobile versus desktop a little differently? Yep. And lastly, the download button is the only item hidden behind the new three dots menu, requiring two clicks to achieve what used to need one click while taking up the same exact amount of space. Remember the stuff I said about making your IA scale up? It applies to scaling down too. So to wrap this up, in immersive experiences, the player is the stage, not the show. To make that happen, we reduced the UI to the smallest possible footprint and got the play button out of the way of the content. We made a flexible menu system that works for lots of functions or just one. And lastly, we made a player that elevates the video content. The projects I shared are the culmination of years of work from super bright, passionate people, use our tools and benefit from our failures, bridge the web app divide, and get the full set of media key functionality with the media session API. You can increase watch time with the picture-in-picture API to help users stay engaged with your content even in busy moments. And lastly, my favorite part, you can get our media controls. They have my blood, sweat, and tears in them for your app with one line of code. And if you'll please welcome Sandra from the Shaka team, she'll tell you how. Thanks Amy. Hi everyone, my name is Sandra. I'm an engineer at the Shaka player team. Shaka is an open-source JavaScript library built on top of HTML5. Basically, HTML5 gives you some media capabilities and Shaka makes it easy to use them. ABR stands for adaptive bitrate streaming and that just means that we adapt to network conditions. If your network is great and fast, then you will see media in high resolution and if it drops, then we will adapt it down. Our demo is an example of a media PWA. We support offline content, which basically means that when you're connected to the network, you can download your media and store it offline. And then once you're no longer connected, you're on the plane or in a place with bed reception, you can go ahead and watch it. We have built-in Chromecast support, so if you have a Chromecast receiver application, you just give us the receiver ID and we will handle the rest. And we have a sister product called Shaka Embed, which is everything that Shaka is, but for native C++ apps. Shaka is pretty widely used in the industry. I have some of our largest partners on the slide, which on the one hand means that every time I'm watching HBO with my friends and it hangs for one second, everyone in the room turns to me and starts at me accusingly because I write the tag that it uses to do that. But we appear to not be doing too poorly. One of our global partners, Globa.com, used Shaka last year to stream Soccer World Cup in Brazil, if Soccer is big somewhere, it's Brazil, and they reached 1.3 million simultaneous users in peak hours, as well as 8.7 million users per day. And other of our partners, Ghana, created a Shaka-driven PWA with offline support and they saw tremendous increase in session time and sign up percentage. They also saw bounce rate decrease. Until now, Shaka did not have a UI layer. When the new Chrome controls launched, we were really impressed by them. They quickly became one of my favorite browser controls and I started browsers all day, so that says something. But the thing about Chrome native stuff is that it's only there for Chrome users, not surprisingly, which is great for Chrome, but then as an app developer, there is at least five other browsers that they care about. And if you want a unified experience for all of your users, you will be forced to create your own UI. Shaka UI was our way of taking advantage of all the work that our research and design teams have done and make it available for everyone on most major browsers. We polyfilled a lot of browser capabilities to make it happen and used a very limited set of CSS to get a similar experience across major browsers. The browsers on the screen, Chrome, Firefox, Age, IE11, Safari and Opera are the ones that we're actively testing on. There might be in other browsers that we accidentally support, but these are the ones that we're constantly looking for. Amy talked about what went into the design of the controls. Development was also a pretty significant effort. It took us a year to implement the UI, make it accessible, build testing, customization frameworks. The five plus K lines is what we have on the Shaka site. It may be even worse for the Chrome folks, but on the flip side, what it allows us to do is if you're using Shaka Player, this is everything that you have to do as a developer to get are the default set of the controls for your application. We will see the data attribute and dynamically construct the layout. I mentioned Chromecast support before, so if you have a Chromecast application, this is where your app ID will go. You just drop another data attribute on your dev and we will see it and use this to enable our built-in Chromecast support. We keep an eye out for the new web features like picture-in-picture and constantly update our controls. Amy talked about the ways that it can really help you drive the watch time and completion of your applications. We adopted picture-in-picture and you will see now all of the supporting browsers. If the browser does not have support for it yet, then it just won't be there. If they go ahead and implement it tomorrow because of the nature of the API and our auto detection, it will just magically appear for those users as well. You also might notice that there are some buttons on the slide that were not part of the Chrome native controls like the captions, resolution, and language menus. The way we're able to do this is because we're building on Shaka and Shaka has access to the information about your content. We can pull this information from your manifest and we know what kind of language is your content comes in, what subtitles you have, what resolution your content comes in, and we're able to take this information and build more relevant controls for your users. We also realize that however well-thought through our design will not fit every application. This is actually Amy's favorite slide in my doc. I think it gives her sense of job security. I actually spent time on this probably much more time than Google would want me to, given that I'm paid to develop software. It actually works. I'm really proud of it. While Amy was talking, I realized that it will probably look great in VR, too. So anyway, it might just, our default design might just not feel good with the look and feel of your application. You might not want all of our controls or want them laid out differently. We have an easy API to customize the layout, and everything is styleable through CSS, so you will see no styles applied through the GS code. Just to remind you, this is what our default layout looks like, and these are some of the ways that you can change it. So code on one side, result on the other side. With this slide, I'm just listing all of the elements that I want to appear on my screen and the order in which I want them to be. By the way, rewind and fast forward button out there for you from the box. So they're not part of the default layout, but we have them implemented. If you want them, you just specify that this is what you want, and they will appear on the screen. For this layout, I also excluded the SIG bar. So this is what the SIG bar bullying is for. You can do that dynamically, too. So if you have add insertion in your application and you want to disable SIGing, you can just on the fly a new configuration object, and it will disappear and then enable it back. For an example of a more CSS-driven layout, for this one, I just changed the font and tweaked the play button a little bit. If you were curious about my wonderful full-screen obsessed layout, it's kind of a mixture of GSM and CSS. So for CSS, I'm tweaking the colors and making play button nice and square. And for GSM, I'm just listing a dozen of full screens. And by the way, each one of them works. Creating custom elements was another important part of the application, because you know your application best and you might have custom elements that you want to go on the screen, maybe something that has to do with your content or something really specific to your application. This slide is paying my respects to the Dragon Queen, but closer to the Earth, it may be a skip-ed button that you want to add to your layout or a purchase button. And as you probably noticed, they can go in both places. They can go just right here in the Controls panel or also in the Overflow menu, is my Dracarys button did. The way it works is we have an interface and we also provide a base class for you that your custom elements need to extend. This example is really simple just for the presentation sake. The only thing it does is it adds a button and then once you click on it, it does something. But in the real world, we give you access to everything that's going on on the player. If you care about the tracks being changed, you can listen for those events and then define behavior for your custom elements, localization event, anything that you want, you just listen for it and then act accordingly. This is how you register your custom elements. This is, I'm just saying that this is a button called Dracarys. This is a factory that's creating it and it's going into the Overflow menu. This is the highlighted part. So if you wanted to go to the Controls button panel, you will register it with it. And once I'm creating my layout, similar how I had a dozen full screens, I'm just saying that I want a language button and a Dracarys button. So I'm referencing it by that name that I gave it before. Accessibility is another important area for us. I'm used to thinking about accessibility as more of a moral choice than a product-driven decision, but actually it's both. Just to use US as an example, there are 56.7 million Americans with some form of disability. Just for reference, that's seven times the population of Switzerland. 7.6 million people have a hearing impairment. Those are the people that need subtitles. And it's about equal to everyone who lives in Hong Kong. More than 8 million people have a vision impairment. Those are the people that need us to get screen reader support. And it is twice the population of New Zealand. And almost 20 million have difficulty lifting or grasping. This is almost two portugals worth of people who need us to get the keyboard navigation right. Moreover, designing for accessibility has vital situational benefits for the rest of your users. If you're designing for a person who can only has one hand to use, you're suddenly benefiting all the new parents who are holding the baby in the other hand. Designing for people who can't see the screen gives benefits to all the drivers who want to use your application. Designing for nonverbal people on the other hand helps everyone who's traveling and doesn't speak the language of the country or just has heavy accent and would rather not open their mouth. For Shaka UI, we have convenient keyboard navigation, screen reader support, and tutorials on making accessible custom buttons which is really easy to do. We partner closely with the accessibility team here at Google to make sure that it works right. Localization is another important area for us. It's important if you want to reach global audience or if your primary audience speaks a language other than English. We support 16 languages. We're pulling user preference from the browser which basically means that a few user is traveling and they're physically in another locale. We will still pull the right user preference. Or again, to use US as an example, part of the population speaks Spanish as their primary language. So if those are your users, despite the fact that they're physically in the US, they will detect that they have a different preferred language and use that for the UI. And it's really easy to insert specific strings and languages if you want to localize your custom elements or if there is an uncommon language that you care about. If that's the case, by the way, talk twice, we might be able to add it to the application. But still, if you want to do it yourself, this is really easy to do. You just format it in JSON, you will label them, you let us know what it is, and then you drop the file into our folder and next time you build, it will be there. This is an example from my title slide of localization. It's an Easter egg that we have on our demo page where the UI is available in Sindarin, which is a language device built to Allke and for his books. And my manager did this in the evening while watching TV with his son, so it's really easy to do. Even if you're not using any of our styles, if you have a heavily customized application, accessibility and localization will still be there for you to use. Some of the things that are next for us on the UI side, there seems to be a lot of interest in having out-of-the-box ad controls, so we'll focus on that. On the player side, we're significantly extending our HLS support later this year. We currently support HLS and Dash, and also adding support for media capabilities and codec switching. Also, some of our best features came from the community we're an open source project. So if you have ideas, if you have thoughts on what we can do, if there is something that you require, let us know. This is our GitHub page. It has links to our demos, tutorials, community discussions. If there are features that you would like added, if something's not working right and you want to file a bug, there is always a list one of us looking through everything that's being submitted to GitHub. And also, we're really fortunate to have a great online community of developers, so even if you discard us from the equation, you will still get great advice and great discussions from the other members. Finally, Amy showed you goofy music pictures, so I feel like I owe you one as well. Come back, Amy. This is a picture of me performing with my band 10 years ago. I taught myself to play bass by watching YouTube tutorials and pick bits and pieces of English by listening to trashy metal songs. You can imagine that vocabulary works great to me once I move to the US eventually. Later in life, I taught myself to code by watching online courses and eventually wound up at Google, so there is literally no way I would be standing here right now if it wasn't for free video on the web. Same. The web connects people to information, but media connects people to stories and inspiration. Putting these two together is a really powerful combination. If you have the opportunity in the future, I hope you'll check out the Media Session API and the Picture API and, of course, Shaka Player. They're working for even more partners than we were able to share today, and if it's not the right fit, we'd also love to know why. We'll be in the back and at the sandbox in a bit. Also, our developer relations folks are really easy to find, or you could just leave a comment on a random subreddit and I'll probably find it. Thank you very much.