 Hi, welcome to Chrome University 2019. I wanted to give you a presentation called Life of a Codec I know this is sort of the Chrome OS track if you look at the The presentations, but this isn't necessarily Chrome OS related. It's Chrome and Chrome OS related But hopefully you'll enjoy it and the organizers of Chrome University asked us to favor breadth over depth for a Chrome University And that's great for me because I'm a PM not a sweet so it fits in nicely so the usual caveats apply If they're technical problems in here, I'll direct you to my engineering team and you can you know ask them how everything works So Chrome Media consumption is millions and millions of hours a day Across a browser and Chrome OS. I wanted to talk to you about how video codecs audio codecs are developed deployed and deprecated So what is a codec? The dictionary definition is here and the important thing to notice is that it has to Compress and decompress receive data. So normally when we talk about codec It comes from coder decoder and you need both sides And codecs are used to compress media for storage transmission or playback and in this talk I'll be talking about mostly video, but there are lots of different kinds of codec So you guys all know one codec at least You guys all should recognize this if you used a computer in the 90s so you guys all know about mp3's and This heralded in you know codec knowledge for everybody. So this is a win amp and you can see Napster in the background And the important thing is that media compression in this case mp3's allowed us to share Transmit music files over the internet much much faster than you would be able to do so if you were trying to transmit raw files off of a CD There are a lot of different types of codecs Google Run some projects and then contributes some other outside projects for all of these video image audio geometry VP9 is a webm project kind of a Google run codec development Webp is similar We have opus for audio and then we have drako for geometry compression. These are all examples of open source codecs That are available to anybody free free on the web Um, we mentioned briefly that you need encoders and decoders and the reason for that is if you want to use a video conference app in Chrome or mirror your screen or something like that You need to have an encoder to scrape the screen or use your webcam. That's why you need both So i'm going to talk about uh, mostly video codecs and when I talk about video compression the first thing everyone thinks of is This tv show silicon valley But it actually is um makes a lot of sense and it actually is it's quite important here because if you look at the credits in the first season Uh, depending on what team you're on in chrome or you might recognize some names so we've got jim matt yawu and germy here and Jim matt and yawu all are in the chrome media team They came from a 2010 acquisition of a company called onto and they've worked on the vp series of codecs vp6 vp8 vp9 Jeremy was instrumental in that acquisition. He works at youtube now And there's a lot of teams of google outside of chrome that work on media compression as well But um, they were chosen to be technical advisors to the show for uh Making the video compression parts accurate or as accurate as possible while still being humorous So why do we need video compression anyway? Um, so this dog is really sad. He wants to watch cat videos on youtube But the problem is that a 1080p uncompressed video at 30 frames per second He wants to see you know a good quality video is 1.49 gigabits per second if you stream it raw So yeah, you think that jpegs do a good job of compressing your images video is a whole other level um And the average uh internet speed in singapore is 150 megabits per second I think it was number one last year and so you still can't even watch anything at 30 fps It's going to be absurd. So small cats are better So we want to use this example of h264, which is an older video codec And instead of doing it 1.49 gigabits per second We can actually get it down to five megabits per second at very good quality And this is representative of what of what youtube might use a bit rate for streaming 1080p 30 So how do we make these codecs? Um, there are two organizations I want to talk about that work on codec development. There are many other groups around the world But two that work on um video codecs. I should say So one is mpeg and you're probably familiar with them because they uh have the greatest hits here They have mp3 of course mpeg one layer three They have mpeg two video which is used for dvds and atc trust real broadcast in the united states and i think south korea A vc which is used in blu rays and H64 which is used on all kinds of internet streaming and then h e vc, which is a newer codec I'm used in high-end consumer camcorders as well as Ultra hd blu rays I meant some newer technologies and one of the things I want to point out is that google does participate in mpeg But mpeg is not a organization that has royalty free provisions at its heart and we'll talk about that in a moment So the alliance for open media is a consortium of companies that also develops codecs We have a video codec and a still image codec right now We might do audio and it was founded in 2015 And this is a huge number of tech companies that work on web technologies mostly And we have released av1 uh last year and avif I think earlier this year the still image format And one advantage of an organization like this in addition to just having a lot of industry Participants behind it though mpeg does as well is that royalty free policies exist in these organizations So all contributors agree in advance to license their patents at no cost So this is very different from mpeg And if you look at the development of open source, which is the bottom versus kind of the closed source mpeg iso world at the top You'll have this diagram of video codec development And it's a little bit different for the two groups So the bottom one is web m which was kind of a google run thing and then it transfers to aom so 2003 is when 264 came out and VP 8 which was that onto acquisition that was open sourced by google came out in 2010 and they were about roughly on par as far as efficiency So open source is about seven years behind In 2013 hivc came out and vp9 open source was released and they were about on par as far as efficiency in the same year And last year we released av1 which is about 30 percent better And we're still waiting for the mpeg world to come out with h266 We don't know what the compression is going to be yet, but they are a few years behind our release cycle at least and You know, we talked about the royalty free policies that aom has but for mpeg It's a little bit different for codec such as hivc It's developed by organizations that don't require specific financial terms ahead of time So even though patent holders might agree to fair and reasonable licensing fees What that means the definition of that differs for different companies And so there's uncertainty About what the fees might be So that's why we focus on open source codecs, especially coming from organizations with royalty free policies So how do these codecs work anyway? So we have this cat and we want to compress him And in h264 so this is 2003 technology We look at blocks and this is actually very much like jpeg work starting I think the 80s when jpeg development started where you look at blocks and you try to divide an image up into blocks And figure out what the blocks the largest block size you can actually have up to 16 by 16 Such that there's not a lot of variation in that block so that you can code it Together and so with 264 you have a 16 by 16 block is the largest block size and you can go down to 4 by 4 There's 17 possibilities. So this is kind of a search problem, right? and The question about efficiency in this codec is how far or how long are you willing to search how hard you're willing to search To make sure you've got the optimal block partitioning And you can do a really bad job and make a compliant h264 video And it'll just have really low efficiency or you can try really really hard and make a h264 video That's a lot more efficient and more compressed than another one 81 released last year has a lot more options for block decomposition So we have 128 by 128 blocks And you have all these weird blocks some are rectangular not square And there's about 10 to the 6 possibilities If you go down All the way down to 4 by 4 So this just means that the search space is much larger And as you can imagine, uh, it takes more software Computation time to be able to do got do a good job of encoding this video However, the efficiency is going to be much greater So at the same bit rate, you'll have a lot more quality or the same quality video at a much lower bit rate There's also the next part part is about, uh Intro prediction, so in the same frame, we're still talking about one frame of the video at this point Where you want to be able to say this block is similar to another block. This is actually, um A blown up version of the whiskers here and you can see that these two blocks are actually quite similar Even though there are two different parts of the image And what we can do with, uh, blocks like this is we can, uh, in newer codecs We can say we want to do some rotation or some warping So that we can try to match up these blocks in a more similar manner But of course this is a video so you have multiple frames So we have intro prediction so temporal prediction And what we want to do is say from frame to frame what's changing Can we just point back to the image before the frame before And say let's just use that block and copy it forward And that's how you save, uh, data So in this, um video if you're watching on youtube, I think this talk is being recorded The slides and the background is going to be updated far less frequently than, um myself who's talking because I'm moving around I'm talking and the stuff in the background perceptually you don't care as much because it's not the center of attention But also it's just not changing as much There's a lot of new tools that have come out in newer codecs. It's not just about block based coding Um in different transforms. This is one that I actually really like That was invented for 81. So this one is called film grain synthesis. So this is a Still frame from do you guys know what movie this is? The godfather right by capola. So this is a digital transfer from the blu-ray That capola himself oversaw the transfer of so he actually was involved in saying I want this original film digitized and put on blu-ray And 1080p So what do you notice about this for so the first thing you might notice is the colors a little bit washed out, right? That's intentional capola wanted to keep the exposure that he did on the film which was intentional in this So if we had saturated these colors, he would not be happy with that Um, but that's not the film grain part. The film grain part is it's hard to see on this slide unfortunately But there are a little bits of noise in the image that are not due to compression artifacts They're actually due to film grain that was on the film when he shot it So these are kind of noisy patterns that appear when you actually use film to shoot something And capola wanted to retain this film or this noisiness in his video And this makes it really hard for compressors and codecs because You have a really hard time compressing random noise, right? all of you guys who studied math should know that and so Um, a very smooth image without noise is much easier to compress So some codecs denoise the image and then compress it, but it's not um, It doesn't have a lot of fidelity to the original image or the original video So I think netflix proposed this tool and it was developed in aom and what we actually do is we Characterize the film grain or the noise in the image Denoise the image and compress it and send it along and then in the container or in the bitstream rather Send parameters for the noise and reconstruct the noise at the end Okay, so it's very crazy But it means that for directors and artists they actually get to display to the user what they intended to show Which is quite interesting Okay, so we've got this fancy codec What are we going to do with it? We want to deploy it, right? And This is uh, going to be difficult because it's new technology and we're going to ship it in chrome So how do we get a video to play in chrome? How do you get a video to play in a browser at all? So do you guys remember plugins? From before 2007 Ude does I know that okay, so what what is this? Flash we just deprecated this right that's great. And what is this one a little less known? Silverlight right so before we um a long time ago You used to actually have a plugin that would decode the video for you And so flash um had a proprietary version of t63 It actually licensed onto technology vp6 for a while Before google, uh, we bought that company and 264 silver light was a microsoft technology So it used it's just for as well as well as vc1 and plugins are terrible um as we all know so Luckily circa 2007 html5 came out and we had the video tag Um which meant that as long as a browser had a codec built in it could decode Video audio or whatever you were trying to display um you would be able to do that Without having a plugin and it also meant that you would write a player in javascript So chrome has this project called chaka player Which is an open source javascript player that can do video playback media playback Encrypted video all of that kind of stuff, but the important thing is that you don't have a plugin anymore Um, okay, so the good news is you don't have a plugin the bad news is that means we have to put a codec into chrome Um, so how do we do that? So this is kind of the life of a codec You write the spec for the codec And then the first thing you can do is you can write software software is easy. It's field updateable It's fast to write But it's power hungry You're never going to be as efficient when you write software as if you could design it into the hardware But at least you can do it quickly and it has some performance issues depending on the machine You're running on it's going to operate differently. So if you're decoding video So hardware is a lot better, but it has a lot longer lead time a long longer development cycles And you can't update it if you have a bug in the field You have to deal with it Um, and then as these two things happen you get a lot of usage in different use cases And then finally over time you can deprecate it So now we just need to write our software decoder and put it into chrome or a software codec and put it into chrome Um, and wait for the hardware to show up. So that should be pretty easy So what's the first thing you care about when you're writing something in chrome? Okay, one of the things we talk about is is speed And um Speed is um, oh, sorry. I want to go back one slide to mention this Most of the time when we talk about new features, especially new codecs, we start out by Doing them on desktop. So if you're familiar with the nomenclature for chrome when we say desktop, we mean desktops and laptops We mean linux mac os Windows chrome os We just mean that it's not going to be on, you know a mobile device like android or ios And the reason we can do that is the heavy lifting can be done by, you know A higher power cpu than if you had to do something on android or ios Especially when battery life is a limitation because it's less of a problem on laptops and it's not on desktops So the first thing we might say we care about is speed. You obviously want to write a decoder. That's really fast Um, so you can have a really good performance for the user But there's a lot of challenges with that and it's not just about efficiency We have to worry about the binary size So one of the things we have to do every time we add a codec Or any other feature that you're developing for chrome You're going to have to look at the binary size of the uh update as well as the uh installer And say is this worth it? And you're going to have to fight a lot of people in chrome rightfully so Who are pushing back and saying we can't take this 500 kilobyte hit The next thing you have to care about is security. It's one of the other tenants of chrome that we talk about all the time But along with security Means that you have to be able to update this codec over time So you can't throw this technology into chrome and expect that you're going to be able to not touch it for a long time You might worry about drm for uh wide vine encrypted videos for example in chrome And then last but not least you care about power Because we saw it said that some of these people are working on laptops and users care a lot about the power efficiency of their browser And we you know get some, uh, Press about that as well In chrome so it's very important to have a codec that uh is as efficient as possible, but also as performant as possible Okay, so our dog is still sad Why why right so our dog is sad because now we've got a codec in the browser It's ready to decode video And there's no video and the reason is that it's only useful to deploy new technology in chrome if someone is going to use it and We need to go and reach out to some of the streaming video providers and say hey, will you use this new technology? Will you use this new codec? Luckily for us up the street We have this company called youtube Which happens to be a pretty large streaming video provider, which means a few things one is that they care a lot about Egress costs and how much the bandwidth costs them so they're willing to adopt newer technologies to compress videos Because they'll save a lot of money on bandwidth The second thing is they care a lot about users all over the world They're not popular just in one region and so reducing bandwidth means that you'll get Higher watch time by more users around the world or you'll get users that have higher definition Content or higher fidelity content at the same bit rate, which means they'll increase watch time So that's great for youtube when vp9 was released YouTube was kind of the first and only for a little while streaming video provider that adopted it There are some questions about if other people were going to switch to the mpeg hvc codec But over time we've actually gotten netflix and facebook on board who are using vp9 as well as and other companies For av1 we're even more excited because we have all those companies on that slide Who are working on the codec and even though av1 launched last year and youtube started supporting it in september We already have hulu and facebook serving av1 video as well as other people who are going to be announcing it this year So now we've got the other side. We've got the streaming providers So now we're good. The dog should be happy Except when he switches laptops His codec is going to work differently Right because the cpu is different. So what are we going to do about that? So luckily there are some extensions in chrome or apis and chrome that can help with this So this is media capabilities introduced in i think chrome 63 or 64 That allow you to check not only if a codec is going to be able to be played well Or played in the browser, but if it's going to be able to be played well Is it power efficient? Is it smooth and by taking information like this from the browser? You can actually figure out if you should even use av1 or vp9 or a different codec on that machine So the dog will be happy But then what happens if he's watching a small video of a cat and then he wants to full screen it We talked about performance and software decoding. He might not be able to watch av1 At a full screen Because he can't decode 1080p and av1. We don't want to have the video stutter. We don't want to have it reload That'll make him sad So then we use another api called change type and this was introduced in chrome 70 last year It's really interesting actually using the regular videotag It lets you switch the source buffer on the fly and you can switch containers and you can switch The video codec So you can switch the entire stream and it's kind of transparent to the user Because the next frames come in using the different codec and they show on the screen So if someone has a video playing full screen And then switches it back down to a smaller view you can switch from vp9 a very performant codec That maybe has hardware decoding to av1 which is more efficient, but is in software And the user won't even notice so we affectionately call this codec splicing for us So now we've gotten our deployment set we wait for hardware to show up for all the other devices that can't do software decoding And we roll it out to all the devices and I should mention that we mostly talked about desktop software decoding But we do software decoding of codecs on mobile devices as well. There's nothing inherently Problematic or wrong about using a software decoder on a mobile device So the last thing I wanted to talk about was codec deprecation We talked at the beginning about how do you get rid of technology in chrome that you've put in that is now So old that we have a lot of newer things that can replace it So this is a problem, especially with codecs and the reason is that People like to use old stuff if it still works and it's really hard to get people to switch to new things So As recent as 2017, this is an encoding dot com report They said that 79 of streaming video was h264 And that was released in 2003. So these numbers are about the same today about 79 80 percent, I would say so 16 years after H264 was released. It is still the dominant codec on the web. Why is that 81 is Much much more efficient. I mean it requires, you know 70 fewer bits to transmit something similar And to put this in perspective The number one selling phone in 2003 Was this So the two things I want to point out here one is this had an amazing, you know a 12 bit screen 128 by 128 display But the other thing I want to point out is that Actually the creators of h264 did an amazing job Right, they invented something in 2003 That was able to work flawlessly until today and still works So first of all, we should give them credit for doing such a really amazing job with h264 when this was the technology at the time But the second thing we should think about is This is the technology at the time. We've come a long way We should be able to go further and develop new stuff and get it deployed But there are some considerations. So one is abandoning users There's always going to be streaming video providers out there who are using older codecs You are never going to get down to zero percent of people using older codecs So at some point you have to say it's more beneficial for everyone to reduce the binary size by removing this codec But with 264 especially it's going to be very very hard With rtc real-time communications, there are different requirements for codecs So a codec that performs well for encoding and software might be great for a lot of Users for rtc who are trying to encode just a low resolution webcam. Your webcams are what 720p So you don't need a hardware encoder. Perhaps you can just use software So someone who's running an rtc app might never want to update their codec because things are working. Okay And it's good enough And then of course there's local playback for chromo s Someone is going to want to drag in that h264 video into the video player and they expect it to play And as long as they have a camera that's capturing an h264 that they might plug into their chromebook This is an issue that we're going to have to deal with and there's a lot more issues that we have to worry about So the end point I want to make is about video codecs But also about chrome in general which is whenever you're developing technology In this case a codec or anything else for chrome and you're including it not only should you be excited and About the technology you're developing excited about the number of users you're going to get But realize that when you get a lot of users and we want that because we want people to be able to use The stuff that you're building in chrome You also have to think about how are we going to sunset this at some point in the future Because at some point the software you wrote or the technology you invented is going to get surpassed And we're going to have to remove it But you have a lot of users out there who are still going to be counting on you So it's a challenge and it's not something we have solved It's something the chrome media team is working on right now trying to Make a roadmap for codecs and codec deprecation But I just wanted to point that out because it's applicable to things other than codecs