 Hello everyone and welcome back to The State of the Web. My guest is Doug Sillers. He's a freelance digital nomad working on the intersection of performance and media. And today we're talking about the state of video. Let's get started. Doug, so thanks for being here. Take me back to the days before the video element. What was it like to have video on the web? So before we had standardization in the browsers, the only way to get video in like a desktop browser was to use a flash plugin. So all the video was being delivered via flash. And then when you start thinking about mobile, you know, we were on really slow networks. It was like 2G, maybe 3G if you were lucky. And it was motion JPEG, which was literally just like small really grainy images being played at 15 frames per second that sort of gave the idea that it was a video, I guess, but it wasn't pretty. What was the user experience like in terms of like having that kind of jankiness of video? I guess because, I mean, we're talking flip phones, right? This is the era before the iPhone and smartphones and big screens. So, you know, we weren't talking high resolution screens anyway. So I think, you know, it never really caught on because it wasn't beautiful, but for the early adopters, it was like, hey, I can get video on my phone. What were some of the limitations of video? What held it back back then? I think a lot of it was networks, right? So the video has to be transmitted, right? I mean, this is the days when we were all getting DVDs in the mail, right? That was the way we were transmitting video to get that sort of quantity over a network. We just didn't have the network bandwidth to do it. And then, of course, we didn't have the hardware, like the chipsets on our computers and our phones just didn't have the oomph to process that much content. So fast forwarding to today, what are the different ways that video is being used? So, you know, when we think about the web, there's lots of different ways you can do video. You know, you've got sort of the YouTube's, which has a billion people going every day, watching billions of hours of user-generated content. You've got the Netflix's of the world where you've got video on demand, watching your favorite movies and TV shows. But what I usually work with are sort of websites that have video, whether it's a background video or maybe a web page that just has a video like, this is our company, here's a two-minute video of what we do. And what's neat about that is there's just so many different use cases for using video on the web, and you should have to think about how you're going to apply the different ways of delivering the video for those different use cases. My feeling is that most websites that need to embed video would probably just prefer to embed it via, like, a YouTube embedder as opposed to, like, self-hosting. What are some of the pros and cons of third-party versus self-hosting video? So, like, YouTube is great because then you've got all the experts at YouTube doing all the encoding and all the delivery and all of that stuff for you. There are a couple of tricks to that in that one is, like, if you're embedding a YouTube video, if somebody flags it on the other platform, it might stop working on your web page. Another thing that happens is when you embed a YouTube player, you're adding 700 kilobytes to a megabyte of JavaScript fonts, et cetera, that come from the YouTube player. So, there is a hit that you take even before the video starts getting delivered in terms of tonnage that's coming down to your web page, which can add to performance issues. And it's also who's paying the bills of that video being served over the wire. Okay, so fair enough. So, if you're hosting that on your own server or you're putting it up in your own AWS bucket, you're paying for that bandwidth, whereas if it's up on YouTube, it's free. So, there is that trade-off as well. So, what are some of the features of the modern video element on the web? So, the video element is really incredibly full-featured. So, it's amazing what you can do. So, you can have a background video and you can turn the controls off so people can't play, pause, or you can add the controls so the users can, you know, zoom forward and back, you know, flip through the video or stop it. You can autoplay videos, which is just fabulous, right? So, that's how you get the background videos playing on web pages. You can tell the player how much to download ahead of time. You can add a poster. So, rather than just having the first frame of the video, you can pick something that denotes really what that video is. There are a lot of features. And then, of course, you can script a lot of things as well. There are a lot of JavaScript attributes that you can call as well. And that's how you end up with a one-megabyte JavaScript player for YouTube? Well, yeah. So, I want to ask you also about accessibility. How can developers ensure that video is able to be seen and heard and read by all types of users? Right. So, I mean, if you think about when you watch videos on a lot of social media platforms, captions are added automatically. So, even if the video's silent, and so when video's autoplay, to make them autoplay on mobile, you actually have to mute the video. It won't play unless the video's muted. So, adding captions is a great way to let people, even people who have the ability to hear, to know that there's a video playing. And, you know, it sort of engages you, like, oh, there's something happening in that video. I want to learn more. And maybe they stick and watch it a little bit more. Maybe then they turn on the volume if they do have hearing. But when you have a video tag, you have a source, which points to the MP4 or the WebM or whatever the format of the video is. But then you can also have a track, which points to the close captioning track. And that will play automatically. And that's built into the standard HTML5 video player. So, it comes on every browser by default. What are some of the things that the video player can't do out of the box? So, I think the biggest thing and the biggest reason why people will use a third-party JavaScript library to play video is to do streaming video. Streaming video is generally not supported in the browser. The top streaming format used today is HLS, which comes from Apple. So, HLS works in Safari, but it doesn't work in the other browsers. So, you would have to add a JavaScript player to make those videos play. So, just like images, there are many ways to encode and serve video. Right. Yes. What's the state of formats? So, when I look at what's out there on the Web today, most videos are being served with the MP4 format, which is H264 codec. So, you have a format, which is sort of the extension at the end, and then you can encode it with different codecs. And so, the MP4 has two codecs that are popular, but everything's H264. And that's supported in every single browser except for Opera Mini, which doesn't play video at all. So, it's sort of the universal format. So, if you're going to deliver video and you don't want to encode lots of different formats, like MP4 H264 is the way to go. You're also writing the media chapter for the Web all-neck. Yes. Have you found anything interesting in there about video? So, there's a lot of interesting things. One is we actually looked, we talked a little bit about accessibility in captions. And we, in our queries, we actually looked for the number of websites that have the track element, which points to a VTT, which is the captions file. And we found a lot of, we found a lot of players that actually have that link. And then when I, so I was, I was really excited because, like, that's great for accessibility. And then I started actually looking at some of the websites, and the track was commented out in the HTML. Or the VTT went to a 404 error. So, like, either they're copying template code, and it's just not working, or they're like, yeah, we'll get to that. And it's sort of on the to-do, and we all know what happens with to-dos on the web. Like, it's just not there. So, unfortunately, from what I found is like the accessibility, I think that's a huge loss because to make the web available to everyone is super important. And video being such an engaging part of web pages, I think it's really important to make sure that you have captions for people who can't hear the video. Did you identify who the most popular codecs are? In terms of video formats, the most popular formats are the MP4, HLS streaming. And then probably the third is WebM, which is a newer format. And so, you can actually deliver two formats. And so, like the picture tag where you serve a WebP and then fall back to a JPEG, you can serve the WebM, which has a newer format, a little bit better compression, so smaller files. If it doesn't, if the browser doesn't support that, it can fall back to the MP4. So, we do see a lot of websites that do that. WebM first, fall back to the MP4 for browsers that don't support that. You had mentioned H264. I'm just curious, what's holding back H265? Patents. So, there's a lot of IP and royalty requirements for every single device. And so, it's not supported just because of the cost. And so, when H265 came out, I mean, H265 is 25 to 50 percent smaller than H264, but because it's burdened by patents and royalties, it just hasn't gotten the buy-in that WebM has gotten. And so, most people are using WebM as the newer format. What types of APIs are available for developers who are maybe self-hosting video or even embedding to understand what the user experience is like? When does the video start playing? Right. So, you create a video, right? I've seen so many webpages where it's like someone said, put the video on the web page. It's like, all right, video source equals paste, right? Push it live and we're good. And the videos end up being really large. So, if you have an image, there's a lot of work out there on image performance. But, you know, if you mess up an image, you might be a couple megabytes. You know, you probably could get it smaller. If you put a full-size video, you could very easily be talking 100 megabytes on a web page and start thinking about what that'll do to people's data plans on mobile, right? I mean, it ends up being very, very expensive. And so, if you start thinking about sort of the traditional tools for measuring web performance like web page test, you can see the video downloading, but you don't know how fast it takes to start playing. And so, I actually built a tool called stream or not. And basically, you paste in the URL of your video and using DevTools, you can change the speed of your network connection. And it will tell you how fast the video started playing. Now, this is a really synthetic test because there's nothing else going on. There's no JavaScript. There's no CSS. There's no other images downloading. It's just that video. But if you're on a, say, you set your DevTools to fast 3G and you start playing the video and it takes 14 seconds to the video to start, nobody sits at the top of the page of a web page and like, there's an image there. I wonder, maybe it'll be a video. Like, no one's going to hang out that long. So, you know, if it's going to take that long, maybe you need to work on some way to make that video smaller so that it can download faster and show up on the screen faster. You've told a story about a sushi restaurant website having video. What was the story behind that? All right. So, looking in the HTTP archive as we're doing the state of the web, I was just looking at web pages that have large video files. And this web page, so you think about a restaurant and you're traveling in a new city and you're like, I want to know if it's open, where it is, and hopefully there's a menu. You know, I'll deal with PDF. If I have to deal with PDF, that's fine. This web page has a video of the interior of the restaurant playing in the background. And it's 113 megabytes. And so, it just constantly plays. And then, due to a caching issue, when it gets to the end, if you stay on the page long enough, because it's like two minutes long, right? Two minutes long is too long for that. It starts downloading again. So, you keep the page open. I had it open for 10 minutes and I hit really close to a gigabyte in DevTools, right? That could be really ugly. So, first you have to decide, do I even need video on this website? Before you can even get to the, how do I optimize my video? Right. So, like, do you really need the video there? And then, you know, there's a lot of conflicting information about how long a background video should be, but pretty much the consensus is like somewhere between 15 seconds to maybe 40 seconds on the long end. So, like, two minutes is way too long. But, yeah, like 100 megabytes on a web page, that's getting really expensive. You know, I was looking at roaming costs as I was going to a different country and imagine if I was paying five bucks a megabyte and all of a sudden, like, what my cell bill looks like at the end of the month because I went to that web page once, right? Sticker shock. So, did you get that restaurant? I didn't. So, it's actually, it's a sushi restaurant in Brazil. So, I actually haven't seen it. I've just found it on the, you know, the HDB archive. Oh, that's fine. So, finally, would you recommend any resources for people who want to learn more about video? Sure. So, I've got a lot of blog posts on my web page that talk about best practices on how to re-encode video, maybe look at different formats, resizing video. And then, of course, once you've resized it and made it smaller, you want to make sure the quality is good. So, I've built some tools that will actually measure the quality if the quality is visually pleasing to the end user because it's great if you make it smaller, but you don't want it to look like a motion JPEG from the 2G days and, you know, Nokia candy bar phones, right? We want to make sure that the video still looks good. So, we can put all those resources at the bottom of the page of the video so that people can look to see, A, how fast their video is and B, how to fix their video so it is faster and then to look at the quality afterwards to make sure that it still looks great and starts quickly. All right, Doug, this has been great. Thank you so much for coming on the show. Thanks for having me. So, you can check out links to everything we talked about in the description below. Thanks for watching and we'll see you next time.