 I don't know if you've ever done coding where you do an awful lot of architecture and wrangling and then people kind of looking over your shoulder go, so what have you been working on? You're like, well, lots, but there's nothing really to see. It's just a lot of machinery under the hood. So has it been for me with the story of offline in this particular app? The media app has two offline cases. One is relatively standard, I suppose, that you go to a video and you say, I would like to watch this one later, and we have to go and download not only the video, but things like its poster frame for the video element, its album art so we can do the media session API, its dash manifest, all those things and you kind of go, yeah, I just get a bundle of things and I'm going to go and get those and download them. And one of the things that in this area that's still to land on the platform for which Jake has an explainer at least is the background fetch API, wouldn't it be great given that list of files that you want to get to be able to hand off to the browser and say, hey, look, these 10 files, would you mind going off and getting them for me and just fire an event when that's completed so that I can decide what to do. I can put them in a cache or I can do whatever I feel like doing with them. Well, that's certainly on the way and it doesn't exist right now. So what I'm doing is when you click on the, I want to take this video offline, I do, in the page, I do a fetch and I populate the cache myself. Now that's quite a lot of code in and of itself because I'll show you why. And the, let me tell you about the second use case. That's worth doing. The second use case is this. Imagine that you have watched episode one of a show that you're interested in and episode two is probably going to be of interest to you. Maybe you've subscribed to the show or something. And what we want to do is rather than getting the entirety of episode two, you're not asked for that and that's quite a lot of space. But what we could do is maybe just get the first one minute or two minutes of the show because that kind of prefetching idea is good because it means that when you hit play, hey, we can just give you a flying start and we can drop back to the adaptive network side later on. And so I've been trying to tackle both of those. Now in this particular video, I think what I want to do is I just want to talk about the service worker and the case of getting all the video. The one where you say I want to take this video offline for later. Let's talk about approaching that problem. Now I'm not going to dive into every little bit of code. There are bits and pieces in there that you take a while to explain and I think it's probably the case if you just look through the code, it'll probably make more sense. But of course, if you do that and you have questions, we have the comments below for you to ask your questions. So please do feel free, as always, to go ahead and do that. Right. Enough talking. Let's look at some codes. Right. I am in the point in my app.js where thanks to the UI, and let me show you the UI. I should have started there, really. Imagine you are, let's see, you've got this video of me and Jake here. And you click this. We don't need this. Let's do that. Do you know what, I'm just going to close that. Yeah, I'm just going to close that. And you click the Make Available Offline button. And we start saying, that was very quick. That was very quick. I wish the internet was always that fast for me. But it isn't. And because it isn't, we have the offline case. Perfect. The bit where the spinning dial, all that kind of stuff. Right, what's actually happening? So in the app, I fire an event that causes this on offline toggle to be triggered. And the interesting bit here is, well, I check whether that we actually have the thing. And if we've already got it, then do you want to remove it, and so on, and we'll delete it. But if you haven't, which is that case that we were just looking at, then we want to add it to our cache. And you can see I have this offline cache class here. And I tell it to add this thing. So that's where the rubber hits the road. So we should head over to that class. Let me find the actual app function. Here we are. And it takes in a bunch of things. If we don't support caching, then we just bail. But we build a set of assets that we're interested in. Do you remember I said it was things like the poster image and so on? In fact, we have them in here in the constants here. When you go offline, when you take a video offline, here's the things that I currently get. I get the artwork. I probably only need one of these, actually, to be fair. Anyway, I get both artworks, a small poster image, and the large poster image. And I pretend. So I get the 720p offline manifest, which is a reduced dash manifest with just the 720p video and the audio instead of all of the various options. But I pretend that it's called dash manifest. So when later on I ask Shaka, just get dash manifest. If it hasn't got it stored offline, it gets the full one with all the representations. If it has got it stored, it's just going to get the 720p one. That's my way of locking Shaka into the Shaka player, into only ever playing the 720p video. So it builds up the list of assets that we want to get. And you'll see I have this special magical chunking thing here. Right. This has proved to be quite the journey. Because I don't know if you didn't catch the episode, the entry, where I talked about the idea that when we make a request for the video, we do a range request, which means we don't ask for one gig of video up front, for example. We say, I want these bytes to these bytes. I want just this range, like 1,000 bytes or 2,000 bytes or whatever. And I said at the time with my prototype, this is going to be a bit of an interesting one later. Because in production, if you've got a large video, you don't want to store it as one big file and then pull it out entirely into memory as like a one gig array buffer that you then slice, for example. So what I do is as the files coming in, when I'm doing this all part of this downloading of, say the video files and the audio files, I have this chunking flag, which basically means on its way through, please split this up into chunks. Later on in the service worker, which if we've got time, I'll show you, I ask for the chunks that we need. So for example, I have, I think at the moment, half megabyte chunks. And so if you asked for a particular range, it might be that I need chunks 9, 10 and 11. And so I get 9, 10 and 11 and I create an array buffer from those three array buffers, one big one. And then I slice out from that array buffer the bit that you actually asked for. So it's my way of making sure that I don't pull out too much data into memory. I chose, in my chunk size, I chose half a meg at random as a sort of a guess that maybe we'll need two, maybe three occasionally and sort of a meg, meg and a half in memory. It's, yes, it's a bit garbagey. No, I don't like it, but it does mean that I'm not pulling out a one gig video into memory every time we make a request. I'm saying it's always a gig, but I'm just saying, you know, think big video and that would be the problem. Now, a longer term solution for this that I'd love to see on the platform is the ability when you say caches.match and you give it the URL that you could give it some options besides say the cache name, but you could actually give it the range as well saying, do you happen to also have bytes 1,000 to 2,000? And or, you know, I assume it does if it's got it in there, but it should, you know, those kind of things should be, it should be able to give you the answer, right? As in, no, I don't because the file is only 500 bytes long or yeah, sure. Well, a lot of that needs specking. It's not a thing that exists today. So it's something I'm actually having to do and it's a bit of a pain point, but there we are. So, wow, it just said chunk true and that required a whole heap of explanation. See what I mean? So what does the chunking do? Well, let's keep going through the code. So we build on this list of assets that we actually want to push and eventually we get to the idea of downloading and downloading is here. And this is probably one of the more interesting bits because what I do is I have a thing with which I call track download, interesting. And this actually looks probably fairly straightforward, I imagine to many of you, or if you haven't done service worker stuff, this'll look slightly bizarre. If you have, it'll look makes sense. We open the cache and we take whatever the request is and the responses and we just pop that in the cache. If it's not set to chunk, if it is, we cache in chunks. So we seem to be doing two things with all our responses, all the things that we've asked to download we're doing two things with. Firstly, we're tracking them and secondly, we're caching in chunks. Now, I better not say the word streams three times in quick succession because if I do Jake will show up and I think we can all live without that, don't you? Okay, streams, that's what we're doing. Imagine that we've made a fetch request and one of the things we get back is a readable stream at least you can in Chrome today. You can also say with the response, you can say get me the JSON, get me the text, get me an array buffer but you can say I wanna read this as a readable stream and in order to handle this when a response is read once it's consumed actually in this world which means you can't do two things with a single stream. You can't both say track a download and count the number of bytes that have passed through the stream as well as cache it into chunks like read the stream and chunk it up as it goes but you can clone a response which is kind of like teeing a stream. If you've not done anything with streams the idea is that imagine the T shape and it comes up the middle and then splits out into two. It's our way of taking the stream and turning it into two streams which is exactly what I'm doing here. So let's have a look at what we're doing. So imagine then we've got say the video coming down the wire, okay? We're gonna do two things with it. One, we're gonna track the download. Secondly, we're going to cache it into chunks. So let's go down to the track downloads. Oh, where is that? Where is track download? I should just look for, there we are. Okay, for each response what I do is I get a reader. I get a reader, you can see here, oh actually I clone it. So that's the first thing I actually do. There we are, I clone it to make sure that whatever I do here on the response means that it's not gonna be consuming the response and stop it being used somewhere else. So the first thing I do is when I get one of these responses, so I fetch the video, comes the fetch comes back and I go right clone it. First of all, clone this. Okay, so that I can now operate independently with my clone of the response. And then I ask for a reader on there. So clone dot body dot get reader, which is a readable stream. And then I can start reading and I get, there's a callback that you pass to the promise when that happens. So you read from the reader and then you say on stream day to do this thing. And it's a bit convoluted actually when you do stuff with this because it's kind of, you kind of call the callback and then in the callback you set up the next callback of the callback. Yes, you do. So on stream data, we just say, well, are you done? If so, you know, if blah, blah, blah, we're completed. Is the general idea. Because one of the things that's the case here is that we've got a byte total. So remember, I'm getting like five or six files. And so the byte total has to represent the entire total of all five, six, however many files. And so each one of these responses is contributing to that total. And when one of the streams finishes, it will, if the byte count isn't the final byte amount, well, we don't need to do anything. If it is the final byte amount, we've actually completed the download of all our files. And so we can basically say, yeah, we're done. We can call this callback, which is something I registered somewhere else that's like, okay, update the UI and all those kinds of things. So remember now we've had, say, the audio file come in or the video file come in, we've read some bytes and we basically keep tracking the length of each of these reads. And that allows us to say, well, how many bytes have actually come in? And we can post back to this on progress callback with the how many bytes we've got and how many bytes we're expecting in total. And as I say, when we're finished, we can call this on complete that says, right, all the bytes are in, we're done. So that's what the tracking part does. And that's a great way of using streams. I say it's a great way. It's a garbagey way, unfortunately, because imagine every time this callback is called, this result here has, it's an array buffer, I believe. It's got lots of bytes in it. And I don't do anything with those bytes apart from ask for the length. I would love it if we had some mechanism with, say, a fetch to say, just give me progress events like you do with XHRs. That would be a nice thing. And I believe it's something once again, that my friend Jake is looking at. Hurry up, Jake, and whoever else is working on it. I have need of it, I need it. Anyway, so that's one part. We're tracking the download. We basically ask for all the bytes that are passing through the stream and we post back every time we get an update, basically going, more bytes came in. And that's what I used to do that pie chart. If we got time, probably won't. If we have time, I'll quickly show you how that pie chart's done. It's pretty neat. And then the other thing, as I said, was caching into chunks. So this was the idea that instead of a one gig video or a half gig video or 200 meg video in the cache, instead what we want to do is we actually want to separate it out into like half meg chunks and then we can build our responses in the service worker later. So the caching chunks, once again, we're cloning the response. Probably don't technically need to do that. We could just consume the response here if we wanted to. Not dissimilar, get a reader. And what we do is we go through reading in chunks. And every time we hit our chunk size, we commit that particular chunk into the cache. And that's this commit buffer here. And then we reduce the amount that we're expecting still to come in by that chunk size and we kind of restart again. So we kind of have this buffer and we fill it up. And then when that's done, we basically pop that into a response and then we reset and go again. And eventually we run out of bytes, which is the result.done. And we commit whatever buffer size we've got left there at the end. So imagine the say, instead of the half meg, we've ended up with like 200 bytes at the end. We just commit a final buffer with just those 200 bytes in. The commit buffer has this. It has one header that I set, which is just the chunk size because I need to know later on about how many chunks were actually, the chunk sizes that were actually stored in case, I've got a part chunk at the end. And I need to kind of know, do you actually have enough bytes to satisfy this thing, the range request that came in? Anyway, nonetheless, we do a cache put of that chunk. And actually on the front end, let me show you what that looks like. If we go to application and we take a look in the cache storage, you'll see I've got the CDS show reel here, which is the cache for this particular episode. And there you go. Look, you can see it's, but down this, the 720p stream all the way, zero, one, two. Don't you just love string-based ordering? All the way up to what's that, 32? Yes, into 32 half megabyte chunks. Cool. So that was the first bit. And you can also see here, by the way, there's the artwork and everything else. And you can see I actually keep one cache directly for this video. That means that later on, it's a lot easier from my UI perspective, because I can say, when I'm looking at this video, and I look at its URL, I just go, do you have CDS slash show reel, by the way, caches? And if the answer is sure, then I go, cool, I'm gonna assume you have that video. And so my add and remove is based at the kind of cache level. So when I say, get rid of this, if I remove this offline copy, and then I refresh the cache storage, refresh it, there you go, you see that, that entire cache is gone. But it didn't get in the way of say the assets that I need for all the icons and everything else. Cool, makes sense. Hopefully all that makes sense. Also, I have the old Chrome Dev Summit one there, because I was running that on local host 8080. Flashbacks of fun. Right. I am gonna stop there, I think. And in the next entry, I will actually show you the corresponding other side, which will be the service worker code. But before I do, I did say, I was gonna show you that dial. Let me just show you this here, right? See this when it goes, wee, and it goes round like that in a dial. How did I do that? Before I go, I'm gonna show you that really, really quickly. It's in the... Where is it, where is it, where is it, where is it? The download progress. Look at this. It is basically, this is what actually gets called by the on progress callback. The percentage value is a value between zero and one, which is the bytes count over the bytes total, okay? And it comes in and we have a target which happens to be an SVG. And that matters because what we're actually doing is doing an SVG path. It's an SVG path, there we are. And you can see here, what we do is we take that percentage and I do a little bit of cause and sign work to figure out what arc we actually need based on the percentage. So one is a full revolution and then it's some proportion of that revolution. So this is something I didn't know before I actually started doing this. I had a sneaking suspicion that with SVG, you could create paths on the fly and I had a sneaking suspicion that arcs might well be one of those things. And you can see here that this is exactly what I'm doing. So I find the target, I find a thing called path.dial which is just a path element inside the SVG. And I go through and I create based on a bit of calculation what I think the arc should be and it's got weird flags like this large arc flag and targets and all that. Have a look at the documentation on how to do arcs inside of SVG. It is slightly bizarre. I'm much more used to the canvas way of doing things. Oh well, but this works brilliantly. So I based on the percentage, I just spin an arc through and I just sort of make a pie chart thing like that. And then I set the attribute and that causes the SVG to update and it all works really well. I'm actually really very happy. So this is a little bit of code here. All 50-yard lines of it does the job just great. And it obviously means in my UI can give a nice bit of progress to the user on how well the download process is going. So there you have it. That is the process of downloading the files, chunking them, using streams, giving progress, updating the UI. And we're not even done yet because we actually haven't even responded to the user's request or the player's request for range requests. Use it, hang on. Let's try that again. We haven't even responded to the video player's range requests coming through the service worker. For that one, you'll have to join me next time. Now don't forget you can subscribe, don't forget you can drop in your comments as always. Thank you so much for all the feedback, all the ideas, all the suggestions, all the comments that you're making. I do love reading them all and I will catch you. Yep, you guessed it. Next time. Hello, thanks for watching. If you enjoyed this video, well, you may enjoy other videos that we make too. So don't forget you can subscribe and you'll get notified when we push out a new video. And there's more videos over there or down there, depending on how you're watching the YouTubes right now. Definitely click on those.