 I'm Jake. I'm Jason. Earlier in this conference, we released, is this a conference, actually? Is this a conference, or is it just like an event? I don't know. Let's call it a conference. Earlier in this conference, we released tooling.report. This is a website. It looks at all the different JavaScript bundling build tools, shows the things they're good at and the things they're not so good at. But in this session, we are going to write our own plug-in. Right. So cruising around the web, we see lots of sites with performance issues. And a lot of times, these can be solved with something simple, like a preload tag for a web font or some selective code splitting. But the thing is, when these are controlled by a build system or even deep in layers within a build system, making those changes can be really tricky. So supposedly, simple changes can start to seem impossible. Yes. But once you're familiar with writing build plugins, sure, that means you can write build plugins. But it also gives you insight into how the build tool works. It makes it easier to debug things when they go wrong. And of course, it means that you can go and help with community plugins as well and do PRs and add features and fix bugs, that kind of thing. Definitely. So we're going to create the same plug-in for rollup and then for webpack, just because those are the two most popular and growing build tools that we've seen. So what are we actually building? We are going to build a service worker plug-in. Of course, we're going to build a service worker plug-in. So the actual subject matter of the service worker plug-in itself doesn't matter all too much. We picked this because it touches on a couple different parts of the build process. So it's one of those things that seems simple on the surface, but there's actually quite a bit of complexity when you dig in. Right. So here's the app. Now, we're not going to go through the whole app, but it has JavaScript, CSS, images, and HTML, all of which are part of the build process. Right. So the bit that we care about for this session is specifically the registering of a service worker. So the URL for a service worker should never change. And this is because we need to be able to go and check for updates at that URL. But it'd be really nice if we didn't have to duplicate that URL here and in our build configuration and on disk. And instead, that was just filled in by the build system. So here's a code for the service worker. We're not going to go over all about how service workers work. There are articles for that. But here are the things we need for this build system. We need a list of URLs to cache. So this should be our HTML, CSS, JavaScript, and images. These are the things that a native app would ship as part of its bundle. Now, we can't just hard code these because the build system is going to change their file names. It's going to make them cacheable. It might also code split as well, which is going to generate whole new JavaScript files that we didn't know about in advance. So because we can't predict it, we need the build tool to fill it all in. Right. So we also need a version for the cache. And this is because when we're installing a new version of the app, we want to use a new cache. That way, we don't potentially disrupt any current version of the application that's in use maybe in another tab. And so because of that, we need the version to be unique for each given set of assets. Yes, and if we don't change the assets, the version number should stay the same. But if we say, like, update the HTML, the version number needs to change. Easy, right? So the challenge is that we really want the service worker to be part of the main build. We want it to potentially take part in code splitting. We want to minify it just like we would our application JavaScript and also apply any other plugins and optimizations that we have to that code. The thing is, it also needs to know about the final build result. And that is only available once the build is finished. OK, here is how I would do this in rollup. We need to get that service worker URL, so I'm going to import it. Now, it looks a little bit magic. It's got this sort of fake URL scheme. This divides people, but I really like this because I want this to look weird if it's a special part of the build. Because it means you look at this code, and you're like, huh, that's odd. And it's like, yes, it is odd. The build tool is going to do something special here. And it also means if the build tool doesn't do something special, it is going to fail really, really early. But like I said, this divides people. Jason, how do you feel about this? I mean, I like this for the fail early reason. Like, that's a nice thing. If you pull this code into a new project, it's going to tell you, hey, this isn't a thing, which is sort of exactly the message you would want to receive. But I also like that on the right-hand side of that colon, it's still just a module path. So it's really clear for you when you're visually looking through, this is the special part. This is the regular part. Yes. So all we need to do now is create the build plugin, I guess. So here's the config, the rollup config. You've seen this before if you've used rollup. I'm going to import this new plugin and add it to our list of plugins. Now all we need to do is write it. Rollup has a one pager on how to write plugins. But it's quite a long page. But I have to say, the API is really well designed, and the documentation is really well written. So let's get started. This is what rollup plugins look like. It's a function that returns an object. It has a name as part of that, and that's for error handling and logging. But everything else, the rest of the object, this is methods that are called throughout the build. They're essentially callbacks. Rollup calls them hooks because they're hooks into the build process. The first hook we're going to look at is resolve ID. This is called for every module that's loaded. So it's going to be called for the main file there, but it's also going to be called for this import and everything it imports, including dynamic imports. The ID you get is the raw unedited import ID. So it was going to start with the dot slash in this case. And in this case, it's going to include our special made up URL scheme as well. The second argument is the importer. This is the path to the module which did the importing. And this is the only time you get this piece of information. The job of the resolver is to return a full absolute ID that doesn't need the importer anymore. So any relative paths should be turned into absolute paths. Now, if you've used rollup before, you'll know that it doesn't support node modules out of the box. You need to add a plugin to be able to do that. It's an official plugin, but you still need to add it. All that is is a resolve ID hook. And it sees these bare modules and it goes, oh, OK, I'll try and find that in node modules. But also, if you want to create path aliases to make them work in rollup, all you need to do is create a resolve ID plugin. So we don't need to do anything with this first import. We just need to handle the second one with the special URL. So here we go. The first thing I'm going to do is just exit early if it doesn't start with that prefix. Actually, we're going to do that sort of thing a few times. So I'm going to pop that in a variable up here because it's going to come up time and time again. So there's an early exit. And it means if it doesn't start with that prefix, we can hand it off to other plugins or the default rollup stuff. Now I'm going to remove that prefix and resolve it. This resolve, this is a rollup API. And what it says is, take this ID and go and find it for me. It couldn't do it by default before, but now we've removed the prefix, off it goes. And that's going to actually pass it back through all of the plugins that do resolving, including our own. But we're going to ignore it because it doesn't have that prefix anymore. But it means if you have a service worker in node modules or if you have path aliases, it'll all just work. So if there's no match, just bail. And that will cause an error eventually because it means that we couldn't find that service worker. Otherwise, I'm going to add that prefix back on so we can pick it up later on. But now we have an absolute path to the service worker. We have achieved a thing. We can try and build it. Here we go. Wah, wah. It's not going to work. But you can see in that error message it's failed because of that special prefix, but the rest of it is an absolute path to the file as it is on my system. So all we need to do is tell it how to load that script. This is another hook, load. It gets the ID as it's been fully resolved. Again, we want to ignore it if it doesn't have that prefix and let other plugins deal with it. And what we need to do now is return the content for this module. So we could just actually make up a module. I'm going to do that. I'm just going to return hello. And now we've done that. The project will actually build. It's going to turn that fancy custom import into hello. This is kind of silly, but just knowing those two things, knowing how to resolve and knowing how to load, you can do all sorts with that. You can do all sorts of cool code generation, loading of a file, that sort of thing. Like it's not necessarily what we set out to do, but you could totally see how this is basically like a Constanced plugin. Yeah, exactly. And I've built Constanced plugins, and that's exactly how you do it. But let's improve on that. So instead of just returning hello, we need to bring the service worker into the build system and return its URL to bring a file into a build. I'm going to use emits file. This is a rule of API for adding stuff into the build. I'm going to say this is a chunk. And that's telling rule of like, treat this like a JavaScript entry point. So it should be processed by all the same plugins that deal with JavaScript, like minifiers and code splitting and all of that sort of stuff. It could also be asset. And if it's an asset, that's like an image CSS, I don't know, WebGL shaders. Anything that's not JavaScript essentially, you can process in this way. I'm going to give it the ID, and that's telling rule of where to find it. So we've removed the prefix again. And I'm going to give it a file name. And this is not something you would usually do. What this does is it overrides the naming system that rule of would usually use, because rule of would usually add a hash to this file. But with service worker, you don't want its name to be changing. You want it to be in a static location. So that's how we achieve that. Actually, I'd say like, this is something that you would offer to the user as configuration. So I'm going to do that. I'm going to add an option object to the top here and I'm just going to like change the variable down here. So it's using that. So that means the user can change the path of the service worker, the file name, all of that sort of stuff. Now what we need to do is get the URL for this file and strap in, it's a bit of magic. Woo, there it goes. So I'm returning the import meta and then this sort of magic string that ends in a file ID. And we've got the file ID from emit file. Rule of will see this and it will turn it into a URL for the resource. So here's what that looks like. Here's the input and here's the output. So it's created this URL object and it does that so it can create a relative URL from this script to the service worker because on the web, if you've got an image tag with a source or whatever or when you're calling service worker.register by default URLs are relative to the page but rollup doesn't know anything about the page. So it creates this script relative URL. You can actually configure this in rollup. So you can tell it, hey, all my stuff is actually at the roots of the origin. All you need to do is just add a slash to the start. And obviously that will clean up the code a bit. It just doesn't do it by default. But also in our build, we've got the service worker. So that bit is working. All we have to do now is deal with these two things, the version and the assets. We'll handle the assets first. To do this, we need to know the full details of the build. Because we need those hash file names and all of that sort of stuff. So we need a hook that is towards the end of the build process. We're gonna use generate bundle. This happens just before things are written to disk. The options program there, it's the output options. It's not very interesting for us in this case. But this is where the party is at. This is full of interesting details about the build. It's a JavaScript object, where the key is the file name and the object has details on the file. So this is an asset. So you can see it's got the file name and it's got a buffer representing the content. But for chunks, look at all this stuff we get. This is amazing. It tells us things like the imports and the exports of the file. I've used this in other plugins for if I'm generating HTML and I'm putting a script tag on the page. I know that that script is just going to import other scripts and I want to preload those things. Well, I can do that here. This will tell me which things it's going to immediately import so I can turn those into preloads. Anyway, loads of interesting info. We're gonna make use of this. In the generate bundle hook, I'm gonna get that service worker chunk just from that output option because that's the file name of it. And then we need to figure out which assets the service worker needs to know about the things that it wants to cache. Gonna iterate over the bundle, get the values and I'm gonna get everything except the service worker item itself. If a service worker caches itself, like it doesn't cause the world to end like you might imagine, it's just a bit wasteful. We definitely designed around that. We were worried people would do that so it's actually okay. But this is a point where people might want to configure the things which are actually cached by the service worker. So what I'm gonna do is add another option here, filter assets returns true by default and I'm gonna call it down here. A slight complicating factor, we want these paths to be relative to the service worker itself. And unfortunately, rollup doesn't give us access to its URL resolving magic stuff. I have filed an issue for that but we're gonna have to do it ourselves this time. So I'm gonna use nodes, posix, path resolving stuff. And the only reason I'm doing that is because posix paths have the forward slash and that's the same as URLs. So we can rely on that. And then down here, this is how I create the relative URLs. This is done by saying, give me the relative path from the folder that the service worker is in to the item from the build. And now we've got that. I can prepend that to the file. So this bundle object is live in that any changes I make here are actually going to be written to disk. So here I'm just taking the code for the service worker chunk. Adding that assets line to the start. I'm using JSON stringify to correctly escape everything. And now that will be part of the service worker. And if we build, there it is. There it is in the output, the assets. We've created a problem though. And this is something that catches a lot of people out when they're doing service workers. It's this, it's easy to think that the service worker is caching files but it's not, it's caching URLs. And we don't want .html in our URLs. We don't want index.html in our URLs. So we need to fix that. Back in our plugin, this is where we were before. How can we solve a problem like this? Regular expressions, of course it's regex. And I'm not going to go through how this works. I see a back reference in there, I don't know. I love back reference. I often say that regular expressions are right only. Yeah, you can't read them again afterwards. But trust me, this is removing the .html and it's removing the index.html as well. This could land us with an empty string, which is a problem, because that will resolve to the service worker script itself. So in that case, just gonna output a dot. And that solved the problem. There we go. Index.html there is replaced with a dot. Okay. Last thing, just need to deal with that version thing. Thankfully, we're most of the way there already. This is going to be a hash. So I'm going to include nodes hashing stuff and then down in the generate function, create one of these objects. I picked char one because I've heard of it before and it wasn't MD5 and I'd heard it was worse. Yeah, this is very secular. I mean, yeah, security doesn't matter in this case. It's just for hashing. Oh, don't quote me on that. Like security generally matters. JGARTABLE, security doesn't matter. Just, what a quote to take out a context. Okay. So now here I'm adding everything to that hash. So it's either the source or the code, depending on whether it's an asset or whether it's a chunk. And now I can get a hex version of that hash and include it in the old output there. And that's really it. That's it. We can see that the hash is now in the service worker. I wanna, there's a couple of caveats here by editing the source right at the end there. That was after source maps were generated. So we have just broken source maps just for the service worker. I've filed an issue with rollup because I would like this to be an easier way to do this. So hopefully they can add something. We could probably work around this by adding those blank lines in a transform step so we can fill them in later because that will, those extra two lines at the top will at least be taken into consideration with the source maps. I don't know, it would be good if those are proper way of doing this. But anyway, that is it. That is the full plugin. So, Jason, how would you do that in Webpack? Right. So building plugins for Webpack works a little bit differently. And to illustrate this, I'm gonna use a fairly similar approach and naming to what we saw in Jake's rollup version. Much like with rollup, the place where we're gonna start will be the bundler config. Now, a typical webpack.config.js file specifies some entry points, which are gonna be the modules where bundling starts, some options for the format and location of files generated on disk and plugins, which is what we're concerned with today. All we need to do is import our plugin that we're going to write and then add it to the plugins array. At its core, a Webpack plugin is really just an object with an apply method. And that apply method gets past the compiler instance. So wait, one second. So this is like, I see you're using classes here. So in rollup, we were using a function which returned an object, but in Webpack land, it's all class-based. So technically, Webpack does not care. It just needs an object with an apply method. But all of the Webpack core modules are classes. And I think as a result of that, all of the ecosystem plugins tend to be classes just because when people import it, they expect to instantiate it with new. You could totally write a Webpack plugin that was a function that returns an object. I was even actually tempted to do that for this talk to make it look more like the rollup plugin, but we'll stick with the thing people actually do. All right, so if you remember in rollup, plugins define special hook methods that get called in response to events. In Webpack, we do the opposite. Your plugin taps into events. And so what we wanna do is we wanna handle a special service worker colon import prefix. And so to do that, we need to tap into normal module factory. And just to preempt, because everybody gets confused with this, normal modules are source modules. These are the code you write, the code you get from NPM, code that you're going to put in your application. There are other types of modules, which is why there's other module factories for things like loaders. Basically, they're more on the infrastructure side of things. So what we're looking to do here is resolve our own import. We use normal module factory. And so similar to with rollup, anytime we tap into something, we wanna pass it a name, which is the name for our plugin. And this gets used for debugging and logging purposes and also when taking performance profiles. So within normal module factory, we can now hook into Webpack's resolver. And this gives us a reference to Webpack's own resolve function, which is the thing that sort of handles finding modules and loading them from disk. But it also lets us return our own custom resolver function, which gets passed a dependency description and a callback, cause it's asynchronous. And so this function, we can call the original resolve function, we can do something custom, we can combine the two. And to make things easier, we're just gonna take those three bits of information that we have and pass them to a new resolve ID method that we're going to write. And that's sort of everything we needed from apply. So our resolve ID method is gonna be called for each import specifier in the app with a description of the dependency, that original resolve function from Webpack and the callback to call when we're done. A dependency description is an object with context and request properties. Context is the directory of the module that's doing the importing. So in this case, dot slash. And request is the unmodified import specifier. So in this case, it includes our service worker prefix and also the relative path to that file on disk. So similar to roll up, the first thing we wanna do is figure out whether this is an import that has our prefix that we want to handle. If it isn't, then we can just call Webpack's original resolve method. It will do what it would normally do since there's no prefix, we don't really care about that module. But if it does have the prefix, we need to remove that prefix and then pass it through resolve. And so if the module can't be resolved, we can just forward that error up. That will break the bill. There will be an error in the console saying, hey, couldn't find module SW slash index.js. If it does work, then it'll pass us back a new dependency object where the request property is a fully resolved disk path. The thing is we don't actually wanna resolve this module to a path, right? We wanna resolve this module to a string somehow, right? This is the string that contains the location of our service worker. And so to do that, we need to sort of, we need to come up with a way of producing a virtual module, even though we're inside of the resolver. Essentially, we want to intercept this service worker prefix to request and then resolve it to a module that we create on the fly that doesn't actually exist that just exports the string URL of our service worker. And then one thing we do need to account for here is when we're working with URLs in Webpack, we need to make sure that we respect any Webpack public path magic global that has been set. And so this is either the output.publicPath configuration value or some value that has been set at runtime. Right, okay, so this is another difference from rollup. So rollup had to do some of the magic around resolving URLs and stuff. Whereas here, you've just got a string, which is like, where in the web route is it? Right, yeah, so like rollup, they have automation here to sort of give you module relative paths. In Webpack, it's a configuration value. And the nice thing is, because everybody uses that configuration value, all plugins tend to respect it. But so here, if we combine these two strings, we'll get the slash, which is sort of our default Webpack public path value. And that means that the result of this import will be the service worker's URL. The thing is, we haven't actually explained how to construct a virtual module yet. We know what we want to do. We don't know how to do it. So first we wanna start with that code that we wanna generate. And you notice here, I've parameterized the URL from that code string. And that's because we want to make that configurable. So in our constructor, we'll accept an output parameter. The developer can pass us the URL to use, we'll store that, and then we can inject it into our code by JSON stringifying it, which will escape it and wrap any quotes. And then the last bit is to take our piece of code and pass it back to Webpack. And to do this, we're going to use something called raw module. And we're gonna import this from Webpack core. So raw module is basically a way for us to provide code back to Webpack in the same module format as it would expect modules loaded from disk. And the reason why this matters is, we're going to use something packed with an object that has a source method that returns a code string and an identifier method that returns some sort of a unique identifier for that module. In our case, based on the code string. And in doing this, Webpack will actually grab the code that we send back to it and avoid ever going to disk to resolve this module. We'll just use whatever we pass. So now we have our import resolving to the URL of the service worker, but that URL doesn't exist because we have not yet generated our service worker. Let's do that. For this, we need a new plugin hook. And so when we need a new plugin hook, we have to jump back into apply. We need information that's gonna be codified into the service worker, those version and assets globals. And to get this, we need to tap into something that happens at the end of the build once that information is available. So for this, we can use the emit hook. You'll also notice we're using tap async here. And this is because our emit hook is going to be asynchronous and we want to hold back compilation until we're done working. So the emit hook gets past a compilation instance and a callback, which we're gonna forward onto a new method just to keep things clean. In Webpack, each compile pass is referred to as a compilation. And the main thing that we're interested here is the generated assets, which are an object where the keys are file names and the values are asset descriptors. I recognize this. This is the same as the rollup bundle thing, right? Like we had in generate bundle. Yeah, it's really similar. I don't think I actually realized until we were sort of looking at these things side by side, they contain really similar metadata. So it has, you know, the dependencies and is this an entry chunk and all that information. It's also for both JavaScript and non-JavaScript assets. And in both cases, there's a source method on the asset descriptor that will return either a buffer with the non-JavaScript asset contents or a string with the JavaScript code. So we need a list of files for the service we're gonna cache. And to get that, we could just take the keys of the assets object. Similar to with the rollup plugin though, it's generally a good idea to make it possible to filter these assets before they get embedded into the service worker just to avoid a huge list. So we'll add a filter assets constructor parameter that a developer can set and provide a function that takes a file name and returns a boolean indicating whether that asset should be in the list. Now we have our list of asset file names. The next step is to calculate a version based on a hash of their contents. And so to do this, we need to pull in the create hash method from nodes crypto module. Then we'll loop over each file name in our filtered list and add its corresponding asset source, which will be that code string or buffer to the hash content. Then our version string is just going to be a hex digest of the combined contents of all of those assets. So that's the magic version global done. Now we can do the assets magic global. So as Jake explained in the rollup walkthrough, our assets array contains file names, but our service worker needs to work with URLs. And so to fix this, we need to pretend any configured public path value to the names in case we're deploying somewhere that isn't the root domain. Right, so this is again, this is gonna be simpler than it was with the rollup case because you've just got the string that you can add to the start. Right, so because everyone knows to configure that public path, we can basically count on it being there. If it's unset, we'll just use slash, but we're gonna be able to resolve these things at build time. So I'm showing some of my biases here, but I think this is way more complicated than the rollup solution, but it's interesting seeing these little details that I'm like, oh, I wish rollup had that. I mean, I know why it doesn't, but that would be so much simpler for me if it just had a public path string or whatever. Yeah, and maybe that's part of rollup adding that exposing their built-in resolution is maybe there's an option you can pass that's like, oh, by the way, like resolve it all against this public path. That might be a convenient thing, who knows. So while we remap all of those file names to prepend that public path, we can also remove any trailing index.html because that doesn't appear in the URLs. We wanna catch the URLs, not file names, and we will use that dot workaround to safeguard against an empty string actually catching the service worker, which we don't want. Instead, we'll have it catch the directory the service worker is in. So that's both of our magic globals prepared for the service worker. Now it's time to generate it. And here we're gonna create one last function to handle compilation of the service worker. So to build our service worker module, we're going to use something called a child compiler, which is essentially a nested webpack compiler. And we'll configure this to output these sw.js file. And one thing to think about when using child compilers is they can have a completely different list of plugins compared to the main compiler. So we can pull in a plugin that generates worker compatible code because we have a service worker target. So there's gonna be a difference here as well, isn't there? Because like, well, actually you tell me because you're using a child compiler and it's like a whole different pass, does that mean that this service worker bundle is not going to be able to share code with the main bundle? It's not gonna be able to code split there. Correct, yeah. So any split points that end up happening in the code generated by this child compiler are gonna be totally independent from the main compiler. They'll be a separate set of files. Also. Okay, that's an important difference between the two. I mean, it's not common to share code between your service worker and main thread, but the rollup version could do that. Yeah. Exactly. So the other thing that worker template will do in this case is if there are split points rather than using a script tag to load those chunks of code it will use import scripts, which is available in workers. Cool. And so the last thing we need is a second plugin also imported from webpack core. And this just lets us specify the entry module to start compiling from, which is in this case our service worker source module. So finally we run our compiler as a child of the main compiler. And this just ensures that if our compiler takes a while to run the main compiler won't finish and terminate the process before we're finished. And then once it's built, the callback will be called with our compilation and also any error. So just like with the admit hook, we are gonna grab the generated service worker asset from compilation.assets and then we can pass that back to admit. If there was an error in the child compiler, say a syntax error in our service worker code, we'll just bubble that up, that will fail the build and print it to console. If there wasn't an error, we do now have our compiled service worker asset. The thing is it doesn't yet have version and assets magic global variables. So to inject those, we need to pull in something called concat source from the webpack sources module. And this lets us concatenate strings essentially, it's very similar to string concat, but with the added benefit of it is able to produce source maps. Ah, yes. Okay. So this is interesting because I broke source maps in my solution, but this is going to just work. Now I could have done the same like in rollup, I could have created essentially that it doesn't support child compilers as such, but you couldn't just call rollup as a sort of separate layer. That would have been a lot more complicated. So it's interesting now, like some of the complexity that you're using here with webpack is actually paying off. Right, yeah. So like the initial bundle of service worker was harder, but as it turns out, it's the same number of lines to support source maps as it is to not support source maps now that you've opted into some of that complexity. So that's sort of an interesting tradeoff between the two. So yeah, so the last step, we've constructed this asset, but it's only just an object in memory. So what we need to do is merge it into the compilers generated assets, like with rollup, this is a live object and anything that we merge into here will get written to disk. And so with that, our plugin is done. So we can call the callback, compilation will finish. And what we get is our generated service worker source. We can see here that we have assets with the generated array of URLs. And we also have version, which is that string based on the hash of their contents. So we did it. Amazing, and that's it. Yeah, and I guess that's bringing this talk to a close, provided that everything has recorded properly and we don't have to do it again, because I can't tell you, this recording has been absolutely cursed. We have done so many takes due to just Bluetooth stacks failing on my machine, recordings not happening on Jason's machine. So who knows, maybe this will be the final record. If you're watching this right now, then I am so happy because it means we don't have to do all of this again. Oh, right. Okay, if you want to see those implementations, they are both on tooling.report, along with how to handle assets, code splitting, hashing, transformations, all of the stuff. And not just webpack and rollup as well. We also have parcel, and we also have browserify. Remember browserify and gulp and all of that sort of stuff. I mean, you might laugh at that, but this is, if you look at the stats on NPM, that is still absolutely huge and climbing. And we did actually find some cases where it really holds its own and was maybe better than some of the more modern things that we deal with. So go on and check all of that out. Thanks so much for watching this, hopefully final take of this talk. Thank you very much and goodbye. Bye.