 Hey everyone, my name is Laura. Welcome to our screencast today on Webpack, a hard source plugin that my colleague Zee Goddard has helped create. I want to say thanks so much for coming and we're really excited to get started. So I'm going to introduce my colleague Matt Seravian, the director of Webapps to take it away. All right, you can ask questions in chat and I believe Matt has also posted a poll that he'll go over that. Thanks so much. I think everyone can hear me. As Laura mentioned, I'm Matt Seravian. I'm the director of Web applications with Bo Koo. My colleague Zee Goddard here is as well and we just want to kind of talk today about Webpack and the hard source plugin and how you can improve your build time. So for about five to ten minutes I'll show you how it's integrated into a project. We can talk about a few gotchas when it comes to adopting it and then the rest of our time together we really want to leave open to y'all to ask questions so we can help you use this stuff. So yeah, there's to that end on the right hand side of your screen there's a polls tab and there's three questions there that kind of will help us get to know how familiar you are with Webpack and hard source today. That'll help guide our content. So if you could take a second and answer those questions that would be great. There's also a questions tab and even if you don't have a question to ask please keep monitoring that tab and the reason is because you'll be able to see everybody else's questions and upvote them and downvote them and that'll help us prioritize them depending on how many we get. So yeah, I'll just get started too by saying we're going to be recording this and so if you want to come back and watch replays you'll be able to do that and if you haven't registered ahead of time we'll put the video up on YouTube and write a blog post about it too so you can definitely share that with folks that you think might be interested. So to start I think what I'm going to do is share my screen and hopefully you'll be able to see what is on it. Cool. Yeah, at a high level it looks like pretty much everybody knows what Webpack is based on the poll but I'll just give a brief overview for anybody that might be watching this that isn't quite sure. Webpack is a front-end build tool and it's dedicated to kind of answering the question why should our build tooling only be bundling JavaScript files. So we want to be able to write code that actually catalogs all of our modules dependencies. So you can actually write something like this for example require theme.style or require image.asset.png and that's really powerful not just because we can have Webpack take over where say something like gulp or grunt would need to do CSS preprocessing or anything like that but also because Webpack gives us all the tools that we need to assess our bundle and chunk it out such that the users only required to download and consume exactly what they need to do what they're trying to do in your application even though most projects don't necessarily get all the way into Webpack code splitting that's really it's promise the improved Analyzation and support of that critical path. You know think about if you're writing a game and it has 20 levels you don't want somebody who's just coming to your game for the first time playing level one to need to wait to download all the assets for levels two through 20 or if you have a large application with many views you can deliver those views and the code that implements them on demand as the user accesses those parts of your application. The trouble is that as we ask Webpack to go beyond the land of say concatenation and minification things that require JS and browser if I used to do for us. I mean at least for me you know before I started using Webpack that's what I was using. Yeah Webpack can do all those things and any other kinds of like asians that you need transpilation whatever it is the more you ask it to do the longer it needs to do those things and if we were only cutting builds once a day or something like that shipped to production maybe that wouldn't matter but now we're writing code like this and this code isn't strictly valid without Webpack right what does it mean to require a theme.style us you know we need Webpack to actually do active development and that can mean that we have to wait around a lot and Webpack has this multi-stage build process and so what my colleague Z's plug-in does is use the hooks that Webpack gives us to dive in and cache those intermediate stages of a module so that Webpack doesn't have to say go all the way out to a file system to resolve it do all of its processing on it. This is great especially for like a third-party dependency that hasn't changed since you brought it into your application or if you have a particularly large app and you're making some small change adding a comma and starting a semicolon you don't want to have to be sitting around waiting for every single third-party dependency to get re-bundled every single time you make a change and Z and I can answer more technical questions about exactly how this works I have the plug-in code here but I don't know exactly how much y'all want to get into that we can definitely do that later so please ask those questions if you're curious how that works under the hood. Most of what I want to show is how Webpack can get brought into a project so this here we'll just take this part away for now we'll remove the cache. This is my Duck Hunt game. Oops pretty simple replication of the NES game used to be using browser 5 and now it's using Webpack and I have a big third-party dependency and that's the Pixi.js rendering engine that handles putting this in canvas or WebGL depending on the consumer and so I changed it over to use Webpack instead the Webpack config itself is relatively small. Oh I'm already running that over there. My Webpack config itself is relatively small. I want to be able to you know transpile my ES 2015 code into ES5 so I have Babel loader in here. Right now I'm not actually using it to load my PNGs but I want to be doing that. I'm you know using it to load some JSON data as well and I'm not yet using it for the audio files but I want to be doing that too. Still my build kind of takes a while. Considering it's such a small project usually my build is about five seconds and let's see how long we get for this run. Oh this one's even longer maybe because I'm screen-sharing. That was 12 seconds kind of rough especially considering I'm not even doing anything with images or mp3s. If I put back in the hard source plugin I'm kind of cheating because I already have an established cache directory here because I've been using hard source on this project. This will go through and this will use that cache so it'll still still take a little time but you can see that's significantly less right. That's that's a little less than half the time. This one only took five seconds. Now if I were to change something in my code let's say we're to add a new state it's going to rebuild that file because it's changed but it's going to use the cached versions of everything else. It's not going to blow away my entire web pack cache that it's using so you can see that module that it just ended up rebuilding. It's just this one that it's changed since the last time it populated the cache and it added a little bit of time but it's still not 12 seconds right. It's significantly less than that and so this is the most simple config that I've found to use with hard source. It needs a cache directory and that's just essentially where it should store its cache data so here's what that looks like. It can be anywhere. I like to put it in node modules.cache slash hard source but this is user defined. It should be absolute. It will accept a relative path but more and more things in web pack really want that absolute path config so it's a good idea to use path.join here. Records path is something that may already be set depending on your project. What this does is this lets web pack keep the same IDs and and store in a persistent way the ID of your applications modules with the ID that web pack gives it. The config hash method here is what determines what these folders are named. This is actually running an object hash on my web pack config which is a JavaScript object. That's going to make sure that let's say I have multiple web pack configs or I change this to add a different loader. That's going to make sure that a new cache is created and this kind of brings up one of the gotchas with this plugin and that's that by default it really doesn't know where to put your stuff. If I didn't have this here it would store everything inside this directory but it wouldn't know when to bust, when to make a new cache. Giving it some knowledge about that and that's what this is. It's going to help distinguish these things. The other thing that it has under the hood is a concept of your environment. This is what the default environment hash looks like. This basically takes your root directory, watches node modules and watches your package.json. What we're trying to do by default is support the NPM ecosystem. If you add a new node module we're assuming that there's going to be big changes that we need to make a new cache for. If you're updating an existing thing in node modules or you're operating a version not just adding a new package we want to be able to bust your cache for that too. You'll notice though this isn't purging anything and that's one of the things that you should keep in mind is that if you have a really large project with really large build artifacts these cache directories can build up on your machine and take up space. It's a good idea to go through and clean them out. There's an open issue right now where we want to look at trying to figure out a property that might be able to help with that logic so that we could do some purging but it's a little bit complicated because obviously don't want to purge things that the user might need. So right now that's a task that's left to to the folks who adopt this plugin to kind of stay on top of. The other thing that you should be aware of when using the webpack hard source plugin is that it doesn't always play well with other plugins. It seems to work well with you know main things that we use often but that's not to say that it's gonna play well with every single plugin. The issues is a good place to look to see what's going on. The only thing that I'm aware of right now is a change was recently made on the offline plugin and there was some interrupt problems that seems to be offline plugin related. There's an open issue about that and the previous version webpack dev server just released a new version about a week ago and the previous version was broken for some reason working with hard source but the current version of webpack dev server works appropriately and it's just kind of hard to track these things down so definitely if you notice this not working with a plugin that you have on your project these open an issue and let us know. The only other thing that you should really be aware of when using hard source is using loaders or using dependencies that when they're bundled aren't done in a deterministic way. So in other words if you have some loader that changes files and it's not always gonna output the exact same bundled version of that file it's gonna change that every single time it runs through even if the file doesn't change hard source isn't gonna be able to cache that. So that's something to be aware of. The majority of loaders are deterministic and then are cacheable but because the webpack ecosystem is so large it's possible that there are things out there that won't work but so far we've done a pretty good job of trying to find those things and then track down what makes them not work so definitely check out the issues list to see if there's anything right now that you may be using that don't work. If you run into any problems outside of that a good place to check is Stack Overflow 2. We try to be active on there answering questions. Let me show you before I kind of turn it over to more broad questions exactly what the adding of hard source looked like in this project. So it really was adding these two dependencies hard source and then node object hash which is what we're using right here in the webpack config to name our config hash so that hard source knows when to bust its cache and that's it. That that thing you can see here when I first added it it was with relative paths but you add the hard source plugin and we required it and that should be all it takes to load this into your project. I'm going to stop sharing my screen now. Let's kind of check back in to see what kind of questions we have coming in. Cool so Andrew wants to know what the difference is between this and webpack's existing cache option. I don't know Z do you want to talk to that a little bit? Yeah I can speak to that a little bit. So I guess one part is there isn't strictly a difference. There are parts of hard source where it makes use of what the cache option does. What the cache option does is it basically creates a memory store which actually keeps copies of your built modules and then when it does rebuild with webpack dev server or webpack dash dash watch it's first like it builds a module it builds a reference for a module looks inside the cache to see if that module already exists and if possible reuses it. Hard source works similar similarly in that it creates those references so there's less work that needs to be done hitting the file system but also that it prefills that cache where possible so that provides a lot of performance and then also the versions that are put in that cache are pre-built. That's the truth like every step of the process and not just that initial file system hit that hard source is trying to optimize for. The file system hit is very real it takes a long time to resolve your dependency especially if it's not an absolute path and it takes a lot of time to get that data but all the other pieces that go along inside webpack for building up your bundled assets are what we're trying to optimize for here and I think there's a push happening to try to get hard source into webpack core is that right? Yeah there's definitely a lot of desire for because it'd be really great for it to be a first-party member now that webpack 2 is out that's going to become a much bigger reality likely in a future webpack major version change because the the webpack the core team is very much planning on having more regular updates so that'll make it more of an even more capable sense like webpack hard source was first released in September and that was still a point in time for webpack 2's main development it could have gone in for webpack 2 but just all the other changes webpack 2 was making it was going to be a reality now but now that's much better and like based the main path is like figuring out what in hard source we can generalize so it's more available to the community to interact with in a deeper level yeah and just to say explicitly hard source does work with webpack 2 and webpack 1 it's just not built-in core to either of them yet well I guess it'll never be built in the webpack one yes yeah are there any other kind of questions about how this might work under the hood or how you might be able to adopt it on your project I mean the integration is meant to be pretty simple like I said there's in the read me it goes over each specific piece of the let's see here we go the plug-in config options I mean we can kind of review those at a high level to talk about what they are just again to remind you the most minimal thing that you need to provide is a cache directory a records path if you're not already using this if this is set elsewhere in your webpack config then you don't need to specify it here and some kind of config hash so the other option here you could use like a process environment variable or something to name your folders the only trouble there is that then you know you're gonna have a bunch of things in there that might not be exactly right so it's it's recommended that you just hash your webpack config and you let the environment variables the environment hash rather which is this here you let the environment hash take care of the other busting logic and that will also create a new you know that will update any existing things in your cache automatically for you without necessarily creating a whole new folder yeah these other ones this is just a different thing so instead of the providing this config where we let a hard source take care of watching these things you can also have a custom environment hash function that handles looking at other things that you may have in your project like a yarn file that don't directly you know impact package.json or your node module structure sounded like we might have had another question come through so how do you think this would interact with the webpack DLL plug-in Zina we're just talking about this this morning actually yeah do you want to talk at a a little more detail about how that might work I know the idea is that with DLL there's potential for having like a dedicated implementation of hard source so that it could take better advantage of what the DLL plug-in is doing right now yeah I mean do you want to talk about how that works right now I think you probably do a better job. Right now there's a change made I think in December to hard source to support DLL plug-ins we're not able to take like full advantage of DLL plug-ins but hard source plus DLL will still be faster than just DLL the change that was needed was if you beforehand depended on a DLL a dependency insider DLL module it would have to rebuild that every time but now it's hard sources be able to be smarter about that and only rebuilds when either well specifically your module changes so like if you depend on a DLL dependency it's fine caches cash works DLL is still able to give you a lot of performance as well that you wouldn't get with hard source alone in the future what we want is a plug-in interface for hard source because one of the troubles right now is DLL plug-in has some internal knowledge that hard source cannot access so hard source needs to expose an interface that you would have then a DLL plug-in and probably a hard source DLL plug-in and the hard source DLL plug-in would be given that specific knowledge that DLL plug-in has so that it can more deeply integrate and provide even better performance while also being smart and not getting in your way a couple other questions here when might I need to provide the environment has config option I think what I would say to that is if you're using yarn you should look at the function that's provided inside the read me of the current hard source version because by default it's only gonna look at package.json and node modules for changes the other time you might want to do it is if you are vendering any dependencies like actually have a vendor folder with depths that might change that you need to be cached specifying that inside you know so extending this default config see there we go yeah extending this default config to include they have under directory and that way any of the scripts that you're including in that vendor directory will properly be cached I should say if you don't do this the only consequence is that you'll have to wait for those things to be processed by webpack right it's not gonna break hard source or anything whoa it's not gonna break hard source or anything like that another question here is sounds like the best way to use this is npm run dev enable cache I don't see must use for QA or prod I think the only things I would say there is it depends on how long your your build time is you know if you have say CI run and that's taking you 20 or 30 seconds to get feedback your CI box could do well to have a cache enabled but prod build it really depends if you don't mind waiting then fine but yeah the biggest impact that we've seen is definitely on active development and it does work with webpack dev server see does the webpack build benefit from using this plug-in if I'm using webpack watch and keep alive configs and touch a file there's this only beneficial between launches of webpack as long as it'll correct me if I'm wrong Z but as long as your your current you know running webpack process has knowledge of heart source so your configs been modified to include hard source it should play well with watch it should play well with webpack dev server so bit detects a change in the file it should you be using your cache for anything that has not changed still so you shouldn't have any you shouldn't have any issue using webpack watch or keep alive configs along with webpack hard source you should still get the benefit yeah I can I can add that to that a little yeah um in part has to do with other options there are some things you can get still performance benefits with say that that are less obvious hard source supports source maps if using say like eval source maps for your dev environment so that each module internally stores the source map instead of a larger source map which is like the normal recommendation for development use hard source provides a benefit there because how the by its name like it's literally keeping kind of like a hardened version in its cache it also includes a hardened version of the source map so when it's rebuilding it doesn't need to webpack doesn't need to go and rebuild as much source back work there's also the other small benefits in that same way that the cached versions are not as flexible as the normal webpack versions and hard source just immediately drops its cached version if it needs to do if hard if webpack needs to do more work with it there's a couple other questions here I want to I'm going to share my screen again just to show that minimal config the other thing I'm going to do is put a link in chat that shows you on github that duck hunt game is an open-source project of mine so you can always reference that so there's a link to that now to just share my screen again yeah the minimal config you have to instantiate the hard source plug-in and you have to give it a cache directory so a place to put its cache data if it's not already specified elsewhere you have to give it a records pad and this is what tells webpack how to persist the IDs that it assigns to your various modules when it's running a build and you have to give it some kind of mechanism for hashing your config to know you know when to bust the cache and what to name those folders that it's going to use and the recommended way to do that is this way by using a node object hash is just something that hashes a JavaScript object and passing it your webpack config this callback is expecting to be passed the webpack config so this here is the minimal config available the documentation in webpack hard source provides sort of multiple things that are you know available so like this says this is optional it is but you probably want to use it otherwise you only have a single cache and things could get kind of screwed up for you so definitely even though it's optional we suggest that you provide config hash all the time environment hash just to review one more time since I think it's relevant to this question you really only need to provide if you're doing something like vendering dependencies that are you know not inside node modules or you're using yarn or something else where you would want to trigger your build based on a change to a file that is not in node modules not in your vendor directory if you're using it and not in package.json so something that's totally outside of your project and again if you fail to provide this and you are using vendor dependencies where you do add a node module you're just not going to get the benefit of caching that stuff it shouldn't break anything can invalidated cache directories be detected and removed all child directories that do not match so theoretically yes there's an issue that's trying to explore the best way to express that in a config parameter which is really the challenge right now so that is something that's manual you are kind of responsible for purging your own cache if you're not changing your config a lot you probably won't have a bunch of different cache directories but it's something to keep in mind and as I said I do think that we'll solve that it's not a problem that's like oh my god how do we purge the cache it's just something we haven't had a chance to do yet I want to add something to that yeah please do the main reason config hash exists as so config hash is an option you don't need to use config hash if you only are going to have one only one configuration with webpack and heart source you can drop that you can drop the square bracket config hash from the paths and you'll have one config one cache directory and that will be busted every time your environment changes through the environment hash config hash exists for webpack environments that have multiple configurations a lot of projects will have one webpack config for development and one webpack config for production or their configs might be generated and one of the most common uses I really say is people using webpack and webpack dev server it's not super obvious but webpack dev server when given a configuration actually makes small changes to it specifically it automatically enables the hot replacement plugin and so if you're using webpack and webpack dev server with the same configuration you want to use config hash because it will give you two cache directories for each of those or it will give you one one cache directory for each yeah I think to that point we might be able to answer Jay's other question here which is how will heart source save time while running mocha tests each time you make a change on a js file which is a major pain it is so usually for projects that I've been on we try to run a separate webpack config for ci or you know any kind of testing scenario and to that end we should be able to cache that as a separate cache directory that webpack config should hash differently if it should have its own cache directory and again you should be able to just run those tests and have webpack know to use your cache files if you're somehow writing tests that are not operating on your webpack builds like if you're somehow excluding webpack from your test environment then there's nothing that heart source could really do I'm not sure exactly how you might be doing that I guess kind of depends on the application there's another question here about I'm explicitly excluding buffer and crypto from the webpack bundle with node my second build with the heart source plugin seems to fail any ideas I think so this is something that we're happy to help debug definitely put some more data about that up on stack overflow and we can take a a little bit more time to dig into what that could be I assume it fails to like fail to resolve module or something like that so we'll have to dig into that a little bit in theory it shouldn't be doing that because it should say oh I can't resolve the module no problem let's go to the file system and get it is another question here that is could we talk a little bit more generally about strategies to reduce webpack build time and size yeah I think the the high level things that I would say is one we want to do another screen test about sort of bundle analysis using webpack's profiler to look and find like okay you're you know requiring some whole dependency theoretically we should be using tree shaking to like toss out unneeded exports but that doesn't always work the way we want it to and we still end up with some unnecessary code in our bundles and webpack's profiler is a great way to identify that and also identify like what is causing your bundle to be so large so like my webpack you know bundle for duck hunt is something like 2.6 megs which is pretty big and the reason for that is a lot of low-dash dependencies that I'm using and the rendering engine and some of its associated dependencies and so by looking at the bundle analysis you can kind of look at that and go okay well yeah this this particular thing that I'm bringing in is doing XYZ for me maybe there's a way I can either live without it or write a first-party implementation that involves less edge cases a lot of times the dependencies that we bring in like low-dash have really powerful functions that are made to solve for lots of possible edge cases when you call them and if you know that you don't have any of those edge cases you can save a lot of code not necessarily the best advice depending on what you're building because you may run into those edge cases so there's a there's a risk there I don't know if you have any other general advice for you know reducing build time and size but I can think of a few things if you haven't used it yet there's a package called happy pack which is like basically utility to work with webpack to do loader building in their own node processes doesn't work with every loader requires a much more manual configuration but if you can set that up that can be a pretty nice boost to build time if you have like a four-core CPU on your machine you can build four times the number of loaders that you normally be able to that also then works well with hard source because then hard source still is able to cache hard sources cache effectively caches all the work of all loaders so you still get that benefit but then if you also have multiple files changing at the same time you get multiple loader multiple modules being built at the same time um you uh Christopher also asked the same question dll plugin dll plugin is really great for providing build time and if you want to consider it for size dll's can be used over a large number of builds so that users only grab like a dll that can be like a month old because it's still uh still works and it can be hard cached on their browser uh the dll like lets you build for anyone who isn't aware dll lets you build a bunch of say dependencies like if you're using like uh react you can get react and reactom all of its files into a single dll and then all of your future builds needing react and reactom will use that dll and it gets a lot of build time performance boost and it gets that caching benefit for the user some other size stuff size definitely gets uh pretty tricky yeah i think like what you're saying previously is it's not just size of your bundle it's like how you're going to chunk it out and deliver it to your users so it is important to manage size but i think thinking about using dll's or async chunks or anything that kind of prevents your user from having to take the entire thing on it once but again it depends on your application whether or not you can actually do that and have your application function because some stuff you really do need everything yeah uh if you're not using hard source on size on that caching stuff too uh the records path which you can use outside of hard source uh is a really great option because it helps make your builds more deterministic so that you can have one build today and then you make a build tomorrow and if you've made no changes it will be a identical um webpack too also does some other things that make this more likely to do without records path but with records path uh you get those benefits you get those benefits with async chunks and you can just produce more parts of your build that are uh more cashable for users yeah cool well i think um we're about out of time now but um if you have any questions you know problems that you run into using this either open an issue or uh go to stack overflow we're going to be trying to keep our eyes there for for general things like hey i'm using this on my project it's not working with xyz and of course pull requests very welcome suggestions on how to make this better if you have any thoughts on that cash purging problem we would love to hear them uh yeah and feel free to reach out anytime if you want to uh find other ways that we could work together thanks everyone hey everyone thanks so much for joining um as matt mentioned we're going to be posting or sending out the recording um of this webinar and posting it on youtube um so yeah thanks so much for coming and we hope you have a great day