 Before we really get into the talk, there's this URL that I just want to share with you. It is bit.ly-cli-build. I uploaded my slides there so that you can basically follow along as you wish, especially if you're sitting in the back. You just want to get the cool animations and embedded movies, which is really sad, but it's better than nothing. So I'm a developer at Dockyard, and if you haven't heard of us yet, Dockyard is a design and software consultancy located in Boston. We do both staff augmentation projects as well as full-stack web app development, especially with Amber in Phoenix. If you happen to be interested in either working with us or for us, let me know, and I'll be happy to chat with you. So quick mention, Dockyard is organizing another Wicked Good Ambercon conference at the end of June. We are returning to Thompson Island, and it's going to be a two-day event, so it should be a lot of fun, and I hope a lot of you guys can make it. If you haven't grabbed your tickets yet, we have a 25% discount code available. It's Ambercon 2016, and it's only going to be valid for the duration of Ambercon. So all right, on to the show. Today I'm going to talk to you about AmberCLI and specifically about its built pipeline. I'm going to assume that you use AmberCLI in your day-to-day work, so you're familiar with the project structure of an AmberCLI-generated app, and you're also familiar with what add-ons are, but not necessarily with the different add-on hooks that are available. So I think we can all agree that AmberCLI plays a major role in the success and growth of Amber. It brings us some very much needed conventions, powerful toolset, so that we don't have to spend time deciding on things like how to structure a project, or how to load modules and dependencies, or how to run tests. We also don't have to think about how a project gets built. AmberCLI has essentially abstracted away all of the build details, and the only thing that we are left with is this very simple AmberCLI build file, which is just a few lines of code. We have access to this Amber app class, and we can use that to augment the build or provide different options from the default. For more advanced customizations, we can turn to add-ons, whether we create them as in-repro or add-ons, or we bring them in through NPM. There are lots of add-on hooks available here that AmberCLI exposes to make this customization possible. Of course, the majority of us could probably go on and build large and complex applications without ever having to tap into the build system, either because we don't need to, or because the customization work has already been done by somebody, and the only thing that we need to do is go fetch the add-on that does the custom work, install it into our app, and then we're done. But what if the functionality doesn't exist yet? So let me give you an example. Let's say that you're building an app, and you realize that for performance reasons, you could really benefit from using web workers. So web workers have to sort of live on their own. They don't really get concatenated with the application code. You could create a worker's directory to house the web worker code. But the problem is that AmberCLI isn't aware of this directory at all. So how do you tell it that you want this code to be included in the build? What if you also wanted to lint the worker code or write it with the latest atmospheric syntax and have it transpiled using Babel? Or even minify the code? Somehow you're going to have to configure the MCLI build to do this. Here's another example. Imagine that you want to create something like Laurentian's Ambermetrics, and you want the add-on to provide built-in support for various web analytics services, such as Google Analytics or Kissmetrics, Mixpanel. Wouldn't it be nice if at build time, the add-on could somehow analyze which adapters were used by the application so that it could exclude the ones that are not used from the build output? So also, again, in order to do that, you would have to customize the AmberCLI build. And if you've never done that before, the first thing you probably would do is go to the AmberCLI website, navigate to the Extended Y section. But chances are that you're not going to find anything too useful there. As you can see, the Advanced Customization section is fairly small at this time. In the AmberCLI repo, there is a markdown document that does describe each add-on hook, which is great if you already understand broccoli. But I think a lot of developers who are just starting out and are not too familiar with the concept of trees, may find it difficult to go from the documentation to the actual implementation. So I want to show you that it's not too complicated. It's just a matter of understanding the basic building blocks. So Joe List introduced broccoli two years ago at the very first AmberCon. It's the library that is used inside of AmberCLI to manage the build pipeline. And it was built on a very simple primitive, a tree. So what is a tree to begin with? Back when broccoli was first introduced, a tree was presented as nothing more than a directory of files. You could define it as a string that represents the path to a given directory on the file system. Or it could be an object that implements the plugin API. Naturally, you might wonder what a plugin is. So the docs may tell you that a plugin exports a function that takes one or more input trees, performs some transformation to the files that those trees represent, and then returns an output tree. So okay, where do we go from there? As it turns out, the plugin API has changed quite a bit since then. So you used to have to define this read function when implementing a plugin. That got deprecated in favor of a method called rebuild, which was also deprecated later on. And then it was decided that all plugins would have to extend from this base plugin class. And then they would have to implement this build function instead. On top of that, new terminology was introduced. And what was known as a tree has been renamed to a node. Even though in embassy li, you will still see references to trees everywhere for legacy reasons. So yeah, for those who feel this way, I will try to demystify everything and for the rest of this presentation, I will only focus on the new plugin API. And I'll be using the term node instead of tree for the most part. But whether you say tree and IC node, it's really the same thing. It's just that some people got confused with what a tree is. Whereas the concept of a node can really help us visualize things better. Because the entire build pipeline can be represented as a directed graph. In broccoli, a node falls into one of two categories. You have a source node, and then you have a transform node. A source node represents a directory of source files. And you can configure it to be either watched or unwatched. So if it's watched, it just means that changes to the directory would automatically trigger a rebuild. Programmatically, a source node can be created by first requiring this broccoli source module, which gives us access to these two classes, watchdir and unwatcheddir. So in order to create a source node that represents the app directory, you would simply call new watchdir and specify the path to the directory itself. On the other hand, a transform node represents a directory of transformed files. It's an instance of any class that extends from this broccoli plug-in class. And so the constructor for the plug-in looks like this. It takes one or more input nodes, possibly some options. And it defines a build function that all plug-in subclasses have to implement. So this is really where you would want to put most of the logic for a plug-in. This is where you would take the files that are represented by the input nodes, manipulate them as you wish, and then write the results of the transformation to the plug-in's output path. So let's say that we have a very simple app directory, which contains code that has been written in ES2015 syntax. And the goal here is to transpile it to ES5. So we can define this very simple build in terms of broccoli nodes. In this case, our source node is the app directory. And our transform node is an instance of a plug-in that will do the transpilation for us. Now I'm not going to actually cover how to write a plug-in, but there is already a large number of plug-ins that have been written. And the one that we can actually use to do the transpilation work is called Broccoli Babel Transpiler. So we just need to create an instance of that, pass it the node that has the code that needs to be transpiled. And what will happen at build time is that the plug-in will read from that input node and it will transpile every JavaScript file that it finds into ES5. Of course, this is a very, very simple graph. It only has two nodes. But the beauty of this is that plug-ins can be chained. So this is what we started out with. Maybe after the transpilation step, we might want to take the results and concatenate all the JavaScript files down into one. That way we can minimize the number of network requests that are necessary to load the application. So to do that, we just need to add a concatenation step to the build pipeline. Again, there's a plug-in that can do this for us called Broccoli Concat. So we just have to plug this in and then its input would be the Babel node before it. And to complicate things only slightly, we can introduce a public directory which contains assets that don't need to be transformed in any way, but they still need to be copied into the final output directory so that it can live alongside the concatenated JavaScript file. So for this, we can use a special plug-in called Broccoli Merged Trees in order to merge the two nodes into one. So I'm going to refer to this last node as the output node. It's essentially the last step of the build definition. And its output is exactly what we want to find at the end of the build. And that's it. That's how you define a Broccoli build. Of course, nothing is in motion yet. You won't actually see any files being written to disk until something actually triggers the build. So in this next section, we're going to see how a Broccoli build runs inside of embassy li. This is the embassy li build file, except that I took everything out that embassy li normally puts in by default. We're going to start from a blank slate and we're going to start by defining our build. So we can bring in the modules that we've seen so far for creating source nodes and plug-ins. Here, we define our two source nodes, app and public, and then the transform nodes, which are really just instances of the Babel transpiler, Broccoli Concat, Broccoli Merged Trees. Once that's done, we simply have the function return that last node. Now, when you run number build or number serve, embassy li will load up the build file. It will execute the function that was exported from that file and that returns the output node. And it's going to pass that output node to Broccoli's builder. A Broccoli's builder is really what orchestrates the entire build. Embassy li will kick things off by simply calling the builder's build function. When that happens, Broccoli will recursively traverse the entire build graph, starting with the output node. It's going to walk down the graph and find the input nodes of that node, then the input nodes of those input nodes, and so on and so forth, until all the nodes have been hit. At that point, the main function of each node will execute. So for source node, Broccoli will invoke a read function in order to read the files from the directory. For a transform node, Broccoli will call the plugins build function instead. And so that's going to happen all the way until we hit the end. At that point, the build is considered complete, as far as Broccoli is concerned. Embassy li then takes over and takes those results of this last step and copies them to the final disk directory. All right, so I want to take the moment to show a quick demo of how this would basically happen inside of embassy li using embassy serve. So here I've got a very simple embassy li generated app, except that I took everything out from the default embassy li build file, and I replaced it with the sample code that we've seen so far. If we now run embassy serve, yep, went too fast. If we now run embassy serve, embassy li will read the build file, invoke Broccoli, and in no time the build completes. And so you can see here that at the end, embassy li even reports which nodes took a particularly longer time to execute relative to the total build time. So this is really helpful if you want to figure out which plugins might be holding things up. If we return to the project view and refresh the tree, you'll notice that we now have two new directories. The first one is dist, which is essentially the content of the app directory, transformed to a single file, as well as the content of public. The second folder that was created is temp. Now this is where intermediate build products of each plugin get cached. And you'll only see this while embassy serve is running. Everything that is inside of the temp directory will get cleaned up as soon as the process terminates. But if you want to check out these subdirectories, you'll notice that they're named after the plugins that were involved in the build. So the Babel plugin, that's what we started out with, it took the app node as input. And so you can see here that the code is in ES2015 syntax. This is what you would normally write your Ember app with. If you now examine the output, the same files are still there, only now the code has been transpiled to ES5. And the results of that Babel step are then used as input for the next plugin down the line, which was the Concat plugin. And it uses them to generate a single JavaScript file along with the associated source map. So same goes with the broccoli merge plugins. Again, it took the results of the Concat step plus the content of the public node and merged everything together. So I also want to point out, since we're seeing it here, that broccoli uses a lot of sim linking for these intermediate steps, because sim linking is actually faster than actual file copying. At any rate, the content of this node should match the content of the disk directory, because Ember CLI copied them there after the build completed. And that, ladies and gentlemen, is an Ember CLI built with all the bells and whistles. So if you're curious, you can run Ember build with the broccoli viz flag set to true. And what that will do is it'll generate a dot file that contains valuable information about the actual node graph. And you can convert that into an image to get a nice visualization of the build. So this is the viz that was generated from the very simple example that we've been seeing. It's a bit hard to see on screen, but if I highlight and annotate it, you can kind of match it up with a diagram that I showed you earlier, except that it's flipped vertically. If you think this is neat, wait until you see what the graph for an actual Ember CLI app looks like unmodified. And here's a sneak peek. So right now, I am focusing in on the output node that gets exported out of the Ember CLI build file. What I'm going to do is zoom out. And I know it's going to be really hard to see, but I just want to give you a sense of the magnitude of this build definition. And this is still not everything. The entire build graph looks more like this. So there's obviously a lot going on here, lots of transformations, lots of plugins. Imagine what your life would be if you had to define all of this in your build file. But luckily, Ember CLI already does this for you. So you just have to know that all the magic happens in this Ember app class, which means that the only way for you to really modify the build definition is by using the hoax in API that Ember CLI exposes. So that brings us to the next and final section, which is on how to extend the broccoli build using Ember CLI. So the first kind of customization that you can do is to add assets to the app. In a typical project, you have a lot of source directories. And these can be represented by all the source nodes that you have on the right. Ember CLI internally maintains this list of nodes. And the most common form of customization that you can do is to bring in additional vendor files using app.import. App here is the Ember app instance that comes from the Ember CLI build file. So basically, if you install a moment from Bower, you still have to tell Ember CLI to include moment.js as part of the build, because even though the Bower components folder is already defined as a source node, it doesn't mean that all the content in that folder will be part of the build output, for obvious reasons. So we do have to tell Ember CLI which specific files to include. And there are two places where we can call app.import. The first one is directly inside of the Ember CLI build file after you create an Ember app instance. The second place is from within an add-on in the included hook. This hook is called when the Ember app instance is constructed, which is also when a project's add-ons are discovered and initialized. So this is helpful if you're writing an add-on that needs to bring in additional vendor dependencies to the app. But we can do more than that. Let's take a look at these three, four hooks. So each source node in the project has a corresponding three, four add-on hook. This means that if you're writing an add-on that needs to augment the files in apps, styles, templates, et cetera, with additional assets, you can have the add-on implement the three, four hook of the same type. For instance, if the add-on contains a folder named custom, you can implement a three, four app function that returns a source node for that custom directory. And that will be merged with the app's source node from the consuming app. But this example isn't that silly because so long as you follow conventions and name the folder app instead of custom, it's going to be merged automatically. So you don't even need to implement the hook at all. This is true for all the other nodes that you have there. One thing to note is that if the add-on contains a file called food.js and the consuming app also contains a file of the same name, the one in the consuming app will always win. And MCLI will actually ensure of that when it does the merging because it's a guarantee that the project will have a way to override add-on files if needed. There's another hook that I have left out so far because it doesn't have a corresponding source node in the consuming app. And that's the three, four add-on hook. Anything that is returned from this hook or anything that you define in the add-on folder of the add-on itself will end up being pulled into the vendor files. In fact, if you were to browse the source code for Ember Data on GitHub, you will find that now that the repo or the add-on Ember Data has been converted down to an add-on, you'll find that most of the core code lives inside of an add-on directory. And this is how you end up with Ember Data code in your vendor.js file at the end of the build. OK, so I've shown you three, four hooks only to tell you that you don't need to use them. But they are actually quite useful. This is the Ember Metrics add-on that I mentioned at the beginning of the talk. Ember Metrics has built-in adapters for various analytics services. But if your users are only going to be using one or two of those services, then why bother loading the vendor files with all these adapters that they're not going to use? If we didn't do anything here, all the content from the add-on folder would end up showing up in the vendor files. What we want to do instead is intercept this node and filter it out before things get written to the vendor file. And this kind of filtering is exactly what the Broccoli funnel plugin allows you to do based on some exclusion criteria. So I'm showing you a simplified version of what Ember Metrics does. But essentially, we can customize the build by implementing this three, four add-on hook. The argument that is passed to this hook is the default add-on source node. In other words, it's everything that is in the add-on folder of Ember Metrics. So to filter the adapters, we just have to pass this node to an instance of the Broccoli funnel plugin, specify the exclusion logic that we want, and then return that instead. So this wasn't too complicated, but sometimes bugs can creep up. Or maybe we just want to verify that a plugin outputs what it is supposed to output. And once upon a time, when I didn't know any better, I would say, oh, I wonder what's in the street. And we just can't to log in. And I would get several screen pulls of this, which isn't particularly useful. But thankfully, there is a better solution. There's this Broccoli Stu module out there that's very useful. It has lots of handy utility functions that we can use, including one for debugging. So going back to the Ember Metrics example, all we need to do to verify that the filtering works as expected is to take that final instance, pass it to the debug function, give it a name, and return that instead. What's going to happen at build time is that the debug function will then create a folder inside the project, after the name that was given to it, prefixed by debug. And it's going to copy the output of the node to that folder. So we can not only see that the correct files are there, but we can inspect the content as well. And this is a huge lifesaver. So OK, we've seen how to add assets to the app. But what about modifying existing assets? For this, we do have these pre- and post-process tree hooks. Basically, at various steps along the build pipeline, a project's assets will be passed through a number of preprocessors. And these preprocessors are really nothing more than Broccoli plugins. So for example, template preprocessors are responsible for compelling handlebars to JavaScript. So sometime after the results of the 3.4 templates hook are merged with the templates from the consuming app, the combined template node will be passed through any template preprocessors that are registered with Ember CLI. And you can, of course, register more preprocessors if you'd like, but that's a topic for another time. What's important here is that Ember CLI will call preprocess tree prior to running the preprocessors. And it's going to call the post-process tree hook afterwards. And the first argument to these hooks is a string that indicates which type of node you are dealing with. So we have the opportunity here to intercept the templates before and after they get pre-compiled in order to transform or augment them as needed. Sim applies to CSS preprocessors, except that instead of templates, you are now dealing with style nodes. And these preprocessors are those that would convert SAS to CSS and such. For JavaScript preprocessors, these are the ones that would basically transpire your code from ES 2015 to ES5 or even compile JavaScript to JavaScript. But this happens in two phases. These preprocessors will run first with the application code, so everything in app. And then a second time with the test code. So to recap, here are the hooks that are related to preprocessors. It's the same hooks, really. You just have to inspect the value of that first argument to figure out what you're dealing with. At the very end of the build, Embers.ly will call post-process tree one more time, but the same with a type of all. And it's going to pass the node that corresponds to the very last node of the build definition of the build so that you have access to everything there. If you have any last broccoli transformations to perform before the build completes, this would be your chance. To illustrate this, the broccoli asset rev add-on that handles fingerprinting uses this hook to open checksums to the file names at the end of a production build. So there are lots of possibilities here, as always. And there are also way more hooks than I could possibly cover in this talk. The ones that I showed you dealt directly with modifying a broccoli build definition. There are others that do not actually alter the graph, but they do still participate in the lifecycle of a build. And those are pre-build, post-build, output-ready, and build error. Those are simple enough to understand because they don't actually interact with broccoli, so I encourage you to go and explore those on your own. What I do hope that you have been able to take away from this talk is a basic understanding of broccoli build, how it fits inside of Ember CLI, and how you can manipulate it further. And so that when you return to the add-on hooks documentation in the Ember CLI repo, you can finally start making a sense of things. And you can hopefully go from this to this. So thank you so much for attending this talk. If you have any questions, feel free to find me at the end of the conference. Just throughout, I'll be around. You can also tweet me at EWIS or find me on the Ember community Slack. And I hope you enjoy the rest of the conference. Thank you so much. Thank you. Thank you.