 Anyways, hi, I'm Contra, and let's talk about some build systems. So this is me. You can follow me at any of these links. My name is Contra. I co-founded a company called Fractal, and we are based out of Phoenix. So at Fractal we spend a lot of time on open source, and we currently have like 250 projects on GitHub to show for that policy, so we're really invested in it. We believe that open source is not only good for the community, but also good for business. So we try to stay true to that message. Before we get really deep, I'd like to bridge the gap between people who are and aren't familiar with build systems. Basically, build systems are tools that help automate your workflow and let you focus on what really matters, because at the end of the day we just want to write code. We don't want to be stuck messing around with these tedious, highly fragmented tools just to get our project working. So compiling, minifying, and bundling any other word that ends in ing, they're all within the domain of a build system. Build tools are a crucial component of a developer's workflow. I know that I wouldn't be the same developer if I didn't have build tools to back me up and help me save time. To be super blunt about it, though, the current batch of tooling is just not cutting it. They don't work well, they waste time, and they cause immense frustration. The problems we need tooling for are fundamentally different than the problems of yesteryear that these tools were created to solve. We're simply using the wrong language to communicate the job that needs to be done. So we have new problems. We need to come up with a new solution. First let's define what this solution should do. It needs to read in some files, transform them a few times, and then save them. Seems simple enough. Now when I ask people to envision that process, we usually end up with something like this. In this example, we're doing two transforms. First we pass multiple files to concat, concat outputs one new file. That sunglass touch at the end. That new file gets passed to a minifier, which then outputs the modified file. And then when all that stuff is done, it gets saved to the file system, those little folder icons. We essentially drop the input in at the top, let it fall through the transformations, and then save the result. This idea probably sounds familiar. It's kind of like basic functional programming. Let's compare that to our current day tools. The primary abstraction in these systems is a task, which describes the input, transformation, and the output. This means each transformation is logically separated from all of the other ones, and your file system becomes mutable state in between them. This mutable state is used to communicate information between these tasks, and basically it slows everything down by radically increasing the amount of IO that's getting completed. It's slow, it's confusing, and it creates a hell constructed entirely of temp folders. So these are some different tools that follow that model. The IO costs that come with the task-based system increase not only with the number of files, but also with the number of transforms. This is super frustrating when you use something like live reload, and your reloads aren't exactly live anymore. You end up waiting like 30 seconds for some results to show up on a page, and it becomes kind of ridiculous. These flaws are a symptom that we're just using the wrong abstraction here. The dialect of a build system should be molded around the way that our brains envision the problem. We still want tasks to logically separate sets of operations, but a lower level abstraction is needed to give us that granular control over transformations. It's clear that a file pipeline can help us fill this gap and solve the problem. So instead of inventing our own data pipeline system, we can reuse an existing one, streams. Streams are a dependable way to compose large systems out of small components that do one thing well. They help separate concerns by restricting the implementation surface area into a consistent interface, which can then be reused anywhere. You simply plug the output of one stream to the input of another and then let the data flow. Here's a great quote by Doug McIlroy. We should have some ways of connecting programs like garden hose, screw in another segment when it becomes necessary to massage data in another way. So in this example, we have A pipe to B, pipe to C, pipe to D. And in Unix, streams are implemented by the shell with the pipe operator. You guys are probably familiar with this. In Node, streams are implemented by a built-in stream module and you chain them by calling .pipe. Node streams have this unique benefit of JavaScript's flexible type system. It means you can have a stream of just about anything. That could be objects, strings, buffers, numbers, functions, whatever you want. And this is super cool because you can model transformations on any type of data, not just text. So we've now lifted that limitation and I think that makes streams in Node super cool. So this becomes really useful when we start thinking about what exactly does a file consist of? The file objects we pass around are more than just text these days. We have metadata like file attributes, which describe permissions, modification times and other junk, source maps, which are used to keep track of changes and keep your code debuggable, ASTs, which prevent CPU waste from reparsing the same files over and over. Like if you uglify something and then you want to go remove console.logs, each tool is going to parse that AST twice. So you're just wasting CPU there. Whereas if you have this file pipeline, they can reuse that same AST and there's no need to reparse it. So we can finally decouple these individual transforms by passing metadata around in these streams. So after being completely unsatisfied with the current batch of build tools, I decided to create Gulp. Gulp has existed for less than a year and it's already one of the most popular projects on GitHub. We currently have somewhere around 800 plugins. So anything you want to do, you can pretty much do it already. I'm also pleased to announce that just yesterday we passed Grunt and GitHub stars. We have somewhere like 8,300 now. So we're now like the most popular build system on GitHub. Super cool. I'm really excited about that. The next version of Grunt is actually being rewritten to use Gulp under the hood. So that feels pretty good. We won there. So the main goal of this project is to provide a common interface around these highly modular pipeline transforms and have a friendly API to tie them all together. The common interface is simply a stream that takes a defined file object and possibly outputs a file object or maybe more file objects. It's completely up to the transform. As the benevolent dictator for life of this project, I make sure that things stick to this. All development of Gulp follows this ideology and we try to keep the API as simple and as node-like as possible to stay true to our catchphrase, which is, if you know node, you know Gulp. We also like to squeeze performance wherever possible. Globbing is lazy, tasks run in parallel, files are processed in parallel, and IO is the bare minimum needed to get the job done. So people have been benchmarking Gulp against other build systems for a really long time and we're always at the top of the list. I have never seen anything beat us in a benchmark. So we're like extremely proud to be able to say that Gulp is the fastest build system out there and those milliseconds really matter. I see tweets like this come in all day and let me tell you there's no better feeling than having people use your open source project and say stuff like this. I actually just added this slide like 30 seconds before I came up because somebody just tweeted this. So yeah, it feels good. I'm sure there's better feelings than this, but this is pretty cool too. Personally, this is why I like to do it. All the love feels really good, I have to say. So let's cover a few of the APIs you'll need to know to get going. First off, we have Gulp.task, which registers an asynchronous function with Gulp. After registering, you can call it from the CLI. In this example, if you wanted to call this, you would type Gulp do-stuff in the CLI and it would run your little function. Just to be clear, when I say asynchronous function, I mean we support a variety of different control float techniques, that being callbacks, you can return streams, promises. I think we just added support for like RxJS, you can use generators, anything you want to use, we're completely agnostic to it. So we don't try to have really a preference there. Next up, we have Gulp.src, and this is our input to the pipeline that we talked about. This is what actually reads the files from the disk. It takes in a glob and starts spitting out the files lazily as they're found. So this means that we don't do batching as stuff is picked up from the file system. It goes through your pipeline and then all the way out to your destination. When you're using library load, you'll start noticing changes immediately. Let's say you have 100 CSS files, you don't wait for all 100 for them to be processed. You start seeing changes immediately as they go through the pipeline. Finally, we have Gulp.dust, which is the output of the pipeline. When files are passed to this, it simply just writes them to a folder. What's cool about this is after the file is written, it gets re-emitted so you can continue the pipeline and add more transforms after that or maybe more outputs. So it's not necessarily one source, one output. It's one source or many sources to as many outputs or transforms as you want. So to put that all together, we come up with this sample task. Basically what this does is read in some JavaScript files, concatenate them into all.js, writes them to a build folder, then passes the file after it's been written to Uglify, renames it, adds .min to it, and then writes it to that same build folder. So at the end of this, we end up with two files, all.js and all.min.js. And what we actually did here is decrease the IO overhead two times because we're only reading in the files once, whereas in something like grunt, you would read them in twice to construct the unminified and then the minified. So my favorite thing about this is with the pipeline, you can immediately look at this and see the entire process that happens to a set of files. When you look at a grunt file, it's not readily apparent what sets of files you're actually dealing with. Every task is at the same level, so you kind of need to go through each one and see, okay, this writes to this folder, this reads from that folder, okay, what reads from this folder. And you have to construct the pipeline yourself by looking through and grokking that big JSON blob, whereas here, it's just right there. Concat, build, uglify, rename, build, done. So let's compare some real code here. We're gonna compare a task runner and a streaming build system with like a little demo project. In this file, we're gonna lint, concat, minify, and then run again if any of those files change. So this first file is a grunt file. Let me see if I can scroll this. Sorry, this is a big grunt file. So you can see it's pretty long. Like I said, everything is flat. We've got concat, uglify, jshint, watch. And then we have like this funky load npm tasks thing that loads in the modules and wires them up into the task system. And then we create a task called default and say do all these things when somebody runs the grunt command. So a couple of things to note here. Every task specifies input and output on the file system. So we see right here concat, source, give it a glob, test, give it an output JavaScript file. And then we see that this does the exact same thing. It reads what concat put out and then this is the output file there. Jshint, no output file here, but it also specifies those inputs. So what I dislike the most about grunt is that your build file is this funky JSON language that defines task and task options. Like I look at this and I just don't think that it looks good. To me, I am a programmer. I like to write code, not configuration. One other thing to note, each task is executed synchronously. So as you add more tasks, your build time is just gonna increase linearly, that's not good. So declarative configuration for a build seems completely wrong. I think trying to shoehorn this into a declarative language leads to a massive over configuration like what we just saw. The grunt file we looked at before was like the smallest grunt file I've ever seen. It was just the bare essentials to do a basic operation. When you start to add more stuff, it becomes super complex and you hit the wall of configuration where you end up with something like this. So basically what happens is you have two choices. You can start writing code to generate your configuration which defeats the whole purpose of having configuration or you end up with monstrosities like this where the config language tries to support every possible use case. Why? That's all I have to say about that. So to me, JavaScript is already this beautiful and expressive language. We don't need to shackle it with all this custom crap on top. I would rather be writing JavaScript than writing whatever this is. And just for reference, this is from Ant. If anybody's worked with Ant before, this specific example was from a Drupal build file. So this is the same thing as our grunt file that we covered earlier. No need to scroll here. It's only like ten lines of code. So you can see with Gulp, we use code instead of configuration. Couple things to note. Concat and Minify are in the same pipeline so the data's only read once. This is plain JavaScript. There's no custom language to learn so the learning curve is minimal. Like I said earlier, if you know Node, you know Gulp. The plugins are super simple and take few to no config options. We enforce this heavily actually. We have like this super strict set of guidelines that we tell plugin authors that they have to follow. And this makes sure that we keep high quality modular plugins and that we don't have people writing the same plugins over and over just because they didn't want to send a pull request. But we have like this blacklist file with like about 80 plugins in it that didn't follow these guidelines and we'll go open an issue there and say, hey, you need to follow these guidelines. Here's the exact thing that you broke. Like whatever, we're gonna add you the blacklist in a week if you don't fix it. And if they don't fix it, then they don't show up on our plugin page anymore. So I think that barrier to entry really kind of keeps the ecosystem clean. So my favorite thing is files and tasks are processed asynchronously. So more files and more tasks doesn't equal longer builds. You can add as many tasks as you want and it's not gonna increase the build time. Having no mutable state is really the secret sauce to solving all of these performance bottlenecks that afflict the task based systems. So I've seen many real world examples where Grunt was taking 30 seconds to build and a Gulp file can do it in 300 milliseconds. So that's like orders of magnitude faster. So removing mutable state to allow asynchronous execution is super, super important not only for a build system but also for applications. So what are some takeaways you can get out of this? Configuration sucks the life out of JavaScript. Streams are awesome and pipelines are pretty dope. Hopefully you've at least learned about the merits of streams. And I have a good document here. If you wanna learn more about streams, you can check out this stream handbook by Substack. I think that you'll be a better programmer just by learning about streams even if you never actually use them. You can never learn enough control flow techniques in my opinion. So for this talk, I put up a special link you can use to get shirts. We usually completely sell out in about ten minutes. So get one while they're still available. I think we're already half sold out somehow this got leaked. Yeah, they sell out really fast. They're hot shirts. I forgot to pack one, so I chose the closest thing that I have. Just whatever red shirt I brought with me. So I'm pretty bummed about that. But if you wanna get a shirt, there's the URL right there. Give you guys a second to take a picture of that. And that was my rant. I left a lot of time for Q&A if anybody has any questions. Hi, I'm a big fan of Gulp and I've been using it for most of my new projects now. And I love it. One pain point that I kind of keep running into is I use Browserify with Gulp. And I know the Gulp Browserify is on a blacklist. And there is a recipe on the Gulp document for how to do Browserify. But it kind of sucks because then you're forced to use Browserify and watchify to watch a change in cash and all that stuff. And then, so in my Gulp fight, there's all these beautiful Gulp watch tasks that are very standard for Gulp. And then you have this blob of Browserify config stuff that's really ugly. And now I am left to depend on watchify for watching my files. And sometimes that doesn't really work well. And so there's a lot of instances where my files are not being watched. And the changes are not being detected. So is there any talks or plans to make this better? Or is it just something that Browserify and watchify have to do? Yes, there are plans to make that better. So currently, the whole problem there is right when Gulp was released, we had somebody come out and make the worst plugin I've ever seen. It just didn't work. It was super buggy. So that was why that ended up on the blacklist. It's not because it's a bad idea or anything, just because the module that somebody published was, I mean, not to be mean or anything, but it wasn't any good. And that kind of goes back to the quality thing that I was talking about earlier. So while you can still use this module, we don't officially endorse it. If somebody opens an issue about it, we'll just tell them, no. We don't have anything to do with this. Don't use this. So we have this really long issue thread going regarding Browserify. And I detailed this. I don't have time to write it, but I left a blueprint basically if somebody wanted to write a good Gulp Browserify plugin, how they would go about doing that. So really, I'm just waiting for somebody who has the time to come in and take that blueprint and actually make the thing and publish it. And then we can get rid of that recipe, which is just a stop gap until somebody does that. So I am excited for when somebody actually publishes a good Browserify module, because we'll be able to tell people to use that. So that's coming soon, I think, though. I think a couple of people have kind of raised their hands as wanting to do it in that GitHub issue thread. So stay tuned. We'll tweet about it from the Gulp account. So are you, I have not heard what you think, that whether you guys are preferring Browserify for AMD or do you, I notice your top plugin here is AMD Optimize with a whip on it. So what are you recommending or seeing with RequireJS builds? So just to be clear, that isn't a top plugin. That list is unsorted. I think it's sorted alphabetically or something. So that's why AMD Optimize would be at the top. So that's in no way like an endorsement from us that it's our best plugin. I use Browserify personally, but I know that one of the pain points with Gulp is dealing with these third-party tools like RequireJS that want to handle their own I.O. So essentially, what we tell people to do is, OK, we can't really make a plugin to make this kind of fit our paradigm. So go and just use the RequireJS module directly. But some people have actually stepped up to the plate, like that AMD Optimize module, where they basically rewrote the RequireJS Optimizer using the Gulp philosophy. And you can pipe files in, and then you get an Optimize file out. And it works really great from what I hear. So if you're going to do AMD, definitely give that a try. I'm not sure how stable that is, since it's relatively new. But if that doesn't work, you can always just use the RequireJS module. Another feature that I'm not sure if you have plans for is that listing of tasks that are defined by Gulp. I think that when I'm writing the build tools for someone else to consume, instead of having them to open the Gulp file and look at other tasks and figure out what tasks do, having a way to, on from the command lines, do Gulp help for something, lists a task. I think Grunt had this feature, and I think that was one of the more useful ones, especially when you're interfacing with a new application. So I don't know if Gulp has that feature planned. Gulp 4. Gulp 4, it's coming. So we completely rewrote the whole task system for Gulp 4. We went away from what we have currently where you specify a function, and then you can specify an array of dependencies that get executed before that task. So if I say, OK, before JavaScript happens, and you have this clean task that needs to run that wipes the folder of any JavaScript files that were outputted in the last run, you would specify your JavaScript task and say, clean is a dependency. And then it would run it beforehand. But we kind of are going away from that model. That's kind of like an ant-like model, or more like Maven, I guess. So what we're doing instead is we have this new composable task system that we're using called Boc. And what that lets you do is just go back to using plain functions without dependency trees or graphs or whatever we have going on right now. Because it's overly complex, in my opinion. So what that lets you do is you'll be able to tack on description attributes to these functions that you register into the task system. And our command line tool we'll have in dash dash help, or I think it's dash dash tasks right now. The ability to, when you list out the whole tree of tasks, it'll show you the descriptions next to it. And any other meta information that you tack onto it. So it makes it completely flexible in that respect. Can I set that out? Gulp4, we said it was going to come out in May, and then it didn't come out in May soon. If it doesn't come out by 2015, I'll punch myself for that. Because it'll be out soon. It's really close. We were waiting on the source maps thing to be done. We have Gulp source maps, which is the best source maps out of any build tool. We were waiting for that to be done, which it's super stable now. Every plugin has it, basically. You can do 10 CSS things, and you have source maps the whole way through, which is insane. No other build system has ever done that before. So we were waiting for that to be done before we move forward. So we're really close. I'd say a month away if I had to give you a hard estimate. Yes? So can you talk a little bit more about how dependency management will work in Gulp4? Because I'm curious about that. I use, for example, cleaning tasks quite a bit. So how would you figure that out in this new system? Yeah, so one of the huge pain points right now with the task dependency system is that, for example, the clean task, like maybe you want to run the clean task before every single task that you have in your build file. Well, now you have to go and add that before every single task. Or you have to make these weird pseudo tasks that run it beforehand. And basically what people have started doing is writing modules that just let you run stuff in series because we run everything in parallel. So there's a module called Run Sequence that solves that. And it lets you specify nested arrays of how things should be executed. So what we did was take that idea and apply it in a more functional respect. So you can type, like, Gulp.series and then pass it Gulp.parallel with some functions in it and another Gulp.parallel with some functions in it. And it'll execute those two parallel sets in series. And you can nest those infinitely. So you can do some really cool stuff by combining those. I think not only is that cool for our task system, but just for asynchronous, control flow management in general, it's super cool. If you want a preview of that, github.com slash faded p-h-a-t-e-d slash bock b-a-c-h, like the composer. And that has some docs describing how the whole new task system works. Also there's a Gulp4 branch, if you're interested in that, on GitHub 4.0. Two quick questions. You mentioned that you contributed a lot of projects. I think you said, like, a few hundreds or something. What other interesting things have you guys worked on? So currently I'm working on an open source genetics analysis library called Genome.js. That's genomejs.com, if anybody's interested. And that's been super fun. That's just kind of a passion project of mine. I'm also working on a lot of WebRTC stuff for doing peer-to-peer video calls and encrypted data transfers and stuff like that. So that's two of the things I've mainly been invested in. We're also doing some work with doing smaller React components. So the React component ecosystem is super new. So it's kind of like this awesome Wild West where you come up with an idea for something cool. It's like Node when it first started. You come up with an idea of something you need. You make a little React module for it. And then, oh, no one's ever done this before. I'll publish this. And now other people can use it. So we are starting to come out with a ton of stuff like that. I actually have this whole framework around WebRTC and React where you can just drop a call in at any point in the page or drop in video streams that are coming from someone else's webcam across the world. So React components, WebRTC, and genetics analysis stuff are kind of my main focuses right now. Then a second quick question. What was the most insane non-build system related use of Gulp that you've seen? Netflix has a pretty crazy Gulp file that they do a lot of non-build related stuff with. I know a lot of people use it for deployment. It's become pretty big for deployment, which was surprising because I didn't even think about that when I was making it. But it actually kind of makes sense. Like you're just streaming files to servers and setting stuff up. So I'm curious to explore that a little bit more and maybe flesh out how to possibly use Gulp as a deployment system because I think that's just super interesting. So yeah, I think deployment and just weird odds and ends stuff are the weirdest things I've seen people do with it. Sorry. So there's one thing I've been trying to figure out how to do better in Gulp that's been driving me nuts. And I've had people point me to a few different plugins, but none have really kind of got at this the way I want. So in my source.js file, I have some files that I want to remain as is. And then in some subfolders, I have a bunch of other files that I want concatenated together. So I might have an all directory or a feature to text directory with a bunch of different files. And currently, I'll have a concatenation task that'll drop all those subfolder things into a temporary file and pull them in later. But is there a better way to in one task? And this may be two in the weeds and we can talk later if it is. But is there a simple way to identify if it's in a sub directory, concatenate it. But if not, just pass it through the stream and don't. So I would recommend just writing some custom logic for that. If there's not a plugin already, just write a plugin. I think one of the greatest things about Gulp is that you can write a plugin and it ends up being like 10 lines of code or something. And it's super simple. Because we have that strict specification, you're a stream, you take in a file, you possibly output a file. That's the whole thing. So I think that if you need custom logic like that, you can just write a little plugin that does it, publish it. Maybe somebody else will get some use from it. If you don't want to publish it, there's GulpTap, which is a module that just kind of lets you tap into a stream of files and do arbitrary logic in there. So you could possibly use that to kind of encapsulate that stuff that you want to do. Yeah, I'm actually playing around with that right now. If you have time after just like five minutes, I think I'm not wrapping my head around tap properly. So cool. Thanks, man. Cool.