 So, this talk is about how Bundler works. How does Bundler work? This is an interesting question. We'll talk about it for a while. So this talk is kind of a brief, hopefully brief, history of dependency management in Ruby and kind of a discussion of how libraries and shared code works. Both how it's worked in the past and how it works now because how it works now is kind of directly a result of how it used to work in the past and trying to fix problems that happened back then. So before we get started, let me introduce myself. My name is Andre Arco. I am indirect on pretty much all the internet things. That's my avatar. Maybe you have seen me on a web page somewhere. As my day job, I work at Cloud City Development, doing Ruby and Rails and web and Ember Consulting. We do web and mobile development and I mostly do architectural consulting and senior developer pairing and training. If that's something that your company's into, talk to me later. The other company that I founded is called Ruby Together and it's a nonprofit. It's kind of like NPM Incorporated but without the venture capital. Ruby Together is a trade association that takes money from companies and people who use Ruby and who use Bundler and Ruby Gems and all of the public infrastructure that everyone who uses Ruby uses and pays for developers to work on that stuff so that rubygems.org stays up. And so that people keep being able to have gems, which is pretty cool. As part of my work for Ruby Together, I work as lead of the Bundler team. I've been working on Bundler since before 1.0 came out and I've been team lead for the last four years. So, using Ruby code written by other developers. Nowadays, this is actually really easy. You add a line to your gem file, gem foo. You go to your terminal and you run bundle install. You start using it, that was actually it. Pretty cool, that's really easy. The thing that I've noticed talking to people who use Bundler and think that it's awesome is that it's not actually clear what just happened. Based on the text printed out by Bundler install, it seems like something probably got downloaded and something probably got installed. But it's not totally clear what got downloaded, it's not clear what got installed, it's not clear where any of that happened. And it's not clear what exactly happened there. Nobody's really sure, how does just putting a line in your gem file mean that you can start using somebody else's code? So, to explain that, we'll need a little bit of history. We're gonna go back in time and I'm gonna give you a little tour from the beginning of sharing code in Ruby up until now and hopefully by the end of it, you'll understand why things work the way they do now. So, I'm gonna start talking about Require which came with the very first version of Ruby ever in 1994 and then talk about SetupRB which came out in 2000 and then RubyGems which came out in 2003 and then Bundler which came out in 2009. And that's what we're still using today. So, Require, the Require method has been around since 1994 with the very, very first version of Ruby that actually came out. And or I guess I should say, I'm sure that Require has been around since at least 1997 because that's the oldest version of Ruby that we have in version control still. It was probably there before that though. But Require can be broken down into even smaller concepts. So, using code from a file is basically the same as just inserting that code and having Ruby run it like you had just written it in the file. So, it's actually possible to implement Require yourself with just a one line function. You say, hey, I have a file name and I want to require it. And you read the file into memory into a string and then you pass the string to eval and Ruby runs it and it's just like you typed that code yourself. And it got run. So, there's some problems with this. This is not how Require works in real life. I'm sure it's totally fine that this will run that same piece of code over and over and over if you require it over and over and over. You like having lots and lots of constants that keep getting redefined. I'm sure it's totally fine. So, working around that is actually also pretty straightforward. You can just keep track of what you've required in an array and not require something again if it's already been required. This, as you can see here, you set up an array. You check to see if the array already contains the file name that just got passed in and then if it hasn't already been required, do the same thing we were doing before, read the file in, pass it to eval. And then add it to the array so that you won't require it again later. In fact, this is exactly what Ruby does, albeit written in C and not in Ruby. There is a loaded features global variable, and it's an array. And it contains a list of all of the things that you've required in the past. So if you ever wanted to know, if you've required something yet, you can actually just check the loaded features array. So there's one more problem with this, which is that right now it only works if you pass it absolute paths. I'm sure you don't mind typing the full path from wherever you are to exactly where the file that you want to require is, I'm sure that's fine too. So the easiest way to allow requires that aren't absolute is to just treat all requires as if they're relative to the path where you started the Ruby program. And that's easy, but that doesn't help a lot if you want to require Ruby files from different places. So say you have a folder full of this library you wrote and a folder full of this application you wrote and you want to use the library from the application. You can't, like, writing relative paths from wherever you started the Ruby program would be terrible. So instead, we can create an array that holds the list of paths that we want to load Ruby files from. In a burst of creativity, I'm just gonna call that variable the load path. And here's an implementation of a load path. If you put something in the load path array, you can then pass a relative path to any directory that's in the load path array and will look for the file. Hey, is there a file? So if you require foo, we'll say, hey, is there a file named foo inside any of the load path directories? And the first one that we find, searching the load path in order from first to last, will require that one. Coincidentally, this is exactly what Ruby does. There is a global variable named load path. And if you put a string that contains a path to a directory in it, Ruby, whenever you require something, will look in that directory to see if there's a file with that name. So you can totally use the load path to require files from somewhere else while you're working with them. And of course, the load path and loaded features can both be combined, but that code didn't fit on a single slide. So I'll leave that as an exercise to the listener. It's pretty straightforward, to be honest. So load paths are pretty cool. They allow us to load Ruby directories, even if they're spread across multiple places. At this point, we could even have, like automatically at the start of every script, we could add the directory that holds the standard library to the load path. And then all of the files that are part of the Ruby standard library, like net HTTP, all of the set, all of those cool things that come with Ruby could just be available for require automatically, and you wouldn't have to worry about putting them in the load path yourself. That is exactly what Ruby does. The standard library starts on the load path whenever you start Ruby. It's pretty great. So this was cool, and for several years, this was enough. People just added things to the load path a lot, or wrote scripts that added things to the load path before requiring things before their actual script happened. The thing that got super tedious about just having load paths is that if you want to get code from someone else, you have to find that code, download that code, put it in somewhere, remember where that somewhere is, put that somewhere in the load path, and then require it. It was pretty tedious. Sorry. So the next thing that happened was set up RB. So we're totally caught up to the state of the art in Ruby libraries around the year 2000. Everyone's still installing shared Ruby code by hand, CP, CP. And that wasn't so much fun. So a Japanese Ruby developer named Minoru Aoki wrote set up RB. And amazingly, even though this was created in the year 2000, set up RB is still around on the internet. The website for this developer is i.loveruby.net, which is pretty cool. And you can even download set up RB, although to be perfectly honest, it hasn't been updated since 2005, so I'm not sure it's super helpful to you. So, how did set up RB work? Well, you, at its core, set up RB kind of mimicked the classic Unix installation pattern of downloading a piece of software, decompressing it, and then running configure make make install. And so Ruby kind of copied, set up RB kind of copied that for Ruby and you would run set up RB set up, set up RB config, set up RB install. And what would happen is set up RB would copy all of the Ruby files. And there was a specific directory structure, kind of like a gem today where you would have library files and bin files that you could run as programs and support files. And set up RB would copy all of those files into a directory that was already in the load path called site Ruby. And that was like the Ruby files that you had installed that were specific to your computer. And so after set up RB using Ruby libraries was actually much easier than it had been. You could find a cool library on the internet, you could download that cool library. You had to untar that cool library by hand, and then you had to run Ruby set up RB all by hand. But then hey, it was all installed, no more manual copying, no more like having to manage all these files. And everything was in the load path, you could just require it as soon as you ran set up RB, it was pretty cool. So after a little while some of the shortcomings of this scheme became apparent too, there's no versions for any of these libraries. And after you run set up RB, there's not even a way to tell what version you have unless you write it down or unless the library author was really nice and put the version into the code somehow, and there's no way to uninstall. Everything just gets thrown into the same directory. So you run set up RB for five different Ruby libraries, and now all of their files are just in one directory. Good luck figuring out which ones belong to which. Because if you delete the wrong one, too bad. And then upgrading, upgrading was super fun. If there was a new version of the library, which, good luck finding that out, right? You had to remember the website where you got it from in the first place. I hope you write all these down. I hope you've written down every website you've ever downloaded Ruby from. You have to go back to that website, and you have to remember which version you have, which, as I said before, there's nothing there unless you wrote it down. And then you have to download the tar ball with the new version, and decompress it, and CD into it, and run Ruby set up RB all. And hope that the new version didn't delete any files, because the old files are still there and you just went over the top of them. Yeah, so overall this was, this probably sounds a little tedious. It was really tedious. People frequently kind of had no idea what was actually happening with their libraries. And it was actually not uncommon for people to be like, this doesn't work, I'm just gonna fix it in my site with Ruby directory. Okay, everything's great. Right, yeah, super awesome. So at some point, some people were like, hey, this isn't actually that great. What if you could just gem install? That would be cool. And so in 2003, Ruby gems came to the rescue and kind of fixed all of the problems with setup RB that were known. You could check to see if a library existed by just running gem list. You could install a gem just by running gem install. You could uninstall a gem, super great. By running gem uninstall. And Ruby gems kept each of these libraries in different directories. So that you knew which libraries you had and you knew how to uninstall those libraries. And you knew how to install new versions of those libraries. And it was all with a single command. There was none of this like, find it on the internet somewhere, download it, unpack it, set up RB it. And Ruby gems had another super cool trick up its sleeve, which was versions. Ruby gems actually kept each version of each gem in a different place. You could install multiple versions of the same library and they could all be in your Ruby because they did not go into one giant folder. They all went into their own separate folders. So there was a folder for Rails 14, or Rails 4.1, Rails 4.2, Rails 5. This was pretty cool. So to make this actually work because required doesn't support versioning inherently, Ruby gems added a gem method that lets you say, hey, I know that I have or I don't really care whether it's installed or not. I need version 1.0 of rack and Ruby gems will check to make sure it's installed. And then put that directory, just the one with rack 1.0, into your load path. So then when you run require rack, you get rack 1.0, it was pretty cool. And so calling the gem method told Ruby gems that you wanted to manipulate the load path to load exactly the version that you knew that your code wanted to talk to, it was pretty useful. Ruby gems also has a way to support versioning even in commands that come with gems. So the rack gem comes with the rackup command. And if you have multiple versions of rack installed, the rackup command could run any of those versions. So Ruby gems defaults to the newest version that you have installed, hoping that the newest version is the right one. But if that's not the right one, Ruby gems actually checks the first argument to the command to see if there's something with underscores on either side of it. And it thinks that that will be the version number that you want. So in this example, we're running rackup from rack version 1.2.2. And only version 1.2.2, if you don't have version 1.2.2 installed, Ruby gems will be like, hey, sorry, I couldn't find that version. You need to install it first. Ruby gems was really, really successful. Ruby grew in popularity a lot, but Ruby gems made Ruby libraries and sharing code grow in popularity a lot. Present day, we have about 100,000 different gems and about a million different versions of those 100,000 gems. That is a lot of shared Ruby code, and that is super cool. So, and you probably knew this was coming. As cool as Ruby gems is, it still has some problems. If you have multiple applications that all use Ruby gems to load their dependencies, this can be problematic. It's really hard to coordinate across multiple applications, because the way Ruby gems works, every machine, or technically every individual version of Ruby. But each installation of Ruby itself just has a set of gems, right? Like you ran gem install, and now there's all these gems. And so if one developer runs gem install foo and starts using foo in their application, and then commits that code and checks it in, and the next person checks it out, and tries to run the application, it's gonna explode with like, I don't know what foo is, where is it, you need to fix that for me. And so it led to an era of basically pure manual dependency management. Starting a new job, hooray, no joke, this literally happened to me in 2008. New job, welcome to the team, here's your cool new laptop. We expect you to have the app running by next week. It actually took me, and I was totally working overtime on this. I think it only took me three and a half days. It was amazing. To figure out which gems to run gem install, I looked in the readme and there was a list, and I installed all of them. And then clearly there were some that some people had just kind of forgotten to put in the readme, and then it kind of worked. But then I wasn't able to get images working, and then some other developer was like, yeah, you have to install image magic. This was before homebrew. It was really, really terrifying. Yeah, so to try and kind of fix this problem of like, do we just put the gems in the readme? How do we even know if we have written everything in the readme? I don't know, try it. And of course you had to get a new machine to try it on because some person, after three years of using Ruby, you've just gem-installed everything and you have no idea what is important and what isn't important, and yeah, it's terrible. So people started working on tools to help this problem. Rails actually added this thing called config.gem, and this is like Rails 2.3 era, 2.2 era, where you would say, hey Rails, I need this gem, and you would put it inside your application rb file. And that was super helpful if you needed to like know for sure that this was the master list of all the gems that you needed in your application, but you could only access that list if Rails was already loaded. So if you upgrade Rails over here, it was pretty bad. So because RubyGems automatically uses the newest version of each gem, just having an older version installed didn't mean that it would be used. And if you like install some gem a month after the other person did, maybe there's a new version, you just get the new version automatically. This is also totally a real life experience that happened to me in 2009. Debug a production server that just throws exceptions sometimes for three days, the other production servers are fine. You can't reproduce this problem on a single developer laptop. Like what is even going on? This is so weird. After three days, I finally thought to look at the output from gem list for the entire production machine. And I was like, this production server has gem version 113. And every other production server and every developer laptop has gem version 114. And that was the problem. There was a bug and but that only that server had this problem. And then like I was saying about Rails versions, you could gem install Rails, be happy, make a new app, run your server. Everything's great. And then you switch to another application that already existed. Didn't get written to use that version of Rails. Got written to use some older version of Rails. And you're like, okay, let's go. Cuz you just didn't have the right version of Rails. And there was no way to like, if you put the Rails version in your config.gem line inside your application RB, then Rails would complain that you had the wrong version of Rails. But Rails had to have successfully started up to tell you that you had the wrong version of Rails. So it didn't actually help. And ultimately, it was actually a significant part of my job as a Ruby developer to figure this shit out by hand and it sucked. Depending on what exactly you did on the team. Some people on my team at the time spent like a quarter or a third of their time doing nothing but figuring out and fixing dependency management issues. And it was really, really, like I felt really bad for them. Sometimes it was me and I felt really bad for me. And then there's one more, even after you have done all of this by hand management, there's one more problem that RubyGems has that is another reason why Bunder was created. And that is activation errors. So an activation error is what happens in RubyGems when you load an application and you start by saying, hey, I need this gem, hey, I need this gem, hey, I need this other gem. And so RubyGems will load the newest version of those gems that can. And so sometimes you'll say, hey, I need this gem. And then this gem will need that gem. And then that gem will need this gem. And you'll get the newest version of that child gem. And then later you'll say, oh, and I also need this gem. And that gem won't work with that gem. So how common can this be really? Well, unfortunately, it was super common. Not like happens to you every day common, but like happens to you maybe two or three times a year. And when it happens, you basically tear all your hair out, delete your entire Ruby install, and reinstall Ruby and start installing gems again. Because figuring out exactly which combination of installed gems was causing this problem was just a total nightmare. So this is a real life activation error. I salvaged this from a presentation that I gave in 2010 about why Bunder exists. So this is a Rails app. And it's loading. And Rails, of course, depends on Action Pack. This is the Rails 2.3 era. Action Pack depends on Rack. Rack is a gem that helps Rails talk to web servers. And Thin, which is a web server, also depends on Rack. So Rack is how Rails talks to Thin, how Thin talks to Rails. But there's a problem. Thin is perfectly happy to use Rack 1.1, which makes some changes to how Rack works. Action Pack, on the other hand, is not happy to use Rack 1.1 and can only use Rack 1.0. And so when you run your server, your server, of course, loads Thin first, because Thin is the server. And then Thin gets to work trying to load up your Rails application. And your Rails application says, I can't actually use that Rack. Sorry, the end. So the conclusion here is that the reason that these activation errors would happen is that RubyGems does what we call runtime resolution, which is RubyGems figures out which versions of which gems it should load after RubyGems is already running. And you say, hey, I need a thing. And it's like, OK, I think this version works. And at some point, if later on you say, hey, I need a thing that doesn't work with things that you've already done, RubyGems just has to be like, well, can't fix that. And so the fix for this problem is to figure out all of the versions before you run your application. You have to know that the versions that you're going to use are all versions that can work together with one another. And so resolving things at install time, which is when you install all of the gems, know that you're installing versions that work together. So hang on a second. You're probably saying, how do we make sure that all of the versions that we're installing work together? Well, that's actually where Bunder comes in. Before Bunder, the process of figuring out which gems would work together was done entirely by hand. And it consisted of gem on install, gem install slightly older version. Does Rails start up yet? Gem on install, gem install slightly older version. Does Rails start up yet? And when the exception stopped, you knew you'd won. Unsurprisingly, computers are a little bit faster at this process than people. And computers are also really good and accurate at trying many, many, many, many, many options until one of them actually works. So this is what Bunder does. You, yeah, I know, Bunder figures out the entire list of every gem and every version of every gem that you need, but that also all work together with one another. This is called dependency graph resolution. And there's an entire academic literature about dependency graph resolution. And it's kind of a well-known, hard problem. It's part of the set of problems called NP-complete. And the totally fantastic thing, and I say this as a person who has to fix Bunder when it doesn't work, is that in theory, you can construct a set of gems and a gem file such that it is not possible to find a set of gems that work together until after the heat death of the universe. Most of the time, we don't have that long to wait. And so we use a lot of tricks and shortcuts and heuristics to try and figure out which gems to try first and hopefully actually finish before you've drunk that cup of coffee or whatever. So we have this pretty large built-up set of tricks over the years. And most gem files actually resolve in less than 10 seconds, which is pretty cool, considering that the upper bound on that is practically infinity. So after finding versions that work together, because this problem is really hard and we don't want to have to keep doing it over and over and over, Bunder writes down the exact versions of every gem that did all work together so that they can be reused by other people who are also interested in running your application. So that file is called the gemfile.lock. This is a little snippet of a gemfile.lock showing you which gems need to be installed, which versions of those gems need to be installed. And as a bonus, the lock file is what makes it possible to install the exact same version of every gem on every machine that's running this application. That means that when you develop on your laptop, you get whatever version of the gem was newest when you were developing, because you can bundle install and you've got the newest version by default or whatever. But because of the lock file, when you go to put that on your production server, you're completely guaranteed that you will also have version 1.1.4 of Robnitz and you won't have to spend three days figuring out why that production server doesn't quite work all the time. It's pretty great. So fundamentally, the core of Bundler consists of two steps, bundle install and bundle exec. So the steps for bundle install are actually pretty simple. They're totally understandable in plain English that fits on a single slide, which is great. I edited this slide for maybe 10 minutes, deleting words. So the steps to bundle install are read the gem file, ask ruwegems.org for a list of all of the gems that we're going to need, find versions of those gems that are both allowed by the gem file, because you can sometimes say, I only want this version or I only want that version or I only want versions greater than that version kind of thing. And then once you've found versions that all work together because you checked, write all of those versions down in the lock, and then install every version until every gem that's in the lock is installed. And that's how bundle install works. Bundle install actually uses RubyGems under the covers to do the installation. And so every bundle is its own little RubyGems isolated install. Every application has its own RubyGems. Thanks to Bundler. And then the next step is bundle exec, which is how we use that application's dedicated little RubyGems instead of the one that just has whatever in it because you ran gem install last year. So the way bundle exec works is it reads the gem file and it reads the lock if the lock's there. It uses the lock gems if the lock file is there. And if the lock file isn't there, it finds versions that all work together just like install would, except bundle exec doesn't do any installing. It just says, oh, do I already have versions that all work together? They do cool. And then bundle exec deletes any gems that are already in the load path because sometimes that happens before bundle loads. And then it adds the exact gem at the exact version that you need to the load path so you can use it, which is pretty great. That's it. That's all bundle exec does. It's once your gems, all the gems that actually work together and their exact versions are in the load path, your application just goes on its way and it's happy. No activation errors. All your requires actually succeed. I hope everything's pretty great. So as I think I promised in the abstract for this talk, here's a bundle exec removing pro tip. I don't really like typing bundle exec. I find it really annoying, but bundle provides a way to not have to type bundle exec all the time and it is to create programs that map to the little copy, like the Ruby gem installation that belongs just to that application. You can use the bin stubs command, bundle bin stubs sum gem, and it will create in the bin directory a program for that gem that only runs the exact version that belongs to that application. So if you have RSpec in your Rails app, you can have bin RSpec that will only run the RSpec for your app. And in this way, you can have bin RSpec refer to RSpec three in this application and have bin RSpec refer to RSpec two in that application. No exec required. It's pretty great. Rails has actually started to do this very thing and Rails four ships with bin Rails and bin rake that are scoped like so when you run bin Rails, you get the exact Rails version for that application and not this application. And when you run bin rake, you get the exact version of rake for that application and not this application. Pretty cool. No more bundle exec. If everyone did this and you can check in these bin stubs, right? So you can take bin RSpec and you can put it and it will be mapped to just that application forever. So no one would ever have to bundle exec ever again if everyone did this. Pretty cool. So now we bundle install. All our gems show up. We have versions that are dedicated to each individual application. But as you probably sense the pattern going through history, that wasn't actually the end. There are still problems that show up after Bundler came out. The biggest problem that was left was that running bundle install just took a really long time. And if you lived really far away from the US, it took a really long time. I talked to some developers in South Africa when I went there to give a talk and they told me about how running bundle install means that they literally get up to start making themselves a cup of coffee that they can finish before bundle install finishes. So to try and speed things up, Bundler 1.1 created a completely new way to get information from RubyGems about gems and that sped things up by like around 50%, which was a pretty big win. We keep working on this. Bundler 1.9 just came out this month. There's a bunch more improvements that we're still working on. Bundler will keep getting better. If you're interested in following along with that, the Bundler website has news announcements at bundler.io and on Twitter we're also bundlerio. So having said all of this, if you use Bundler, I would totally love to have your help working on Bundler. It's an open source project. We're super, we have dedicated a lot of time to making it easy for people who don't know how to do open source to help with Bundler and to start working on Bundler and to kind of get into open source that way. It's a project on GitHub at Bundler Bundler. It's on Twitter. If you are interested but don't really know where to start, you can totally email the Bundler team at team at bundler.io and we'll get you set up. On the other hand, if you have a job that means you have money but not time, join Ruby Together and give us money and we'll work on Bundler and it'll be better. As Ruby Together grows, we're also gonna be tackling bigger community issues. We wanna add easy to use gem mirrors so that you don't have to go all the way to rubygems.org for your office or for your data center. We wanna add better public benchmarks. There's a project called Ruby Bench that's starting to do that and we'd really like to expand it. There's a bunch of other things that Ruby Together is working on that'll be totally cool. If you want Bundler or Ruby Together stickers, I have a giant pile so find me later. That's it.