 So, this talk is called How Does Bundler Work Anyway? And it basically came from realizing, as I talked to people, that a lot of people are like, oh, I know how Bundler works. You copy and paste this line from RubyGems or read me and then you run Bundle install. That's how Bundler works. So it turns out that if what you care about is what's happening when those things happen, you actually have to understand what the problem is that Bundler solves. And to understand the problem that Bundler solves, you have to understand the problem that created the problem that Bundler solves, and then you have to understand what created that problem and then what created that problem. So this is going to be a brief history of dependency management in Ruby, hopefully brief. But before I tell you about how dependencies have worked forever, let me quickly tell you who I am. My name is Andre Arco. On the Internet, I'm pretty much always indirect. That's my current avatar on all the Internet things. You may have seen my picture on a website or something. I work for Cloud City Development, mostly joining teams who could use someone who has seen how that decision can go wrong and can warn you to avoid it. Also pairing with people who are like junior and mid-level and talking about how Ruby works. If your team could use one of those people, maybe talk to me later. I co-authored the third edition of the Ruby way. The first edition of the Ruby way was totally my favorite Ruby book ever in like 2001. And I basically learned Ruby from it. And so the Ruby way third edition is updated for Ruby 2.2 and 2.3. I'm sure it'll last at least a good six to nine months before it just turns into something that you use to prop up your monitor. The other thing that I do is called Ruby together. It's a nonprofit. It's kind of like NPM except we don't have any venture capitalists. And basically we take money from companies and people who use Ruby. And we pay developers to work on projects that everyone who uses Ruby needs to have work. So we have some companies have just agreed with us that this is a good idea, including Stripe and New Relic and Basecamp. And what Ruby together does is it pays for work on things like RubyGems and the RubyGems.org server that everyone gets gems from and Bundler. So I've been the team lead for Bundler for about four years. And that's why I'm talking to you about Bundler today. So let's share some Ruby code. How hard is it to use code written by someone else? Well, it's actually pretty easy. You just put this thing in your gem file and then you run Bundle install and then you run Bundle exec. And ta-da! So that was so easy, but what actually just happened when you did that? Well, it's complicated. Something got installed, something got downloaded, what got downloaded, where did it go, how did you get to it later, I don't even know. So history time, let me take you on a little dependency history tour. But by the time we finish you'll at least know, you know, not just how it works but why it works the way that it does today. And as you could maybe predict from anything that involves computers, the answer is legacy reasons. So starting, here's a little preview of what I'm going to talk about. I'm going to talk about require, which is how Ruby lets you load code from files. Then we'll look at setupRB, which was the first way that you could install code into Ruby so that you could require it without telling Ruby where it was. And then we'll look at RubyGems, which was the first way to easily download someone else's code. And then we'll look at Bunder, which lets you have multiple gems that all work at the same time. So require. The require method has been around since at least 1997, which is the oldest Ruby code that is actually in version control. I went back and checked. Presumably required did not exist at some point, but Matt wasn't using version control yet, so I don't know when that happened. Even though require is that old, it can still be broken down into some pretty small concepts. So using code from another file in Ruby, at least, is basically the same as copying and pasting the contents of that file in place of the line that says require. That's pretty straightforward. So you can actually implement the require function yourself naively with just like one line of code. You say, I have the path to a file. I would like to read that file. I would like Ruby to eval that file as if it was code, done. I've implemented require, pretty sweet. So as I said, a little naive. You'll probably notice the problems with that naive implementation, so if you call require more than once, your code will run multiple times. This is usually bad, especially if your code, say, creates a class because it defines a constant or whatever. So it's actually not that hard to implement a system that checks to see if you've already required something. I even got it to fit on a slide. Global variables are usually pretty evil. In this case, it's probably the only way to track things that you've required that make sense. So let's call that global variable loaded features for no reason in particular, and then you can see that you basically just say, hey, did I already do that? That file name? Well, if I didn't go, if I did, hey, we're done already, sweet. Turns out Ruby provides a global variable named loaded features that contains a list of all the files that you've required. Good work. So then the next problem that you may have already noticed just from looking at this code is that you have to pass absolute paths. If you say require rails, it will say there is not a file named rails. So that's probably okay if you know where every single file on your entire machine is, but I'm assuming that you don't want to have to know that all the time. So the easiest way to allow requires that aren't absolute is to just treat every file name as if it's relative to the directory that the program was started from. That's really easy, but unfortunately it doesn't work well if you want to require Ruby files from multiple directories, so let's implement something that lets us require from more than one directory. I guess theoretically that would probably look something like a global variable that's an array that contains a list of directories, and then you would loop over the list of directories, seeing if the file that's being required is present in any of those directories. I even got that code to fit on a slide. So we'll call that global variable load path for no particular reason. And then we'll say, hey, let's just find the first entry in load path that actually contains the file that we're requiring, and then we'll do the normal eval thing. So it turns out Ruby provides a global variable named load path, and when you call require, it iterates over the load path looking for the file that you just tried to require in each of the directories contained in the load path, and now you know how require works. In real Ruby, unlike my slides, both the loaded features feature and the load path feature are combined into a single method. I couldn't get that to fit on a slide, but you feel free to do it yourself if you would like. So adding the load path is pretty cool. It lets us find Ruby libraries, even if they're spread across multiple directories. We can add a directory that has the Ruby standard library in it, and now it's very easy to load net-http. You just say require net-http and it finds that there's a directory named net and there's a file named http.rb, and it works. Pretty great. So this was awesome. And then Ruby developer said, hey, I wrote some code, but someone else wants to use it. And that's how we got set up RB. So require, state-of-the-art in Ruby libraries, probably circa the year 2000, right? From 97 to 2000, require was good enough. And around 2000, everyone's installing shared Ruby code by saying, oh, this guy, he emailed me the Ruby file, and then I put it in a load path, and then I required it, and everything's awesome. That's really tedious, as you can probably imagine if you're thinking about sending people files so that they can copy them into places. So the next piece, or I guess the next solution to the obvious problem, it's called set up RB, it was written by Mianaro Aoki, and it's a script that automates copying Ruby files into a specific place. So in Ruby, when you install a copy of Ruby, it creates a place called, wow, now I'm blanking on it, site lib, thanks. I was gonna say site features, and I knew that was wrong. So site lib is where you can install things and they'll automatically be on the load path. Ruby just puts site lib on the load path, and you can put whatever you want into that directory. So set up RB is a way to automatically put things into that site, site, which is a really weird way to describe a specific installation of Ruby, but it works. So amazingly, set up RB is actually still around on the internet. I love the website, it's actually i.loveruby.net. You can even download it if you want to use it, but no one has touched it since about 2005, so I probably wouldn't recommend it. So at its core, set up RB is basically a Ruby implementation of the classic Unix trinity of commands configure, make, make, install, except for Ruby files, so you have set up, config, install. And so moving forward in time, this is now how you use Ruby libraries. You find a cool library, you download the library, you untar the library, you CD into the directory that you just untarred, and then you run Ruby set up RB all, and it copies the Ruby files into your site and now you can require them, and everything is great. So as you, actually, so this was before rubygems.org. And there was a thing called the Ruby application archive, which is sadly lost to the internet as of a year or two ago. But it was amazing, this is about when I started using Ruby for the first time and I was so excited that there was a website that I could go to that was like, here are things that people have written in Ruby that already do stuff, so you don't have to write them yourself. Without the RAA, and honestly, even after the RAA existed, sometimes using libraries meant happening to be able to find the person who wrote the thing's website where they had posted a tar ball that had their Ruby code in it. So this was great, you could find a library, you could download it, you can install it, it was a big improvement. You may have thought of some possible problems with this so far. So a big thing that became problematic as time went on was there's no versioning whatsoever. What version of that library do you have? No idea, maybe they put it in a comment. If they didn't put it in a comment, you could just start downloading old versions and trying to guess, it was amazing. How do you update when there's a new version? Well, you hope that you bookmarked the URL where you downloaded it the first time, or that they put the URL in a comment, because good luck, and then you had to find the new version, download it, decompress the tar ball, and run Ruby SetupRB again. And then you'd have the new version, but since there was no uninstall, what you would actually have is the files from the old version, and then the files from the new version copied on top of the files from the old version. And then the next time, you'd have the files from that version copied on top of the files from the old version. So you also crossed your fingers and hoped really hard that they hadn't deleted any files as part of the new version because that got really messy really fast. And finally, this was kind of my favorite part of how this was horrible. All of the files went into a single directory. So if you had two libraries that both defined cool thing.rb, too bad, whichever library you installed last was the one that overwrote the previous library. So I don't know if this sounds tedious to you. It was pretty tedious. I was there. It took a lot of time. There was no really good way to know if new versions came out. You overwrote your old versions with new versions or your new versions with old versions by accident. And honestly, if someone changed something in a new version and you were like, oh, I want this cool new thing that's in the new version, but then you installed it and something was different than every single Ruby script that you had on your machine that needed the old version no longer worked. And you had no idea if that was going to happen or not. So that was also kind of frustrating. So in 2003, RubyGems appeared and specifically tried to address the shortcomings of setup.rb. So it provided a single command line utility, the gem command, which was pretty awesome. It let you download and install in a single command, which was pretty awesome. It let you uninstall in a single command, which was very awesome. And it let you see what libraries existed in a single centralized place, which was super, super great. This really revolutionized sharing code in Ruby, honestly. It increased the speed at which people were willing to say, hey, this thing is cool. I'll let other people use it. And it was actually a big part of why I thought Ruby was so cool as I started using it, because RubyGems was brand new at the time. And I was like, wow, no one's written things that do lots of things that could be useful. I'll sit down with my friends and write a thing that does a thing, and it'll be great. And we'll make a gem, and it'll be super cool. The last most awesomest thing that RubyGems added was multiple versions. So setup.rb put everything in the same directory, and it was impossible to tell what version you had, and it was impossible to have more than one version at the same time. RubyGems actually adds its own patches to the require method so that you can have multiple versions of a single gem installed at the same time. If no version of that gem has been added to the load path yet, RubyGems will add the newest version, but if you want to, you can explicitly say, no, no, I actually want this exact version. RubyGems put that version on the load path because that's the one I need. So you can call the gem method with a version, and RubyGems will put that version on their load path, and then when you call require, that's the version you'll get. Pretty great. And there's a little-known RubyGems feature slash horrifying hack that lets you put a version in underscores as the first argument, and then you'll get that version of the gem. I don't know if that's really relevant to any of your particular lives, but while debugging gem versions in Bundler dependency hell, sometimes it's important. So I care a lot that it exists. Thank you, Eric. So, RubyGems made installing, upgrading, creating, using gems so easy there is this huge explosion of libraries. So today, there are almost 100,000 different gems, and there are almost a million versions of those 100,000 gems. This is really cool. As you may have noticed by now, somewhat predictably, once we started using this, we noticed some problems. The new problems were, what if you have more than one project that uses Ruby and needs gems? So this turned into a big problem, I wanna say, over, it took several years for this to emerge as this is clearly the biggest problem with RubyGems. So the explosion of gems and gem versions meant that if I ran gem install foo and started using it, and then a week later, I wanted to share this with my friend and said, oh yeah, you have to run gem install foo before this will work. My friend might get a completely different gem than the one that I got when I ran gem install foo the week before. And that's really not good. And then if you ever wanted to deploy this into production, you would have to run gem install foo on the servers and hope that you got the same code, because, yeah. So, setting up a new machine, this is a thing that actually happened to me. I started at a Ruby company in 2008. They said, here's your new work laptop. We're really hoping that you'll get the app to boot in less than a week. I was really ambitious. I worked really hard. I was visiting the office and didn't really have anything to do at night. I got it working in only three days. I literally spent three days trying to boot the app and then trying to diagnose what was missing from the error message and then installing gems and then trying to boot the app again and then trying to figure out why it didn't work. Three days of that. It was pretty amazing. Once the Ruby community recognized this as a pain point, people started trying out different ways to solve this problem. The most popular way to solve this problem was something that Rails added called config.gem. So you would put in your Rails application, config.gem, I need some gem with this, and maybe with this specific version if you cared, maybe you didn't. And there was another gem called the gem installer gem that you wrote a list of gems and then the gem installer gem install all of them for you. The underlying problem here was that, well I guess it was, there were multiple underlying problems. The first underlying problem was that Ruby gems automatically uses the newest version of each gem and so just having an older version of a gem installed wasn't actually enough for you to be able to use the older version of the gem. And also a true story that actually happened to me. Three days of debugging why something worked on every single developer's machine but failed mysteriously in production, turns out that the production server had gem version 113 and every developer had 114. There was no way to even be told that. You just, at one point someone was like, wait could this possibly be and ran gem list on the server and was like oh my god this isn't a version that everyone else is using. It also led to things like version conflicts. Like god forbid you have two Rails apps rather than just one or god forbid that you're a freelancer and that you have two clients who each have a Rails app because at some point they may not have the same version of Rails and this is how you get into really really caring about that underscore hack that lets you get a specific version of a single gem. So you have two Rails applications. You say oh no this one's too old. Oh I'll solve this problem by upgrading them so that they're both on the newest version. And then the next person who pulls your code is like hey why did you break the app? And you're like oh no it's an upgrade. You have to upgrade all your gems. Run gem install again for all your gems. Which gems? Oh well I think the readme has a list. Is the readme list up to date? Well probably. And then if that person had some third app that used Ruby on their machine they would be like oh well I guess I can solve this by upgrading all of the apps too and then you just repeat the cycle. So ultimately a large chunk, and I am really not kidding, a large chunk of being a Ruby developer was like figuring out why your shit did not work when gem versions did not match. And it kinda sucked. Just fixing that problem would be enough of a reason for Bundler to exist. But there was another even more frustrating problem. Which is you would ask RubyGems to put, I mean RubyGems calls it activating. But what that actually means is RubyGems just puts that gem, a specific version of a gem in the load path so that you can require it. And then later on some other piece of code says hey RubyGems I need this gem at this version. And RubyGems would say no actually you can't have it because a different version already got activated by some completely unrelated code too bad. Runtime error, your server no longer works. It was super great. I've actually, oh man, there was one time where our production server worked perfectly fine unless someone tried to do the one action that somehow like depended on this gem and it would always activation error and then the server would die. But we had something that automatically restarted the servers. So we didn't actually notice until we were debugging the ticket about why that particular Rails action didn't actually work. Then someone was like oh it's an activation error. Why didn't we know that? So at this point maybe you're thinking surely this wasn't very common, it sounds very complicated and difficult to produce this. I wish. Basically by the late RubyGems era, so probably like 2008 to 2010, basically every Ruby application with more than a handful of gems would get into this weird situation where certain specific circumstances would produce sorry no you can't. This was exacerbated by certain specific versions like Rails 2.3 depended on Rack 1.0 and not higher and Thin, which was a popular app server at the time depended on Rack 1.1 or higher and if you needed them both at the same time too bad. So the moral of activation errors is not to do runtime resolution, which is what RubyGems was trying to do where you say oh I'll just wait until the app is already running and then use the gems that I need because by then it's too late. And instead do install time resolution which ultimately just means that before you actually even install any of the gems you say hey is it possible to make all of these gems activated at the same time. So install time resolution, hooray, but wait how does that actually work? So this is the kind of underlying problem that Bungler has as its core problem that it exists to solve. How do you figure out which versions of all of the gems that you need actually work together? Each gem depends on other gems, those gems depend on other gems, et cetera, et cetera. So before Bungler, as I have just described at sad length, this process was done entirely by hand. If your versions did not work together you said oh no, an exception and then you tried a different version. And then you said oh no, an exception and then you tried a different version. And then as soon as the exception stopped you went out for celebratory drinks. So unsurprisingly computers are better than humans at this kind of tedious and repetitive work. So that's what Bungler does. Happily Bungler is also by virtue of having the computer do it a lot faster than humans. So thanks to Bungler Ruby developers list the gems that they want and then Bungler finds what versions of those gems work together all at the same time. This problem is called dependency graph resolution. It's an example of dependency, I should say, dependency graph resolution is an example of a well-known hard problem, which is I guess another way of saying you might not ever be able to find an answer no matter how long you spend looking for one, which is super great. In theory it is possible to construct a gem file such that it takes into the known hard problem first to find which gems to use. This picture is a small piece of the graph generated by a gem file that has nothing inside it except gem rails. To give you an idea of why Bungler needs to do a lot of work. So most developers do not have until after the heat death of the universe to find out if their gems work together. So Bungler's resolver uses a lot of tricks and shortcuts and heuristics to try and come up with a list that will work or is at least a possible solution even if it is not necessarily the best possible solution. And so we have a pretty big library of tricks that we've built up over the years and most gem files actually now are resolved just within a few seconds. We had, I think the last kind of pathological case that we had was a gem file that took like maybe a minute to resolve and we got it down to 10 seconds in the latest release and I think all of our known horrifying gem files are down under 10 seconds so that's pretty cool. Once it has found versions that all work together and can be activated at the same time, Bungler writes down the exact version that works for every single gem and that list is called gemfile.lock and that's what makes it possible to always install the same version when I give you a copy of the code or you give the production machine the copy of the code or whatever. Amazingly it means that the CI server and the production machine and every developer always have the same versions of every gem and it means that most of the time the same code means that you get the same results which is great. So at the end of the day how Bungler actually works boils down to two things that mostly everything else is just like some aspect of those two things. So the first thing is bundle install which is figure out the versions of all the gems that work together, download all those exact versions, write down the lock file so that we know what those exact versions are and install all of those exact versions so that they're available. And then once they're all installed the second half of what Bungler does is bundle exec which basically just says make sure that the Ruby code that I'm running using bundle exec has access to the exact versions that are written down in the lock file and no other versions. So bundle exec reads the gem file and the lock if it's there, it uses the lock gems if it can, if there are versions already installed that will work then it will make a new lock file on the fly and then it deletes everything that's in the load path and puts only the gems that match the versions that are in the lock in the load path and then lets your application do its requires and everything works out great we hope. Most people don't like bundle exec I mostly don't like it because I don't like typing it so one solution which is something that I did for a long time is to just use your shell to alias the letter B to bundle exec and then you type B space before everything but another thing that you can do is create commands that are specific to your application and the version of the gem that you need for that application so here's say you can avoid completely avoid using bundle exec use the bundle binstubs command on the gem that you want to run the specific command from and then Bunder will create a file in bin that will make sure to only use the version that is written in your lock and so this particular bin RSpec will use the version of RSpec that's in the lock for this application but if you have bin RSpec in another application it will use the version of RSpec that's locked for that application and so this is a way to get around bundle execing ever and means that you always get the thing that's relevant for your local application context. Rails has actually sort of endorsed this pattern by making it part of a default Rails app starting with Rails 4, bin Rails and bin Rake are both created as part of every new Rails app and if you run those commands you will always get the exact version of Rails and the exact version of Rake that go with that application but you could do this for literally any code with literally any gem using the bundle binstubs command which is pretty cool. So that gets you more or less up to the present day. Even after Bundler was created the pattern continues. Bundler 1.0 solved activation errors which was awesome and created a new problem which was that running bundle install took multiple minutes. So to fix that we created a new version of Bundler that could download less information but still resolve. That was Bundler 1.1. There we continue to work on Bundler. We're up to Bundler 1.10. It has some cool new stuff. We have even more cooler newer stuff in the planning stages. We'll continue to keep making things better. I definitely recommend keeping an eye on bundler.io where we have a news thing that talks about new things as new versions of Bundler come out or you can follow bundler.io on Twitter. As part of those plans it would be super, super cool if you could help us since everyone uses Bundler. We would love your help. You can tweet at bundler.io or you can email us at team at bundler.io and there are people on the Bundler team who set aside time to pair with new contributors so that you can get up and running and kind of know what's going on and we would love to have your help. If you are a company and you're thinking to yourself, if you are a company, if you work at a company and you are thinking to yourself right now, that sounds great except my company doesn't have any spare time for me to work on Bundler. That's totally okay too. Make your company give money to Ruby together and then we will do it for you and you won't have to think about it. As Ruby together grows, we actually have plans that extend beyond just Bundler. We're going to start tackling other community issues. We've been working on a project mostly in stealth mode that's called gem stash and it's a server that you can run in your office or in your data center and it acts as a local cache for Ruby gems. So you can run gem stash in your data center and point all of your production servers at it and they can download and install the gems that they need super fast because they're already present in the data center or you can put it in your office and all of your developers in your office can install gem super fast because they're already in the office, it'd be great. We'll have the first release candidate of gem stash out tonight. Yeah, it'll be great. That pretty much wraps things up. I'm happy to take questions. It looks like we have about seven minutes for questions if people have any. Given that this is Bunler, I may answer the question with, I don't know, I'm sorry. Yes. So the question was, what are some of the interesting heuristics that you use to reduce the amount of time that it takes to solve the dependency graph problem? Most, I mean, none of them are actually that interesting to be honest. It's mostly just like, well, we come out with something that finishes much faster if we, and the if we is things like start with the newest version of everything. If you have a specific version number, start with that and go down from the newest backwards in time. If you are way down a rabbit hole, try going backwards and changing the version of the thing that depends on the thing that depends on the thing that means that you can't find a version that works, stuff like that. So the question was, when do changes to Bunler become new releases? We last released probably four or five months ago at this point, and there have been some changes since then. Usually the answer is when we have enough time to update all the documentation, write a blog post about what's changed, and then answer people's new questions because the new version always means that people have questions. The plan for 1.11 is sometime in the next couple of weeks, I hope. Sure. So the question was in Bunler 1.10, it added a section to the lock file named bundled with. This was really contentious with the minority of people who actually pay attention to and care deeply about whether their gem file dot lock is changed according to get or not. It turned out, so we thought we were being really smart and we specifically wrote the bundled with feature to only change the gem file dot lock when you installed new gems. So we thought this is totally fine, it's only gonna change the lock if you're also changing a gem or adding a gem or whatever. It'll be cool. Turns out, spring the command that ships with Rails by default and runs copies of Rails in the background waiting for you to want them to do things. Well, spring actually runs bundle install and Rails. I will never think that I am as smart as I thought I was. Again, basically is the summary of that. So we released a later version of 10.6 that actually has code just to work around spring and says, well, even if you run bundle install, we can't assume that you actually mean it because it could just be spring doing it automatically and so we won't change your lock file unless you actually changed a gem and then we'll add it. So that's what wound up happening. As to why it exists, it's because we want to do things like, well, I guess actually, as to why it exists, it's because we totally screwed up in bundle 1.0 and didn't put it in and realized approximately 10 minutes after we released Bundler 1.0 five years ago that it should have been in the lock file the whole time. Unfortunately, Bundler 1.0 had a bug that meant if we added it to the lock file, Bundler 1.0 would throw an exception. Five years later, no one uses Bundler 1.0 anymore and we can add it. So the question was, who owns Bundler and how is that determined? The answer is a lot of people. More or less, everyone, how to put this? So Bundler the project is licensed MIT, which means you can do whatever you want with this as long as you don't claim that you made it yourself. At various times, people have worked on it while employees at Engine Yard, while not employees anywhere at all, while lots of other things as well. Ultimately, the project only works because everyone who submits code to Bundler is implicitly agreeing to license their submission MIT along with everything else that's in Bundler so everyone can keep using it. As for ownership, it depends on what you mean by ownership. Oh, like who's allowed to merge pull requests? The Bundler core team and Bundler contributors team, which roughly means anyone who's had probably two pull requests merged in the past gets the ability to review code and merge pull requests. We have a merge bot that tries merging to make sure that it's actually possible to merge and then runs all of the tests after merging to make sure that all of the tests still pass and then actually commits that. And I think just the core team right now has the ability to trigger the merge bot. So the question was, how is unifying Bundler and review gems coming along? The answer is, we are making progress until Ruby together has more money, it will continue to be very slow. The progress we've made so far is that Ruby gems and Bundler now share a dependency resolver, which is super great. At one point they had two completely different implementations of dependency resolution, which meant that sometimes it was really confusing because Ruby gems tried to install this version and Bundler tried to install that version and why don't they agree? So that's the biggest thing that's unified right now. And ultimately a lot of the code in Bundler is actually just calling out to code in Ruby gems. So I would say that like today now that the resolvers are the same code, the overlap is probably around 40%. We'll keep making progress. Okay, so Eric the maintainer of Ruby gems just said that Ruby gems is gaining more Bundler like features to try and make them more similar so that one day they can be the same thing. That's ultimately the goal. It would be nice if they were the same thing. Yes. Oh, you're right, red light. Okay, I'm happy to talk to anyone who has questions afterwards, but we're out of time for now. Thanks.