 why it is annoying and stuff that we did about that and other things that we are planning on doing. My name is Andre Arco. I wanted to know if it's indirect or if Porter also has indirect or flexed via media center company. Terence, I'm on GitHub as home. On Twitter as home02. This is homeless taken and I work for Rocket. The story of Bunder starts in the bad old days of Ruby when you had applications and your applications had gems. But wait, which gems again? And this is when you usually looked at the project and you said, gosh, I really hope that whoever wrote this read me, you remembered to change the gems that are listed when they change the gems that they actually use on their machines. And people realize that this sucks because you can easily spend hours or days trying to figure out which gem and which version. And wait, why does this machine have this version and this machine have that version and which one is the one that actually works? I don't know. So people thought that they could fix this. So on Roku's Aspen bamboo stack, in order to install the gems, we had dot gems. And so like if you wanted to install Sinatra, you just list Sinatra and it would get passed to gem install. And then we would basically just iterate through the list and install the gems. But this would become problematic when you would change your dot gems file. It would go through and install whatever you listed, but the gem versions might change. So if you didn't pin them down and all the dependencies of them, you might be in for a surprise when you deploy them. Yeah, so Rails, of course, also tried to solve this. And it didn't work that much better. The problem, of course, with Rails's solution was that by the time Rails was loading your gems, some of your gems were probably already loaded because Rails was loaded. And that didn't really work out that well either. So this was a really big need in general. And even though there were people trying to solve it, there wasn't really anything that worked super well. So into that world was born Bundler 1.0. And it was a huge ecosystem revolution as far as getting your apps and your gems to actively be working in development and then actually working in production with the same code. It was really, really great that you had the same code everywhere that you were running your application. The biggest thing that Bundler 1.0 did was once you had run it in development and you were sure that it worked, you ran it in production and it actually worked with the same code. That was pretty sweet. People being people, of course, once that problem was solved, it wasn't actually enough that that problem was solved. There was slides that are cut off on the left. That was very helpful. So people had Bundler. The same code was running everywhere. It wasn't actually enough. People, of course, said, oh, that's a solved problem and now it's not interesting anymore. I now have real pressing needs. You know, why aren't my new real pressing needs already solved? Gem files could be prettier, for example. This was the age, of course, I'm thinking about this. So we actually said, well, please do investigate that problem. And DSH came back with a border question. And his complaint, of course, was that if you have rails off of GitHub, you have to type git so many times that this was totally unnecessary. So we actually have a patch written by HHH that allows you to stop typing git so many times. So this is a fantastic improvement. 1.1 will just massively increase developer productivity thanks to DSH. Slightly more seriously, there were also problems like now that you had a bundle and your gems were locked down to a specific version, when new versions came out, you wouldn't actually know. In the battle days, all you had to do was deploy to find out that new versions were available. And that was no longer available as an option. So we actually added to Bunder 1.1 a command called bundle outdated. This is actually a contributed feature. We've been maintaining it, but somebody just sort of came out of the woodwork and said, I feel like writing this feature. Here you go. It was really cool. This was pretty much the first time that that ever happened to be with any significant feature. So bundle outdated basically does what it says on the tent. You run it, it talks to RubyGems, and it says, oh, by the way, here are all the gems that you have in your app that you could upgrade if you wanted to. And you didn't even have to deploy to find that out. The next thing that people started complaining about was orphaned gem files. I think this was actually mostly a problem with curaboo. Yeah. So this was a problem when people would complain that their select slides were too big, because over time you would change your gem power, update your gems, and we would keep around to all the gems when you used dash, dash, pat, and they all had isolated and installed in Indirectory. And so curaboo would not clean these up, because bundle didn't have this functionality. The alternative was to rebundle every time, but I don't think people wanted that to happen with bundle 1L. That's pretty slow. Yeah. So this was actually my first contribution to bundle 1L. The other part of the team was to add a quantum clean method, and basically you'll go through and look at your existing gem file.lock and figure out what gems are actually in use, and all the other gems and gem purges that are not, and we'll go and clean those up and remove all the files associated with them. So you have the smallest possible subset available of the gems that you need. Yeah. So this feature also actually, it turns out, once it was written can come really handy in development. If you're using, if you have multiple applications, and you have each application's gem, so you can bundle install dash, dash, pat, you can actually run bundle install dash, dash, clean to activate the auto clean feature. And from that point forward, whenever you install your development, gems will be cleaned up, and you won't end up with, you know, your most heavily developed app with a gem bundle that is slowly taking over your hard drive. So there's a problem that some people reported with one point on the wire. After a few months of changing gems and adding gems and leaving gems, their bundle has grown to be 10 or 20 times the size of their actual app. So auto clean comes in pretty handy if that's your development setup. Pretty cool. We don't do it automatically for rollback reasons. All right. So like if you have a catastron and you need to roll back, and you want your old gems around, we initially had it implemented that way, and then someone complained and realized we broke some bird. So we rolled that back. So the next thing that people started complaining about once bundle 1.0 was actually in wide use was sub-shells. Interestingly, like you would think, oh, you know, if you're shelling out from your bundle application, you probably want your sub-shell to have the same set of bundle gems available. And that's totally true right up until the point that you realize that there are many extremely popular command-line utilities that you don't realize are actually really... People started running up against this when they wrote rate tasks for their application that showed out to, say, homebrew to install Redis because it was an application dependency and suddenly homebrew exploded because it wasn't in the bundle. That turned out to be a pretty big problem. So in 1.0, we tried to solve this, and it turned out we didn't actually solve it. And in 1.1, the same method bundle.gov with vnend is still present, but it actually works correctly and completely strips out everything that bundle.org changed or did, and your sub-shell from inside with vnend will be completely free of bugs and you'll be able to use homebrew or use the gist command-line tool or whatever else it is that you were planning on using from the command-line that happens to be Ruby, but isn't in your bundle. It's not the best solution ever, but we had to come up with something that let you sub-shell both inside the bundle and outside the bundle, and that's what we have so far. This is also used a lot for people who have CI apps, I guess, that have a jump file ready and you're trying to execute something within another jump file, so you don't have to keep overriding the bundle or the score-jump file, the environment variable, and try to essentially do the same thing. Yeah. So this totally was a surprise to me, but something that happened pretty quickly after 1.1 came out was we started getting pull requests at top-level bundle commands like a bundle grep and bundle app and bundle RAK all in a row because everyone now that they had a bundle wanted to search their bundle and only their bundle. People, I guess, had never really tried to do this before or had something set up that let them search through all of their install Ruby gems, and suddenly now that they had bundles, people were like, whoa, you should let me search only in my bundle. And of course it seemed like a bad idea to accept pull requests that created three different local searching commands depending on your favorite search utility. So we kind of had a little pow-wow and we came up with something that lets you just extract out, or not extract, but just ask bundler for the paths of all the gems in the bundle. This turned out to be really useful because it means that you can just pass that to act or grab for your tool of choice and very quickly search through specifically one of the files in the gems that are in your bundle. It's also actually pretty easy to just turn to a function and then you can, once you've done that, just back your way through your bundle or, you know, e-grab the pipeline. And that's actually both useful and fast. So that was a sort of unexpected development of having a bundle out there that, you know, I didn't see at all for sure. The next and somewhat larger problem that people started asking about and it just sort of ramped up over time. And then for a while this was almost the only question that I got asked about bundler was, what do I do if ops won't let me install bundler as a system gem? People had, for whatever reason, lots of ops teams that said, oh, we will install the Ruby package for you and we will install the RubyGems package for you and now you don't have sudo. Good luck. And people came up with, sometimes, really tacky workarounds for the fact that they couldn't install a bundle gem at the system level. So unless, you know, you're this guy and you can just install it that way. But most people, sadly, were not able to do this. And so bundler 1.1 also includes a feature that basically amounts to self-loading bundles. It's pretty easy to use. You can just call bundle install dash dash standalone. This feature was guest contributed by, give it a click. And once you've, once you've installed your bundle at the standalone bundle, it doesn't actually require a bundler to exist from that point forward. It just creates a file named bundler setup, bundler slash setup dot rv that you can require inside your app. You know, very similarly to the way you require bundler slash setup when the bundle gem is available. But instead of loading the Ruby gem, it's actually a really straightforward file that just sets up the load paths for all of the gems in the bundle that just got installed. This is really cool in the sense that, like, you can create a packaged version of your app or a deployable package that doesn't care about the system level stuff as long as Ruby is available. It's not even actually invoking Ruby gems itself. It's just setting up the load path for you and saying, now you can require whatever you want. So, let's get into one of the biggest features probably in bundle one one, was the speedup of bundle install and sort of the process we did to get it there. So, just to give context and examples, we're just going to have this really simple jump file sources for RubyGems.org and then just have this not your jump. So, just looking at the latest version of bundle one one when you run a bundle install, you get the infamous fetching source index for RubyGems.org and then you have to wait and you're still waiting and you probably still hang in there and then eventually it finally finishes and just for a single Sinatra gem with its dependencies, like it has four dependencies or something, it takes about 18 seconds and it should not take 18 seconds to install five gems. So, just going back to that source index like let's look at what it actually does. So, we need to build up the index of all the gems available to us to actually resolve. So, the first thing that bundle does during that phase is it needs to fetch the modern index for RubyGems.org and this includes, this is 4.8.gz, so you can download this for RubyGems and we also need the pre-release specs in case you're using pre-release gems. So, let's go ahead and download it. I did this while running the slides and looking at the current index, it was under megabyte. So, anyone on a decent network connection it does not take 17 seconds to download one megabyte. So, clearly it's not the network traffic that is slowing us down here. But, let's look into the actual spec of the modern index itself and see what is contained. So, it's just a martial array of this tuple of the gem name, the gem version, and the platform that's running. For most gems, this is Ruby, but you have stuff like Java or MS, you win 32x86 for windows and other things like that. And then, we also need to, one thing you notice is it doesn't actually include any dependency information. So, that's all contained inside the gem spec file. So, the resolver actually makes a call out to fetch each of the gem specs that it needs to fetch this information to try to resolve the gem file that you've given it. So, if we just go look at this gem spec file we can see that it returns an array of gem dependency objects that contain similar information. What kind of, what the name of the gem is, what the requirements are, if it's a development or runtime dependency. So, bundler 1.0 will actually, it downloads the whole index and constructs it in memory. And, when it came out on older slice boxes that only had two megabytes of RAM you couldn't actually run bundler because it wouldn't take more RAM that was available on the box, than the box had itself. And so, this only gets worse every day as people release more gems and more versions of each gem. So, the problem's only gonna get worse. So, let's take a look at what the latest version of on the 1.1 does. The first thing you'll notice is that we don't have that source in Xline. We're fetching the gem data from RubyGems.org. And then the second thing you'll probably notice is it's a lot faster. So, we went from about 18 seconds to a little over three. So, I guess that's a little under one-sixth of the time, or a little over one-sixth of the time that was needed. So, it's pretty huge speech improvement. So, at the heart of that, if we go look at the fetching gem metadata, what it actually does is that you get points on RubyGems.org. And basically, you pass a comma-separated value list of all the gems that you need. And it will return the first level dependencies back. So, if we just write some Ruby code, it's pretty simple. Let's marshal load, open that URL for Sinatra, and it will return the name, the version number, what platform it's running, and this is pretty similar to the modern index as well as having gem spec information there. So, if we just go through a quick scenario of that gem file running through 1.1, if you pass debug to any, if you set debug to any, anything vulnerable output, a propose, some proposed debug information, and so just quickly stepping through this, the first thing we're going to pass all the gems in that gem file. In our case, it was just Sinatra. And then that point will return the first level dependencies, and then we pass those back through, and you sort of just iterate through until you have an empty query list where you don't have any dependencies you need to specify anymore. Bunder keeps track of gems that you already passed through, so we're not repeating queries that have already happened, because we don't need to relearn what is unique for that gem. And this is the actual implementation that came out in Bunder 1.1.3.5, and I thought that was all you needed to get to work, but turns out that's not the case, and everything's always more complicated than it actually seems to be. And the issue that was brought up was that, well, what happens if I have a private gem repo and I have this gem there? So if we have the gem foo, and it depends on bar, when we fetch the information and we fetch for bar, we don't actually have the whole graph anymore, like we did in 1.0, so we wouldn't have the information of the dependencies for BAS 1.3, and so when you actually go to do the resolve, Bunder would say, sorry, I don't have enough information to complete, and it would just exit out. And obviously this isn't good at all. So we ended up having to implement some edge cases around using that point. The first edge case is just like, so let's just say that the private gem repo implements the API as well. This case is probably the simplest one, since we can naturally just query both endpoints until we have dependencies, and then we have the full graph that we need to pass the resolver, and we don't have to make any more network calls there. But this probably isn't the case for most people who have a private gem repo setup. A lot of people are probably just using the gem in a box and just complying with having the moderate index around and not actually building out the API as well. So in this case, we have API endpoints for root gems. And now we need to figure out what the unmet dependencies are. This means that we have to actually go and fetch all of the gem specs from the private gem repo because a moderate index like we saw before doesn't have any of the dependency information. This is great if you don't have a lot of gems on that server. Because, you know, just making those network calls is not that expensive. But if you have a ton of gems then you have to fetch every single gem spec on that server. So we had to change this code because let's say root gems API endpoint is like 500ang then we're going to try and fetch every single gem spec on rootgems.org and that's probably bad and would seriously increase the S3 bill for root gems for that downtime while the API is down. So basically what we do is we just fall back to our index and you're sort of left with bundler 1.0 speeds which seems acceptable since people have been doing it for a while now. So we actually allow you to bypass the API endpoints in case there's problems or you just like waiting a long time for your bundle to install. You can actually just pass this dash dash full index here and we'll just fetch the source index. And we actually do fall back here. So if rootgems.org is 500ang on the API or for some reason it's not working instead of forcing you to run the previous command again we just rerun it for you in using the monitor index. Alright. So now that bundler at least version 1.1 is significantly faster and bundled install on the launcher is so slow we wanted to talk a little bit about stuff that we have coming up in future versions of bundler. The absolutely the biggest thing that we have as a priority in future versions of bundler is a shorter release cycle. We've been working on bundler 1.1 for a little over 18 months now and while it contains totally cool stuff that is forever in internet years so our main goal with future versions of bundler is to get the new cool stuff cut down into individual releases and get it out and have people using it in a much smaller amount of time. So here are some of the features that we're trying to get into soon upcoming versions of bundler. First one is the Ruby version check which is basically just the idea that your application is your application is written against specific versions of gems but in most cases your application is written against a specific version of Ruby as well it could be written against 187, it could be written against 193, it could be written against you know Rubinius but most of the time when you want to make sure that the same code is running in deployment and in development you also want to make sure that the same interpreter is running in deployment and development so the essence of a Ruby version check would just be that bundler will happily tell you when you have attempted to load your bundle on the wrong Ruby interpreter and your results are now unsupported or unified since rbm and rbm do not share a place for you to declare which version of Ruby your application should use we're kind of hoping that this can be like a application-wide declaration and it would be totally awesome if rbm and or rbm took advantage of the fact that there was a single place where it was declared to know which version of Ruby to invoke we'll try talking to them hopefully that will also become easier in the future the next feature that this is probably the largest point complaint in the people who are already using the fast version of bundler if you're developing I mean now that bundler exists people have taken advantage of how easy it is to use gems with their application and turn a lot of things that used to be in the lib directory into gems of their own and this is really really cool it's increased reusability with that code it's increased other people's ability to attribute code but the main downside to breaking out of stuff that your application if you're developing a gem and your application at the same time it's a gigantic pain in the butt because what you have to do is change your gem file to say oh by the way the code that I actually want to use while I'm developing this gem is a path object and in deployment that doesn't work because you don't have a check back copy of that gem so you end up either doing a horrible dance of editing your gem file continuously or discovering that you didn't check in your code or push your code to get a run bundle of big people and then restart your development application either way this slows down the cycle of development it seems like a bad thing so each of us are currently discussing and trying out different implementations of his local gems and fundamentally what it does is it just lets you say I have a gem checked out and I'm going to be editing it and if it's there use it and if it's not there go back into the fall back on to get repository and install as you normally would this does amazing for productivity in the middle of developing a gem and your app at the same time it has the potential caveat that you develop your gem and your app and your gem and your app and then you push your app to production and the code in your gem it doesn't come along with because you didn't check it and push it release it whatever so we're working on ways to make that less of a problem and at least be able to give you a warning that says probably aren't going to get the code that you're thinking you're going to get but it doesn't actually go into the log file the local version stays in the gem file at least in our current implementation that we're testing and last feature that we're actively working on for near future versions of Bummer is sort of the holy grail of Bummer not having to bundle it back at all yeah yeah so the reason why you have to bundle it exact is of course that when you're in your application you want to rate to be the version that your application wants but even though you're there you might want to use another command that isn't part of your application and we can't provide you with a bundle no exact command to be inverse so the plan and hopefully this will work out but it seems like it should is that Bummer will provide smarter auto bundle invoking bin stubs and you will be able to either manually put those bin stubs into your path there will be a specific location that you can either add to your path yourself or that RBM can add to your path automatically for your bundle and those smarter bin stubs will say oh rake is in your bundle so you're going to get your bundle version of rake but gist is not your bundle so you're going to get the ruby jams version of gist and that actually solves this problem we think so keep an eye out for the ruby's version of Bummer coming soon that lets you stop bundle exacting because we would love to get people testing it that pretty much covers everything that we had about Bummer at the moment if you guys have any questions one final one final, yes one will be final it will be as soon as it's humanly possible we really want to get it out of the door as a final version we have a couple of known very obscure edge cases that we can iron out and make sure are working and then we will have a final rc Jeremy Hangerner here will be able to write one of the private gem servers stickler and I will be implementing the dependency api the private gem server stickler obtained by Jeremy Hangerner will be adding the gempenter api so you will be able to install your gems more quickly even if your private gem server is stickler that seems cool seems like you're always dependent on there being a network connection is there a way to bundle and install yes no network connection is totally feasible you run the bundle pack and it fetches proactively every gem that you need and shows it into vendor patch at that point you're done you can run bundle however you want it doesn't pack get rebos though there is a patch under testing that packs get rebos that network connection has not been as big of a concern of other users there's a small vocal very small vocal minority but no one was willing to test the get packing patch so it's been languished for a while hopefully that vocal minority will get on and that patch will end up in a near future version of vocal yes path goes in your lock there are lots of people using path in their lock because they have put the gem into say, vendor gems or somewhere else that has a path that is specific to their application and that needs to be locked exactly so the local option is a development only path basically and that's why it has to have a separate API