 So the topic I'm going to be talking about today is RubyGems in general, and the impetus for this talk is that Carl and I just spent a large amount of months writing a wrapper around RubyGems for your own application called Bundler, which is going to be part of Rails. And so we got very, very familiar with a lot of RubyGems. The first pass what we did with Bundler is just to wrap RubyGems as much as possible and not have to worry about trying to push things upstream, but long term we want to try to push a lot of what we did back upstream and make it a sane thing for other people who are doing similar things to what we're doing, so sort of just a feedback of what we've done and how we think RubyGems can improve to make what we've done saner. So first of all, I've noticed that in all the presentations I've been at, there's this huge dead zone, so I've just not put anything in any of that spot. And yeah, I don't know how feasible that is for people, but it seems like a good idea if people can still tweak their presentations. So obviously RubyGems was released in March 2004. I think there's more history before that that I'm not aware of. The public history of RubyGems says that it started in March 2004. It started with version 0.2, so I think there's a good chance that something happened before that, but I just wasn't around to see it. RubyGems is good. Despite I'm going to enumerate a list of problems about RubyGems, RubyGems is actually really good. It may be the best packaging system for a language that exists, certainly up there, and it should not be... There are people who are attempting to say, like, we should have something entirely different conceptually, and I don't believe that. I think that RubyGems right now is a pretty good conceptual fit for what we're trying to do with RubyGems, but there are some specific problems that are not related to the concept of RubyGems, but are more related to implementation or specific details that I'm going to talk about. RubyGems, like I said, worked well for a long time, and I think we have to look at the bunch of years that we've successfully been able to grow the Ruby community with RubyGems as evidence that starting from scratch is silly if somebody would want to do that, and that there's a lot of... This is something I've learned from Rails. There's a lot of embedded knowledge in these projects that have been around for a long time that look disgusting and gnarly inside internally that is very hard to reproduce again without going through the pain again. Merb did this. Merb essentially tried to reproduce a lot of what was in Rails, and what we ended up doing was relearning all the lessons that Rails learned because we didn't have the knowledge embedded in the code. There's a lot of value in looking at code that already exists, and even though it's not the best code in the world, it still has information in it that's hard. Does this work on Windows and how? That's something that if someone started from scratch, they probably would just fail to do until the Windows people bugged them for another couple of years. That's sort of the introduction is I think RubyGems right now has a lot of information and it works really well conceptually, and it's a good starting point for the future. There's a bunch of problems that exist in RubyGems. This is my least favorite of the problems. I think everybody has experienced this problem, and if you go and Google for this... If you go and Google for this problem, you get stuff like this. The problem is somehow related to my having multiple versions of the active port gem installed. I could just try removing the newer versions of all the gems I have duplicates for, but I'd really like to know what's going on, and then never any response to those threats. Because it's conceptually complicated. It's something that once you understand RubyGems works, it's perfectly reasonable what's going on, but it's conceptually complex, and it essentially is resulting from people having a lot of gems installed in their system that are not related to their environment, the application. There's gems in the system that are conflicting. Here's another example. Somebody reported on a bug tracker that they had can't activate RSpec, already activate RSpec 1, 2, and then the result of that was state change from do to resolve can't reproduce it. Because it's a problem that's related to somebody's system. This problem was the impetus to doing bundler to begin with, and actually the precursor to bundler, which is part of Merb, which is just that there is system state that exists in people's RubyGems that is messing with other dependencies in ways that are extremely hard to understand, and certainly the masses of people using Ruby have incredibly hard time figuring it out, and people who are writing gems that have dependencies that are generating these problems almost never can figure out what's going on. That's the biggest problem I think in RubyGems right now is just this usability problem to do with dependencies. Second related problem is just the usage of system gems in general for applications. You might have a dependency in your application for launchy 0.3.3, and it requires configuration bigger equal to 0.0.5. I might have launchy 1.1.0 in my system, and so I'm developing my app, sorry, configuration. I'm developing my app all the time, everything is great, and then I go and give it to you who happens to have an old version of a configuration on your system, and I was depending on some newer feature in my code, and all of a sudden everything breaks, and again, in a way that's not clear to you what just happened, because you don't actually know anything about configuration. You just installed launchy, and when you did gem install launchy on your system, it said, oh, you already have a perfectly valid version of configuration, or there could be any number of reasons why you may end up with 0.0.5 in your system, and then weird things happen. That sort of the symptom is weird things happen. You can never figure out what the weird things are unless you actually know the problem, so this is chicken and egg thing, like how do you figure out what the problem is, what's generating these weird problems. Another related issue is if someone releases a gem between the time you do a test and you actually deploy, so let's say I'm using a version of Noko Geary, and I say my application says I want Noko Geary bigger or just Noko Geary, so I have in my manifest somewhere that tells the deployment server to install Noko Geary, and then in between the time I tested and the time I deployed, they released a new version which broke something, so even though I tested, or ran while I test, I put it on staging, they tested it, everything's great, when I actually go to deploy, the way a lot of people do deploy is that they just reinstalled the gems on the remote system, and now they've updated something that they didn't mean to. Now what people usually say when someone brings up that issue is just hard code Noko Geary, and that actually works for Noko Geary, but it doesn't work well if something depends on something else, because people are not usually in the habit of hard coding the dependencies if they're not conceptually hard coded, and so this is just a general problem of how exactly do you specify your dependencies, and as long as people have dependencies of their gems that are soft, then you have this problem of you tested something, someone doesn't release a new gem and it broke something, and you don't know why that happened, so this is not a problem with RubyGems per se, it's a problem with the way people tend to use RubyGems to deploy, obvious problem is RubyForge being down, which now means gemcutters down, but that's a problem that happens, right, you have, your deploy system requires RubyGems to be up, or I give you a repo, like you check it out of Git, and you go to set it up, and you can't get the gems out of RubyForge because they're down, so that's another thing, it doesn't happen that often, but when it does, it kind of sucks. Another problem is edge projects, and that just means a project that has not yet released a gem of some kind, and an example of that is right now we have action pack 3.0.pre, it's on Git, and rack mount is on 0.1.0, usually the solution that people give to floating gems is just release pre-release gems, which you can do now. The problem is that it's kind of difficult to make sure that you have synced up pre-release gems if you have more than a few dependencies, so there may be a bunch of gems that are being worked on together, on edge, and making sure that you always have all the pre-release gems with the right dependencies set up so that someone can always gem install action pack, let's say, it's kind of difficult when really all you want to say is all the stuff works on head together, right, so there's a bunch of repos, they may be in Git, or Subversion, or some other version of the control system, and they're conceptually being worked on by the same people for the same purpose, and so what you really want to just be able to say is all this stuff works together on the latest release, and you might release some pre-release gems occasionally for people who are not developing to try, but when you're developing it gets pretty messy, and this actually has been a problem for us in Rails. We started adding more dependencies and trying to keep the ones that are on Git in sync has produced problems and has been the impetus to some of the solutions that I'm going to talk about. Another related issue is Git dependencies with gem dependencies, so you may have something that hasn't been released yet, but it has a dependency that's a gem dependency, and so you want to make sure that the general dependency resolution process takes into consideration things that are not gems yet. So again, this is not really a problem of RubyGems, because RubyGems is not intended to solve this problem, but it's a problem with the way people use RubyGems in development right now. So right now when people are developing their gems, they often end up with Git things that they're using as dependencies, and they often have gem dependencies, because they're just pre-gems, right? They're just gems that haven't been released yet, and so making sure that everything is set up correctly, and all the things that you actually need are installed turns out to be a chore, and it works okay if you know exactly what you're doing, but as soon as you don't know exactly what you're doing, it's not really possible to do correctly without people having problems. The reason I know this is that in Merb, before we got a quasi-viable solution, pretty much nobody was able to check out Merb from Git and get it running without trying to follow a bunch of steps that work very hard, and often didn't work. So that's just a general problem of when you end up with a lot of stuff in development. Most people don't actually have this problem because most people use gems, but when people are developing gems, this produces an issue. An example of that would be, I have Rails, some other library that's not Rails decides to use rackmount, which Rails uses, and they're like, I want to use rackmount 0.0.1, and I want to be able to say, I want to use Rails with some lib, and I want to make sure that the rackmount that gets installed is actually the one that works for both of these dependencies. Right now, there's no way for the sumlib, whatever the library is, to tell RubyGems like, I need rackmount 0.1.1, so the other thing that Rails needs, some version that might be looser, make sure you get that one. Sure. Right, so this problem is, you have Rails, which says, I want rackmount bigger equal to, or tilde, weird operator, sperm operator, 0.1, let's say. And you have another dependency that has not been released as a gem yet that you are going to somehow have in your load path. So that might be by checking it out manually and adding if your load path in your rake file or something like that, or somehow you're doing it, but when someone goes to install this in reality as a gem, it's just gonna be a regular gem dependency, it's gonna work. But in development, I have no way to make sure that RubyGems knows that this is a dependency of this git repo, because it's not part of RubyGems. It's just a git repo now. So yes, there's ways that you can work around it, but there's nothing that, this is a common problem of, I have a git repo that I'm using in development, it's not released as a gem yet. Eventually it will be, but right now it's just a repo. Another related problem is Rails and RubyGems. The way Rails works with RubyGems right now, for a variety of reasons, is that it's outside of RubyGems and hacks RubyGems, so you can't, if something requires an active support dependency, even Rails is outside of the RubyGems system, so the active support dependency that, let's say active merchant might have, is not connected to the active support dependency that Rails has. So Rails is active support dependency RubyGems never sees, or might see later or in a different order or depending on how we're hacking RubyGems. So this is actually a Rails problem, not a RubyGems problem, but it's a problem that exists in the world right now. That Rails loads itself outside of the RubyGems system and outside of the dependency system. So that's not good. General problem with RubyGems right now, the tests are very coupled to the implementation, not gonna say much about this other than to say it makes it kind of hard to submit patches, so it's just a thing that exists and needs to be resolved. RubyGems works pretty good as a library, but it's not really built as a library. That's been improved in various releases, but there's still a lot of work to go to make it so that the things that people are tending to do with RubyGems are really easy to do. There's obvious specific examples that I'll bring up later. Runtime speed, the runtime right now is very, is coupled to the rest of the pieces of RubyGems, so there's things in, for instance, the source index that are there for indexing that make the source index slow and hard to refactor that make RubyGems take a second or 1.5 seconds to boot up even more on Google App Engine or JRuby and making the runtime fast would be good, it would solve a lot of problems. So there's some specific attempts at solutions, so projects that are solutions. The first one was Geminstaller, and Geminstaller just tries to say, it tries to solve the deployment problem by saying you can put in your manifest, your list of all your gems that you need. What people realized was that initially when starting to use Geminstaller was that if you specify only the top level dependencies, then you still have the problem of what if Nokogiri depends on something else and then that gets deployed, so when people use Geminstaller today, they end up having to specify all the dependencies and all the child dependencies, which means that you have to know a lot of stuff that you probably shouldn't have to know and you have to do it manually, but it works as a solution, Pivotal has been using it for a while, it works fine. Doesn't solve some of the other problems like your dependency on RubyGems itself at deployment time and it doesn't resolve the dependencies for you, but it solves the, how do I make sure that whatever I tested in my deployment server, in my staging server actually works in deployment. There was an early attempt at solving this problem in Merb, which was Thor and Merb Geminstaller, and what this tried to do was figure out your dependencies for you somewhat recursively, it used a pretty poor algorithm, but it was better than other attempts to do so at the time and what it did was it packed your, it vended your gems inside your repository automatically for you, so the Merb solution was essentially, you say, you list the gems that you need, you run Thor and Merb Geminstaller, it packs your gems and then you could deploy the packed gems to your remote system and then unpack them in the remote system, which solves the dependency problems and knowing what gems you're using solves a bunch of problems. It's still unfortunately because the dependency resolution wasn't good enough, so it ended up resulting in people having to still put their dependencies in an order that was saying, figure out through manual dependency resolution and that kind of, that was the intention was to avoid that, but it didn't actually work in practice. Another solution to the whole problem is just to say, Ruby gems are really just another kind of package, so we'll just take all the gems, unpack them, put them in one directory and let you have an easy way of twiddling the low path, so like eliminate Ruby gems from the picture altogether and what the cool thing about that is that it solves the speed problem because there's no actual Ruby gems runtime that's taking up any time, everything is just in one place. The thing that's not good about it is that it assumes things about the actual, the running of actual gems, namely that they don't require the Ruby gems runtime, that isn't always true and is not true in a lot of cases in specific ways, so a lot of people rely on features of Ruby gems, some very specific ones that make the rip approach not work very well, but would be easy to shim. Right. Another solution is just to have your package manager like Debbie and say, we're gonna take your gem apart, put it into our package system, we'll handle dependencies for you using our native packaging system. Again, that doesn't work so well because people are relying on Ruby gems, they're relying on the directory structure of the gem that they're deploying, so Debbie and people have long had a gripe with Ruby gems and that's because they aren't able to do this, they aren't able to take a gem apart, put the bin somewhere, put the lib somewhere and I'll go into why in a little bit. And then the solution that we now have is Bundler which tries to solve all the problems I outlined. I'm sure it doesn't solve all of them 100%, but it solves a lot of them, a lot of percent and it handles like the get, specifically the get problem. If you may have a gem that's not released yet problem in a bunch of ways that's convenient architecturally sound. The real problem with solving this problem, so it's actually kind of easy to release a solution that looks like it solves a lot of the problems. RIF is the example of that, but when you actually start looking at what people are doing in gems, it turns out that there's all these confounding factors that make it very hard to just say, we'll make a simple solution that solves the runtime problem. So there's a lot of other problems, there's like the install time problem, but the runtime problem is confounded by people doing weird stuff in their gems. So here's some of the weird stuff that people sometimes do in their gems. So first of all, people rescue from gem load error, even though their gem load error inherits from load error, so you can happily do require something, rescue load error, that works fine, that will work whether or not you have RubyGems or some other load path managing system like RIP or Bundler or something else, but people still for some reason do rescue gem load error. And if you do rescue gem load error, and there's no RubyGems installed, everything explodes unless you have shimmed rescue gem load error. Even weirder, people rescue from gem exception, which means they're doing some arbitrary RubyGem thing. Usually they're actually not doing that. Usually they're doing some very specific thing, but they're rescuing from gem exception. People think that instead of putting their dependencies in their gem spec, they'll do them at runtime. So instead of saying that my gem depends on foo, they'll say gem foo at runtime, which means that instead of RubyGems installer being able to resolve dependencies for you, you get runtime exceptions, like that gem does not exist. There's one case where it's actually needed, which is there's no support for optional dependencies, so you can't say, I want JSON, I want JSON, I don't need JSON, but if JSON is here, it has to be this version. Rails has this problem, this results in us having to do this, and we can shim this system. But a lot of people who use gem just use it in lieu of actually putting it in the gem spec. And they do this because they're not having to always release the gem spec, and they're doing development stuff sometimes, but this is essentially something that asks the RubyGems runtime for information that not all runtimes will have, right? So RIP and Debian don't have this information, so these gems just don't work on those environments at all. This is a more common thing that people want is gem.ruby, like what's the command that will execute the same Ruby that I'm running? This is actually valid and probably should be something that Ruby gives you, it should be like Ruby.self or something. That tells you what the command is, and it's actually pretty easy to shim because it's just like a few things in RB config that you have to concat to get the thing, but most people don't want to go dig in to figure out what it is, so they just use gem.ruby, I do, it's valid. Kind of sucks. Gem.dir is, gives you like the directory of the gem, I think Sinatra uses that. This came up recently after we shimmed a lot of these other things. There's a few people that use gem.dir and gem.path, it's like, well, I don't really know why you're using that, but people do. Gem.loadedSpecs, I wanna know if this gem is already loaded for some reason, and gem.sourceIndex to be able to search for gems. So all these things are not things that you really want conceptually need to be doing inside of your application most of the time. There are a few cases for a few of these that are valid, but the vast majorities are just people using RubyGems' features for no obvious reason. And the thing to realize about your package is that before there was RubyGems, there was Ruby, and you were able to take your package and add a load path and it would work, and RubyGems is basically just automating that process, so you can package it up, and RubyGems will handle that for you. But there's no reason why, if you reverted back to the old system of just twiddling the load path yourself, why your package should not work. Your package should still work, and if it doesn't work, you're doing something wrong. And the reason why this is taking on some additional urgency is just that there are people now who are trying to do this. There's Debian who's been trying to do it forever, and everybody just laughed at them, but now there's, or grumbled at them. But now there's RIP, which is trying to do it for real, and there's, Bundler would like to be able to do some more intelligent things that we can't do because people are doing strange things in their gems. Even worse strange things are this idiom, which is, go get all my gems inside of, all my Ruby files inside of my current directory, which works fine if you could assume that you control your current directory, but there are package managers that wanna take all the gems and put them in one load path. RIP is an example of that. And if you're just requiring all the Ruby files, then you can't, your Ruby can't live, your Ruby files can't live next to somebody else's. People do this a lot in their binaries. They'll go up one level and find Lib and then require your library. This means that you can't take the binaries and put them somewhere else from the Lib path. So basically what is essentially happening is that people are relying very much on the development time structure of their libraries and not giving latitude to the package manager to do same things. And it would be one thing if there was a really good reason for this, but in practice, you can rely on whatever package manager you're using, if it's RIP or RubyGems or Debian package manager, you can rely on it to put things in the load path that belong in the load path. So if your things are in the load path, you can actually just say require my library. You can rely on that working. And if in development time, you don't have a package manager yet, you can set the load path yourself. That's very easy. And it basically gives a lot of flexibility to both people like RIP who are trying to do interesting things and RubyGems in general in the future that might wanna do interesting things. So we're kinda stuck now because packages are doing very specific things. And even worst problem is people who put server.rb in their load path. So now they go and they say require server. And who knows who else did that? Like very generic names like initializer. So people put really generic names in the root of their load path and then it could easily be clobbered by some other gem that gets installed that does the same thing. So this is very bad. You should always have a directory with your package name inside of it so if you have your gem, you should have a your gem directory. You can put server in there and then your gem.rb can say require your gem slash server. Very easy solution solves the whole problem. And yeah. Okay, so with given all of those confounding factors, we went back and we wanted to solve both the first set of problems without running the foul of the confounding factors. So the confounding factors, we decided when we started the project they make us emo, but they're like, all right. So our philosophy is the project has to work today. It even has to work with Rails 2.3 which does really weird gem overriding hacks. It has to work for gems that exist today in real life. And you could instead build a system that works with magical unicorn gems that we hope everybody writes. But in practice that will result in a lot of people having to spend a lot of time tracking down a lot of gem authors and changing gems. That doesn't actually work. So we decided to stick with make it work with gems that exist today. So things that we actually did. First of all, automatic dependency resolution. That just means that instead of resolving your dependencies one at a time as you specify them or as you encounter them in the installation process you tell RubyGems or Bundler the list of all the gems you actually need and we will do it at once. Get you all the dependencies that you need. And that completely solves the unable to activate this or already activated that problem because that happens because you're doing them one at a time. If you do it all at once, you can't have that problem. Now you either have a, it works or you have a conflict. Like there are these two things actually conflict with each other. And here's an example of like how it looks in your, there's a gem file and you say, here's the gems. So here's the top level gems. This is in gem cutter. Has these requirements in top level. And then when you actually run gem bundle you can see that there's like net SCP requires net SSH rack maintenance required rack 1.0.1. So it went and downloaded the additional gems. And there's more that says, we'll say installing those gems. But the idea here is that you just say what gems you need at the top level and we do a somewhat sophisticated attempt to figure out what the resolution is. For git, we basically just said git, you can treat git as, if git has a Ruby gem in it you can treat it as an unreleased Ruby gem. So as far as the gem resolver is concerned it's exactly the same as a gem. And so you can do, this is part of Rails gem spec. You can say gem arrow 0.2.3 and then you can point it to git repository. And what that means is that inside of that git repository is something that is equivalent to the gem arrow 0.2.3. And we will even take that into consideration when doing dependency resolution. So if the Rails gem that may actually be released as a pre-release gem says that it needs arrow 0.2.3 the fact that you said that this git repo is 0.2.3 will correctly resolve it together. If the git repo actually has a gem spec in its root then you can leave off the version, you can leave off 0.2.3 and we'll pull that out of the gem spec. So you can specify everything explicitly or we'll just pull it for you automatically. We also support directories of gems. And basically the way that works is that you can say Rails is in the Railsize path. And then what Rails will do, what the bundler will do is it will check us here, there's any other gem specs anywhere inside of that path. And it will create essentially another gem source. So the equivalent of what is a gem source in your system. So you have a gem source in your system, you might have a remote gem source. So we'll make a equivalent of a gem source that represents all the gems that are in your git repo. And then when we go to try to resolve the dependencies against the gems that exist, we'll include the ones that happen to be in paths that you specify. So this is just abstracting the idea of a source from being something that is just a remote source. It has these requirements to an API that you can specify. Here's how you can get a list of gems and here's how you can download them. Essentially how this works is that there's a gem resolver. The gem resolver takes in a bunch of dependencies along with a bunch of sources. And here's the sources that are available but they're just an API so you can specify it. However, you can specify additional ones if you'd want to. And you can also specify for a given source, here are gems that have to come from that source. So you may have like 10 sources, but you might say I would like rails to come out of the git source. I don't want you to pull a newer one that happens to be on Rubyforge. I want this one from git. In the current version of Bundler, the DSL does this automatically for you or will really soon. If you say, I think it does do it now. If you say gem whatever, git, and you pointed a git repo, it'll automatically lock that gem to that git repository. What was happening was that people were specifying git repository and then it was still pulling versions from other sources. So we added this feature of being able to lock a dependency to a source out of an actual need that people were having. But it would be cool to be able to do it in general. Like, yes sir? Can you lock to a tag or a specific shot? Yeah, so to get supports pretty much any ref. So tag branch or shot, yeah. Ruby, the Bundler is item potent. So if you go halfway and then do the other half of the way on the deployment server, it'll still work. Yes sir? Does this work in 1.8 and 1.9? Works in 1.8 and 1.9 and JRuby, except the JRuby cannot compile gems. So the parts of Bundler that handle compiling native gems obviously don't work on JRuby. But it works on all Ruby's that we know of. Yes sir? Can you say item potent? I have to disagree because there could be an ex-cubal Ruby code in the gem file that's definitely not item potent. Can we get a Bundler format that is pure yaml or pure text that can be parsed to create the same thing so I can bundle some of that without loading arbitrary Ruby codes? Yeah, so we should talk about it. I know you guys are working on that. But assuming that you don't have arbitrary Ruby code in your gem file that does random stuff, assuming that you're using the DSL in the gem file. I don't know, I don't do crazy things. If you're like a hosting provider, then it matters that people are not putting arbitrary executable code in there because it could be an exploit or something. So that's understandable. When I say item potent, what I just mean is that if you run through the entire bundle process and then you get ignore or remove the expand. So what the Bundler does is it goes and it resolves your dependencies, downloads all the gems and then expands them including installing native dependencies. What happens if you delete the expanded versions is that if you run gem bundle again, it doesn't go and read download all the gems again because it notices that it has them in the cache and it just re-expands them. So what that means is that you could ignore the expanded versions in your repo and push the entire repository, run gem bundle on the remote system and not have to worry about remote dependencies or spending time re-resolving because it'll notice that everything already still works. There's some other cool features. You can disable Ruby gems. And what we've done here is we've shimmed certain really common things like gem load error. So if you rescue gem load error, we've shimmed that and we've shimmed the top level gem method. But there are still tons of cases like Rails 2.3 cannot run in disabled Ruby gems mode because it still does stuff to monkey patch Ruby gems. So this is something where it's like, this would be a good mode if everybody could run in this mode but we know that today a lot of applications won't be able to run in it so we're making it an optional thing where you can say this and hopefully long term it will be something that most people will be able to do most of the time. More likely something that you would want to do is disable system gems, which just says when you require something, don't include the system gems. Only include the gems that you specified and bundled in your application. And this just means that you can't get into a situation where I'm on my machine, everything seems to work but when I actually give it to you, it doesn't work because there's a gem that I've installed in my system that you haven't installed in yours. Google App Engine is actually already starting to ship with disabled Ruby gems mode so they've taken the bundler, they bundle, they take all the gems that are expanded, put it in a jar and then make it available to your application and they're running without any Ruby gems at all. So that works fine as long as there's gems that don't do strange things and they're the ones that reported the Sinatra using gem.there or gem.path us. Okay, the architecture is really simple. Add some cool stuff. There's basically four big pieces. There's the CLI which is what happens when you say gem bundle. The environment has sources and dependencies and can install cache or list the listed dependencies. That's basic, the environment is basically just the, it's the equivalent of the manifest so it's the object version of the gem file. There's a source which is just, it has gems and you can download a spec and that's any remote system, you can have a remote system or a local directory or anything that supports that API of having a gems method and downloading specs. And there's also a repository which represents the local place where it's cached and so you can have a repository that's your system gems or a repository that's in your local, in your application itself and that has a path and gems and it can install a list of dependencies for a list of sources. Obviously the resolver can resolve so it doesn't really do anything other than take a list of things to resolve and give you back a list of things that you should download. So the nice thing about this is that it already works today. There are projects already using it today including Rails itself. There are gem cutter, it uses the bundler on Rails 2.3 today and there are other applications that are using the bundler on Rails 2.3 today and in Rails 3.0 it will be built in. I know I'm abusing some of this space over here. In Rails 3.0 it'll come in, it'll come built in so you'll be able to do script bundle and it will do all this for you and what we've been able to do is eliminate all the monkey patching of what we're doing with RubyGems and all the work we're trying to do to like re-implement features of RubyGems by just using essentially load path twiddling. The nice thing about the bundler is that all it really does in the end is create a file for you that sets up the load path and optionally will require RubyGems. So depending on whether you've enabled or disabled RubyGems it will require RubyGems but it puts all the information about the path twiddling in a file that will always work. So with all that said I just wanna talk a little bit about what I would like to see in a future version of RubyGems based on what we've done. The first thing is just conceptually there's like this glob that's RubyGems now and it's actually four things. I have at least four things and the server's already broken out in gem cutter. The indexer is actually separate and the installer runtime are really two separate things. So the code that actually runs RubyGems should be optimized for performance and startup time and that may conflict with installation time needs and so the code that actually goes and installs gems turns out to be actually separate and we've proved that because Bunder actually does that. Bunder installs completely separate from runtime and it works and the architecture of RubyGems right now can accommodate that not without much work but it would just be nice to make it explicit. Like this part of the code is the part that installs. Here's the runtime. Anything that's in like the source index object. There's the source index object that's used for runtime and there's the source index object that's used for indexing or installing. What ends up happening now is that there's some objects that like the source index one sticks out like a sore thumb that has a lot more in it that's used at runtime purely to support these other environments. So getting to the point where we have mainly getting a sleek runtime would be good. Architecture, I like what we've done with Bunder. I think the separation of concerns is good but obviously RubyGems already has a pretty sane architecture in a lot of places so that's fine. I think there's some things like the repository object that are really convenient and that makes sense to be patched to RubyGems. For testing, it would be really helpful if the tests in RubyGems were testing the API that was meant to be used, exposed, and the CLI and specifically avoided trying to test internals. That's actually really legacy. Like the fact that it does, that it tests internals right now is legacy in a big way but it makes it kind of hard to make modifications that don't modify the public API but modify internals. A problem that I've had a lot is that the source index object exposes its internal representation of gems to the tests. So source index has a gems method on it that returns a hash but nobody uses that in reality. It's not the API but the tests use it to be able to figure things out. So it'd be good if the tests were actually testing the public API of the source index and not the internal representation. Basically makes it impossible to make the internal representation more efficient without a lot of rewriting. I really like our source API that we did in Bundler which basically just extracts the idea of what a remote source is into an API that can be reused and re-implemented. So I think RubyGems already tries to do this with the fetcher idea but it's not exactly the right abstraction and I think what we've done in Bundler shows that it works in practice for a lot of different cases. So we have a directory source, a directory of gems, we have a git repo, like we've already made it work in a lot of cases that people have actually needed so I think it works. As I said earlier, I think RubyGems should be places where it's not easy to use it as a library that should be resolved and again that's something that people have been working on for a long time so it's kind of lame for me to throw it out like dude you should fix that. But it is the case that there are parts like the factures in particular are very hard to use as a library. There's a lot of hard coded details and looking at them in terms of how people have tended or would want to use them is good. I think this is a no-brainer. I've spoken to Eric about this a lot actually over as I've been working on the problem and have not felt comfortable with the solution enough to get it back into RubyGems but RubyGems should have dependency resolution built into it. It should be able to take a list of gems and a list of sources and give you back a list of the dependencies, a list of the actual specifications that you should download and that should be built into how RubyGems works. There's a ready API for that called GemLock in RubyGems that is just for giving it a list of things and it will give you back basically a list of gem commands you can put in your application. That might be a good place for it but that should definitely be part of RubyGems and runtime performance easy to, actually pretty easy to fix. There's a few possible solutions. One of them is just go to disable RubyGems approach like try to find a way to make it so you don't need RubyGems at runtime at all. There's the rip approach of trying to collapse everything into one directory structure which requires community help and then there's just less invasive approaches of just improving the on-disk format and the internal representation of gems so that the source index is not very slow. Right now the source index, if you say I would like MerbGrader equal to 1.0.1, it has to loop through every single key in the entire source index to check this because if the key is Merb-1.0.1 so it has to go through each key and check to see if it matches Merb and see if it's the right thing or it might now go through all the values and check to see if they have the right name but the point is that it has to go through every single entry. There's no, it would make more sense to have top level keys which are the names so if you want MerbGrader equal to 1.0 it can index in, get the list of things that are Merb and then have a smaller space to search in. What this also means in practice now is that you have to load every single gem into memory so that you can search it and that's the cause of it taking a second and a half to boot applications that use RubyGems. Virtual environments are a useful thing. RIP did that with RIP and VirtualM. We kind of like Bundler handles that by giving you a local virtual environment. Being able to say like I would like to use my name thing. This is already using RIP. RVM has something called name gem bundles. Bundler does it, it seems to be an obvious need getting into RubyGems would be good. Being able to specify extra gem spec properties. This is kind of a minor thing. That's one way to do it, just say like you can specify whatever you want. This is a more likely scenario. Just say like here I can put in some extra properties so other random people who are using RubyGems can check to see if some property that they're agreed on is there or you might be able to provide more details but just some extra space to put things in. The idea of per application requirements in general, like my application actually has these dependencies as opposed to it just being there's a big system gem that has everything in it. Bundler handles that by bundling it but some notion if it's just something like the gem file or something else that lets you say this application has these requirements and could hook into the rest of the system, the dependency resolver, the runtime to actually do things usefully for your application. The API that we've built for, that we've done right now in Bundler, the three pieces of it, I see the dependencies piece of it being a thing that is okay to have as a separate piece like the gem file and the DSL that can be managed by something like Bundler but the resolver and the repository concept I would like to see in RubyGems. So the idea of, like I said, dependency resolution and also that there's an object that represents the actual location on disk where your gems are stored that you can ask it for all the gems, you can ask it's location, you can ask it to store a gem in there as opposed to it being a raw API would be cool. Optional dependencies, I wanna explain what this actually means and what people actually need optional dependencies for. You wanna be able to say this, I can use JSON but I only want greater or equal to 1.4. So what ends up happening is that people don't wanna specify JSON as a dependency because it's not but there's some code in their application that allows you to opt into JSON behavior but it only works with some version and the only good solution for this right now is to use gem in your application, to say gem JSON greater or equal to 1.4 which means that you can't actually use other package managers that don't have gem. So being able to say this explicitly and have it be something that downloads that is there during resolution time would be cool. Yes? Because you have no idea what version. Right, so if you just say require JSON, you don't know that it's the right version but that if you didn't care, that would be fine. So optional dependencies where it's just like I may need JSON, there's a good solution for it right now. Right, if you do that then you're relying on RubyGems runtime which is not ideal. There's a bunch of people that would like to not and basically like what I would argue is that gems should be things that work just fine without RubyGems. They're just Ruby packages and the RubyGems runtime makes it easier to manage the low-taps. If you start relying on RubyGems features at runtime then you can't do something else. Yes. Google App Engine is an example somewhere where they actually can't use RubyGems, it's way too slow in their environment. So, virtual dependencies is also controversial. It's essentially being able to say I required, my app requires JSON, this app provides JSON without it being a name. An example of this might be that MIRB admin says I require MIRB core and then like some other thing says I provide that so you could end up with being able to, the obvious case is namespaces on GitHub but there could be other cases where JSON pure might want to say I provide the JSON API. In general, there's a thing that we need to do as a community that can't really be done as part of the effort in RubyGems which is just better best practices like people stopping to expect the RubyGems environment being available in their library so that we can actually have people experiment with other packaging systems. I know obviously RubyGems as a packaging system is here to stay and it's the way that most people will continue to develop things in the future and will be the most robust solution but there are definitely environments like certain deployment environments or development of certain types of applications where it's convenient to be able to say this is just a package, it's made up of Ruby, it uses the RubyGems API, RubyGems packaging format to store it but at runtime I don't actually need RubyGems, it's just a bunch of Ruby code and getting to the point where that is actually viable requires an effort by all of us to just, if we notice that people are using RubyGems features, rescuing gem load error, doing dirt glob and then requiring entire sections of code or putting server that are being their root, any of these sorts of things just telling people not to do it. Gem is one and it enriches my environment as a developer and as a result I have a very facile plugin system that I can use via gem find files and it's made my code better as a result and by taking that out means that I have to reinvent the wheel for a plugin system every single time I make a new package that is expandable. We should talk about that because I have apparently like 30 seconds but I think that's an argument worth having. So best practices I think we can all agree not to use RubyGems, we can have a longer discussion I'm sure there will not be a blog war about this. We want to get to a point where people are able to actually experiment with things to improve real scenarios that exist in real life and real applications and that is something that is still at this point a somewhat unicorny thing not because of gem plugins but because of people rescuing gem load error which I think I saw Eric shaking his head when I mentioned it as an obviously bad practice. So there might be some cases where we can discuss on the edges that are still where RubyGems you kind of need RubyGems if you wanna support this feature but I think for the vast majority of cases people do not need RubyGems and it would be a better world if package managers that were not RubyGems not because they want you to not use RubyGems on your system but because there are specific use cases for doing something different we're able to treat RubyGems as packages of Ruby code. Thank you very much. I don't actually have time for questions but. Thank you.