 title is a little bit of a lie, but hopefully the talk is still interesting. Solves the problem without solving the problem. As I said, I'm Andrew, I work at Stripe, I'm on our product infrastructure team, which broadly speaking is responsible for the core developer experience in our main product code base, which is almost exclusively Ruby. And that for us is not just about developer tooling, it's also about kind of some of the core abstractions and how the code fits together, how people are actually authoring code on a day-to-day basis. To give you a sense of what we're working with, this main body of our product code, you know, the entire API is in a single monorepo. It's a few million lines of Ruby, and it's kind of a macro service architecture. So the API itself is a single service. Most of the logic that goes through, say, creating a charge of Stripe happens within a single service. And one of the goals on our team is keeping the iteration loop and development very tight. So I edit code, save the code, run a test on the code, and don't make a test request in development. I want that to be very, very fast. And with Ruby, that's not slowed down by something by compiling. That's slowed down by just loading the damn million lines of code every time you hit the save button. So originally the word on here was not reloading, it was compiling. Our problem was reloading. People were waiting 30 seconds, 35 seconds every time they hit save before they could run a test, before they could make a request against the API. So that's a frustration drag on productivity. And that actually comes out of the fact that the code base itself was this kind of organically grown monorepo with really tight coupling between all of the components, which meant that if you touched one component over here, there was really no clear way to separate it from having to reload all of the components all across the code base. And that's a pain in development because it makes the iteration loop slow. But it's also a pain because you have to understand every component of the code base in order to understand what you're doing here because they're all so deeply entwined. And so I'm going to talk a bit about a project that sort of helped us solve both problems. Before doing this talk, I wanted to see sort of how unique is this problem to Ruby. So I did a completely synthetic benchmark that is, you know, a lie, but a lie that tells my story. And I found that Ruby, at least in my benchmark, is really slow at loading code. I basically generated, I used some templating to generate a few million lines of more or less equivalent Ruby, Python, and JavaScript and loaded them up and profiled a bit and said, okay, what are the times here? Python was actually the slowest from a cold start, but they have this PYC thing built in that does byte code caching and so it ends up being the fastest for kind of incremental work, which is most of development. JavaScript was the fastest from a cold start. I'm going to make things up and say it's probably because V8 is a really great VM and Chrome kind of has to optimize for this, like gets handed a whole bunch of JavaScript over the wire, go run it fast. Ruby is pretty painfully slow here, which explains the bind we got into, of people waiting 30 seconds to reload code. But there's actually something kind of cool with, I think after Ruby 2.3, you can use an instruction sequence to precompile to byte code and then actually load that up. We're not doing that yet, but there's this library called BootSnap by Shopify that does some of this and some other load time optimizations that are pretty cool. And with some of those at least again and some synthetic benchmarks, that makes Ruby quite a bit faster. I had this theory that our heavy use of DSLs and Ruby's tendency to use a lot of DSLs and metaprogramming at load time was to blame here, that it was just running a lot of code, but at least in a synthetic benchmark, that really didn't seem to be the case. It was really just loading a lot of code is slow. And what we did over the course of about six months is we took the code loading process at Stripe that took about 35 seconds and brought it down to five seconds by building an auto loader for our entire code base. And this had the additional advantage that developers never have to require a relative anymore. You just type a reference to something and it kind of magically works for you. You never have to declare anything new when you want to reference something new. So I'm going to talk a lot more in this about how exactly we made that work. So if you look at this example, this gives you a sense of why things were slow to load before. So I want to boot the API here and call the make a charge endpoint in development. I have to load all three of these files, because I'm going to need them at some point. And so I'm going to kind of follow all these required relatives as I load these files and have to load, you know, till to the entire code base because it was tightly coupled. But really to boot the API to like get listening on a socket, I don't need to be loading either of these two files on the side. I only need to be loading API.rb. So for the charge request, I don't need to be loading this refund.rb file. And so the key of making our code loading fast was to just do a lot less of it. Don't do the code loading you don't need to do. And this actually ends up being kind of a theme. My job is finding ways to unhave problems. These making code load fast is hard. Doing less of it is easier. And so what we ended up doing is writing an autoloader that auto-generates stubs that look a little bit like this. This simplified view, I'll get into kind of the details of some of the things we end up having to do to handle some edge cases later. But the basic idea is we have a build daemon that's running in the background as you develop, and it goes and generates these stubs every time you save a file, for example. One really nice property of autoloaders is this stub only changes when you change a definition, when you like add a class or move a class to a new file. And so for the most part, this build daemon can just kind of keep running and keep reloading things, and it's not actually changing much that often, but we can just optimistically let you keep coding, let you keep running tests, and only slow you down when something really relevant changes. So for the most part, you can just ignore this thing running in the background and just have your code sort of magically load when you need it. And actually generating this file is not terribly hard. Once you parse all of the files in your code base into an AST, finding the definitions isn't all that difficult. They're pretty clear nodes in the AST. The names aren't all that hard in most cases. And so you have the definition information needed to know that a thing called charge is defined in this file called charge.rb, and so tell Ruby's built-in autoloader, like if somebody tries to reference this charge thing, go over there and load charge.rb. It's really nice that Ruby provides a built-in facility for autoloading, otherwise this project would have been a whole hell of a lot more difficult. But we want to do a bit more than that. We didn't want to just understand definitions in the code base, we wanted to understand all of the references in our code base as well. And so the idea is you can see on the left here these definitions of the module out, the module mid, the module in, and these various values of mod val. And we wanted to not just understand those definitions, but also understand these pointers, understand that when I reference mod val on the right here, I'm actually pointing to this one at the top left. And we wanted to do that across the entire code base. And that gets us a couple of things. One is it gave us a lot more confidence that the analysis work we were doing was correct. Because if we could go through the whole code base and find that something that really should be getting hit in production all the time isn't resolving, then we clearly have a bug somewhere. And we found lots of bugs actually both in production code and also in our static analysis code in the process of doing that verification. This also let us build, add a lot more things to the build daemon that I'll talk about a little later that let us replace metaprogramming that's hard to analyze with automatic code generation that lets us sort of switch over to an autoloader safely because it understands what we're doing there. It lets us do preloading. So if we understand these references, then we understand every file that a service could possibly use. If I give you an entry point file, I can traverse that graph of references and say this is the complete set of everything that will ever be read in the lifetime of this service. And that's cool because at the beginning of deploying something to production, I can just load all of those things so that you never get a request that hits this weird, cold path that has to load up a whole bunch of files and is awful and slow. And finally, towards the end, I'll talk a little bit about some of the tooling we've built on top of this. You can build interesting things when you actually know the complete dependency web of your code base. So how did we actually do this? How did we do this resolution? The first thing we tried and the thing that I spent several months of my life on was loading up all of the code in the code base and poking the Ruby VM and being like, hey, if you're in this context, what would this resolve to? It turns out that arbitrarily loading a few million lines of code is hard. There were all of these, like, ordering assumptions and how our code loaded that had just been, like, hand-coded into where the required relative sat before. There were required-time side effects. So we had these cases where somebody would be in development just kind of hacking on some script. They wouldn't put a script garden. They'd just sort of hack on some Ruby code in their source tree, and all of a sudden they're like, what's going on? Why is this code executing? Where are these files being generated from? So that was not good. So we switched to trying to do this mostly statically. So doing this mostly from a view of just we have an AST of all the files in our code base. How do we actually get to resolve references from them? And that meant mimicking how Ruby does reference resolution without actually running this code through the Ruby VM itself. So let's talk about how Ruby does this. Ruby, one of the first concepts that I really struggled with in Ruby is this concept of nesting. This is a Ruby audience, so I wouldn't be surprised if a bunch of people aren't really comfortable with this. This still bites me, like, at least once every three months. The strange thing about nesting is it means that these two cases on the right, ModVal actually resolves to a different thing depending on whether you split that out and mid-in on separate lines or leave it on the same line or on the left where it's all on three different lines. That sort of structure of the code that a reference has nested in affects what it resolves to, because it walks up that chain. It walks up, is it in mid-in? Oh, yeah, I found one there. Is it in out mid-in? Nope, there isn't one in there. I can't resolve that. And so this was one of the ideas that I struggled the most to really internalize. But something that's actually pretty easy to deal with through static analysis is to see the structure of the code on a profile basis. The other important concept for resolving references in Ruby is a little more familiar, which is ancestors. And after you get past that sort of left-most side, so if you're resolving a long chain of things, this, like, child colon colon val, a child is going to use nesting in ancestors, but everything after this is going to resolve using ancestors. And so if you look again at this thing on the right, val resolves to foo because mix over here is a parent of child, is an ancestor of child. And so Ruby just walks the list of ancestors and tries to resolve that component. So that gives us enough to talk about kind of a rough sketch of what we do in this build daemon to generate our auto loader. The first thing we do is parse all the files in the codebase. This is actually pretty nice because it's very parallelizable. You just go and spool up a bunch of processes and have them spit out a selected portion of the AST or selected information from the AST of each file that describes what we need to do this resolution. So we collect information on all the references. So what are the names, what nesting do they sit in, all the definitions and any of their ancestors. So we try to parse out includes to figure out where you've included modules, parse out the parents of classes. So we get a sense of what are all the definitions in the codebase, what are their ancestors. We also do some special parsing for odd edge cases like aliases. If I assign a constant to another constant, I need to know how to sort of follow that along as I do the resolution. And we parse out require statements too. I'll talk a little bit about how we deal with gems in a second. So once we have that, we have this sort of parse, we've parsed every file in the codebase, dumped it into a SQLite database that just sits in a build output directory. We then construct an in-memory representation of all the definitions. We call this the def tree, the definition tree. And at that point, as I said before, we need for an auto loader definitions. And so we go off and we split off in parallel. We build the auto loader, and once that's done, people can start loading code to their heart's content. But the fun part is we keep going and do some of this reference resolution. And the way we do that is we just loop through repeatedly and try to resolve as much as we can. So sometimes resolution actually itself requires resolution because maybe I'm aliasing something, but first I need to resolve the other side of the alias to even know when I'm aliasing through. And so we just repeatedly go over all the unresolved references and kind of try to make one more step towards resolving everything. When we can't do that anymore and we've run out or not resolving anything successfully, we then go and require all of the gems we've found by parsing those required statements. I told you that it wasn't quite true that we do this all statically. Gems aren't easy to analyze statically. They could use a bunch of C code. So there might not be any ruby to statically analyze. They could be doing crazy metaprogramming that we simply can't analyze. And so we cheat a little bit here and then we go through all the gems and assume correctly that you can just require your gems. And then we sort of poke them with a stick with const to get and say like, hey, do you have this constant? Yes, I do. Great. And again, we go through this loop until we either resolve everything or no longer can successfully resolve anything new and later throw an error at you and say like, you've made a mistake. There shouldn't be unresolved references in our code base. I'm going to go through an example here of how this works. This is an example of sort of using purely nesting-based resolution using a definition tree. So we go and construct this in memory tree and then we walk each step of the nesting. So the first step is all the parts out, mid, and in. And so we walk down from the root, we go to out, we go to mid, we go to in, and then we look there and we see, is there a mod val here? Nope, there isn't. Okay. Go on to the next step of the nesting and then we walk down the tree again, so we've resolved this reference. So let's go to a bit of a more complex case here. So there's two parts to resolve at the bottom left in child, colon, colon, val. There's the child part and the val part. And the child part resolved is much like before through nesting, so you can see we try these two different nestings. We find the one that actually works that actually has a child in it. And so great, we've got that first part. And now how do we resolve the val part in the context of child? So that one resolves via ancestors. So we look through kind of one by one, each of the ancestors of child. Note that you are your own ancestor. Like if you call ancestors somewhere in Ruby, you get yourself on that list. And so you start with yourself and then you sort of work your way up the ancestor chain until you find a successful resolution. So those two components, nesting and ancestors, and that way of walking through that, cover just about everything with regards to reference resolution that we do. But there are edge cases, lots and lots of edge cases. Taking this and starting with this in a code base from scratch probably wouldn't be awful because you can build the code base in a way that's kind of safe for an auto loader, safe for static analysis. You'll discover that it's not safe because things won't load when you expect they do in development. We didn't have that luxury. We had millions of lines of code. We needed, at some point, we were going to hit the deploy button and we really, really, really hope we didn't screw it up. And so there were a few strategies we had to go through to make this safe to kind of make the code base work. And there were three broad categories. One was where we could, we handle the edge cases, actually make them work. We ban a lot of things with RuboCop. We love, love, love RuboCop and any complex pattern that we think is hard for humans to understand and hard for machines to understand, we ban it. And when all else fails, we have to fail loudly at run time so that at least you don't have kind of subtly undefined behavior you have sort of predictable failure. So I told you before that the auto loader stub I showed you was a lie. This is something that's very close to what we really do. And I'll walk through each of the parts of this and talk about some of the edge cases we handle in here. And this is an example for the most part of, well, no, there's some bailing loudly and some handling things in here. So you notice in the middle here is a pre-declaration. In all of our auto loader files, we pre-declare the constant. So if this was a class, we would actually be pre-declaring the class name with its parent. And the reason we do this is that if Ruby gets really angry, Ruby's auto loader gets really angry if this file finishes and that constant is not defined. And so if you're in some kind of weird dependency cycle where you've already started requiring this db.rb file at the end of this file, this thing has to be defined. So we pre-declare everything which helps a lot with dependency cycles. But what happens if the parent class for this is actually defined in a gem? You need to be that ready in order to pre-declare it. So at the beginning, before we do anything, we pre-load all of the gems that are actually required in the file that we're going to load later. This on auto load bit is a nice callback hook because it makes logic before anything starts auto loading before we actually create any constants. We do this, for example, to catch unexpected auto loading. So if you're running in production and you try to load test code, this is going to raise and fail loudly instead of doing potentially really awful things in production. Getting down to auto load map, this is kind of where the meet actually happens. This is where we go and call out to ruby's own module.autoload. It's going to go ahead and require this file because otherwise, if charge is already partially defined somehow, for some reason, ruby's auto loader will never fire because it's essentially like it's, I think, internally actually uses the same machinery as constant missing. So if there's never a missing constant, you're never going to load this file and you're going to get weird in between behavior. And finally, when we actually load, is where we actually require the file that really defines the behavior we wanted in the first place. And we do some fun things in there with method, I believe, method missing and a few other hooks to detect cases where people are trying to use classes that are partially loaded. So if you have two classes that have weird require time dependencies on each other and one is depending on some mix-in that gets required partway through the other one, we'll raise loudly because with an auto loader, you can't guarantee it gets required in a particular order because it's going to depend on how things get hit in code. So that's what our auto loader actually looks like when it gets generated. One of the things that we found a lot of in our code base are places where people depended on require time side effects. They sort of assumed that particular files would get required when an application booted and depended on, say, registering instances of a gateway with some router class. And then when I actually tried to run our code with the auto loader, is all of the charge requests failed because all of the different ways we have of talking to card networks were defined in these registration mechanisms that just assumed you'd required the file and they got to register themselves back with the thing that told you where to route each charge and that doesn't work if you never actually explicitly reference the thing. So the easy solution here is just force people to write a list. You want to write another handler, and add your thing to the handler list. It's nice and explicit, but it does require making sure you change things in both places and can lead to some kind of subtle mistakes. And so one of the things we built into our build daemon, the same thing that does the static analysis and builds the auto loader, is the ability to output other information as a consequence of the reference resolution we did. So you can output a list of all of the subclasses of a particular class, for example, as generated code. And so your router can have all the initializer, for example, that just references this and says, look, I want all of the handlers. I want all of the subclasses of, like, production handler. And then somebody doesn't have to think about kind of adding those manually. And so we were able to repurpose a lot of the static analysis to do code generation that makes it both really easy to use the auto loader and safe to roll it out. And some things are just not okay with an auto loader. If you're using const set, we are not going to be able to figure out what constants you're creating, if you're using const get, if you're using object space, we're going to be able to do that. We wrote Rubikops or you, I think we had to write Rubikops for these to ban this stuff outright. It took several months with several people working on it to actually remove all of the cases that this stuff was used in the code base. But that was time well spent, in my opinion. Like, these sorts of constructs are just as hard for humans to think about as they are for machines to think about. And so I'm sort of happy to be rid of them. A few more we ran into is there were cases where we reopened classes. So we used Sinatra for all of our routing. And the way we did routes in a lot of our Sinatra apps was we just had, you know, say 40 files, each of which reopened the base app and added some more routes. And we had a list of requires that just had to run in just the right order to make things behave properly. And by enforcing that you are not allowed to reopen classes, you are not allowed to reopen constants in our code base. We can force people to be really explicit about, I am referencing this particular thing here. I am say including it here in this order. And we can analyze that and understand that. Another thing that this one really threw me for a loop when I found an instance of this in our code base and it took me a while to figure out. In this middle section, you would really think, you would really hope that Val there tries to resolve to the other foo, but it doesn't because the second module there, the foo on the left actually resolves to the other foo. And so really this resolves to foo colon colon bar. And so we had some weird places in our code base where I think people were actually monkey patching core models that they didn't think they were touching because of weird resolution. So some of this stuff, we just I don't want this in the code base anyway. And finally, there are some things we just can't accept dynamically. So as I said, our ability to understand the ancestor chain is really important to resolution. And so if you're kind of dynamically prepending modules, dynamically doing includes, that makes it really hard for us to analyze effectively. It essentially reduces to the problem of running the Ruby code in the first place. If there's a case in here, it looks like where we're assigning a constant to an LVAR and then passing that out. There weren't that many cases where this was being used. It tends to be used in cases where people were getting a little too cute. And by banning it, we made it simpler for humans, made it simpler for machines, could actually run an auto loader. So we have this thing. We rolled it out safely. The whole site didn't fall over. It actually went pretty well. A few weird services got a little sad, but we fixed them quickly. So the question is, what have we used in this case? We've used it in a lot of ways. We've used it in a lot of ways. We talked about the auto loader. We got to fast boot times, delete and require relative, pretty much everywhere in the code base. But we actually found some cool use cases for this. So if you look at this sort of set of implementations and set of tests, if I want to run tests for child here, test child, I'm going to have to run both of these tests. If I change parent, I really have to run both of these tests, because child depends on parent. On the other hand, if I only change child, I really don't need to run the parent tests, because nothing that happens in child should impact the behavior of parent, unless you're doing something really weird or require time, which we want to ban. So what this let us do is write a selective test executor such that when somebody pushes somebody to CI, we run a set of dependent, reverse set of dependencies of all of the tests that depend, could possibly depend on the changes you've made. And then only run those tests. And what this means is that for about a quarter of our CI runs, we run less than 5% of what is otherwise like a multi CPU day test suite, which both saves people a lot of time and saves us a bunch of money on EC2 instances. This picture, so at various points, various people on EC2 could have said, wouldn't it be really cool if we could use GraphViz to visualize all the dependency relationships in our code base? We now call this project untangling the Gordian knot. This is what it looks like if you actually try it. I think this was after massaging it a little bit. Our code base is a bit tightly coupled. It's hard to reason about. And what we're starting to do here, we've got two paths. One is we've started to manually pull apart pieces of business logic that we think are really important core models and write some ad hoc rules and tests that prevent you from recoupling them, prevent you from putting them back into the Gordian knot. And we've also introduced a package system whereby you can declare that everything under this folder, everything under this namespace is in a package and only the things declared in my package.yml can be imported and I'm only exporting certain constants within my own package. So we provide a facility for teams within the company. I think we probably have about 40 of these packages having released this maybe two months ago. For people within the company to sort of enforce their own modularity within their own code. We've also done some fun tooling we built on top of this that lets you introspect the code base. You can say what are all the places where charge is defined? Only one and we'll spit that out of the static analysis. But possibly more interesting, what are all the places this thing is constant as referenced? Or what are all of the paths you can follow through files to get from one dependency for another? So what are the places where we could kind of find a cut to split apart some dependencies? We also use this for a bunch of ad hoc refactoring tools. So if you want to rename a constant, we removed a top level include that was causing gross things in the code base such that they still resolved correctly without doing any manual work, which is really nice. So where does this go in the future? The short-term answer is we want to get to the point where we can do this, instead of doing this in the background over the course of 20, 25 seconds, we want to do this in 200 and 250 milliseconds and have this power editor tooling. Your jump to definition, things like that are all powered based on whole code-based static analysis that's done incrementally. The thing we really want to get to eventually and that we're sort of starting to poke at the edges of is type checking. And so in some sense, all this static analysis work, all this practice, running a build daemon in the background, is going to practice for us doing much more serious static analysis of our code base. And so as I said, I cheated a little bit here. The real answer to how to load a million lines of Ruby in five seconds is don't. The answer to how to run your tests faster is to run fewer tests and probably run them faster, too. The answer to how do you get new developers to successfully reason about millions of lines of code is don't. Split it up into modules, make life easier on yourself and everyone else. If you have any questions, I'm at AGMetcuff on Twitter, which means I will have to actually figure out how to log into my Twitter account, but I'll do that for you. You can put the ad on the other side. It's at Gmail or Android Stripe or find me after. Thanks.