 And also, thank you to all of the organizers of Go Ruko. This is my actually first time speaking at Go Ruko. It's my first time giving a programming-related talk. So thanks for having me. Maybe this is why the conference is ending. OK, guys like me showing up. And usually, they say it's hard to follow a great talk. Like, Andy's, it's also even harder to follow a talk when you're trying to desperately dab the tears out of your eyes here. Titleless talk is I made a huge mistake how we got services wrong and what we learned along the way. I think this talk is a version of a talk that's probably been given a lot of different times and different flavors. So this is going to be our take on this topic. It's going to tell the story of how a growing startup tried to break apart its rails monolith and some of our learnings along the way. The talk will be broken down into four talks, four sections, a little bit about me, some high-level discussion around, like, how do you break down a rails monolith? When is the right time to do that? We'll end with five concrete tips or five sweet tips, and then we'll wrap things up. My name's Kelly. I live in San Francisco. I've been there for three years. Before that, I was living here in New York. Glad to talk New York versus San Francisco at the after party. I live with my fiance and our dog. We're getting married next summer. We're having a Twin Peaks-themed wedding. I've been riding rails since about 2005, and I work at a company called Gusto, which does payroll benefits in HR software for small businesses in the US. We'll get into that in a little bit, but right now you're probably wondering, let's talk about Greta, our dog. Figure tender love is going to burn like 10 minutes talking about his cats, so I'm going to burn a slider to talk about my dog hair. So Greta is a very fashionable dog. She likes to keep up with the latest trends. She recently decided to redo her hairstyle and go with an ombre coloring. She's also a very worldly dog. She likes to travel. She likes to have new experiences. So she recently brought back a kimono from Japan. That's not why we're here. We are here to talk about, I would say, rails projects that push the limits of the framework. I think rails doesn't scale as one of the great lies that's uttered by commenters on Hacker News, but there are times where you need to reach for different things, and that's what this talk is about. There's a caveat here, which is you shouldn't take everything here and immediately apply it to your day job. You want to think about it and have a discussion with your team. So let's talk about Gusto. Gusto currently does payroll for 1% of small businesses in the United States. We move over $1 billion per month. Our Rails code base is one of the largest that I've worked with. It's about a million and a half lines of code between the front end and the back end. That's including our tests. We have over 80 engineers, and we mostly work out of the same monolith. We do this all with the Fisher Price framework, that is Rails. One and a half million lines of code is not a small project, especially when you have a language that allows you to be as expressive as Ruby does. The domains of payroll benefits in HR are domains with a lot of incidental complexity. This is complexity that is part of the problem, and it's not complexity that is our fault. Payroll is a careful balance of time, geography, money, and people. There are tax jurisdictions that are as small as the city block or as large as a country. The IRS really gets on your case if you're even off by a single cent, and obviously everyone wants to be paid on time, so unfortunately for Scott's talk, we do have item potent jobs, so you're not gonna get paid twice on Gusto. Sorry. And because there are people on the other end of our software, we are all human, and we are all constantly making a mistake. I'm probably making a big mistake right now. This talk is about mistakes. So all of these aspects need to not only be able to be applied and they change lives, but they also need to be rolled back and corrected when mistakes are made. So reports, it's for this reason that when given like a very important trade-off in application development, we will always choose correctness over performance. We need things to work correctly. The IRS needs things to work correctly. The country needs things to work correctly. So this is like one of the trade-offs that we make, and this is an important thing to keep in mind when you are looking into what is essentially a domain-driven design. So let's talk about how things go wrong, and when things get way too big, like if you are at work and you're like, our Rails monolith is huge, like congratulations, that is a great problem to have. You have a company that exists, and it's useful enough to keep around, so great. So a lot of things that work when you're just five engineers in someone's apartment start to break down when you get to 50 engineers or 80 engineers or hundreds of engineers. So I want to talk about the Swamp. Swamp is another name for a ball of mud. This is when that Rails monolith gets so large that it becomes very slow to develop, and startup times can take tens of seconds. Your test week takes tens of minutes, maybe hours, and you find yourself starting to play whack-a-mole with different parts of the app. Ship one thing, another thing breaks. You fix that thing, and then something else breaks. And you start to kind of outgrow some of the paradigms that Rails gives you out of the box, like an app models folder, or an app services folder, or an app controllers folder. And over time, it becomes difficult to make any changes at all. You might say like, our team is growing, but it feels like we're moving slower. You can feel like you've got your head stuck in a Kleenex box. She was fine, don't worry. So I want to talk about our Swamp. Like, what does Gusto's Swamp look like, right? Like, our Rails monolith, which we are still in the process of piecing apart, right? A lot of people will get up here on this stage for, I think we're kinder than that. They'll tell you like, we switched to service oriented architecture, everything was great, and we're done. As we should have learned from Andy, this is a process, this is a dance. So our Swamp has like four rough domains. Got payroll, benefits, HR, and infrastructure. Exactly what they do isn't too important. But this is the Swamp that we live in. There are 666 models. Don't read into that number too much. And you'll be working like this and someone might show up or someone might come back from a conference and they'll say, let's extract a service. This is a huge Swamp that we've got. And you might say, cool. So let's see how that might work and let's see how we might have done this. So in our case, our team decided to, let's say extract this HR domain into its own service. So an application. And we chose this because HR is conceptually different than everything else, but it's still related to what you do. So in the world of gusto, right? You might keep someone's name, social security number and how much they're paid in an HR service, but the payroll service might be in charge of actually paying that person. So we're like, okay, let's Rails new up, HR v2. We've got a Brand Spankin' new Rails app, Rails 5.2. Maybe we can track master, getting really crazy here. And then you're gonna sit down with your team and you're gonna slowly connect the existing domain to this new app. But as you do that, you're like, oh, huh, there's a lot of stuff in HR. There's a lot of stuff there. Huh, yeah, we missed that. And so you get this, it feels like the goalposts are constantly moving away from you, right? And you're not isolated, you're not just a bunch of engineers banging on code. You need to work with your team here. You have PMs, you have designers, you have the business asking, hey, when is that thing going to be done? And you're like, well, soon, right? And this goes on for a little while and the team's getting burnt out. So you say, well, you know what? We've done this pretty well. HR v2 is, it's got 90% of things moved over. Let's call it quits, project's done. So you say we're done. But then you've got a different problem, right? Which is, we set out to extract this one thing from a Rails monolith and now we've made the problem worse. Rather than there being one place to go look for HR information, now there are two places. There's HR v2, which is really fast, but it's still starting to get slow because you're like, oh, there's a lot of stuff in HR. And now you have this appendage of old legacy HR and chances are it's doing something very important, which is why it's there to begin with and why you're choosing to keep it around. And this is exactly where tribal knowledge comes from in companies. Because now the question of where are names stored or where are social security numbers stored, you have to ask someone. It's not just evident from the names or the structure of the code. There's gotta be a better way. We think we are on the right track, but we by no means have the answer. So you might say, tell me more. So to do so, I wanna first make a distinction between applications and services. I think Scott's talk was great. It did a good job of breaking down and trying to provide nomenclature to these things. I should make a note that this nomenclature is going to be a little bit different, but I think the spirit is the same. So you might say applications and services aren't those the same thing. What's so different about them? So here are the distinctions that we make between applications and services at Gusta. So an application is something that has its own process. It's its own app. It is like that rails new, right? A service might just be a module of code. It might be running in process. It might be running out of process. You might be sending parameters to it via method calls. You might be using something fancy like GRPC or whatever is gonna be cool tomorrow. Usually applications have their own database. Services probably share a database with the monolith. Applications can scale independently where services scale with the host app. Applications might be in another language. Services are going to share the same language here. And for us, we have a rule that we create a service first and then an application. So rails new is one of the last commands we run in this process. So let's take a look at how that works. Let's go back to our swamp. The first thing we're gonna do is we're gonna sit down with the teams who are in charge of these domains. We're gonna sit down with the payroll department. We're gonna sit down with the HR department. We're gonna say conceptually, what is payroll? Conceptually, what is HR? Like what are these things doing? What should they be doing? What is the difference between the expected and the actual in terms of how our application is structured? And we wanna get everyone on the same page. We wanna get them bought into the vision that there is a reason why and there's benefit to breaking these things apart. For us, we found that it's much easier to maintain these applications. Even simple things like routing pages becomes a lot easier once these things are split up. And then once we have like the contours or these bounded contexts defined, we're going to start drawing explicitly drawing edges. So this is where you move out of using the database as an interface in a normal Rails application. And you say we're going to explicitly define every single time payroll talks to HR and vice versa. Every time HR needs to talk to payroll or whatever your two domains might be, we want that to be an explicit operation and not just they're both reading out of the same database table. And when we're drawing these lines or traversing these edges here, we're using value objects. Or we have a rule that if data is coming across one of these lines, we are not passing around an active record instance. As we work, these interfaces are going to become relatively more hardened. It's going to be clearer how these two things interact. And then over time, the decision of should this run on its own box, should this run on its own like a fleet of containers, that is a small change. Whether it's running in process or out of process, whether it's running on the same host or on a different host can be configured and reconfigured. When you're doing this, you will introduce new failure modes, which again, Scott's talk is great to cover those things. But we found this to be a good litmus test of did we get the boundaries between our domains correct? If changing a transport layer is a very easy operation, we feel pretty confident in those domains or in where we drew those seams. So in the Rails keynote this year, DHH talked about this concept of conceptual compression. I think this is one of the great strengths of Rails. You don't have to think about this stuff for a long time. Rails is one of, I think, still continues to this day to be one of the most expressive ways of building a web application. Much like Ruby optimizes for developer happiness, I think it also optimizes for developer time. And you get a lot of concepts, just curled up into this nice little ball so you don't have to worry about writing SQL statements or sanitizing SQL statements or validating data. You get a lot of that for free. Everything is kind of curled up into a little ball here. But we found as the application grows, as you get deeper into your domains, you need to figure out which parts of Rails you need to start breaking apart. Which concepts do you need to expand? And the four or five things that Active Record handles for you. How do you start breaking those apart and being a little bit more selective about what you choose to use? So these next five concrete examples are some of the things that we found work for us. They may not work for you. Have a discussion with your team to figure out if this might make sense for you. Our first recommendation, mind and avoid circular dependencies. Although we do not have like import and export statements in the Ruby language, we still draw dependency graphs with every line of code that we write. And so for us, as we grow our Rails applications, we actually have started to question bi-directional relationships. So from a code perspective, this is a very simple, very simple like two Active Record models that you might see in an application. We started to ask the question, do we really need that other direction? Or whenever we have an employee, are we already operating within the context of a company? We do this because bi-directional relationships, by their definition, create circular dependencies. These circular dependencies are what a ball of mud is made out of. If you took a microscope and zoomed into a ball of mud, you just see a lot of Active Record relationships and other things like that very coupled together. So we ask ourselves, can we drop one of those lines? Not always perfect, but nonetheless a good discussion to have to improve the design here. Recommendation number two, whenever you are communicating between services, use value objects to traverse those edges. So let's take another look at a very simple service class. So this is a service class that handles what happens after a company signs up. We're gonna send an email using the company mailer class and we're going to track some stats through the stats tracker class. And this looks like perfectly normal Rails code. And actually I would say it's a perfectly fine Rails code for small applications when speed is more important than maintainability. But if we look at this in depth, we'll notice that we've actually coupled company mailer and stats tracker, the two classes here, tightly to the structure or the shape of our company Active Record. And so the problem here is that now if we ever want to change the shape of a company, we need to go and make changes in several different places. And that has a name that is called Shotgun Surgery. So instead, we always ask ourselves the question, when is the best time to bail out of Active Record and just go down to pure values or value objects? Sometimes this might just be strings and integers, sometimes we will actually build up plain old Ruby objects for poros to represent what we are trying to pass around. And now we only need to make a change in one class should the shape of that company Active Record ever change. Third recommendation, callbacks, try to avoid them. Callbacks are incredibly powerful in Rails applications. You would kind of get the sense that it's hard to write a model that does anything interesting without them. But their expressiveness comes at a cost. So I want to talk about that cost and I want to talk about why we say generally avoid them. So let's look at, again, company model here. After a company signs up, we want to send a welcome email. Let's ignore the fact that we might be using a third party mailer behind here and we're tying the ability to create a company to is that other service online. Also never mind the fact that we should probably be using an after commit with a background job. Just bear with me here. So we look at the dependency graph that we just drew with that Ruby code. Again, circular dependency. And this one's even a little bit more sinister than even bi-directional relationships because here we have two different layers of our application, like the model layer and the mailer layer. They're now coupled together. They know too much about each other. And again, this is not a problem immediately. This is a problem when you go to change the thing. So instead, we reach for service objects. We've got a lot of service objects. I mean, one and a half million lines of code, we have to, you gotta write those somehow. So we reach for composable service objects. And here, so rather than having the company model know about this mailer layer, we have something else, a create company service object that knows how to create a company and it knows how to take care of that email that we wanna send afterward. If we look at the dependency graph of that, we have a new node. We have three nodes now, instead of two. But the trade-off that we've decided to make is that it's better to add a new node to a dependency graph than to create a cycle because the cycles are what makes it hard to change. Fourth recommendation, services first, then applications. Try to get these boundaries around parts of your app drawn and hardened before you type that Rails new command. And when you type that Rails new command, make sure there is nothing left over from the previous service. Fifth recommendation, move slowly. This stuff takes time. For one of the pieces of the app that we applied this to, it took us six months and 500 pull requests. We set a vision of where we wanted to end up. We didn't know really how we were going to get there or necessarily how long it would take. But we had the buy-in of our team and of the company, and we were able to gracefully do this and never miss to beat. We were still transacting every single payroll through the system while we did this process. Always move incrementally, no matter how bad the code may be. Rails newing a microservice is akin to a big rewrite whether you realize it or not. You have to stick with what you have today rather than starting to build, trying to build something from scratch. So in the words of Kent Beck, you'll always be looking to make the hard change easy when breaking apart a Rails monolith. That is going to be hard. You're going to have to unlearn some things that you've been doing for years. And then you want to make the easy change. You can't do this without a great team. And this is not something that you can break easily down into a series of stories or points. So it'll take a while. Trust your team. And good luck.