 Our last speaker of the day is Jack Danger-Canty. He works at Square, he's a full-time, or I'm sorry, a part-time Rubyist and a full-time feminist. He moved to San Francisco last year and is firmly ensconced in the mission. And since he works at Square, I almost put him in at the first talk of the day-to-day as punishment for breaking everyone at the party last night. I figured that might be too mean. Anyway, he's gonna talk about some of the things he's learned at Square for building big, complicated applications. Thanks, Jack. Thank you, Josh. My name is Jack Danger, I work at Square. Did people go to the party last night? Did anybody go? You're welcome, and I'm sorry. Just to get that out of the way. Also, my slides are gonna be a little dark. I apologize, I'll put them online right after. So as I said, I work at Square. We have a lot of problems. We have a lot of hopeful solutions to those. We work with iOS, Android, networking, hardware. We have electrical engineers. We rack our own hardware in data centers. We do a lot of high-availability work. And basically, if you're good at anything computer-related at all, we would really like you to join the team. We're growing fast and we hope to grow even faster. Today, I'm gonna be talking about, my talk was actually called MegaRails, as in when Rails gets very large. But I've renamed it to Monorails, or Monorail, because as I've begun to do more research and talk to more people who are in the same boat, the word we all use for monolithic Rails app is Monorail. It's so consistent that I began to ask, what else is consistent? And it's surprising how steady it is, the progression of a Rails application from its humble beginnings with the rake generate something, script generate something, to where it ends up. There's a pretty common pattern. I'm gonna walk you through that and show you what to avoid. And if there's two things that can get you to take away from this, it's one, that Rails applications do go in a particular direction and you can't avoid it if you're thoughtful. And two, that the problem is primarily about ownership. And that's the thing we're paying attention to. So if you'll indulge me, imagine you have not a Monorail, but a nice little Greenfield Rails application that has a few controllers, people can sign up, you just got featured on TechCrunch, you're going okay. Now let's fast forward three years in the future. It's 2015. You're attending GoGarucco and you just left a marathon night of rescuing your Monorail. You have this Rails app now in multiple data centers. In each data center, you've got a $30,000 MySQL box. And I'm not even exaggerating. If you get a Fusion I-O card because the I-O load is too high for one box, you can get more life out of your MySQL database. That card costs 15 to 20 grand. So your MySQL database machine is incredibly expensive, as expensive as the whole cluster. You now have around 1,000 controllers. And if you haven't worked on a big application, you might think that number's too high, but a lot of Rails applications now are well in excess of 1,000 controllers. And who knows how many models. You've got test coverage, but you don't know how much because there's no way you could run Archive because it would take seven years or most likely segfault halfway through. There's just no way to know how well covered it is. And more importantly, you have sad developers. Your 22 developers are coming to work each day and talking about how they're not actually want to work at a big company anymore, because you have 30 employees and they want to work on something nice and small, something where their efforts have meaning, where they can actually make a difference. And you've got a slow test suite. Your tests are, maybe they're exhaustive, but when you have a critical error you need to get fixed in production, it is slow and it is hazardous to get it out there because running your test suite is just a gigantic pain. It might even be somebody's full-time job. You might think that you can avoid this, that this isn't gonna happen to you. I don't think that that's the case unless you've been through a couple of times. I think that I would probably fall into this trap again. But before I explain what I think the problem is, let's take another look at that first day, that Greenfield application and the decisions that Rails makes and helps you make in terms of what defaults you go with and how you organize things when you're getting application up and going. Oh, and if you're curious, if you think that this really couldn't happen to you, these are great companies full of brilliant people and they've all run into the problem. And there's 100 logos that I don't even have on here. Each of these companies survived a monorail. It came, it grew, they managed it, and eventually split some services out of it. Everybody did it in a slightly different way. They knew they had to. Most of these companies actually still have one giant monorail that they're trying to recover and the heroes of the organization spend their day, the people who will be writing compilers otherwise, spend their day pulling pieces of Rails 2 code on running on 187 out and into some sort of service that is manageable. Oh, and a sidebar. This is not a talk about can-rails scale. That is not a question. Like that's not a question you should ask. Can-rails scale, like it's semantically invalid. That's not a real question because if you're asking, can I get a lot of customers on Rails? Yeah, sure. Heroku, more dinos please. Ta-da, you're done. Rails just scaled. That's not a problem that you have. So let's not ask that question. Real problems that you have about scaling are scaling your data, scaling your code base, scaling your customers, and scaling your feature count. So data, that's mostly a solved problem. People have been doing it for years and you can do it too. It's really a matter of getting familiar with your data, getting to understand it and know it, finding out what it likes and what it hates, what shape it needs to be in, and where it wants to live. For you to do the kind of things with it that you need to do with it. It's hard, totally possible though. Scaling your code base, get and GitHub, fix this. Scaling your customers, I don't mean marketing efforts to get customers, I mean managing them. This is primarily about communications, engaging with your customers, and logging so you can explain what on earth just happened to your customer and to their account. And then scaling your feature count, Rails is actually great at this. You want a new feature, you put a new path with a controller of the same name and a model of the same name and a database table of the same name, and it works. It's a Rails engine taking a vertical slice, just throwing it right in. It's wonderful. Rails is better at scaling your feature count than most things. What we don't talk about is scaling your developer headcount. That's one thing for which there is no default way to manage in Rails. There's nothing, no tool off the shelf that lets us do this, and there's no pattern that we've identified in our community that helps us do this. So let's go back to the beginning. This is you on day one, Greenfield Project. You have one product, and it's implemented with one application, and you probably have one developer. Maybe you have four, probably one. That's great, it works fine. You get another developer, she joins the team, and you say, hey, commit code on your first date to our app, to where our code is, the one place where our code is. So she sits down, she commits code, and she joins the team, and now you're going at twice the pace because you have two people. That's a great pattern for the early days. That's how you should start. You need an admin interface, add a nested set of controllers under slash admin, and now you've got a bunch of admin views. That's great, and only let a few users in there. And then you need some analytics to spot trends over time. Well, put that in slash trends, although let's be real, you're just gonna put it into the views of your admin controller stuff because you just wanna have the graph right there, so you're just gonna write in the view. Rails can do anything. Let's make it do everything. Now that sounds like I'm saying it's tongue in cheek, but let's be real. You wouldn't know if your product was worth spending your finite time on Earth building unless you got to a point where you could test it in the market. If you can't get there, then you have no idea if you're wasting your time. So Rails helps you get there quickly. And with a product that you can actually keep building on, the reason we have monorails is because people get there like, actually what we got is a little technical debt, but it's good enough, let's keep going. You don't have to do a rewrite when you realize, yeah, let's keep going in this direction. So Rails can do anything. We make it do everything. This will work too. You subclass action mailer for no reason because you only do it once. And then you make a bunch of methods that are just copy, pasta, one into the other. Whenever you want a new mail sent, and this is not good software design, but it works fine. You get seven, eight methods in there, you're sending seven or eight templates and there's only ever called for one place in your code each of them, so you could probably just put it there. But this works and you get going, you have email support, it's going fine. Now users are a little hairy because you're throwing a bunch of stuff there, but when your users suddenly have phone numbers, you add a phone number field or phone field to the database table, and you add some sort of validation on that. And maybe they have a Twitter account or a Facebook ID, you add that there too. And now you have 40 columns in your users table which is hard to manage, but all the data's right there, it's convenient. It's so easy to just add modules or some sort of set of methods inside the user.rb to manage data that is all localized right there. It's the most convenient way to place it. It's the easiest way to get a feature out the door and if you're still in the phase where you're testing whether your product should even exist, that is the right place to put it. And Rails makes it very easy. Let's say you finally decide to take that time series work out of your admin views and you put it in some sort of Ruby module, congratulations. You extract it, you put it into slash lib, now you've got some sort of time series generator engine. That's great, and if you work at an amazing company that you have specs in speclib and another sidebar, if you're at a company that doesn't let you write tests for everything, you should say, I write tests for everything or I quit. And then either you write tests for everything or you get a much better job from any of the people in the hallway here. There's really no excuse. You'll probably get paid more too because if they don't value testing, they don't value you and your professional skill. So we're up and going. We've got a Rails application and obligatory cat slide and most of your features are there. They're not all organized the way you want, but it's working, it's actually bringing in some revenue, you're all set. So what is the problem? Well, the problem is that Rails is optimized toward the beginning experience. It's optimized toward getting up and going quickly. So any piece of software has trade-offs. Any good piece of software has trade-offs. Bad stuff tries to do it all. So MongoDB, great for some stuff, bad for other stuff. The stuff it's good at, it's amazing at. Postgres, great for, okay, most things. So your software, your software that you're using Rails, probably, is optimized to get you up and going quickly. It takes your first 30 days into account and says let's make that magical and does a really good job at that and then gives you support for the work later on. It actually works pretty well, but it's not optimized for three years in. In fact, it's very heavily not optimized for three years in. I'm gonna walk through just a couple examples of directions Rails could go from the beginning. And if you ever work on giant enterprise projects, maybe they would have picked the right column. But Rails will always choose the left. You begin with one database. And you put all your code there because you need to do joins or something rather than many databases where you have to connect across different active record connections and manage some sort of multiple thread pools to connect across who knows how many machines. The problem is that even in a single database, if you've seen how Rails does its queries, when you do eager loading or something, it's gonna select a bunch of records from one table, grab those IDs, then go do a select another table with those IDs in the query and grab another set of records, grab some, select some IDs or some information, and then go do a third query. Then it conjoins those together and presents it to you as if it was some complex join. Sometimes it'll do joins, but often it just does the eager loading and multiple queries. So if you're hitting the database multiple times anyway, why couldn't the first query be in one database? You get those IDs. And then you go look up, say you have user IDs. You can look at those users' locations sorted by distance from a certain point in Solar or React or some other thing. Like it's not hard to do. It's actually very, very trivial. You just can't use the defaults of how active record wants to behave. My SQL or Postgres, if you have 10,000 lines, and I don't mean to be a hater, but we ran into a lot of pain. So I'll just share that pain with you and you can make your own decision. My SQL or Postgres at 10,000 records, they're both fine. Everything's gonna happen in less than a second. You won't notice a difference. But Postgres does things really well in situations where my SQL does not do them properly. For example, if there's a guy, Nolan Evans at Square, and he was working on a project with an analytics system, he'd collected probably a billion records at the time. And he sent out an email to the engineering team one night saying, I'm starting World Plan 2280. It was a JIRA ticket or a tracker ticket or something for it. To add a new column and change the default type of this other column in my database with over a billion rows. So he said that, I read that up like, oh, I hit R in Gmail, begin to reply, begin with email, done. Oh, okay. It took 0.28 seconds or something because he used Postgres, which does not need to do an entire table rewrite, copy and write to alter some of the defaults on tables or to add a new column. My SQL, however, would need to copy all that data over, rewrite it, block the whole database the entire time you have downtime. It's tough. Action Mailer. In the beginning, you use Action Mailer. In the end, anybody want to take a guess for what's in the right? Anything else? So it's not that Action Mailer is bad. Action Mailer is fine. It's a nice templating system. It just has no business being in most Rails applications. You should have an Action Mailer application where you have a Ruby project that includes Action Mailer, maybe it's in Sinatra or something, and takes some data in. And a Rails project that tries to send email. It's, as Copeland was saying earlier today, about since our mail service, at some point you'll realize, we got this big app, we don't know what to pull out, and then you realize everything involving email template generation is just BS. It doesn't need to be there. Both because you have two application.html templates, one of which is for the web, one of which is for that crazy subset of HTML that works inside email. And maybe somebody's trying to maintain both of those, but that's really hard. Those should be different jobs because anybody who works on HTML email should be given a break. And that should be done in a different process or a different project where they can have their own deploys because templates maybe need to go out all the time. Maybe you have copy changes that are real lightweight, they should be able to deploy at will, and not have to deploy, say, your API at the exact same time. So lots of data in the user's table, good idea in your first 40 days. Really bad idea later. If you have a user, or it might be called accounts or person or whatever you call it, to identify a human being in the real world, whatever that is, it should contain some sort of unique ID and then the minimum necessary authentication data. And that's it. Anything else is a feature that relates somehow to that user and is probably, if you grow infinitely, it will be in a service somewhere else eventually. And in the meantime, it'd be nice to have in a different table. Lost of data in users. Yes, it's supposed to be lots of data in users. Not one person I gave this talk to before ever caught that. That's very interesting. So features sitting in slash lib and specs in speclib. That's great that they're extracted not in your admin views anymore. Congratulations, that's really a big step. However, if you have specs and you really should have specs, then you're running the specs for this extracted lib code every time you run your build, which is silly because it's in lib because the library, which means it's a dependency, which means it's not your application. So if anything's in lib, it needs to be extracted. And if it's not in lib, if it's in lib, it needs to be extracted into its own private gem. If it's in lib without tests, it needs to be given tests and then extracted and then put it into a private gem. Nothing should be there for more than 10 days or so. And if you have a file called something like objectpatch.rb, I recommend just deleting that because that's, or add it to Yehuda's list of bad ideas I made for good reasons a while ago but delete this after 2013. So validates, presence of, validates uniqueness of, those are for generating error messages. Those are for not for validating your data. If you have an application developing initially on your laptop in development mode, all you need is a left column. If you have a data with anyone else's, if you have an application with anyone else's data, any person who is not you or employed by your company, you need DB validation constraints because otherwise you'll have to corrupt data, which means you've just violated trust with your users, they've just given you something or you're supposed to record something and then you lost it, you dropped it on the floor. And particularly if you have customers who pay you, they call you up, they say, what's going on? A bad answer is I don't know and we lost your data. You have to send it to them. Default logging, Tom did a great, or David did a great job early this morning about talking about how Rails will log what it does. Your application needs to log what it does too because you need to know what's actually happening. So log every significant action. This is not part of the defaults. These don't happen by default. Here's the one. You need to analyze your data. In the beginning, sure, do that in your database. It's not too hard to make SQL do multi-table joins and perform some sort of time series stuff, especially since you now extracted this time series generator engine into lib and now a private gem. The problem is that data is both in the wrong shape because it's in a place that's designed for online transactional processing. It's what's not sorted by day, for example, or grouped by day. It's also interfering with what's going on on your site right now. So you should probably extract it somewhere else, either a replica that sends your data from the main database to another one or ETL, which if you're not familiar, it means extract, transform, and load. Now Xavier Shade taught me at a lot of the square, pretty much everything we need to know about how not to do this because he saw how we didn't do it and fixed our habits. If you have any questions about data, by the way, talk to Xavier, he's usually over there, is he still? Anyway, track him down, he's amazing. He set up stuff with replicas and with star schemas and with ways to take your data out of the transactional database and put it in a form where querying actually works, where you can group things by interesting trends. But we didn't always do that. I joined about a year ago, and just after I started, I saw this amazing admin interface that had been built before I joined. You can hear squares laughing because I know where this is going. There's some rich pages. They have lots of trends and charts and graphs and so much data coming up that may or may not have been generated in the admin pages' views. And they're connected to the main database because we need real-time information. We cannot have any delay at all, not even the delay of a replica. So one day we do a deploy and our admin pages are on a different host, a completely different host. So those passenger instances, we don't mind having a little bit of downtime when we deploy, we don't do a funny business with a load balancer, they just go down for a few seconds. But they connect to the same database as the rest of our code did. And we were coming up from a deploy and our passengers were all idle. And one guy decided he needed to see this page refreshed so he hit command R. Nothing happened because Rails has a slow boot-up process. So he commanded R, command R. He must have played a lot of video games as a kid because he had that thing like 60 times. He just kept trying to reload and when it finally tried to come up, Rails, all these Rails instances booted and then ran the most expensive page on our entire site simultaneously. And this guy was able to take us down which is a really bad thing because a lot of things shouldn't be able to take you down. One guy hitting refresh. Yeah, I would call that a weak point in your architecture. So that is fixed. And it's many times over fixed. But it's something that Rails let us do and get away with for a long time. There's a reason why these Rails applications hit a point and fail. Melconway wrote, organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations. Put it another way, the shape of your people defines the shape of the products they make. So all these examples they just gave you were of just normal growing pains of an app growing up. You know, use new data types, use new databases, clean things up, move stuff around. And that's I think where we've been distracted and thinking that's what causes these big apps. That's what causes the pain of these applications. But it's a total red herring. Those are problems that need to be solved. And those are problems that we as a community are getting better solving. But those are the technological problems which we're good at. There's an emotional problem going on that I think from the people I've talked to and certainly at Square, we haven't addressed very well. That problem is ownership. If your team is responsible for building something, then success for your team should be dependent on the success of that product and failure for your team should be the failure of that product. If you produce something and it is a failure, it does not do what you were supposed to make it do. You should know that that was your failure and take ownership and adjust it and make amends. And if you stay up all night finishing a feature, you'll do it because you know that your effort combined with your skill makes a success. There are no random variables. Computers don't just throw away register values. That doesn't happen. It's pretty deterministic. If you put in the work, you will get the thing you're desiring, if you have even the minimal skill. The problem is when the process of you building something is intermixed with other people's processes of doing unrelated tasks, then you have no control over the quality of your product. You're not gonna put in effort and you'll put in all the time, all the skill, all the labor, you'll produce the right thing and it will fail for unrelated reasons. So how could you possibly want to work on this? How could you possibly take ownership? How would you possibly put in extra hours on this project? Conway's Law says that if you get, say, three teams to develop a compiler, you'll get a three-pass compiler because each team needs to do something concrete that they can deliver. Just like if you divided the entire company into individuals and told them all to do something amazing and they were responsible for the success or failure of the thing they built, you would get as many products out of that endeavor as you would have people, which is why it's best to group them in teams so they can get one big product. Now, Rails can totally do this. Rails can grow and then split up such that people can work on it in groups and not step on each other's toes so that they can believe that their energy, combined with their skill, produces success. But that's not how we're doing it right now. That's not how we're doing Rails. That's not how we're doing most large Ruby applications, so I guess which are Rails. Because there's no definition between mine and yours. There's no definition between what I'm responsible for and what I am not responsible for. And a little bit of that is because in any Ruby object, you can just do anything to any other Ruby object, but we don't actually violate the rules that bad. Nobody's patching object in the middle of a method on your sub-feature. Nobody does that. Nobody does that in a code-reviewed company, at least. The trouble is it takes foresight, so Rails can do it, but we need to know what's gonna happen and we need to prepare for it because all those things I just talked about, all those things that Rails does good when it's a young project and then does not optimally in an older project, those are things that we think we're working on, but what we should be working on is ownership, where at a certain point, you will hit this problem. I can tell you that companies that have a monorail, they don't hit this problem gradually. They hit it immediately. It happens at a single moment. And it's the moment that you split your big team into multiple small teams. The second you make that HR management decision, you have a monorail and you have an unwieldy piece of code that you can't manage. You'll say, oh, we need services, but it's too late because you can't cut services out of something that is not defined by role, by task, by feature, by section. I have no idea what slide is there. So to close this talk and to close this conference, I'd like to walk you through a three-step process that every large application has to go through and most of the ones that I've talked to have done a really good job for taking ownership over pieces of the code in a way that a Young Rails app did not have and extracting them. And while it's three steps, only the first one works the same for everybody. Everybody does the second two steps in a completely interesting way, according to their users, their application, their needs, their uptime needs, and their customer's expectations and their data. The first one, and this is the most important, is to build a service interface inside your app. You need to build internal APIs. And not for everything, but for anything that you think maybe someday shouldn't live here anymore. So the easiest example I can come up with is email. Instead of applicationmailer.send or notifications.deliver, whatever it is your most apps do by default, you should give the bare minimum information that is necessary to perform this task with no indication of how it is to be done. Email to person using a template with an arbitrary dictionary or hash of data. That's all you need, and that's all it should be given into this interface. Now, if you start like this, then you'll reach a certain point where you need teams, and they'll say, oh, we need services, we should cut some things out. And some team will say, oh, we'll take everything on the other side of that interface, and we'll just fix it. And then nothing you do will ever change with regards to this, but on the other side that will just get cut out, and maybe they'll have a mailing service, maybe they'll send straight to the send grid or other online third party, doesn't matter. It's no longer your concern, because there's a line between your responsibility and not your responsibility. Another example, this is made up, there's a need to make a small slide, and I wanted to show you some of the things that we're trying with at Square. The Rails application that tends to be made, sometimes people split the API out earlier, sometimes they don't, but it may end up with marketing front pages, so for a signed out experience, a signed in experience using HTML, and then some kind of API. And they're like really clear definitions between these parts of the site, but they're all just in the same Ruby process. So one way to do it is to have some sort of front end that looks like it's maybe just part of the regular Rails application, and you think you know what's going on inside, starts doing this instead, where in some module called marketing, it does who knows what with the minimum necessary information to generate a request. And if you pass in the whole Rails request object and the necessary data for that particular request, something can happen inside of there, but it's an omega mess, where whatever happens inside of there is no longer your responsibility. Somebody could be in charge of that, and then maybe it's happening in a different process, maybe it just gets refactored, who knows. Maybe the tests just get cleaned up, but it's not your responsibility anymore. Step two, usually looks something like this. You either copy the app via get clone and delete the parts you don't need on either side, or you re-implement the functionality from scratch. But the important part is, there's a line somewhere in the left block there, to the left of which nothing changed, and you can count on that, because that's some of their team's work, and their code shouldn't have to change. Now, in reality, with the way we write Ruby app Rails applications, you're gonna have to modify a lot of files to do some operation like this, but that's somewhat inappropriate. You should have to be deleting or moving or adding files, not modifying them to pull features apart. Step three, you're gonna have to do something with your data. Maybe in the beginning, they both talk to the same database, and then you replicate it across, and then it talks to the next database on its own. Now how you do that is entirely up to you, and I appreciate you making as many blog posts as possible about how you do it, because Airbnb did it one way, Groupon did it one way, Living Social is doing it another way, Square is doing it another way. Everybody's got an interesting solution, and we need to learn from each other about that. But the most important thing is that we recognize that there's a real human cost to not understanding the need for these service interfaces early, and for not knowing that as soon as we change our group structure, we're all gonna be in a lot of trouble, because suddenly we'll go into work, and rather than creating the future, and building new things that we imagine, we'll be maintaining somebody else's code, whether that's ours or someone else's, and just debugging it slowly for a long period of time. And Square sponsored the party last night, not just because we like recruiting people, because we need help, although we like recruiting people, but we sponsor a lot of things. We don't sponsor anything like GoGuruko, because GoGuruko, people here, you guys are like our family. We struggle writing Ruby apps together for years. There's something that actually connects us, and we deserve to go into work every day and be really happy. Ruby was optimized for happiness by Matz. Rails was optimized for happiness by DHH. Mega Rails, the kind of Rails apps we're building now, we're doing a very poor job optimizing for happiness. So we need to build some conventions around that, so that we can continue to going into work every day on a thousand controller project, and know that we're gonna have a good time, because we know how to tackle this together. My name is Jack Danger, I work at Square, and I thank you for your time.