 My name is Jordan Reign, I work at a company called Clio, I'm a developer there, and I've been writing Ruby code and been a Rails developer for about nine years. Our company's based in Vancouver, Canada, and over that nine years I've had the chance to upgrade apps from Rails two to three and three to four and four to five. And I can still remember the Rails two upgrades, or Rails two to three upgrades rather, and how many breaking changes there were. And it was painful, but thankfully things have gotten better and there's fewer breaking changes for every version as we get them. On top of that, we have deprecation warnings now that are getting better and better and in a lot of cases they're telling you exactly what to do to make sure that your app will work with the next version of Rails. On top of that, we now have beta periods that can go for months where we can test out the newest version of Rails or we can run master and all of these ways in which the upgrade path has become better. And yesterday when Eileen in our keynote mentioned that Rails isn't dying, it's maturing, looking at the upgrade path, this sure seems to be the case. At least on paper. In practice though, it still can be really, really hard. And it's hard for us at Clio, and when I got a chance to talk with some other companies about this, it's been hard for them as well. And I was chatting with somebody this week and I learned that there is a lot of consultants who will actually go in and do this for companies because some companies just can't afford to do it. They don't know how to do it. I don't know how to get out of the code base that they built for themselves. And it had me asking this question, with all of the improvements that the Rails core team has done, why is it still so tough to upgrade Rails? I wanna dig into this over four parts, starting with a story of two Rails upgrades. The first one is a small startup and one day, one of the developers is scrolling through Twitter and sees a tweet from GHH. Rails 5.2 is out. She has been meaning to try out active storage and she doesn't have much on her plate so she decides to upgrade. She spends the rest of the day upgrading gems, running the tests, making them green, and sends it off for code review. The next day, her and another developer work through all the major flows in the app and get it running. And then the end of the day, they ship it. 24 hours later, they're running the latest version of Rails. The next story is a different company. It's a bigger company. It's been around for many years and they have a larger team of developers. They also have thousands of customers who pay them for a service. When they see the tweet from GHH, in their Slack team channel, they don't get excited. They actually feel dread because now they are two major versions, two minor versions and one major version behind and they're now running out of maintenance. So they can't defer any longer and they put together a team and they start the project. Bit by bit, they upgrade their code base and if they can ship any of the changes, they will do that but it takes them still three months to finish. Once they're done, they haven't done this for a long time. They haven't deployed a new Rails for many years and so they're a bit uncomfortable with the idea so they schedule some downtime just to make sure. They push out the deploy and it sticks. They're now running the latest version of Rails. It may not seem fair to compare these two upgrades. The startup only had a couple gems. They didn't have much in their code base and so they didn't have to change very much code. And they didn't have that many tests and so they had very few failures as well. And when they came time to deploy, their app wasn't mission critical so they could take bigger risks than just ship it. The other company on the other hand had many gems to upgrade some of which were no longer in maintenance. They had thousands of lines of code to go through and figure out if they were working or not and they had many, many tests that would pass or that would be failing but they didn't actually realize that right at the start because their test suite wouldn't even boot. And when it came time to deploy because they had many people rely on them, the deploy was very scary and they couldn't take any risks. But both of these companies fundamentally did the same thing. They upgraded their gems, they changed their code, they tested it and they deployed. There's no question that one of these companies had a lot more to do at each of these steps. But the end result was the same. They were now running a new version of Rails. The question that I had was, is this inevitable? Is this inevitable part of a growing company and an aging code base? And that brings us to Clio. Our upgrade paths look a lot like that second company, right? And we've been around since 2008 building a product for lawyers to help them run their business and help them do their job. And when I started to look at our next Rails upgrade earlier this year, I started to ask this question, why is it so difficult? What have we done in the past and what can we do to make it better? And to do that, I went back in history and looked at what we did. And to bring you up to speed, I'm gonna give you a quick look at our upgrade path and timeline. This timeline starts with Rails 2.0 in 2007 because that's the version of Rails we ship with when the product first launched. The very same day that 2.0 came out, our founder upgraded the app to 2.0. Halfway through 2008 when 2.1 came out, again the very same day we were up on 2.1. 2.2 comes out a couple months later and we lag a little bit three months behind but we're back on top. And then 2.3 comes out and we lag a little bit more nine months this time. And then 3.0 comes out and then 3.1 and then 3.2. And then I guess we feel some pressure because in 2012 we upgraded to 3.0, 3.1 and 3.2 over a five month period. And then 4.0 comes out and then 4.1 and then 4.2. And again, 2015 we go 4.0, 4.1, 4.2 over a six month period this time. And then 5.0 comes out. And then 5.1, 5.2, too fast. And when you look at this timeline there's an obvious pattern, right? And the interesting part to me when I look back I was surprised to find that the part that was interesting was not 2012 and 2015 when we were actually doing the Rails upgrade. It was the space in between. It was these gaps. And I found myself asking what were we doing during these gaps? Like why didn't we upgrade the app? What were we thinking, right? When you look at it like this it seems obvious. But then when I look back and thought about the business and thought about what we were doing during that time it turns out we were doing quite a lot of stuff, right? We were growing as a company. We were hiring developers. We were building features and functionality. We were adding customers and we finally started making money. So we made a lot of good business choices during this time. But we also made some questionable technical ones. And that's what I wanna dig into because I think the key to making Rails upgrades easy is in these gaps. For those of you who haven't been on a Rails upgrade project before I wanna show you a live action photo or a photo of what it feels like to be on a Rails upgrade team. When you upgrade your framework it has this amazing impact where you have to pay off some technical debt that you've been accruing over the years. When you open your gem file or when you open a file that hasn't been touched in a couple years you find some pretty crazy stuff. So when we look at the gem file and we start to think about what the problem is for us what we found is there was a lot of old gems. And I think it's very tempting to add a gem to your gem file when you need some piece of functionality and be content to leave it there and ignore it for the rest of your company's history if you can get away with it. But there's a problem with that, right? Code decays. Code has this way of browning and shrinking and withering to the point where it's going to cause you problems. And even though those gems aren't directly in your Git history that code is still running on your production servers. And if you leave it it will cost you. And this comes down to the idea that dependencies are not free. And this is not a new idea, we know this but sometimes it's easy to forget. There's a really interesting blog post on the Joel on Software blog from 2001 where he talks about the Excel development team's approach to dependencies. Finally, dependencies and eliminate them. And they took this to an extreme where they built their own C compiler for the Excel project. This seems ridiculous if you think about it now especially for us in web apps. But he went on. Oh, got to stop doing that. The Excel team's ruggedly independent mentality also meant that they always shipped on time and their code was of uniformly high quality. So they acknowledged the cost of their dependencies and they decided that they could build it better and cheaper themselves. Now this isn't to say that we should open our gem file and empty our dependencies. Empty it every gem we have. But we need to acknowledge that cost. It doesn't make sense for me to build my own web framework. I'd rather use Rails. But I need to commit to updating it. And this applies for all our dependencies. And if you don't do this, if you neglect to do this, the problems compound on themselves. I'll give you an example from our code base. We integrate with QuickBooks Online. And to do that we use a gem called QuickBooks Ruby. It relies on another gem, OAuth. And we've been using OAuth in production for many years before we started using QuickBooks Ruby. For reasons we also pinned it to an older version of OAuth. And so the gem hadn't been upgraded in many years. This was fine because QuickBooks Ruby worked with it. Until six months later we needed to get a bug fix into our app and needed to upgrade QuickBooks Ruby. Unfortunately the new version of QuickBooks Ruby upgraded the OAuth version. And so it didn't work until we upgraded that gem. So we were faced with this decision. We could not upgrade either gem, leave the bug in our app, or we could upgrade both gems and it would entirely blow out the scope of this otherwise simple fix for a bug. So when faced with this choice, we chose option C. Let's fork the gem and downgrade the version of OAuth it uses. And at the time this may have seemed like a good idea because we got the bug fixed and we didn't blow out our timeline. It looks something like this in your gem file. And we deferred the cost but it ended up coming back to bite us because every single time we had to upgrade this gem somebody had to go in, find the fork, merge in all the codes from upstream, make sure that it still worked with our version of OAuth and then upgrade the SHA in our gem file and ship it. So when a developer is faced with this kind of upgrade it's much less likely that they're gonna do it. So we would only upgrade this gem when we absolutely needed to. And this comes down to a simple idea that the more rigid your dependencies are the more difficult they will be to maintain and the more resistance developers will feel when they need to upgrade them. So they'll punt. Another thing that we did while during those gaps was that we merged a lot of code. And then we merged more code and then we hired more developers and merged more code. And a funny thing happens when you ask a developer to do something. They look through the code base and pattern match to find something that does something similar. Like copy it, tweak it and put it in the next place. And if your code base is full of calls to unmaintained gems or deprecated methods or undocumented private Ruby calls or Rails calls those bad patterns are gonna propagate throughout your code base. And the minute you copy and paste that you now have created a new task for yourself. And it's not such a big deal with changes like this. You can simply find and replace a for filter with before action. But some changes are more difficult to get out. Adder accessible and Adder protected Adder protected were notoriously difficult to get out of your Rails app. They hadn't impact across any controller who used these models. And when faced with this option upgrading from Rails three to Rails four we chose to add protected attributes because it gave us a little bit more runway to get it out of our app. But deferring this choice just made the problem worse because we kept propagating it around the app. And even when we pulled it out of our models we found that this gem added functionality across the whole application when we tried to mass assign things. So pulling it out became a huge hassle. And I know that we weren't the only ones because this gem has been downloaded five million times since it came out. And to put that into perspective active record since 5.0 has been downloaded 15 million times. So I think this is probably a pretty common pattern that we've done. And it makes me feel like we must like protected attributes at least one third as much as active record. The other thing that happened during these gaps is that we started to disconnect from the Rails community. We started to solve problems that had already been solved by Rails. And in some cases been solved by Rails in a better way. This happened when active record enum came out in Rails 4.1. When we needed to upgrade, when we needed to map an integer in the database to a string in the app we built our own version of it. And we didn't just build one version of it we built two or maybe three people all over the app doing a similar thing. And they behave similarly and they offer the same functionality but they were a little bit different. So when we finally got up to Rails 4.1 and could use it we still had these different patterns peppered throughout. And when active job came out in 4.2 we were already heavily invested in another solution. So we skipped it. And it's okay to build something yourself if it's core to what you do but if you're doing it because you're behind you're just wasting effort. And on top of wasting effort and energy you're also making it harder for new developers to come to your company. When they start to onboard and they try to do the things that they're used to doing at other companies they may find that they won't work. That they have to check what your Rails way is. So after looking into it I found that these gaps they make me feel like this. They demoralize you and discourage you from doing the things that you maybe ought to. So let's close those gaps. So the good news here is that most of the things that you need to do to upgrade Rails you can do at any time and you can do them in very tiny steps. There doesn't need to be a long-lived branch and it doesn't need to be a special group of people. And for us we did this in three specific ways. The first was to keep the gem file healthy. If we go back to that rigid gem file from earlier there are a couple things we can do to make it easier to work with. The first thing to do is to drop version constraints. There are reasons that you might want a version constraint but make sure that you have truly a good reason for it and if it's a specific reason that you can get rid of do so. If you have to upgrade open your gem file to upgrade a gem, it's probably too rigid. The other thing is to kill your forks. If version constraints make your gem file rigid forks are in order of magnitude worse. They make it difficult to go back and upgrade the gem the developer needs to think about why did we fork this in the first place? What changes did we make? Are they still applicable? And that will look like this. It feels so much better. And then the other thing and this is the hardest thing is to create a habit and create a culture of updating gems eagerly. Not waiting until you need a feature from that gem. Not waiting until there's a bug fix that you absolutely need to add. Because you won't always know when there's a bug you're experiencing. There are some tools to help with this. Bundle outdated is one. When you run it, it looks something like this. It will give you one line explaining a gem that has been outdated. Now, unfortunately, when I ran this, I got something, sorry, let me zoom in on one of those lines and explain it a little bit more. The first thing you get is a gem name. Then you get the newest version available. The version you have installed and then the version you requested if you've got a version constraint. And then the group it's in. The problem is when I ran this on our code base, the output was overwhelming. Now, this is from discourse and this is about 100 gems that are out of date. And when I ran it against ours, I got up 250 that were out of date. So where do I start? What's the most important thing? What's unmaintained? What's old? To help with this and inspired by bundle outdated, we wrote a script called bundle report. And it uses the functionality provided by bundler and Ruby gems to kind of introspect your gem file. It's got two modes, the first mode is outdated. And as you might imagine, it outputs something really similar to bundle outdated. But with a few differences. The first one is that it'll output the gems from oldest to newest. And here we're pulling off the top five oldest. So you can immediately see that these gems probably need to be updated. If we zoom out on one of those lines, you get some slightly different information. You get the name of the gem, the version that was installed on your local host. You get how old it is. And then you get the newest version available and how old that is. And this slightly different set of information lets you tell what is oldest and what is unmaintained. In this case, it's a seven-year-old gem, probably should update that. And the newest version available hasn't been updated for four years. So you could probably guess this is an out-of-date, unmaintained gem. You probably want to get off of it. And if you really need it, maybe you should take over maintenance of it. The second is focused specifically on Rails upgrades. And you use it like this. You set a Rails version that you want to upgrade to, and it outputs something like this with three sections. The first section tells you what gems will not work with Rails 5.2 and tells you what version you need to upgrade it to to get it to work. The second section gives you a list of gems that do not work with Rails 5.2 and also do not have a newer version that works with Rails 5.2. So these are gems that you either need to ask the maintainer to update or drop them from your app. And then the final section just tells you how many gems you have to go before you can run that version. So let's keep the gem file healthy. The next section is to outlaw deprecations. And as I mentioned earlier, deprecations have gotten better and better. And any most major changes you can catch if you're running the latest patch version of the previous version of Rails. In this example, it's telling us that we shouldn't pass an active record model to find. These will typically be printed to your logs, but you can send them elsewhere. One thing you can do is raise. And if you're anything like us, this is not an option because if you did this, your tests would fail and your app would break in many, many ways. So this was never an option on the table for us. And so to get ourselves to a place where we could turn this on, we wrote a small deprecation tracker. And with what's available with the active support, this is actually really small and simple. You could do this in-house. It's about 150 lines of code. And this ties into RSpec and it will track what deprecations come out of each test file. You use it like this, run your whole specs in save mode and it'll generate a list of deprecations. And then when you're on CI or local, you can run it in compare mode. So if any of those deprecations change or they're added or they're removed, it'll raise an error. And this was really important for us because we had thousands of places in the code that called deprecated methods. But by doing this, we instantly stopped anyone from inadvertently or intentionally adding new deprecations. So immediately, you've planted a flag that says things will not get worse. This is what it looks like on CI when it fails. It'll give you a diff of what changed. And people can fix this on their own. They don't need to be told. They don't need to be, you don't need to check people's PRs for this. And this had another effect, which was we now have a to-do list of all the test files and what deprecations come out of them. So everything is broken down by test file and the messages you get. And with this list, we can do other cool things like display the most commonly occurring deprecations. So you know where to put your energy to have the biggest impact. Or you can run all the tests that have deprecation warnings. Or if there's too many tests, you can filter it by a certain red jacks. So if you wanna fix one type of deprecation across your whole app, if that's, you're in that mind, the headspace, you can just run those specs again and again. And once you've fixed those tests, you can save them to the whitelist. And this allows people to go in and change as little as they'd like, merge it, and slowly you get down and down and down before you can actually turn this on. There are no deprecations left in your app. So that's a lot of deprecations. The last one is to dual boot Rails Next. Rails Next was, I first heard about this from Shopify Post on their Rails 5.0 upgrade. And I immediately fell in love with this concept of Rails Next. And it means whatever version of Rails you're gonna go to, whatever version of Rails is next for you. If you're on 4.2, it's 5.0. If you're on 5.0, it's 5.1. And if you're on 5.2, it's master or a beta or an RC. And this is a really powerful idea. The idea that your developers can very easily boot up the app into the next version of Rails. This is how we do it. And it means that our developers can take commands and the commands they know already and run them. So here we could start the server. We could run our tests. We could save a white list of all the deprecations in the next version of Rails. And this lets people go in, find a problem, and change it. And ideally this means they're taking an old deprecated thing and they're replacing it with a new thing that works in both versions of Rails. You can merge that and you can just ship that. And the code base just got better. One of those patterns, those bad patterns, is now gone. This doesn't work in all cases, so when it doesn't, you can branch. Now this does make me feel a little bit uncomfortable, but there are ways to pull these out once you've finished your upgrade. There are a couple different ways to do a boot. The way that we do it is this. Create a sim link of our gem file. We add this method to the top of our gem file that basically checks what gem file name we're using. And when we're using gem file next, we can return true and choose different gems based on what version we're in. This feeds two different gem file locks so that you never have any churn between when you run bundle install on the old version and the new version. And the vendor cache is also separate. This is how you set it up if you don't want to manually run those commands. And this is what that script looks like. It's really simple. This is a little bit broken down for the slide, but it does two primary things. It sets environment variables for bundler and then it runs a command. If the command starts with bundle, it just runs it, otherwise it'll wrap it. And so your developers now can take commands they already know and prepend them with bin next and they can run it in the new version of Rails. So it's no longer a daunting thing to get this going. And so when you see something like this, when you see an RC come out, you can in five minutes try to boot your app up and maybe tweak a couple of things and try to see if it works. And on top of that, once you have this in place, you can run this on CI. You can run it on every pull request so you can even just run it nightly. And you can see how far you are from the next version of Rails. How much work do you have to do? Does your app boot? How many tests are broken? And if you check this periodically, you can catch bad patterns as they come into the app. And actually Eduardo yesterday had a great talk on Rails upgrades where he talks about ways that you can actually mark tests as failing in the new version of Rails. And I think that's a really great way to do this where you can perhaps actually block pull requests when somebody introduced a new failure on Rails. So these three ideas, these three strategies, they kind of break down to one thing. Make it easy to do the right thing. Make the path of least resistance for developers in your company, the path that they ought to do, that they feel drawn to, but sometimes can't do if it's too much work. The examples in this talk are available in a gem. It's called 10 years RailsConf 2018. It is intentionally a mouthful because this is not a kind of gem that you wanna put in your gem file and commit. This is our internal scripts that I put into a gem on the way over here. But I wanted to have something that people could run, that people could try, and take it and tweak it and use it internally. So to close, what happens if we apply these ideas to that other company? The next time they come to upgrade their Rails version, instead of many gems, they're only gonna have a few because they've been keeping up to date on their gems. And because they have held the ground and have zero deprecations, they're gonna have a lot fewer code changes. And because they've been running it on CI with the next version of Rails, not only will their test suite run, but there will also be far fewer failures because they've been able to nip problems in the bud before they become and propagate throughout the app. And the deploy, there will be less code, but it's still gonna be terrifying. There are ways to make this easier, but I didn't have time to get into it in this talk. So the more you apply these ideas, and there are other strategies as well that you can do, the more you blur the line between Rails upgrade time and every other time. It becomes a persistent gradual process. And I think this is worth our effort because the things that make your code base better and make your code base a better place to work and the things that prepare it for Rails upgrades looks less like this and more like this. The things that make your app easy to upgrade also make it a better place to work. So what has this meant for us at Clio? Well, this is still a new shift for us, but over the past month or so that we've been applying these ideas, we've been able to remove 33 gems, update 27 other ones. We have reduced our deprecations to zero and we have fixed almost, well, about 90% of the failing tests that are in the next version of Rails for us. But it's an ongoing process. We're not on the final version of Rails yet, but we're committed to this idea. And I think the most exciting thing for me has been that this process has now shifted from a team of a couple of developers every couple of years to something that everyone can take a part in on your team. So as you hire people and as you grow as a company, your Rails upgrade team grows with you. So every Rails upgrade is hundreds of tiny changes that you can do any time. You don't have to wait. I have been, my brain has been in Rails upgrades for a couple months now. I am super excited to talk with anybody about the troubles they've had or the things that helped them. So please come chat with me afterwards. Thank you for listening.