 history how to upgrade a big Rails application to Rails 5. This is a history how Shopify launching the Rails 5 upgrade to production. So first I'm going to introduce myself. I am Raphael Franca. You can find me at Twitter or GitHub as Raphael Franca. I'm a member of the Rails core team and I work for Shopify as a production engineer. You may know me more because every single superstar of the Rails organization you enter you're going to see this as the last commit. So I'm also the self-proclaimed Rails maintainer because I usually do all the releases. So this is like the history how Shopify releases Rails 5 to production. But why look at Shopify? Why Shopify is so especially that you have to look on it? Because Shopify started around the same time as Rails. So the first initial commit of Shopify codebase was in 2004 even peered to the first stable release of Rails. So Shopify started at 2004. Rails 1.0 was only released in 2005. And also this application was never rewritten. So we are also still using the same application that started 13 years ago. So this is a timeline cooperation between Rails and the Shopify codebase. So we can see that Shopify followed the Rails versions really closely. In this example like in Rails 2.0 Shopify upgraded to 2.0 just after the release. The same case happens with the Rails 3.0 was just close to the release. But for the Rails 3.2 we took a little more time to upgrade. Like it's almost one year after that we upgraded to Rails 3.2. And we also skipped the Rails 2.1 because it was too mature in that application. So this is when we actually released the Rails 3.2. And for the case of the Rails 5. Like Rails 5 was released in June of 2016. And we started the upgrade of the Rails 5.0 right after the Rails 4.2 was merged to Shopify. And we only finished it in 2017. It was more than one year after we started to upgrade Shopify to Rails 5.0 that we could actually deploy it to production. Also Shopify is a big application. When I say big it's really big. Maybe it's the biggest Rails application that you have in production right now. We have more than 374 lines of code. It's just counting lines inside the app and lib folder. It's excluding kind of test code. We have more than 100,000 lines at all models. And we have more than 2,000 classes inside the mod directors. And also we have a lot of code in the controllers director. And our test coverage is good to application as this size. We have one of the three lines of tests for each line of code. And Shopify is also in the lessons to Rails versions. If you don't consider that Rails 5.1 is probably going to be released today. So how do you upgrade the application to Rails 5? Well, it's simple. You create a new branch. You make all the changes necessary to upgrade to Rails 5. You merge it. I don't know what's happened later in the profit. It's simple. Well, that's not our case. Shopify is a really massive beast. We have some hundreds of developers working every single day in the same code base. Creating a branch to upgrade the Rails version and keep it up to date is hard because of all the conflicts that may happen because someone changes something or people could be introducing new bugs every single commit. We need a different way to use and test the same code base with two different versions of Rails. So we come up with a strategy to make the application possible to work with two versions of Rails. The booting, in our case, was the best solution because it allows us to run the same code division with two different versions of the framework in both tests in development or even production. The solution is not hard to implement. You create two different gen files, the gen file dot next in the normal gen file. You use environment variable that's already a feature of the bundlet to install the gen's. This works just fine. It's the recommended way by the bundleting to do this. But in a big application, two gen files are also really hard to maintain since people could upgrade one and forget to upgrade the other one. Also, the version that is used in one gen file should be the same as the version in another gen file. So to solve this problem, we did something that I'm not proud of. We did a hack to share the same gen file. This is the beginning of our gen file right now. It's a huge monkey patch inside the bundlet internals that I'm not proud of this code that I wrote. But forget about that and focus on what's important. We have some conditioners inside the origin file like this. If you pass an environment variable called Rails next, it's going to run with Rails five. If not, it's going to run with Rails five two. And you could use this environment variable to install your chance or even start your Rails console or your Rails server. This approach is good because it's keep possible to do a bootie. It also address the issues in the press release approach like having two gen files because you don't have data size of forget to update the other gen file. Of course, data size of this is having to do a monkey patch in bundlet. So I try to come up with a better way. I never tested this in production yet, but you could use a feature in bundlet called involved in file. So you have a gene file with Rails five. And let's say you have also a share engine file. And you have an audition file with real size one, and you have the share engine file with all the genes that you want to share. And the coffee is less but Brute Rb, you added that three lines of code to change the gene files depending on the environment variable. Data size of this is you still need to use that in the command line to start the chance, but to start the server, you only need to change the Rails next environment variable. So when I do the dual booting, it's now possible to run my code base with two different versions of Rails. So what we did, we created a parallel CI build that was running with Rails for two and Rails five. Given that that's now possible, I can now start to working on the app itself. So what we did, we started to upgrade all the penises. It's important to always have the penises to work in both versions. And because it's easy to test that behavior is correct. So what we did was really simple, we make sure that all the versions are ready using the same versions that work with both Rails for two and Rails five. And when it was not possible to have a version that work with Rails five, we contribute back to the penises. Some of the penises takes a long time to upgrade to newer Rails versions. So this is a really opportunity to give back to the community. I have one example here. I had to upgrade the J action pack XML parser to support Rails five. And this was a good opportunity to also simplify the code of the penises itself because in Rails five, we had some architecture simplifications and that made the code of the plugins easier to write. So in this case, we had, this is the entirely code of the shame, like you have a bunch of lines of code that I don't want you to focus on. But after the Rails five upgrade, we could change the chain to be just that. It's a simple call that's called hash from XML or return name to hash. So that was a good example of how upgrading your Rails version can make your dependencies simpler. So after that, after we upgraded all the dependencies, we started to fix all the tests because we had thousands of tests failing. So what we did was for each broken test, we created a branch, we got the test, we fixed the test and we deployed to production. It's as boring as you can imagine it. But I have a, this is the list of the output requests that were created between the time we really upgraded to Rails all in production. As you can see, it's huge. And there is nothing really fancy on this task, which has created a bunch of PRs and fix all the things that were broken. Sometimes you can take days to fix one, to fix one single test. And sometimes one single code change can fix hundreds of the tests. Let's talk about our biggest challenges on these upgrades. The first one was the protected attribute genes. If you don't know what protected boot was, let's say that you have a model users that have attributes as attributes named password in admin. By the way, if you know about this, these attribute APIs actually working so you can define your attributes like this if you want to raise your application. So let's say that you have this model and you have a users controller in the admin space, name space. So you find the user by ID, you update the attributes that come from the form and you head directly back to the health path. And in your form, you have naming password and admin. If you, in Rails 4.2, actually in Rails 3, to that work, you need to add a new method called attribute accessible to tell Rails that those attributes are accessible through Mesasite. And let's say that now your feature is changed and you want to give users the possibility to update the old name in password. So you create another controller that does basically the same thing, but the form is different because you only want the users to be able to change the name in the password, but not the admin flag. What kind of problem does this cause? Well, something like this may happen. If you don't know, this was a hack in 2012 where a guy called Homo Kov could commit to Rails repository without having access to the repository itself. So how that guy made this happen? What he did was really simple. He created a new SSH key using his own SSH key in the DHH user. So GitHub did not have support, not support, but protection to this kind of attack in that time. So it was possible to a person add a new SSH key in the user's, any user account. And it was really simple. To mitigate this kind of problem, what you could do, you could add a new attribute accessible call to say that the admin flag is only allowed when you are doing the attribute assignment as an admin. In your admin controller, you say that you want to predate the attribute as an admin. So that was in Rails 3.2. And in Rails 4, we changed the way that all this protection works with the stronger parameters. It brings the protection closer to the source of the input, in this case, the controller. So it's easy for you to remember to protect your data before sending to the model. So how stronger parameters works? It works similarly, but you have to filter all the parameters before sending to the update in the controller itself. So we started a huge task to remove all the attribute accessible using Shopify because Rails 5 would not support the attribute accessible anymore. And we had more than 150 pull requests in three months of work to remove this feature from Shopify itself. In some case, we was just simple as moving the attribute accessible calls to the controllers that are assigning attributes to that model. But in some other case, we were founded that maybe we were missing some kind of abstraction in the framework. So in some case, like we had some massive forms with a lot of attributes, we actually created an abstraction called form object using the drive validation chains to filters all the attributes before sending to the model. Another challenge that we had was controller tests. We had a lot of controller tests like this. Like we created a request to action, sending a title, and in this case, I empty hash as tags. And in Rails 4.2, when the request comes into controller, the tags, the tags parameters were actually empty array, but in Rails 5, the tag parameters was not empty array anymore. It was actually blank. So why this change of behavior between those two versions? So let's say that you have a code like this. Like this is a valid code, but it's not as as as a sizing what's actually happening in the browser. So you have the post, you do the request, you set the parameters, and you send the parameters to the controller. But in Rails 5, what we did was we actually encode the parameters as the browser would be done. Like you cannot send the empty array to an application using the browser itself. That's invalid by definition of the encode of parameters of the browser. So in that time, we had no way to fix this regression because there is no way to you to tell Rails to not encode the parameters as the browser because what we are trying to test is if it's possible to send this information as JSON to the application. We had to open a poor question in Rails itself to make possible to set the content type with an as option in the controller test. What changes is now it's possible to you to pass an as option in the request itself and both Rails 4.2 and Rails 5 is going to behave in the same way because now it's not encoded the parameters as a browser but encoded the parameters as a JSON request. Speaking parameters, that was another thing that gave us a lot of troubles because since in Rails 5 actual parameters does not inherit for hash anymore a lot of code that we're doing type checking with hash broke. So in Rails 5 we had parameters like this in Rails all actually in Rails 5 parameters are not inherited for hash anymore. This change was to improve security because in many places that are not just models are vulnerable for mass assignments like URL helpers, no active record models like active resources and to avoid these problems we made parameters to not inherit for hash. So what's happening in Rails 5 now is like you have a parameters with name Raphael if you call parans.2h you get an empty rash because you did not filter anything if you call 2h you get all the parameters and if you do the filtering properly and call 2h you get what you filtered. So we had to fix a bunch of places that were relying on the behavior that parameters is a hash so we had a lot of a bunch of code with this kind of checks like if parans is a hash do something and this will not happen anymore work anymore so what we did was to avoid that parameters entering the model layer so in the controller layer we call the 2h model or in some case we leakage the abstraction inside the models and do this is the type checking us and parameters. In my opinion that is not an ideal solution it's caused a lot of pain in our codebase and it's also a pain in a lot of other codebase so we need to think in a solution that would improve security but keep being pleasant to work with. So I think six days ago yeah six days ago I opened this pur request in Rails itself to improve the appropriate path of strong parameters it's the change is now if you call 2h without filtering your parameters you get a exception. This is good because usually that is exactly what you want you don't want to send a filtering parameters inside your models or inside other parts of your application. If you call 2h and safeh of course you get all the information in the parameters but if you call 2h doing the filtering you get what you want. So in my opinion this is going to help everyone to get easier upgrades because we suffer a lot with this feature and it's also going to improve security. So how the road to production looks like in an application like this. We have Shopify running with both versions of Rails. We could make sure that all the tests are passing with both versions but how to deploy that to production. It's not just like deployed to production the Rails have container and that's it. So what we did was we deployed to production in a small space like a smaller set of production sizes we're using the new versions of Rails and we had to write some compatibility layers because Shopify needed to run in both versions in productions and there is no way for you to tell that one request is only going to hit one version of Rails. Let's say that your user is going to the checkout page and as soon you click in checkout your request that was served by a Rails R2 application is going to be served by a Rails 5 application. So we need to make sure that there is no difference between two requests to different versions. So in the Rails R2 era we had to do some monkey patches in the Rails itself to make possible to work in both versions. In this case we created the monkey patch to generate CSFR tokens in Rails R1 that is compatible with Rails 4 too. The difference is that in Rails 5 now we tried to create the same kind of compatibility layers inside the framework itself so you don't need to do that in the application too. So this is one of the examples in Rails 5. We created a legacy YAML encoding to parameters because like I said parameter was hashes in Rails R2 and they are not hashes anymore in Rails 5 so we had to be able to read the YAMLs, the encoded parameters in previous versions in the new versions. So we did what we called the weather rollout that we deployed to production just a small percentage of the servers. This is the message of our bots deploying to production with 50% of the servers running in Rails 5. So what we did was we deployed to production, we find bugs in productions, we fix those bugs, we hold back to 0% of the new version and we deploy later with the bug fixes again. We also did some benchmarks so this is our bot responding to me to a profiling of five seconds in one of the production machines so it gives to us a static trace with all the methods that are most encoded in the server in that period. So what we did was we provide different servers to compare the results and see if there is no performance regression between the two versions. If performance regression is found we reverse the deploy, we fix the regression in the framework and we deploy again. So this was the time that I actually did the deploy to 100% of the servers. It was 7 p.m. almost 8 p.m. at night, the same at 22. So I was really brave to do that but at least it worked. So after the deployed production we had to start the cleanup because we end up with a lot of feature toggles inside the awkward base checking if the Rails version is 4 or Rails version is 5. We had to first remove all those conditionals and we also had to remove all the duplications. In this project, a small team of four people working on it, it's not possible to a small team of four people remove all the duplications that you have in the Shopify code base. So what we did, we need to help from everyone in the company. Our first approach was to consider that printing duplications while you're running tests locally well enough, but turn out that people are really good at ignoring duplications. So we created a wide list of test files that have duplications and asking teams to remove all the duplications of those test files that are part of the components. And the last thing that we do after removing all the duplications, we upgraded the configuration to match the new defaults of Rails. And after that we start the preparation of the new upgrades. This kind of project never had end because Rails is always released a new version so we're always trying to keep track of the Rails version. So for the future, what we want at Shopify is to avoid monkey patch at all costs. That means that upgrades are easiest because we are only using features of Rails and the only way to do that is to keep all the number of dependencies smaller because as much as more dependency you have more likely to have monkey patch on the dependencies, we are going to keep the parallel side running the entire year and my goal is to keep tracking off the Rails master branch forever. It means that Shopify would be running in the last version of Rails forever and of course things will be breaking and it's not what we want to make everyone's concern. Everyone at the company should be concerned about keeping the application up to date. We also want to think more about backwards compatibility inside the framework itself. It's part of the Shopify giving back into community to pay me to work at the Rails framework to make it easier for everyone to upgrade this application. Because Shopify loves Rails and we want Rails to succeed and we want everyone to be able to use the last version of Rails without any problem. That's it. Thank you. The question was we have a white list of the applications if we configure the Rails to Rails into the application inside that file. We use the Rails application behavior but we have a special code that records all the duplications and if the duplication don't match the list of recorded duplications it fails the test. If there is no duplication in that file anymore and you have recorded it, it also fails. We are planning to open source that hold too. Maybe next month. The question is the recruiter said that there are ops in Frontend Developers at Shopify if I am in the ops team he has part of it like we don't have the ops, the kind of ops organization. We have what Google calls SRA and Facebook's call production Henry is kind of developers with ops background. Yes, I work in this team so my job there is to make tools for everyone so it's open source tools or any kind of production tools. More questions? I will be here if you have more questions you want to talk. Thank you.