 Hello, my name is Kyusha Trov. I came from far Russia. I am a Finland now. I work on Capistrano as a maintainer and I'm a Rails commuter too. I work at this awesome Rails shop called Evil Merchants. We do Rails development, consulting. We developed a group on Russia since the early days. We are doing a lot of work for eBay, which is my current project. And my topic today is about performance regressions in Rails. I'll talk about what is the performance regression, how to cache them, how to solve them, what are the tricks to avoid them. And about the tool I've been working on last few months to cache performance regressions in Rails automatically. And about the state of this tool and the future. Before we start, let's define what is performance regression. Performance regression is a situation where the software still functions correctly, but it takes more time to compute or it consumes more memory when compared with previous versions. And what is the performance regression in Rails? Imagine a situation where after an upgrade to a new version of Rails, fresh new version, shiny. The page load is increasing from 250 milliseconds to four seconds. That's quite huge, right? That's exactly what happened in Rails 3.2.13 because of some bug in sprockets. And now I have a story why we should care about performance regressions in Rails and why is it a problem for all of us. Right after Rails 4.2 release candidate one was released. They are a discourse team. They have quite a huge app. They decided to upgrade their app on this release candidate one and just to try Rails 4.2 because they were really excited about all those active record patches by Aaron, they expected some performance boost from adequate record. But what they found actually is that this Rails 4.2 release candidate one was two times slower than Rails 4.1. That was very surprising. There was some performance regressions, obviously. Luckily, this was just release candidate one and after applying a lot of patches on Rails by discourse team and by Rails team, they made Rails 4.2 the last release candidate, number three, as fast as Rails 4.1 and even a bit faster. Okay, now it seems like a pain in the ass for Rails committers, Rails contributors, but not for usual developers who only have some small or bigger Rails app. What is it a problem for everyone? Because faster, the faster is framework, the faster is your app and by speeding up active record with adequate record patches, Aaron Patterson made your apps faster just because active record is usually tied with the typical Rails app. To get a better understanding, I have some examples from Rails commits affecting performance and solving the performance regressions. Let's review them. But before we do that, let's think about what kind of metrics do we count on when we talk about performance regressions in Rails. These two metrics are timing and allocations. What is timing? Timing is the amount of time some method or some piece of code takes to execute. By reducing timing, we make our methods faster and we make our app faster. Here is an example of code, very small example, how to measure the timing. It's quite fast, straightforward. And we want to reduce number of allocated objects in Ruby interpreter memory because garbage collection is an expensive operation in Ruby. And by allocating less objects, we leave less work for a garbage collector. Here is an example how to measure a number of objects allocated in your code. We get statistics from garbage collector and then calculate it. And a lot of performance issues can be solved just by keeping in mind a set of tricks, how do basic data primitives in Ruby work. By these basic primitives, I mean strings, hashes, blocks, arrays, and so on. So we will start from commits, from rails that are working with strings, having some issues with strings. First commit by Aaron Patterson. Now I zoomed the diff. So the problem of this code was that it allocated, okay. The problem of this code was that it used plus method to concatenate to strings, while in fact, interpolation is way faster, really way faster. Another problem this thing solved is that we don't allocate one more string before we concatenate them. We just use the interpolation once. Here's another example by Sam Safran, who is actually in this room. So the problem was that this code was called on every active record attribute read. Imagine that. Reading attributes is the core functionality of their active record. And on every attribute read, we allocated a new string. And by moving out the string into the constant, freezing this constant, Sam reduced a number of allocated objects on the discourse front app by 5,000. That's 5,000 objects just by moving out one constant, one string to the constant. Here's another commit. It's not from rails itself. It's from the rack gem. It's not rails, the euro, but as you know, rack gem is used by rails a lot. So the problem of this piece of code was that it used, on every request, it allocated new variables, new strings, like content type, content length, and there were many of them. So on every new request, we allocated a bunch of new strings, and then use them. But using the same trick I described in the previous slide, moving out these strings into constant, we reduced number of allocated objects on one request to rack by 40%. 40%. That's almost a half of all objects inside the gem. We reduced it just by moving out these constants. This was quite a popular request on GitHub. Another example, it is from active support. It's working with time formatting. And just to replace the time pattern, we used gsub four times, allocated new string every time. But in fact, we could use only one regex to regular expression to do all the same things. So we don't need four gsubs, right? Just one regular expression. And this is the comment by my colleague, Ravel, his nickname is Brinopia. The thing was that in Ruby 2.2, in previous versions of rails, we used regular expressions for string replacement like this. Because replacing with regular expression was faster in Ruby 2.1. But it's an interesting case, because in Ruby 2.2, replacing strings with strings is now faster, 1.5 times faster. And this quad string method, it is called on every when you make a new query to your database. So if you have, let's say, 100 queries on some app, it's called 100 times. And now we have some examples working with hashes. Here is a piece of code from URL4 helper. And to override just one hash options, we allocated new array and code merge just to override one hash value. But as you can see, we can just use the hash set method without allocating new array without using merge. And here is the same trick by Aaron Patterson. We don't need to, as I said, we don't need to allocate new hash. We just set the value of the same hash without merge. Just right. It's here. So Aaron has done a lot of search performance patches in ActiveRecord. And he's doing a great job. And one more example about working with hashes. This piece of code is from ActiveRecord connection adapters. And it was called from there. And when we wanted to iterate over all hash values, we allocated new array with all hash values by using the values method. And then we iterated over it. But if we think there is a method called each value to do the same thing. And by using this method, we can avoid allocating new array in memory. We can... And this array may be quite big if the hash is big. It's good to remember that we want to optimize only hot code. By hot code, I mean the code which is called frequently on every request. Many times on every request. And Rails code is like code in the Rails, in the framework. Usually all the code in Rails is hot because it's framework used by thousands and thousands of different Rails applications. And it makes no sense to optimize the piece of code which is called once a month from a job. You won't benefit much, right? And now you have some idea what kind of performance changes I mean here in my talk. And what if we track these changes to see the Rails performance, the overall Rails performance. This idea to build a service to track Rails performance changes was suggested in the Rails base camp. And I became really interested because it's a great idea to how we can optimize, how we can save time for Rails contributors. Obviously, to track these changes, we need some kind of benchmarks to benchmark the code. Are there any existing benchmarks in Rails? The answer is yes. There is a great benchmarks right inside the discourse application. This is not a Rails benchmark, this is just a benchmark of discourse. And discourse app is using Rails, the framework. So in this benchmark, discourse is setting up the database, populating this database with records, starting Unicorn in production environment, and making a lot of requests using Apache Bench. And then this benchmark brings timing and memory usage. It's a great benchmark. It was used even by Koichi to optimize garbage collection algorithms in MRI. And this benchmark is used every time discourse upgrades on the Rails version to see if this Rails version is faster or not. It's a great approach, as I said, but we also want to measure performance of Rails components separately. I started from making a list of these components. You can see it on my slide. I didn't include active support here because active support is something very stable. We don't change it much. And it's almost frozen. And some parts we change a lot. Active record, action controller, action view. I wrote some benchmarks. I run them against a few major Rails versions. You can see this version on my slides. And just a second. And I visualized this common, these benchmarks on the static page using D3 JavaScript framework. I will give a link to this static page later. You can see these benchmarks. The higher is the bar, the battery is performance. We use iteration per second here to measure. You can see the trend how Rails becomes faster and faster, right? Now I will talk about what these benchmarks are doing. So here is an example of benchmark of active record finders. We create one record. We create one record in the database, the user record. And then we are trying to find this record using the find method. We are trying to find it actually 100 times. This is what we call microbenchmark. You can see some special helper called benchmark.trails. And let's check out what this helper is doing inside. So this helper disables garbage collector if it's necessary. It warms up the code because it's really important to warm up the code, to warm up the interpreter before we run our benchmarks. Then it makes some number of iterations. This number can be 100 or 1,000 or few thousand. And then we calculate the mean timing and object allocations. Those two metrics I mentioned before. Here is another example of the benchmark of create method in active record. We create one record inside the benchmark block and then see the result. Now I have a story why is it so important to have a correct benchmark and to run it them in the environment close to production. Here is the full app benchmark. By full app I mean some small Rails app with few controllers, few models. And in this benchmark we are making a few requests to this app. Here is the final version. And we can see that Rails 4.2 is a bit faster than some previous versions. But the first version of this benchmark looked like this. Obviously I didn't believe it because it was some mistake. And I decided to discover what's happening inside with a tool called stack prof. This is still made by Aman Gupta. This is a tool to see calls inside the Ruby stack and see some statistics about these calls. Here is an example of difference between stack calls in 4.1 and 4.2. And what I discovered is that the environment of this app that I run was not close to production environment that we have in Rails. And after fixing it, the benchmark started to look closer to the real life, right? Now we have some set of benchmarks and it's time to automate running them against new versions of Rails. We love automating things in Ruby community, right? There are great examples of automating. Examples like Travis CI that automates built metrics and tests and Hound CI that automates code review and automates review of the code for Ruby guidelines. So we can build some service to automate running these benchmarks against new versions, new comments of Rails. What do we want from this service? Just as I said, run benchmark for every new commit in Rails, run benchmark for every new pull request and report results of this benchmark to the pull request author. And to see some trends in Rails to see the charts. I started from making a prototype of the service. I built it in January. It was very small and simple. It could make some things like fetching a new commit from Rails and running only two benchmarks against it because in January I had only two benchmarks. Now we have more. At some point, I felt like I'm building yet another Travis CI with all those infrastructure and a lot of code, just like reinventing the wheel. And just a month before I started development, there was a service called RubyBench made by Sam Saffron and Alan. Please raise your hand if you heard about RubyBench. Okay, nice. For those of us who don't know about RubyBench yet, RubyBench is a service to automate running benchmarks for Ruby, for MRI. So in every commit to MRI repo on GitHub, it runs new benchmarks. It has nice infrastructure powered with Docker. And it has sponsored hardware because it's extremely important to run your benchmarks on a dedicated hardware. Because if you run these benchmarks in cloud, the CPU in cloud is shared between many instances. And if someone else is mining Bitcoin on another instance, it may affect your benchmark results, which is bad. And after getting in contact with RubyBench authors, we decided that I will just embed real support into RubyBench. And now let's see how it works. On every new commit to the GitHub repo, on this example, it's ruby repo. GitHub makes a web hook request to our web application. This web application is starting a background job with remote connection to our Docker server and where the Docker container is created with this exact version of Ruby with all the benchmarks used for Ruby or for Rails, depends on the repo. And right after the benchmark suite is finished, the result is reported back to the web app where we can see the updated chart. Here is the UI of RubyBench. And there is a... You can choose the benchmark on the sidebar on the left. And on the right, you will see the chart for this benchmark. You can zoom it. You can check out the comets and so on. Now I have some information about the architecture of Ruby Bench. Obviously, we have three components. You can find all those three components on GitHub. There is a web application. There is a repo with Docker files and there is a repo with a benchmark suite. Why do we use Docker here? Because Docker is a great way to isolate environments, to isolate dependencies. Otherwise, it would be just messy to keep all those trunk rubies built inside one RBM or RBM user. We have a benchmark suite repo with a lot of micro benchmarks for Rails for Ruby. And we update the benchmark suite sometimes. But now there are like 300, maybe less benchmarks, micro benchmarks for Ruby itself and about from 10 to 20 benchmarks for Rails. And we are working on adding new Rails benchmarks. Now we benchmark ActiveRecord, ActionController, ActionDispatch, Router and Views. What about the web app? It's just a simple Rails 4.2 app run on Heroku. We're using delayed job to run remote connection to our Docker server in background. And we use high charts JavaScript library to render all those nice charts. Let's talk about results of Rubyvenge and about Rails support there. So now we have a set of benchmarks for Rails. We run builds for Rails for every new version of Rails. We are working on support on per-commit benchmarks. I think we will launch it in next few days. What is exciting is that since November there are already 250,000 builds for Ruby, builds of micro benchmarks. And what's even more exciting is that there was already one performance regression of the Ruby code. And this performance regression could be solved on the next day after the wrong commit was made, because we knew exactly which comment caused this performance regression. What are our plans about benchmarking Ruby and Rails? We want to implement the pull request benchmarks for every new pull request Rails just to report the performance back to the pull request author. We also want to make weekly reports and in this case, some contributors can get an email like in this week. Active record became 5% faster or maybe not. If the performance remains the same, we simply don't need to send this email. And we also want to implement JRuby support, because we already have a set of benchmarks for Ruby. And we can just run the same benchmarks against JRuby. So today we talked about what is the performance regression, how they look like in Rails, how they can be solved, how to find them, how to track them, what are the tools. We talked about building a benchmark suite and automating the tracking of the performance. But wait, what if you are not a Rails contributor and you are not planning to be ever? How can all these tricks help you to make your app faster and to avoid performance regression and just to avoid performance regression, right? To track your own app performance. First, you can start by studying how do Ruby primitives, basic Ruby objects work and how to treat them properly. There is a great book called Ruby by Pat Shaughnessy. I mentioned this book in my talk because it covers all the data primitives, how do they work, not only in MRI, but also in Rubinius in JRuby. If you are interested in more tricks like I showed about writing fast Ruby, you can check out this talk by Eric Michael Sauber. It's called Ruby. And so next, you can write a discourse like benchmark for your own Rails app. This benchmark can set up the application, set up the database, populate it with some records, start server in production environment, make a bunch of requests with Apache Bench or some other tool to some lists of endpoints that are important for you, and then print the page or some other page. And then print timings and just run this benchmark after every comment on Travis and have some idea of the performance. You can also subscribe to the Rails Weekly mailing list. It's also called This Week in Rails. In this mailing list, Rails contributors write about what happened in the Rails repo and what are new features or if we fixed some regression in Rails or fixed some issue. Here are some links about my talk. The first one is about the static page with all those Rails benchmarks since Rails 3.2 and until Rails 4.2. Another one is the initial prototype of the service to benchmark Rails. I call it Rails Perf. It's an open source from GitHub. And you can also check out Rubybench components on GitHub in our organization. Here are my user names on Twitter and GitHub. Please follow me. Also, please don't hesitate to contact me if you're interested in this topic about Rails benchmarks, about Rubybench. If you want to contribute it somehow, we will be more than happy to discuss it. You can also check out the Twitter and website of my company. And thank you so much for coming to this talk.