 I mean, I guess, I guess if we get stuck in Minneapolis, that's where we're supposed to be anyway, so. Right, Evan? Is that why you were late? You were scouting locations? Okay, hold on, I gotta do this real quick, okay. This is actually for expenses, to prove I was here. Computer, don't we? Is that on? Are we on? On? Okay. Hold on, one sec, hold on. No, one sec, one sec. Geez. Okay, okay, all right. Okay, there we go. All right, let's start, let's start. So this is a just-in-time presentation, not to be confused with just-in-time. I'm Aaron. Okay, so I'm an extremely, extremely horrible procrastinator and instead of working on my presentation, I kept putting stuff and doing actual work and putting stuff in Slack and Eileen suggested that I title my talk Things I've been doing besides preparing my talk because I get a lot done when I am supposed to be doing something else. Anyway, I get extremely, extremely nervous about giving presentations and I'm always paranoid that I'm going to go way, way too fast and I won't have enough slides. So the other day, DHH made this tweet. He said, I'm just going to say it. I hate foo, bar and baz as prototypical variable names is time for a change. And this made me laugh a lot. So I made a little video about it and I tweeted about it and I'm like, there's one slide done, yes. This is what I'm doing, working on my presentation and then, and then, and then DHH liked it. And I'm like, yes, there's two slides done. I'm getting like four or five slides out of this. So I think the presentation is going okay so far. Anyway, oh, hello there. My name is Aaron Patterson. I may look different on the internet than I do in person. This is what I look like online. If you don't recognize me, if you don't, some of you may have noticed maybe a cat picture coming up on your screen. I don't know if you recognize this, but Jason and Kevin, I'm not sure why. You should have accepted my airdrop. I'm not sure. I had sent this one too, geez. Gabrielle, come on. Just trying to send you photos of my cat, please. Anyway, I work for a very small startup company called Github. I don't know if you've heard of it. It is the only legit company I've ever worked for. I really enjoy using Gith, but I will not force push it on you. Now, I make this pawn at every conference and my coworkers told me that I really need to branch out. But I had to say to them, look, look, I'm just really committed to these Git-based puns. Not even flogging me is going to stop me. In fact, even if you reflogged me, that wouldn't stop me. Anyway, if you don't like these Git-based puns, we could switch to SVN, but I just don't know what the Git-diff would be. How long do we have left? All right, so, I love titles. At Github, I'm a level five engineer. I am level five, and despite the fact that I have played Xenoblade for over 90 hours, I still don't have enough X-speed level up to level six engineering. So I'm going to be working on this every day and my boss is here in the audience. I'm going to be working on Xenoblade every day until I can get up to a level six engineer. So, I guess I should go to real content. No, not yet. I love local businesses. I love local businesses. Before I visit any place, I look up the best local businesses for that place so that I can go help support them. And I was inclined to learn more about Pittsburgh. Come on, come on, come on. Anyway, so you may or may not know this. Hines is a local business, so I bought some ketchup, there's hashtag local, yes. These are really exciting. I wanted to support local businesses even more, so I picked up some Hines Texas barbecue sauce. That is also hashtag local, this is exciting. Then I went and picked up some Hines beans and these are actually amazing because there's 57 varieties of beans. You may not know this, there's 57 different varieties. But anyway, I thought this was really neat, but unfortunately I looked at the back of the can and they're actually made in England, so they're hashtag not local, so I apologize for that. I was trying real hard, I'm not sure what the deal is there. So anyway, I love local businesses and I hope you got the point. Pittsburgh people, yes, yes? Come on, I know there's like five people in the audience that got this one and I'm sure you're laughing real hard. Thank you, Evan. The end of Rails comes where I waste one of your hours. So the day before yesterday I learned, I've got 99 problems and I guess they're all yours. Anyway, I really, really love coming to RailsConf and it's really exciting for me to be here. I like to visit everyone here. I feel like it's kind of a homecoming for me. I get to see all the old faces and new faces, meet new people, see other folks on the Rails core team, do a lot of mingling. So I want to catch up everybody on what I've been doing over the past year. So this past year I actually, I became an uncle, which is exciting for me. I've never been an uncle before. This is my niece. This is me touching her face. Surprise, my sister, let me do this. Since this past year, I actually became extremely famous in Japan. This is true and if we have time at the end of the presentation, I will tell you exactly why. But I made this tweet right here. It is one character, one character and it garnered 81,000 likes, which from a social media perspective is like a really good deal. Like characters per like is like very, very efficient here. This is worth my time. So any of you social media managers out there, look at this, look at your likes per character. I mean, this one's like great. Anyway, this is well worth your investment. I decided to become a thought leader again. Every year I want to be a thought leader but it just doesn't work out. But the only role model I have for thought leadership is DHH. So I decided the first thing that I should do is get a head shot like his. So I've got this new head shot. And you may notice the resolution is a little bit off on mine but that's because I've got an older Mac. It's still a 480p camera. So I'm sorry, not quite there. I'm still working on it. But anyway, I saw he like, I follow him on Instagram and he has this amazing desk. So I thought, well, I want to have an amazing desk like this too. I'll set it up like I can become a thought leader too if I have the right desk. So I want to show you, like I put together mine, like this is DHH's office and here's mine. It's exactly the same. You can see it's perfect. In fact, one thing I noticed when I took this photo, there is actually that photo. But really, I'm actually really, really good friends with DHH. Just check out this selfie that we took together. In fact, he invited me over to his house the other day so I went and I took a photo and this is it. Okay, okay, that's a little dark, isn't it? All right, this is just a test slide. So I was doing a test slide here. I just wanted to see if this would actually work. I don't know if you can see it. There's something right there in the slide. It's right between the S and the W. I'm not sure if you can see that right there but it is a zero-width space that is character that's between those two and we're actually gonna talk about that. We're gonna talk about that in a bit. This is just a test slide. Anyway, so this tweet really annoys me. So I have really weird conversations in my head. Like, I think I have a problem focusing or something. I don't know, I have weird conversations with people in my head. I think, okay, we're gonna talk about this and this is how the conversation is going to go and I kept replaying it over in my head and I'm like, I have to get this down somewhere so I made a video out of it. This is the video I made. Anyway, so this is what's going through my head. I'm having this conversation in my head. Unfortunately, DHH is the last one to say something. So anyway, I'm thinking about this. I'm thinking about this foobar and baz as prototypical variable names and then I'm thinking about code in Ruby and I don't know if all of you know this but a valid identifier in Ruby is anything that is not a space basically. So this code works. I'm defining a method that's named smiley face and I'm calling the smiley face method there at the bottom. So if I run this, it'll work and print out neat. This also works, I've got a method with smiley face and it takes keyword arguments and the keyword is a green heart. Defaults to a train and you can call it and passing keyword arguments and it prints out that little trolley thing there. So this is what I was thinking about is foobar and baz because they're really just representing something and I thought, okay, well what else can we do with this code? Here is another example. Can anyone see what's going on with this code? I'm gonna wait for you to type it out on your laptop. You can give it a try, it actually works. This really works. No? Okay. All right, so I'll explain it. Right here at the very top, the name of this method is actually a non-breaking space. So there's a non-breaking space right there and down at the bottom there's another non-breaking space so I'm calling the method with a non-breaking space and if we look at this in Vim you can actually see it. On the right is the source code in Vim. You can see the non-breaking space character there but when we cat the file and actually run it, it totally works fine. So let's extend the program just a little bit. We're gonna do one extension to it here. In this case we have a method with the name, the name of the method is a non-breaking space and we have a parameter given to the method that is a zero-width joiner. So right there that's our name, our parameter we're calling puts. So here it is again in Vim, you can see like right there this is of course this is Seattle style, there's no parentheses. So you can see in Vim we've got the non-breaking space and the other space and it totally works fine. So what I'm proposing today is that instead of using foo and bar, we replace foo with zero-width joiner. We replace bar with non-breaking space. So all right, with that let's write some production ready code. I dare all of you to put that in your app code tonight. Wait, don't do that, don't do that. Don't, don't, okay. All right, so today, surprise, surprise, we're going to talk about performance. I always give performance talks. We actually just had our performance reviews at work. My boss is in the audience. We, you did a great job on my performance review. Thank you. Anyway, so we're going to talk about performance today, but I don't want to talk about the speed of your code. I want to talk about the speed of your development, like your actual development getting features done and actually making that website that you need to ship. Now the thing is I was thinking about this and we're always getting faster. We're learning new tools. We're learning new technologies. We're learning new techniques. But unfortunately, it seems like as time goes on, your team velocity gets slower and slower. It takes longer to add a new feature. It takes longer to ship your code. So why does this happen? This seems paradoxical. Who is this mystical man? Why are we concerned with his months? But seriously, why are we slower? Why are we slower to ship these features over time and what can we do about it? Well, there's different ways that we can deal with some sort of situation like this. One of those ways to deal with this situation is to rewrite. We want to rewrite our code. We want to chase that dragon, that dragon that we felt, that high when we wrote Rails new and we can get all of those features out the door really quickly. We want to throw away all that code and say, okay, I'm going to begin again and feel that exhilaration when you can ship new features quickly. But if you're doing this all the time, if you're just rewriting something, all you're doing is going down the same path over and over again. Maybe the weather has changed, but the places you're going are exactly the same. What good is this velocity that you get if you're just running down the same path over and over again? Now, the day before yesterday I learned that if you don't want to deal with these hard problems in your program, you can just hire someone to fix it. Hire someone better than you to fix that scaling issue in your application. Now unfortunately, I cannot afford that. So I have to pick a different solution. I choose to refactor. We could choose to refactor our code. We could make performance improvements and pay off technical debt, et cetera, et cetera. But the problem is it seems like we have these two different arcs juxtaposed against each other as application developers. It seems that we either have to be software writers or we have to be a PhD in computer science to work on an application. Now in my opinion, there has to be something in between. We can't just only be software writers. We can't only be PhDs in computer science. We have to go deeper into the application. We have to go down this cliff so that we can actually solve these more difficult problems. Now, as we learned a day before yesterday, I am the only one who can fix this. And we've recently learned that that doesn't actually work out. The truth is that you're the only one that can fix your application code. So the generic features that Rails provides gives you a great bootstrap. You can get going early. We give you all the things that a generic web application would require to ship a feature, but as time goes on, your application is getting more and more unique. You know your business logic. You know your business. I don't know your business. I can't help you with that. So I think this is one of the reasons that application development starts to slow down over time is we have to get better at our particular domain. So those generic features in Rails give us a huge, huge boost. And I think this is why I like to focus on Rails a lot. This is why I speak about profiling and speed improvements with Rails is because I want people to be able to take their Rails application and live with it for a long time. I want them to have a successful business, not have to run into these scaling issues as often. So I love Rails and I want Rails to continue for a very long time. So this is why I study these particular things. So today we're gonna be talking about profiling and tuning. We're gonna specifically look at runtime and memory management which sometimes I like to call time and space because it makes me feel like I'm on Star Trek. And it makes me feel a little bit more important than I actually am. So let's look at some stuff. We're gonna look at types of profilers and how to use them, how to build them and basically how to improve our application performance with them. So there are two types of profilers that I typically deal with. One is called an exact profiler and one is called a sampling profiler. And to figure out which profiler you need when you're profiling your application you just need to ask yourself a couple of questions. If you're asking, the thing that you're trying to solve if you're asking is how many, how many of something, how many method calls, how many object allocations? If you need to know an exact precise number then you wanna use an exact profiler. Now if you don't care about an exact number like let's say you need to know is this what is the slowest method? What percentage of time does this particular thing take? You wanna use a sampling profiler. So if your question is how much, use a sampling profiler. So in order to figure out which one you need just ask the question out loud. What is the thing that I'm trying to do? What am I trying to solve? So let's take a look at exact profilers. We're gonna actually implement an exact profiler so that we can understand the way that it works. This is an example of an exact profiler written in Ruby. It uses the TracePoint API. This TracePoint basically what it does is it hooks into the call method and every time a method gets called once we enable the TracePoint our hook method will actually get called. So we listen for the event and every time we get an event we just keep track of that and increment the counter. And you'll see at the bottom of this code here it outputs the result which says, okay, benchmark me was called once, fast was called a thousand times and slow was called 10 times. Now if this code wasn't so simple you might look at this and think, oh, the place I need to optimize is the fast method or the method called fast. That's the thing I need to optimize, it's called the most. But obviously when we look at this code we can tell the slow method is a slow one. So this is the particular case where we wouldn't want to use an exact profiler. This is misleading us. If our goal is to speed up this code we don't want to use an exact profiler in this particular example. Now the other profiler is a sampling profiler and this sampling profiler, what it does is it stops your program at intervals, regular intervals and it asks the program what you're thinking about. So every so often, it asks what you're thinking about and just records it. So every so often we have a program here that's running so time is along the bottom, our program runs, it stops, we record what it's doing and we keep doing this at regular intervals until we're done sampling it. Now the idea behind this type of profiler is that let's say you have a slow method. If your method is slow then the probability of being in that method any time you stop the program is higher. So we can tell that if we're in this one particular method frequently, if that shows up frequently within these samples that's probably where our slow code is. So we can write a sampling profiler and Ruby here's an example of it. What this profiler does is it starts up a thread and it asks the main thread at an interval, what are you doing, what are you doing, what are you doing every so often and you can see at the bottom we have results from that and in this particular example we have almost 6,500 samples inside the sleep method and maybe one sample inside the slow method. So we don't know how many times the sleep method was called but we do know that when it's being sampled it's very frequently in the sleep method. So 99% of the time we sample, we know it's inside the sleep method so that's the thing that we need to target. Our fast method didn't even show up in this profiler. So when to use exact versus sampling? It depends on what you're trying to measure. Again if you're trying to get absolute counts using exact profiler, for example the number of objects in the system or the number of times a method was called or API or whatever. If you're trying to reduce time or space then you want to look at a sampling profiler. If you're not sure, use a sampling profiler. Sampling profiler is what you need about 99.963% of the time. That is with one sample. Check my math, it's wrong. So let's take a look at some runtime profiling and tuning. We're going to talk about tuning your runtime. Profiling your runtime and tuning your runtime. So my favorite tool for doing runtime profiling is a tool called Stack Prof. This thing uses MRI's built-in sampling capabilities so it doesn't spin up its own thread. You can actually use a timer inside of Ruby. Ruby has a timer inside the C code, not inside the Ruby code, that lets you sample every so often. So Stack Prof just uses that built-in capabilities and it looks very similar. The API looks very similar to the one that we wrote earlier. We just wrap up our code inside this block, execute it and when you execute it, it saves a profile to a dump file. So I've specified profile.dump here. And when we're done with that, we can just run Stack Prof on that and it dumps out all the sampling information. And this looks similar to the information we got in our profiler, just much nicer. So we get kind of the same information, 999 samples. Again, we can't tell how many times the slow method was actually called. We just know that it's in the slow method very frequently. So we can use this to benchmark Rails boot time. So let's benchmark the Rails boot process. Let's talk about the Rails boot process. So let's take a look at how a breakdown of the Rails boot process. If we look at it like this, where time is moving along to the right there, we can divide it into a few different sections. So at the very beginning here, that's where we hit enter. We hit enter right there. Then we start requiring files and once we've loaded and required all the files, we start what's called compilation. We're compiling the app. It's not really a compilation phase. Essentially what it's doing is taking all of your rack middleware and putting it all together and then handing that middleware off to whatever your web server is. In this case, we're using Unicorn. So after compilation occurs, we hand off the application to Unicorn. Now after that, when you get your first request, it has to compile the views and then actually execute your business logic. So if we look at this from a little bit higher level, we can say that these steps here, these requiring files and app compilation, we can say that that's our time where we're not ready for requests yet. We're spending time here and we're just not ready to respond to a request. And now the first request that comes in, it's gonna spend time compiling our views and then executing the business logic for that page. And then the second request is only going to execute the business logic. So if we wanna profile these things, now we know what to talk about, where we're actually looking. So knowing this, we can figure out what we need to benchmark depending on the task. So what we're gonna do now is we're gonna look at benchmarking a script, just a startup script. This is an example here. We have, we just load config environment. This is how you would benchmark that very first slice before we're actually compiling views. This is going to load all of your applications, information, all of your, I don't know, models and stuff if you're using production mode. Maybe it's lazily loading and development. But this is how you can benchmark that overhead for your scripts. Now if we look at the output from this, you'll see, okay, it looks very similar to our previous output, but I wanna call out one thing in here. You can see that we're spending some time in garbage collection, right there, GC. So we can see what percentage of our time is spent in GC and we're gonna look at how to improve that time a little bit later. So runtime tuning, now talk a little bit about tuning your runtime. Now the thing is Ruby will try to execute your code as fast as it can. So there's not really much runtime you can, tuning you can do. The best thing that you can do is to write faster code. Ha ha ha ha ha. Obviously we want the Rails framework to be as fast as possible, but at some point we can't make the framework any faster. So you need to look at your own code to see what's slow about it. And the most advice I can give about that is say like, okay, use these tools, find where your bottleneck is, look at it and say, well is there a faster way to do the same thing? Can I do this thing in a faster way? Is do we need to even do this thing so much? Maybe I can only do it, I only need to do it once or twice. Or maybe I can defer this until a later moment. Maybe I don't need to do it right now. So these are questions that you should be asking yourself as you're trying to tune your application. All right, let's take a look at memory tuning. We talked about tuning runtime, let's take a look at space, actually using memory in your application. But before we get to memory tuning, we need to talk a little bit about memory, how memory is handled in Ruby. And I hate this so much. This made me so mad. How did David know that I am going to talk about garbage collection in this presentation? Am I that transparent? Why? He said that in his talk and I'm like, no, how do you know? Anyway, so let's talk about a garbage collector. The GC has two responsibilities in your application. One, it frees memory and this is the thing that most people think about the garbage collector doing. It's freeing up your memory. But the other thing that it does for you is it actually allocates memory. I think people don't think about this as much and I think the reason is because of the name. It's called garbage collector and you think, oh, I'm just taking stuff away. But it's actually in charge of allocating your memory too. So what is MRI's GC? I'm going to tell you what it is. We're not going to go through all of the algorithms and stuff but I just want to give you a high level overview so that when you have time later you can Google these words because that's really what's most important is knowing the key words to Google. Seriously. So Ruby's garbage collector is a mark and sweep garbage collector. It's generational, has incremental marking, lazy sweeping and uses a free list for allocation. I feel really, really bad for the translator. I'm sorry. Anyway, so the generational part of this just is a optimization on top of mark and sweep. It makes your GC faster. Incremental marking and lazy sweeping mean that we have shorter pause times. So when MRI's garbage collector is concurrent it is not parallel and what that means is as your program is running every once in a while it has to stop and do something. Now these incremental marking and lazy sweeping mean that those pause times are very short and what that does is it increases the throughput of your program. So you can actually run stuff through it faster. So one question I think is interesting is when can the GC collect garbage or when does the GC collect garbage? When does that happen? Well, there's three main places. If you just call GC.start it will collect garbage though it's not guaranteed. And none of these are guaranteed. These are just locations where it could collect garbage. That's why I said can. So if you call GC.start it will do a collection. Maybe it's not guaranteed but it will, don't worry. Now another case is when you allocate an object. If you need to allocate an object the GC may do a collection. Or if you allocate memory via malloc the GC may do a collection. So allocating objects and allocating via malloc are two very different things and we're gonna talk about the differences between those in a minute. So this leads us to what is an object in Ruby? In Ruby an object is a 40 byte chunk. So it's a chunk of memory that's 40 bytes wide. So let's say we have some code that looks like this here at the bottom. It's a string that says OMG, this is a very long string. Okay, now this string is obviously too large to fit inside of a 40 byte chunk. So how do we actually store this? How can we have a string that's this long? The way that it actually works is rather than storing that string inside of the Ruby object itself we have a string object that points at something else. So we have a string object, a Ruby string object that points at a C string object. Now the garbage collector is in charge of allocating these Ruby objects. And malloc, the system malloc is what's in charge of allocating these strings. So we have a Ruby object that points at a C object. So these are, in this particular case we have two places where a GC could have happened. Now Ruby doesn't allocate one object at a time, instead it allocates a page. So an object is a 40 byte chunk and when you say give me an object it pulls that object out of a page that contains many objects. So Ruby won't actually, or the GC won't actually allocate one it'll allocate a page and then it'll give you one of the objects inside of that page. So in this particular example we allocate our Ruby string. Our string is just a Ruby object inside the page and it points at a different string that was allocated via malloc and in this case we have two places where the GC may have run. So slot is a location where a Ruby object may or may not be and this slot term is important when it comes to tuning the garbage collector because we allocate in terms of slots. Well in terms of pages and those pages have slots and this is the way that we tune it is by the number of slots that we have available. So we can control the total amount of free space in the program and the way that we control it is via this term slots. So let's look at some tuning parameters. GC tuning is done via environment parameters and we're gonna take a look at some of them and what they do. So this is the first one we'll look at is called Ruby GC heap init slots. I don't expect anyone to remember this. I promise I will put these slides online so you can just Google it later. But we can see how it impacts the process here. If we run Ruby you can see the number of available slots with no tuning is about 1800. We have about 1800 slots available. Now if we say init with 600,000 slots and we run it it has about 600,000 slots available. So with this particular parameter we can say I want this many objects or this many free spaces declared when the script boots or when the program boots. So let's take a look at that profile that we looked at a little bit earlier. This is an example here we have our app boot. This is the default case that we saw previously no changes you can see how I ran the script at the bottom of the slide there. Now it's spent about 16% of its time in allocation. 16.5% of its time in GC. Now if we tune the slots and say we wanna init with a bunch more slots we can do this. We just say let's set that environment variable to 600,000 you can see the command at the bottom and in this case we only spent about 8.5% of our time in the GC. So we can see kind of an impact on that when we do go to boot our application. In the top one there's no tuning and it took about 4.1 seconds and in the bottom example it took about 3.5 seconds and this is a default application, default Rails application where I just randomly chose a number and the nice thing about this is that we didn't have to change any of our application code and we were able to decrease the boot time just by saying hey I want you to prepare a bunch of room for allocation. So we didn't need to change our app code at all. You can use this if you measure the objects that are allocated in your system you can come up with a good number for this and then decrease your boot time just by tuning this particular parameter. The other thing that we can do is we can control growth and these are very long, very long. So we can control the growth of our heap via these particular parameters and again you don't need to remember these the important is that we're applying a ratio to the amount of free space that we have in our application. The GC tries to maintain a certain amount of free space for you to allocate into and we can specify those ratios so we have a minimum ratio, a maximum ratio and a goal ratio. So we have our min and our max and we're trying to hit the goal so kind of a window there. Now let's take a look at a test program. We have a program here, this is our test program and it just allocates a whole bunch of objects. It allocates 600,000 objects. Now what I want to do is I want to study how many times the garbage collector had to expand its heap while we were running this program. So each time we allocate a new object we may have to expand that heap and we want to look at how many times we actually had to expand the heap. So if we set our goal to particular values we can look at or we can, excuse me, we can study how many times it had to expand the heap by checking the GC count. So if we tweak the goal numbers a little bit and then graph it we can see a different behavior and this is an example of that behavior. Our goal, the blue line, our goal is set to 0.2 and our green line, the goal is set to 0.7. Now the interesting thing about this is we're able to hit that 600,000 mark in fewer GCs with a larger goal. We're essentially telling the GC okay I want you to be aggressive about allocating memory, then allocate it and then expand into that. Since we're aggressive about allocating memory we're able to hit that goal or hit that 600,000 mark sooner. So unfortunately this means that we have more memory growth. We're not being conservative about our memory growth. The blue line, we're more conservative about memory growth but we have to spend more time in GCs or using more CPU time. So it really depends on what your application is doing and what you're trying to tune for. If you want to tune for fewer GCs you may want to increase your goal. If you want to tune for lower memory usage you may want to decrease your goal. And you can actually try this at home. Again this slide is literally just for being online later so don't write this down or anything, please. Like you wrote down the example code with the non-breaking space in it. All right so let's take a look at some memory profiling. I saved this for last because I actually have a story to go along with it. At work, somebody pinged me at work and we have an application at work it's called the GitHub Enterprise. And basically what it is is it's a version of GitHub that we ship out to clients and they host it in their data centers. It's just I guess stored in a container or something. I don't really know all the details but one of the people on the team came to me and said Aaron I've noticed that as time goes on the application that we're shipping is consuming more and more memory. It's getting larger and larger and do you know, can you help with this? Do you know why this is? So I said okay well I'll take a look at it and see if I can figure out what the problem is and the first tool that I reached for was Stack Prof because it was my favorite tool to use and this is an example of using Stack Prof with object allocations. We can use a sampling profiler with object allocations. So I ran this code and I required our environment and took a look at the output from that and this is what the output looked like. It told me that we were spending, this isn't actually our application, this is a sample app but it said oh we're making a lot of allocations inside of the require method. Now the problem with this type of profiling is that even though we made a lot of allocations in the require method, those objects aren't necessarily retained. They may not live for the life of the program so if they don't live for the life of the program we don't really care, it's not contributing to our bottom line so to speak. So in this case we actually needed to switch to exact profiling. This is a case where we said I need to know how big is the heap, I need to count exactly the number of objects that are in memory right now. So this is a case where you want to use exact profiling and to do that I use object space, this is my favorite tool for doing exact memory profiling and the way that I use this is via heap dumps so this is a way to get a heap dump and I will describe it a bit here. This dumps your heap to a JSON file and in this case I'm just dumping it to a file named mydump.json. This line up here, the second line, what it does is it enables allocation location recording. That's fun to say. Any objects that are allocated before this line we don't know where they've been allocated. The garbage collector does not keep track of where they're allocated. After this line when we call this method it tells the GC hey every time an object is allocated I want you to record where it got allocated. So anything that's allocated after that line we know where we can find out where it was allocated and that information will actually be inside the heap dump. So after I enabled that I said okay let's require the application boot it up. After we've required the allocation I say okay I want to do run a GC, I want you to clear out any objects that are temporary that aren't going to live for the life of the program. Then dump out the heap. So once the heap is dumped out we just have this JSON file and each line in the JSON file represents one object. One line is one object. And I know this is hard to read so I've made it a little bit nicer here. So this is an example of an array. We can tell it has the type, it says array. It gives you a list of all the references so those are all the objects that it's pointing to. We can actually see the allocation location of that object. And finally my favorite is the memory size of that object. And these JSON files make for a very extremely easy analysis. So for example let's say I want to know how many objects are alive? Just how many objects do I have in the system? Very easy I just count the number of lines in the file and I know. So this example had about 185,000 objects. Done. Okay so how about I want to know what file is most popular? This is one tool I really like to use, JQ. If you don't have the JQ tool, get this, it's amazing. Basically it's like a really great grep for a JSON. So I can say hey I want to extract the file, just the file field. I'm gonna sort it and get unique them. Then I'm gonna sort those by the number of counts and then just get the top 10. When I run this I can see okay well the top allocations came from activesupportdependencies.rb. So I know it came from that file. Okay I focused in on that file. Now I can use JQ again and say well I only care about allocations that are inside that file. So I want you to filter out all the allocations for that file and then give me the line. Then we run through the same, basically the same command again, sort them, get the unique count, sort by account and then get the last 10. So we can see here okay the most popular line inside that file is line 283. So I know okay I need to go look at that file, look at that line and now I can kind of get an idea of why we're allocating so much stuff. So we can also answer other interesting questions like how much memory does activesupport allocate? So I just say okay give me all of the records whose file contains activesupport, extract the mem size from that, pipe that to a little awk command and sum the total and we can see that okay activesupport allocated what is that like around three megs. Okay so back to our performance problem at work. We ran into a thing, I want to talk about hidden references, this is the thing that we ran into. We're trying to find what has references to what. Basically this heap dump contains all the information of the entire object graph, we can make a graph out of it so let's talk about references. This is very simple, we have an array, we have references to things, this is obvious. We can see okay we have an array, it points at two symbols. Now here's another example of some references in a program, we have a lambda, this function foo it returns a lambda and that lambda sums the two values that you passed in. So the variable m contains two references and I promise this isn't a trick, there are no non-breaking spaces in this code. So all right here's a question for you. Here's a similar code, the foo method returns a lambda, how many references does this lambda have? You don't need to answer, I will tell you the answer, just think, how many references does it have? It actually has two references, the reference is the number a and b and there are two reasons why this lambda has references to those variables, one Ruby doesn't do what's called escape analysis, so it doesn't look at the lambda block and say oh I'm not actually using those things, there's no reason to keep track of them and the other thing which is basically an implementer's nightmare is a thing called binding. So if you ask the lambda for it's binding, that binding object is the environment in which the lambda was created. You can actually ask the binding, hey what local variables do you know about? It says hey I know about a and b, they are local, when I created, when the lambda was created and then you can say hey binding, give me the value of a and b. So we actually have these references, this lambda contains references to a and b but you can't really see it when you're just reading the code. So I was able to find these particular locations, unfortunately I didn't understand this code, like I could point to it and say hey that's the problem, we're allocating a bunch of stuff there, but I couldn't really fix it, so I worked with other folks on the team and they actually rolled out fixes for this that eliminated those hidden references and we were actually able to drop 200 gigabytes from our memory from our production application. So this was like really awesome and I praised the person who did this a lot, it was amazing, a really amazing change. So I wanna talk about some tools that I want. And then maybe some thought leadery stuff and then we'll wrap up with another story and maybe two stories depending on my time. So let's look at the Rails boot process again. Unfortunately so far all we've been doing is looking at analysis of scripts, like maybe running DB Migrate or other scripts that you run with your application. We haven't looked at anything else. What about app compilation time? I mean when we run this it's easy to know, okay this is how much our scripts are spending inside of just loading the application, but what about that actual compilation time? I wanna know how long does it take to restart a web server? I wanna optimize that, how do we deal with that? So how do we deal with benchmarking the rest of these things? We only looked at that one. So today we're gonna take a look at some new rack up options that I've added in order to deal with this particular situation. So rack, we merged this into rack a couple of weeks ago. This is one of the things that I was doing instead of working on my talk. In newer versions of rack you'll be able to say, hey rack I want you to dump the heap before you or after you compile the application but before you give it to the web server. Or I want you to profile the application up until you give it to the web server. So we're going to build this tool interact for you so that if you wanna know why is it taking so long for my application to restart or be ready for requests we can actually have this built into rack and give you some information. So you know how to identify problems in your code and we've written the benchmarking code for you. So this just leaves us with two other issues, our warm up time and our actual and our run time. So we're able to profile script boot time or able to profile compiling applications but we're missing two components. Now last year Eileen spoke about system tests and this is a really important thing. So besides the fact that now you can do entire system tests this is an important refactoring on Rails itself it means that we can actually run real requests through the application. So something that looks more like a request that you would see in production. Before we had system tests most of the stuff, I mean we on the Rails core team knew this and maybe most people did not but a lot of the integration tests were basically just BS. And I wanna say that, I'm trying to say that in the nicest way possible. They would actually run the data, run these requests through only your application it would not consider any middleware that you had in your application and this is the advantage of the system test is that they actually run the data all the way through the middleware into your application and then back out. So you can get a more real world idea of performance of your application. So now that we have that, now that we have that system in place for running these system tests. Sorry, I said I'm a little scatterbrained, right? Okay, so now that we have the system in place we can run real requests through and actually benchmark those. So we have a script for doing this at work. It's called this, this is bench app. We use this for benchmarking our actual requests at work and this is what the output looks like. It's very easy to read. So I wanna tell a little story about using this tool. We had a, this is amazing. This was amazing. We had a page at work that would allocate 13 million objects per request, 13 million. And I mean, this page, this page would return in like a second. So anytime anyone tells me Ruby is slow I'm like, what? We did 13 million allocations and we're able to render that page in a second. Like this is pretty good. Anyway, I was able to get it down to about 300,000 objects per request. Using these tools so we could monitor an actual request going through the application and get a real benchmark with it. And I wanna share the graph with you because it's extremely satisfying. So this is the graph and you can see where I pushed the production there. It's very, very nice. So the thing that I wanna do, the thing that I wanna do with Rails 6 is that I wanna make performance easy. So we make, we Rails, we make generating applications easy. We make connecting to databases easy. We make writing routes easy, generating views, working with forms, writing tests. All of those things. But I think that now it's time for us to do the same thing with application performance analysis. I think we're falling down in that particular area of Rails. Yesterday, yesterday Eileen said something that I really, so first off, her talk was amazing, amazing. I said to her, your slides look really, really great. And she said to me, you should go back to school. And I was like, whoa. Look, I, yes, I did drop out of college, but that like, that hurts. But yesterday, she said she wanted to make Rails scalable by default. I think that this is really important. This is a really important thing, but when I think about making Rails scalable by default, I don't wanna make just the application itself scalable by default. I wanna make your team scalable by default. I wanna make it so that not only is it easy to get started, it's easy for people who don't know much about Rails web applications to get started. That's the way it is today. We can easily start a Rails application, but I don't think that the Rails team is providing as much support as we can mid-career. So that's where I wanna start improving is making sure that we can actually provide tools for you to move from beginner on to more advanced. So for those of you like me who cannot hire somebody to fix their performance issues, you can just, instead of rewriting everything in Rust, you can say like, let's take about five minutes. Let's just take like a couple minutes here and run this one command and see what it says our bottleneck is and maybe we can just fix that instead. So I wanna do for application generation or I wanna do for performance what Rails has already done for application generation. And I think the first thing that we're going to do is start by taking the tools that we use internally for doing at GitHub for doing performance improvements and pushing those into open source, pushing those upstream. So I wanna tell, I have two minutes left. I wanna tell one more story. I have one story to end with and this is not serious at all so you can sit back and relax. So I need to, for those of you that weren't here last year, I need to do a little bit of catch up. So last year, I gave a presentation that was much more of a waste of time than this presentation. So last year I spoke about crap data and it was basically what it was, the talk where I presented this data here that I had acquired, crap data. I talked about doing analyzing this data and different types of analysis that you could do and 40 minutes later, I revealed that I had acquired this data by a Raspberry Pi and a motion sensor that I had put together where I had mounted the motion sensor above a litter box as such and my cat would get into the litter box and then come out and I recorded all this data and made a log and this is, well I guess the cat made a log. Poop jokes, yes. So we had about 100 grams left over. So it was literally crap data. But anyway, so the reason that I built, the actual reason that I built this thing is because last year, around this time, we were moving, we bought a house. Like the day after I got home from RailsConf last year, we had to pack up all of our stuff and we moved. So we got a house and my wife said to me, you have a closet full of electronics. If you do not use these electronics, we will throw them away. And I said to myself, what can I do with a, what can I do with a scale and a Raspberry Pi and a motion sensor? And this is what I came up with. So let's talk a little bit about home automation because this is the first thing, I mean the first thing that you do as a new homeowner I think, well at least this is the first thing I do as a new homeowner is I'm like, wow, I've got a house, I need to automate everything. I need to automate all of the things. So we have a toilet in the house as you do and unfortunately we have that like blue water in it. Okay, it's got like that blue, clean water in it and the thing that's really bad about that stuff is it's basically tied pods for cats. They like, except that unfortunately my cat isn't as smart as a 13 year old and he tries to drink that stuff. So I'm like okay, I need a way to solve this issue. The way I did it was I decided to automate my house. I bought one of these. This is a door sensor. Basically it's a magnetic sensor and on the inside it has a little reed switch. So that's our magnetic reed switch on the inside. Now what I did is I took a tilt sensor, this is a tilt sensor. I put my hand there for scale and basically I bypassed the reed switch with the tilt sensor. So we have this Z-wave sensor where I've mounted, imagine this is a side view. I've mounted the tilt sensor on it like that, bypass the reed switch such that when the Z-wave sensor tilts like this, oh this is our on position. So when it's like that it's on and then if you rotate it like this, it turns off. So it's as if you've opened the door, for example. Then I decided to hook it up to a siren and I took a video to demo. I think it's running, we'll see. So that's the siren there. There is a toilet. When I lift the lid, you can see the siren will come on. When I close it, of course, yes. Good job editing Aaron, yes it shuts off. So now I can tell when the toilet is open and closed. So if nobody's in the bathroom, we know to close the toilet so that the cat does not go in there. Now aside like one happy accident from this is that I actually have a log of all the times the toilet is open and closed. So I decided that Cambridge Analytica doesn't have enough of my information so I'm quite, I'm quite. All right, anyway. So I mean, I wish I had implemented this earlier so that our cat didn't drink any of that blue water and get sick, but the truth is hindsight is 2020. Thank you very much. I hope I can see you again next year.