 We're gonna get started. The unenviable job of speaking last today, which means you're tired. You want to go to sleep. You haven't had enough cake or all your coffee is crashing. So, and I'm nervous. So let's do this. Turn to the person next to you and say, I promise you I will stay awake for this entire talk. I'll wait. Okay, that's a lot more conversation than I was expecting. Now, turn to your second choice, the guy that you didn't want to talk to, and say, did you know the guy on stage is homeless? It's not true. I don't know why you'd say that about me, but we'll get to that in a second. Hi, my name is John McCarty. I work at Heroku. I help people fix things there. And I'm not homeless. What kind of I am? This is my house. All right, the link is bit.lybit.ly. Slash Rails comp 16. There it is, it's on the screen. Go ahead. Thank you. All right. All right, so where was I? Homeless, joke, no. That's my house. All right, so, and there's a reason I'm telling you this. So yeah, I actually, last year, about a year ago, a little year in change, I sold my house and I moved into a 180-square-foot trailer. And I've been living on the road for about a year now. It's, that sounds kind of scary to a lot of people. And then you throw in three kids and a wife. And a hundred, yeah, that's what everybody goes, oh. So yeah, that's my family. It's really cool, we get to travel around. I work remote for Heroku. We're about 50% remote. 60% of engineering is remote. 50% of the entire company. We get to go to some really cool places. It's up in Washington State last summer. This is my home office. There's my desk chair there, right by my solar panels. It's my roof. And I tell you all this because I love traveling, which is gonna set us up for what we're gonna see here. I'm so tempted to hit play. Do I dare? Okay, all right, I don't know what slides come in next because I'm mirrored, but we're gonna work with it. I'll tell you all that because I love to travel and in the light of today's, my talk today is basically we're gonna cover some really common things that we see at Heroku Support. All of these issues that we're gonna talk about, they're face palm moments, yeah. No, it's fine, I'm good, I'm good, we'll roll with it. They're face palm moments, they're these moments where you think, my app is so slow, what's going on? It's probably Heroku's fault. And you, hi, I'm totally, I'm good, yeah, I'm good. Oh, don't unplug it, it was okay. So they're face palm moments. No, it's supposed to be that. That's cool, it's cool. But I can't tell you who made these mistakes. Some of them are me, that I made early on, riding rails, but a lot of these are really common customer issues that they ride in and things are going right. And then you just kinda start to poke around, you ask some questions. And a lot of times they're like, oh yeah, it's this really simple thing. So, but I can't tell you who they are, so we're gonna tell this entire story using a made-up company. And I like travel, so we're gonna go with a travel theme. I think Airbnb is a fantastic site to use it all the time. But they haven't really looked beyond the horizon, literally, into space for space rentals. So we're gonna take an advantage of that and today I'm launching Airless B&B. Okay, it's my new startup. If you are a VC, some talk to me afterwards, you're also not a very good VC. So there's our logo. We are launching now, thank you very much. And things are going okay. In the last 10 seconds, we launched. And just poking around, seeing how things are going. And it looks like my Rails app is doing about 40 requests per second, or is capable of, if you wanna play around with AB, that's fun, people send us AB results all the time. 40 isn't that great. And so I start to poke around and I'm wondering what's going on. And, well, let's go ahead and boot up. Let's see the server when it boots up. This is what's going on here. Anybody knows what, when I just do that in previous to Rails 5, what I'm doing? Yeah, I'm doing this right there. Shocking, shocking amount of people running Webberkin production. Hopefully that's not you. If it is, if you don't know what web server you're running, it's you, you're running Webberkin. It's a face bomb, okay? Don't do this. Let's talk about Puma real quick. Puma is the web server that we recommended at Heroku. We used to recommend Unicorn. Unicorn is great for a long time. It's susceptible to slow request attacks, so we kinda stay away from that. And there's a lot of other great things that Puma will do. You can look at the benchmarks on their website, they're fantastic. And we'll talk a little bit about to the, kind of the ping that Puma gets is that it's threaded and if you're using MRI, which doesn't really handle threads, what's the point? But in reality, most Rails apps are IO bound. And so while your app is outgoing to the database, it can take advantage of threads, even in MRI. So Puma is fantastic, really easy to get started and throw it in your gem file, boot it up, really simple. And this is, so this is just a home controller. I'm not doing any kind of context switching. And immediately I get double the throughput, okay? So if you're wondering why you shouldn't do this, it's mainly because WebRick has zero concurrency. Technically does, but in Rails it has no concurrency. What that means is, is it can only handle one request at a time, which is really terrible. Because while one request is coming in and getting processed, if another one comes in, it has to wait and it's blocking, with Unicorn or Puma, you're able to fire up multiple workers and handle multiple requests at a time. So that is a fantastic thing. We're done, great, it's a great talk, run Puma. Yeah, thank you, thank you so much. Or is it? You're gonna see a theme through all of these and a lot of our questions that we get at Heroku actually start with, my app is running really slow. And that is one of the hardest places to start helping someone debug what's going on because almost every single thing I talk about for the rest of the talk starts with, this is really slow and you have to start poking around and figuring out what the reasoning is. In this case, I started poking around and I saw this, this is actually a splunk graph. The red shows me how much memory I've used and the blue shows me something that I don't wanna see which is my app using Swap. Now Swap, the first time I learned about Swap, somebody gave me a Linux book about this thick and I'm not gonna cover that. So let me give you a really quick explanation if you don't know what Swap is. I didn't when I started. So your operating system is going to have a certain amount of RAM and then after that once that fills up, it's gonna start saving stuff to disk. Your disk is way slower than in memory so when your app starts going to Swap, you're gonna see a big performance hit, okay? There's a couple different, right, sorry, thanks. Card. Memory enemies, there's two main things that we usually see. The first one is, I'm just gonna give you two links and I'll give you the slides after. Memory leaks are kinda nasty and are gonna take a much longer time to talk about. Go visit these two links. The first one is a library by Richard Steeman, one of the guys I work with at Heroku. It's a fantastic little thing to run against your app and you can actually see if you do have a memory leak. So if you are concerned about that, look at that. He also wrote a fantastic blog post over at CodeChip on debugging a memory leak. The second thing, hopefully number one doesn't apply to you, the more common thing that we see actually is really simple fix. People are running too many of those workers so they've got Unicorn, they've got Puma and they just, how many workers should I run? I don't know, five is a cool number, has a little hat on the top, but they don't think about what does that number actually mean, especially when it applies to memory. So if you look at a Rails app, a typical Rails app is gonna run out of the box right around 180 to 200 megs. Okay, so if you've got a gig of RAM, you go, oh, 1,024 divided by two, I'll run five workers. The problem is, now you're right up against that limit, you have no headroom. So if your app needs to actually use, allocate more objects, let's say you're doing any kind of image processing, that's gonna balloon up and all of a sudden you're gonna start going to swap. So the better way to look at that is to just cut back on your web process, and see how your memory looks after that, and then start to adjust from there. But when you start going into swap, you're gonna have a bad day. Really easy to check this out actually, is go on to your server, type freedashm, you're gonna get that number, which is your total amount of memory, so we got eight gigs. Your operating system has allocated, so it's basically saving 511 megs for swap, and right now I'm only using seven, so just ignore the other stuff. These are the numbers you care about. When that number, that seven, starts getting really deep, you need to be concerned about it. Obviously there's a lot of other tools that will help you visualize this stuff, but first place to look when your app is growing slow, check your memory usage, and adjust accordingly. All right, moving on. I've got people who wanna pay me, which is awesome, found a great place on Mars. One bedroom. The problem is, we've got a bad guy, and he discovered a unprotected part of my website, and has taken the money, and now shit's on fire. This is why. Again, more common than you think. So we put up an SSL cert, and everybody goes to it, all the links that we post have the little S at the end. Why are people going here? Because they did, and you didn't tell them otherwise. And so therefore they're going to this checkout page, and it's completely unprotected. Really, really, really, really easy to fix this in Rails, actually, and we'll talk about how to do that in Rails and Nginx. The first one is this. This is it, this is all you have to do. I don't know why this isn't on to by default. Go into production.rb, go to force SSL, and flip it to true, and you're done. Chip it. This will take any request that comes in on port 80 and immediately redirect the entire thing to HTTPS. Now the problem with this, though, is that it's going all the way into your app. It was kind of a lot of overhead for just doing a redirect. So you can do this a little bit higher up in Nginx. Listen on port 80. If somebody comes into 80, send them to 443. You can do this a lot higher up. Nginx is a much better place to do this, and we're moving on. We got a few more to cover, we got 25 minutes. All right, we are going to change our logo. The rocket ship didn't test well. We're going to change it to a comet. Super easy to do, so we change it, and we push it, and our logo's gone. Nobody knows where it went. Is that an image? That should be IMG. I have, I'm a developer. Designer probably did that. Why did this happen? Our friend, the asset pipeline, yay. It gets a bad rap, and it's unfortunate, because I love the asset pipeline. If you ever had to deal with asset caching in like before the asset pipeline with the question mark crap, it's so much better. So this image, my logo here that I put in is actually going to turn into this MD5, MD5 digested, file name, which is going to let us do cache busting a lot easier. If you don't know what's going on here, basically what the asset pipeline will do is it looks inside of your image, and if it has changed, it will generate a new file name, which is how we do cache busting. The problem is, is that somebody did IMG, and they just direct linked to the logo. Instead, again really easy fix, use the image tag helper. This one isn't actually as common as the CSS version. This is way more common. I see this a lot more. So when you've got your CSS, stop using the URL and linking to assets because this will not use the asset pipeline at all. Very few people actually seem to know this, but there is a bunch of CSS helpers inside of CSS, STSS that Rails provides. Image URL will actually do all of that handling for you. There's also, if you've got a fonts folder, it'll do font URL, there's actually just a base asset URL as well, but make sure that you're using this wherever you can. Teach the rest of your team to always use designers too. Like, on my old team, this is kind of what we would work with our designers whenever they were doing any kind of front end stuff. It's just to make sure that they use this because it's really tempting just to plop in the old URL. Oh yeah, one more thing, the pre-compile list. Thankfully Rails, I think, forget which version it is, where it'll actually warn you if it's not in the pre-compile list. Actually, it was like 4.1. Rails will warn you if you try to render another JS or CSS file and it's not in this list, but be aware of this list. So if you're trying to, by default, it'll render application.js and application.css. If you want another one, make sure to add it to this list so it gets pre-compiled. That's another thing we see people run into sometimes, is it works locally and then they'll push to production. It hasn't been pre-compiled and it'll throw an error. All right, we get a bug. And somebody asked me, is it happening a lot? I don't know, it happened once or twice. Well, let's check the logs, right? Check the logs, what's what we do? We've got a production.log. The problem is we have multiple application servers. This is a problem because now I have multiple production.logs. And I've got really, really big production.logs. So we've got a little problem here. If you've never read about 12-factor apps, I highly encourage you to visit this URL. Let's take a look at what it says about logging in a 12-factor app. A 12-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage log files. Instead, each running process writes its event stream unbuffered to standard out. What this means is your app should not be sending stuff to production.log. It's really easy. I've seen a lot of apps where, usually not on Heroku, that will just by default, it's like we're only gonna be running one server forever and who cares, we're never gonna scale. And so they just, we'll get to it later and just dump everything into production.log. Lots of times they don't even do log rotation so this thing just starts growing out of control. In your Rails app and your Ruby app, use standard out, okay? If you log to standard out, now you're doing what a 12-factor app should do and you're just sending everything to standard out and getting out of the way. This allows you to consume standard out in a much better way. Hadoop, Hive, Splunk, you can use an add-on like log entries or paper trail to consume those logs and have a much better understanding of what's going on inside of your app. All right, we're moving. We're shipping. Ship, ship, ship, ship. Everybody wants to ship. This is our deployment script. Go ahead, find it. Sleep for. So, this is an actual restart script somebody gave me. I took over a little, a friend's project and the former engineer said this is our restart script. What do you think happens when a user comes and we are sleeping? What do you think they see? An Nginx 503 error. It's terrible. The argument was we shouldn't do restarts because you could have two different versions of your app running based on a unicorn restart. That's a whole other conversation, but this is bad. What we need is an actual legitimate way to deploy our app that doesn't involve me SSH-ing into a server and running this. So, you may have heard of Capistrano and I hope you've heard of Mina. If you have not, I'm gonna tell you about Mina. It is my favorite, sorry, my second favorite way to deploy a Rails app. I'll tell you about my first one in a second here. Mina is fantastic. I started deploying with Capistrano a couple years ago and then I found Mina. Mina is really similar to Capistrano if you're used to using Capistrano. Take a look at this. There's a great plugin architecture for restarting things like Unicorn and Puma and Sidekick and there's great plugins for this. But what's great about Mina is it's really, really, really fast. It does the same kind of idea where it will do a shallow clone out of Git, make a new version folder and then do the sim linking and all that and bundling and db migrate, all that stuff for you. But what it kinda does even cooler is versus Capistrano, which fires off every single command in its own SSH tunnel. Mina will look at the entire thing, bundle up the entire script, open one SSH connection to all of your servers, individual, and run the whole thing on the server. So, it's way faster. If you've never seen it, take a look at it. This is what it looks like when you deploy. Oddly similar to something else, I'm gonna show you. It will make the path, clone the Git repo, do all of the business and then right at the end, it'll sim link and restart the server. There's another way. This is what it looks like on Heroku. Create your app, Git push Heroku Master. Whether it's this or whether it's using Mina, you need to have a good way to deploy if you want to ship frequently. The teams that are usually unable to deploy quickly are teams that struggle with these deployment practices. They think they'll get to them at some point and they never do and they get stuck because everybody's too busy doing something else and then by the time they need it, it's almost too late and they can't deploy quickly. So, have a way to deploy quickly so you can ship. Next problem, somebody got into all of our S3 images and swapped them out. Someone nefarious has taken all my beautiful Cosmos pictures and put in something less than appealing. How could this have happened? A long time ago, someone decided to put our S3 creds into the Git repo. Then, something happened, maybe somebody forked a part of the app, maybe somebody took the app and put it on GitHub and they deleted it but it's still there in your Git history. There was a couple years ago where you could actually, I think you can still do this sadly, there's a search command, be good, but there's a search command that will let you go through all of GitHub and look for S3 creds and they're still out there. People have S3 creds all over the place. Shneems did it one time, right? Yeah, that guy right there. How big was your S3 bill in the morning? Three grand, yeah. So, don't do this. There are so much better ways of doing this. There is something called secrets.yaml inside of Rails now. This one works pretty good, I have a better way, but if you like this version, you can go this route. Throw your keys inside of secrets.yaml, please get ignore this file. And then you can use it inside of Rails by calling it thusly. Personally, I much prefer Envars. This is where all of your keys need to be in Envars. You don't commit them, they are different per environment that you're running on. Really easy to do in development, you can use this gem called .envrails. You make a .env file, get ignore it. You've got your key values and then on Heroku, it's really easy, Heroku config add with your key value. And then in your app, you just call Env and whatever the name is, okay? Really easy to do and this is the right way of storing credentials. Please do, it's really tempting, especially when you're going really fast at the beginning of an app. You're just like, I'll just give me the crowds, I'll throw them in here and I'll deal with it later. Start the correct way by using Envars way up front. All right, 15 minutes, so I gotta hurry up because I wanna do questions. Okay, I'm gonna hurry up. This is my database graph, or this is my app graph in New Relic. The orange chunk right there is my database, this is no good. What on earth could be causing this? When we are yet a small little app and I was looking up users by their username, this call took 2.3 milliseconds. But now, the last 20 minutes of us being up and running, we have jumped to 231,000 users and suddenly that call is 51 milliseconds. Who would like to tell me what's wrong? Indexing, thank you. So that little thing is now this big thing and that's only gonna get worse because we do not have an index on username. This is one of my own. I bit this one really hard early on and this is really, really easy to do. So database indexes basically tell your database, this column is something that's important to me. I want you to index it because I'm gonna make calls against this column frequently. It's so easy to do in Rails. You basically just create a migration, add the index on the table and the column that you want and all of a sudden this 51 milliseconds, actually it's faster than the original one, it's down to one millisecond with the same table load. One asterisk here is composite indexes. I didn't know about this one either. Suddenly I wanna look up by the one thing and then something else. That original index that you made no longer applies. You need to create a composite index. Again, really easy to do in Rails, pass an array of column names and you may now have a composite index. All right, two more. I'm gonna blow through these, awesome. We're so close to a billion dollar valuation. It's gonna be great. My first problem that I have here starts seeing again, the app is a little slow and I've got this controller here iterating over photos and then in my view I am spinning out the photo in the username. Except this is what my log looks like. So I'm getting all the images and then I've got all of these user loads. What's going on? It is a N plus one. Didn't know about this one either. These are really easy to ignore while you're writing an app because again, your development environment isn't taking on the load that your production environment will. So all of a sudden what works locally is getting thrashed by the amount of users that you have. An N plus one basically is what happens is when you call one object that has a relationship you start iterating over this and calling the associated object. It's really easy to fix in Rails. All you have to do is instead of going just photos.all or photo.paginate or whatever you wanna do includes, it kind of messes with you because it include is actually a standard library. Method includes will tell ActiveRecord that this is an association that I want you to load and what happens is Rails is this really smart thing where it goes from this to this. So now it will make only two calls. Get all your photos and then do an N and find all of the user IDs on those photos and grab them all in one shot which is way better. In development use a gem called bullet. This will show you when you've got N plus ones. It looks like this in your logs and it will actually tell you add this on your finder. So if you've got photo go it says right here there's an association called users add this to your finder include users and life will be magical. Last one. My users are complaining that this thing's slow when they're doing certain functions specifically adding photos. And if you look around you'll actually find plenty of cases where you can see where milliseconds will actually affect users' happiness. People will bail on websites because things are just a little bit too slow. So this is actually pretty important to us. Now I take a look in New Relic and I see my app's pretty snappy except this one thing called photos controller. It's taking almost five seconds so I dig into it and I see this. I'm spending almost 60%, 66.1% of my time dealing with S3. So what's going on here? This is what's going on. So a request is coming in it's hitting the controller. Now it's creating this photo it's going out to S3 and waiting for S3 which means the user is waiting for us to do something that user doesn't really care about yet it wants to upload the photo and get on with what it was doing. So it's now got to wait for S3 before it returns a response. So we're gonna talk about background workers here just a second. Let's talk about another use case let's just do SMS. SMS is a little simpler we don't have to deal with image magic if you follow that one. So let's talk about SMS. So let's do we got this we have a user object we're gonna send them a message with an SMS provider. So the request comes in hits the controller we got to wait for SMS provider before we return a response. This is a great example for when you want to use a background worker actually an even better one. It's a little bit simpler to talk about is mailers. If you go on to the Rails guides you actually can see I think they use in the active job explanation they use mailers I'm gonna talk about SMS here but mailers is a great example anything where you have to wait for an external service to return a response for something that's non-critical sending an email sending an SMS if it can wait a second send it off to a background worker don't make your users wait for it. So instead of user dot send an SMS in our controller it's actually gonna do the job we're gonna call perform later which is the active job method you can use sidekick whatever else you want the active job is uses perform later and you pass it your arguments. And now instead of this it's this. So we go to the controller it goes and fires off or adds the job to the queue in Redis which is super fast and gets the user back to what they're doing. Now the SMS job can do whatever it wants. This is what a basic job looks like. So it's inheriting from active job base you just need a perform method and the arguments and then you can do whatever was costly before uploading a photo sending an SMS job sending an email whatever it is this is really really small bite size code and saves you a ton of time and what's even better on the app provisioning servers Heroku side you don't need as many app servers now because they're not all waiting around and consuming resources they can actually get the user back and go let a worker take care of that which means you actually need less servers so this is a great thing for the bottom line as well. That's it we sold we made a billion dollars off of a terrible idea but mostly because we fixed all the problems that we talked about in this talk we're actually pivoting we're gonna be building a Tinder app for cats it's gonna be good I don't steal that so the bottom line here is if you are new or if you've been around for a long time when you are running a Rails app in production you will break things the important part is when you do fail on something when something does break learn quickly and continue to learn there's so many things that are changing in Rails there's so many things that are changing in the state of the art of the servers that you're running don't get discouraged keep learning and you're gonna be great thank you very very very very much.