 out we have this amazing panel on performance so I'm going to introduce Sam who's the moderator. Sam Saffron is a co-founder of Discourse, creator of the mini-profiler, memory profiler, mini-meme and mini-racer gems. He's written extensively about various performance topics on samsaffron.com and Sam loves making sure Discourse keeps running fast. Enjoy. Hey is everybody still awake? Okay I'm just gonna have the panel introduce themselves and then my I've got a bunch of questions prepared but also we're accepting questions and there's a little chat thingy so you can just log in and type your question and I'll triage it while stuff is going on and if anything really interesting pops up or uninteresting pops up or if it crashes then it's back to the traditional way. Anyway let's start. My name is Nate Bergepeck. I'm a independent performance consultant. I work on people's Rails apps to make them faster and I blog about it online at SpeedShop.co. I am Rafael Fresa. I work at Shopify as a production hearing. My job there is to make sure that Shopify runs smoothly in our production systems and I'm also a member of the Rails Cloud team. Hi I'm Eileen Yuchitel. I work at GitHub. My job is sometimes performance but not always but I've given a few talks on performance on speeding up integration tests and other stuff in Rails and I'm also on the Rails core team with Rafael. Hey everybody my name is Richard Schneemann. I go by Schneem's on the internet. I work for a startup in San Francisco called Heroku. They're kind of up and coming. Some of you might have experienced some issues with them in the last couple of minutes here. So yeah I do performance work like when customers come with an issue then it's like okay well how can we make this a little bit faster and I'll take a look at their apps. I wrote derailed benchmarks. I don't know if any of you all have used that one as well as I blog. I'm on the blogosphere so yeah that's me. Thank you. Cool. So I think we should I'd like to get started talking a bit about how to get involved in this whole performance thing. So I guess the question to the panel is you know I've I haven't done anything publicly before and I haven't done any performance work. Where do I start? We're gonna do this like down the line every time. All right so I'm in the pole position. What? You want to arm wrestle for it? Yeah here we go. All right I lose. Well I just talked though like two seconds ago so you gotta. So I mean I got started in performance work because I used to work at an e-commerce company and it was clear to me then and it's still clear to me now that fast sites are sites that make money and sites that people enjoy using. So I've kind of always had that very like customer focused approach of like you know the the speed of the website is a primary primary component of how your users or your customers interact with it. So I mean I got interested in it for that reason because I wanted people to buy more stuff on this website and you know it's it once you start digging into it it just like became like it's not often in programming that you can get like these very discrete problems which when you fix them it's like oh I made this 10 a 1 that's really good right like it used to take 10 seconds now it takes one second like that is better you know like it maybe that's just like the one route that someone hits like once every blue moon but like it's faster for that dude now so that's cool. So that concrete impact of like performance work became very addicting to me and then you know you just keep doing it and keep doing more of it I guess and you just sort of start picking up the skills I guess. So I mentioned I work for Heroku one of the things that I do is when a ticket comes in for Ruby someone is using Ruby and it can't be handled by our support staff it'll it'll get escalated and it'll end up on my desk and people occasionally will say hey I'm using Heroku and like I'm not getting the performance I want and generally whenever I consider I consider performance issues a bug and like generally when somebody is hitting a bug of any kind on Heroku they're like oh Heroku is breaking all of my stuff well so first of all 99% of the time unless it's like right this very moment that's not true and so instead so I mean I do spend a good amount of time helping customers get to the right solution but also a good amount of my job is also proving essentially that it's not that it's it's you know it's not me it's you and so that's that's actually what inspired D rail benchmarks is I was was getting a couple of performance issues that I just could not could not debug on Heroku and this was before we had so we have something called PS exec right now where you can actually SSH into a dyno and get you know some extra metrics and stuff that way but before we could do that the only way to debug a performance issue was to reproduce it and reproduce it locally which also happens to give the benefit of now I can say oh the performance issue also happens locally like not my issue but no and and then I got into this great community met a bunch of other people who care about performance and it's just like even even today people come to me and say hey this I'm experiencing this behavior on my app like why is it happening like this and I was like I don't know that's a really like you know that's like telling Sherlock Holmes somebody I don't know stole a cookie out of the cookie jar and you know it's just like I have to dig in and get deeper yeah is this is this something like only advanced developers can do I'll just move because I remember you got you got started like a lot with the performance work just you pick to the mini profiler jam back in the day and you worked on this giant commit so so how does it like how do you what made you decide to just I guess get started with this and like get out there and do this kind of work well the first time I got involved with performance was because I crashed a staging database I was trying to insert like a ton of data for the sales team to use for their demos and I didn't really know I just knew I was like oh well active record knows how to do it I'll just shove data into the database and then I used so much memory that it crashed the MySQL instance of the staging database so I had to like figure out well why why is it slowly is it rails fault is it my fault is it MySQL's fault and I used that to figure out ways to insert or update or delete data better so we didn't have a whole memory bloat problem so for like the insert records I used like direct MySQL batch insert versus when I was deleting stuff I used delete all instead of destroy all because I just needed to like blow the database away that led to my first talk on active record which is all about really how active record is not it's not it's not active records fault because it can't be like oh by the way you should use this instead and I'm gonna just do it for you because then your callbacks aren't run and like other things happen to your life that make you sad that are not just performance so you don't you know you don't want active record to take over for you and then like because of that I started getting involved in open source stuff and well I you're the mini tests I'm not may test a mini profiler just there was like this big problem it wasn't like a problem but it was I don't remember what I what I worked on it was just something that you were like I need help and it was like the one open source project that said I need help across the top of it like in big bold letters and I was like cool this guy's desperate enough to let me touch his code base awesome so I first started with performance work as I was working as a consultant in Brazil and one of my one of the things that my job required me to do was to work your other people problems and usually all the classes that we had had major performance problems because of any reasons like sometimes was database sometimes it was front-end performance and nobody my team wanted to work on those problems so I kind of felt my plate and I enjoy it because like Nate said it's the kind of problem that you actually see the value being delivered to the users right away because you can see that in your machine too so that was when I started I can say that I did not had any kind of contact with this kind of work before doing that so I had to learn how Disney worked for me a lot of different tools and yeah I don't think that you need to be I experienced developer to work with performance you can just know how the system works like let's say that you're working a rubric application that if you know Ruby well enough to write the application Ruby it's possible to you to also know how to do performance or Ruby application yeah I just I just wanted to add that like as far as experienced developer like I don't think you have to be an experienced developer to do performance work but I do think you have to be curious because performance work can involve so many different areas of the stack most of which you probably have no experience with you know if a web request takes five seconds right if that's all the information that we have there's literally like 20 layers right that that are involved in that five seconds it's like the network the application server your application code the Ruby VM the kernel like there's there's there's and there's so many shades in between of all those different layers right so like I found that this work has made my knowledge pool like much I have like a very wide and shallow knowledge pool now because when you're doing performance work the requirement is known you know it's this this thing is slow and I have to make it faster but oftentimes it's like that solution could be anywhere so you just have to be curious and you just have to not be intimidated I think yeah so one one just generic piece of advice that I've really taken to which when somebody asked me like oh how do I do thing this thing well there's a term there's a saying that says writers right if you want to be a writer you should go right like how do I become a writer while I write if I want to do performance work well you need to do performance work and just like being generally interested in it if you don't know where to start like you can see what other people are doing or other people are talking about like what tools exist out there like I I got my my even my first what things could possibly even be optimized like I got from like adequate record from a keynote and it was like oh hey there's this thing called like memory allocation when you like create a new object and like if you do that a bunch like that's like bad and there's cases where where like at the library level maybe we can optimize that a little bit and that doesn't have to be the case for everyone but like I also went to Nate had a front-end workshop and it was like here are discrete examples of you know here's a problem and then here's a here's a series of potential solutions and it's just kind of a iterative process of doing the same things like over and over and over and like like he said exploring and going deeper cool the chat rooms actually all alive and full of questions so I can go off my question list to the chat questions one question that I'd like to actually talk about here is is tuning the Ruby GC still needed in 2x I remember 1 9x defaults did not work well with Rails a question by Tony and I'll start answering it in kind of a non obvious answer is that if that's the wrong question to be asking because what you should be doing is measuring and if you are measuring and you have some sort of graph that you're trying to make the line go down and you twiddle with something and the line goes down and you know that there's success but if I tell you yeah you need to edit this kind of GC setting and you go and edit it and it actually does damage to your performance and you don't know then you're in a very serious situation at discourse we do edit some GC settings but the ones that we do edit are all based on feedback that we got from graphs that actually showed that changing that setting helped do you regret publishing that blog post that listed all those GC settings that you use because I think I wish I knew the amount of people that have just copy pasted those I'm sorry I regret I don't know I don't regret information is good I don't regret information being out there yes you brought something very important about performance that I think a lot of like when you first see a performance problem like your first reaction is I want to fix this but you can't fix anything with performance until you measure it just like you can't fix a bug without actually knowing what's causing the bug you can't just like you could just shove a rescue in your controller and you're like sweet I fix the bug but you didn't really like oh yes it doesn't throw an error anymore but it's still there and so that's the biggest thing with performance is making sure that you have a measurement before and after you fix something because otherwise you don't know you actually fixed it and there was a incident there was one time Aaron Patterson and I were trying to fix a performance issue and we luckily were measuring because the profiler said it was fixed but the benchmarks said it wasn't and so we had just gone with the profiler like speed that that number is no longer high after we benchmarked it and figured out it was still the same speed and slow so that's why you have to measure before and after or else you don't know you actually changed anything for the better so on what was that conversation we had earlier about I think the three of us you're talking about we were talking about setting GC the max slot growth limit yeah so there's a I think it's like GC growth it's got the words growth and factor in it all yeah so it a it pays to know what changing one of those settings is actually doing like you don't want to just like this is a blog post that's been sitting in my queue for a while I want to go through each GC tuning variable and explain exactly what it does but the important one though one of the important ones that we're talking about is so each Ruby object gets a slot in the Ruby VM this is an R value C structure and the there's three variables I think which control this how this slot these slots grow what one is a factor so each time we run out of slots we multiply the amount of slots we have by this factor and that's the new amount of slots that we have then there's two very two other variables which can put a max and a minimum on that number so by default we realized it was 1.8 I thought it was less than that so and if I have a thousand heaps slots and I decide I need more I will grow the heap to 1800 and the problem is that most people's applications have a million heap slots when they're in this steady state you know after running for six hours or whatever and then you run out of slots at some point major GC happens and now you have 1.8 million slots which is probably 800,000 more than you actually needed what you really needed was another hundred thousand slots so setting those can sometimes reduce the amount of free slots that are in the Ruby VM you would know that this is a problem if in for example if you have New Relic New Relic's Ruby VM tab will show you how many used and free slots you have at any given time and if you have these huge numbers of free slots it's the yellow part of the graph on New Relic then maybe you have a problem with this setting and you can also a lot of these things you want to like you don't want to just start optimizing GC until you yeah you benchmarked and you say okay yes this is the thing this is the thing I'm having problems with you could maybe tell if you have this problem if you look at your your memory graph and it's you know it's kind of going up and going up and then just all of a sudden you see this giant wall of memory and you're just like okay well you know what after your server has been running for like whatever six hours or something and changing and changing this value changing this growth factor will help with that I've got a recommendation and the actual name of the growth factor the actual name of the GC setting on an article on the Dev Center for Heroku under if you like search memory use Heroku I think it's our R14 maybe I have a little section on that with a recommendation I think it's like 1.01 so it grows by like 1% instead of like 80% so yeah I mean if you're looking for specifics discourse is actually open source and anybody can download it and install it and we have an official Docker image that controls the whole installation so you can just look at the source there and see what we're using and it's pretty well documented why I change whatever GC settings I change so there's only one really that I play with which is capping the growth at instead of it growing exponentially making it just grow by a hundred a hundred thousand slots at a time there are more questions here what does great performance mean to you is high RPS fantastic CPU resource utilization sub 100 millisecond request by Danny what is fantastic performance for you I guess yes yes that is fantastic it's when I when I think about performance I think about the end result so a lot of times we like to think that you know the service says that it's able to render stuff to generate stuff in a hundred milliseconds everything is good and like we can go home now because our work is done when in fact like we have all of these little JavaScript snippets that we've collected from all these places and we put that we didn't think about client performance at all and every user that goes to our site actually experiences a five-second delay because we didn't think about a CDN we didn't think about deferring a JavaScript we didn't think about minifying a JavaScript possibly and so on so for me like yeah I mean there's this hundred millisecond thing that you know Amazon strive for but there's also you know when when I think about performance I don't just look at the server side when I think of great performance I think of Wes Anderson movies that is good yeah those are those are good so thank you Evan thank you low so I I mostly focus on the server side of things and and I consider good low request variance and so what that means is that it's even if your average is like oh we're serving requests in like 10 milliseconds but one out of every like you know that means maybe most of your requests are like one millisecond and then one is like 10 seconds or 20 seconds or 30 seconds like that's really high request variance and and that that ends up having a lot of impact I mean yes that one person has to wait for that 10 second thing but if you can also get this like this queuing effect where slow requests will end up behind that or fast request will end up behind that slow request and then it'll feel a lot slower and like a lot of these things like tuning you know GC or like you know modifying your server settings or something like it kind of doesn't matter if if your end code is just slow and so like my big my approach when I end up with when I'm working on an app that is slow is I want to know these slowest endpoints and that's where I start I mean the slowest endpoints that have a lot of traffic and then getting getting those down and generally you not only do those become faster it actually speeds up a little bit of the rest of your server as well yeah it's just like queuing theory 101 that it's way easier to load balance something where the response the time in the system is has low variance than something that has high variance now there are jokes in the chat room as well so I need to filter them out no there was a there's a good question yeah Laura said do you ever find yourself preemptively over optimizing your code for performance before there is need to consider it yes well it's like performance I think this is kind of why sometimes measurement will go by the wayside is that or improperly like benchmarking and profiling is like I think I don't know if you guys feel like this but like the the solutions are like such like candy to me like getting to implement you know let the latest like little trick or whatever like I love that so like yeah sometimes I'll just do things to do it and and whether or not I really needed to so you know as long as it you're not actually spending that much time doing it it's not really that big a deal but when you get like you know 24 hours deep on a yak shave to like you know make whatever the new thing cool hot anything was work with your app it's like that was a waste of time time out request expired yeah okay so when you start learning like things that you can improve for like in your app performance wise there's you're gonna have the urge to just do that every single time you're like oh I know it's faster but if you're doing that push that upstream to rails or to a gem or to Ruby because if you're doing it over and over again chances are someone else is doing it over and over and over again too and so then everyone can benefit from it when you push it upstream to rails or whatever you you know if you do something we had a someone change finding batches recently on rails something you could add limits yeah and so like now you know maybe somebody was doing them their app a lot so now everyone can use it because it's in grails so you said you said push it up so I guess this is for a fail so like there are what a thousand open PRs for rail 5 and like how do I how do I manage to get my PR now into rails 5 to and my performance fix into there what what can I do to make it more likely that the change that I submitted will be reviewed by the team and actually merged in what what steps can I take I think the the first thing that you need to do is actually prove that there is a problem in the framework so if you are measuring your application and you put that data in the issue or the poor request description it's very likely to you to have this PR measured because we know that someone is having that problem because what we try to avoid in rails is premature optimizations like there were a bunch of PRs frozen strings everywhere in the framework but we don't want that like we only want to solve real problems not to change all the code base to be slightly faster but not really so you're saying if I overall like cut down on 10 allocations in a rare case then it's less likely to be accepted for me than me cutting down 50,000 allocations in the general like high yeah and that is what you're looking for and also say different patches are every single pull request has an overhead even if you're just adding dot freeze to like a string on one line well now suddenly if I'm blaming doing get blame then that you know that might mess up that it's you know it requires overhead from from us I've done I've done a bunch of rails performance stuff in the past a lot of those freezes are my fault I'm sorry but when I did it I submitted I said here's here's the benchmarks here's how to reproduce here's the steps I took like and not not only that like not only does that keep me honest it also gives other people eyes and to look at it and then previously I said if you're interested in doing performance stuff then find other people doing performance stuff and hopefully if you if you hear oh somebody so-and-so like yesterday there's a camel got three three times faster right that's a that's a thing and and the first thing I did was was be like okay well how I messaged him that and misspelled it horribly and then like a bunch of people have saved it and and but in your pull request if you have here's my benchmarks then you you can go and try and and do that I act and like I've actually had a case where people have done that and not gotten the same results and then and actually it turns out that you know the they didn't push the the fix or whatever and it's like oh that was actually really helpful to know that not everybody got the same thing so yeah benchmarks I have a correction from smiley face huggy face koala saying there are only 700 issues in rails that are open now I'm just apologizing only seven anyway okay there's a question by Nicos should you refactor before trying to optimize a code bar about like the tension between fast code and pretty code so where do you where does the panel see themselves in this I guess continuum I'll start fast code that's that's actually two separate questions I think in general you just you know I've every time I touch something I try to make it better so if I'm touching it for a performance reason I'm also going to try to rewrite it to make it you know more readable or whatever like we just we literally just had a thing on Puma where we have an issue where Puma processes are accepting too many requests and the like conditional and the code that was like causing this behavior was like really confusing and we actually like reading through it with Evan we're like what is this until less than and like couldn't even like process it all at once so we re rewrote it at the same time like the actual change was like just removing it unless from one line but then we rewrote the conditionals we actually understood what the change was doing so I don't think there's any tension there I think you're kind of getting that like I mean we don't really do this in Ruby but like when people like inline stuff that you would like in you like unroll for loops or whatever like we don't really do that kind of stuff I don't do you feel like we have a tension between kind of so like on the other on the other question which was I don't remember but it's the one that you just said no about what was that question the tension between pretty code and fast code so like knowing a bunch of performance tips like a bunch of micro performance optimization sometimes that micro performance optimization is not as pretty as what you would normally do in in Ruby like you know recently there was a thing on like don't use case statements there's like a blog post like stop using case statements very large grandiose you know thing and it's like coming from a like performance work it's like a case statements are actually optimized by the by like the the Ruby VM in some cases and like yes it might actually be harder to work with and not as pretty but there there might be case there might be some times when you just want to make that thing faster or or it's like oh I can move move this object allocation outside of the loop but it's you know it's not as like it's not as nice it's not as pretty and so I mean I I get that but I I tend to only optimize for one thing at a time ideally I want it to I want it to be correct and then I want it to be pretty and then I want it to be fast like that's kind of like make it correct make it you know good to work with and then and then make it fast and generally when I do it in that order I'm fairly happy with the end results I think I think freeze is a good example of this because adding freeze everywhere makes your code ugly but then it sometimes makes it faster so it's usually like you have to and that's why you have to measure is the performance impact so much better you know the method in rails is called only once like ever on boot there's no point in freezing it because it's not gonna actually it's not like all multiple times to build up your allocations so you don't necessarily need that in the same way if you're like calling it all the time and you want to keep your allocations down another question what might you do to tackle a major performance degradation from a framework library language upgrade that you only find with live traffic and actually a referral has been going through a big upgrade at Shopify and like yeah how do you how do you tackle any performance regressions that you get from a major upgrade so just so give it like let it be like the rails team want to know those regression so the first thing you have to do is like open an issue please we want to fix that as soon as possible because what we want to avoid is like this kind of regressions and the only way that we have right now to know that is when people get with live traffic we have some some projects like the whole benchmarks websites that runs the rails some rails benchmarks to see if you had a regression or not but that don't get all the kids that's happening so if you find a major regressions please report to the framework so we can work together to fix those three questions I so at this course does a lot of those reports like I remember when we released the way I saw too he did a bunch of reports saying that we were allocating more objects or some code path was slower than before and we fix all of them I agree reporting reporting is the key but like how did you pick up how do you pick up that you've got a problem for example like you've just upgraded it did you know before did you measure everything before did you deploy and then look at monitoring how do you good question so we at Shopify we have some monitors working and we also every single new version of the framework we do is all these holdouts so we put the new visual only a small subset of servers and we measure the performance is different between them and if anything is enough the graphs are higher because of the new facial we usually use some profiling tools like strike proof to see what's the difference in the code path like what methods I've been calling now that was not been called before and we get that report we refer to the change back in the servers and we get the report and we try to fix the button so I kind of want to answer a subset of that question just the like how do you deal with a performance problem if it like only happens in production so most of the time I'm able to isolate it a little bit like say okay this endpoint is slow like this one specific endpoint is slow and then if I can't reproduce it locally like well what is different between my server and and locally like one big one is data right like if you if you don't have data locally that is you might be doing like a select star and without a limit and if you only have five elements in your database it'll be really fast if you've got a million elements you know it's it's like it's really painful and so like your database your redis your data stores and and then make it might be the conditions it might not just be that endpoint and with data in your database it might be that all of those things but only one one specific user is logged in and so that's when having like a little bit of logging where if you do something like paper trail or something and you say oh yes here here this one request like this very specific request is itself slow I'm going to try and reproduce my local environment as much as to simulate that request as possible and that's gotten me a really really long way granted you sometimes you spend all that effort and it still can't reproduce it locally but nine times out of ten like I find some really low hanging through that way if I had a nickel for every time a developer said it's fast on my machine like that yeah so there's a ton of different ways to make sure that you're experience it experiencing the same things that your users do the big one that most Rails developers and shops don't do is work in development with production like data you probably can't do that if your Shopify or GitHub but most of us are not Shopify or GitHub so just making sure that when you know you call user dot all that in development that might return 10 rows but in production it's going to return 10,000 so just exposing yourself as much as possible to the same conditions that will occur in production on front end this is always network and CPU we all have really nice MacBook Air's probably some fat fiber connections at our offices or at home Chrome dev tools lets you throttle your network connection there's also I mean there's a ton of tools for throttling network connection but throttling your network connection slowing down your CPU Chrome will do that for you and and making sure you experience your website the same way your users do cool I just wanted to make another comment about like major framework upgrade something that both Eileen and Rafael do is they're able to run at the same time both an old version of rails and a new version of rails so they have the ability to deploy it into production with the new major framework change see if it works but doesn't work well we'll just use the old version for now and do you think this is something that more I guess shops should practice more shops should follow with major rails upgrades or is this something that is very niche to GitHub and Shopify well one of the problems with GitHub is we're like really far behind so we kind of have to do that like have the multiple versions because it's going to take too long to do an upgrade to actually get to the point where we can deploy it without having two different versions in the codebase most times actually in fact theoretically could have more than that we just don't do that more than two at once I think that if you have a big codebase it makes sense to have multiple rails versions going just because of the whole how how long the upgrade process is going to take because otherwise you're just going to constantly be falling behind and there's no way you don't want to work on a branch like by yourself that you're constantly rebasing because then you're fixing other people's conflicts all the time and then you're going to go like extra crazy not just rails upgrade crazy but you're going to get rebase crazy and so that's one of the things that we do in a GitHub is we have a conditional that if and and this is how I recommend to do it the version that you're currently on is your if statement so that's like the special case and then everything else should fall through so that way you keep moving the code that's special up to the if case and everything else should fall through and then you know when it's broken for that new version and that it's not like you're not confused about well when am I running rails for code or when am I running rails three code like it's always your top conditional and then falls three so yeah I believe that doing this kind of dual booting as we call is better to upgrade because it's make easy to you to deploy to production if find bugs but also I think it's a good practice for the entire community to be able to do that early like we always try to release rails in some kind of good piff like every six months but we release release candidates and a lot of different things everyone's try risk and it's usually in small applications but the real problem is only happens when people upgrade the major applications like when Shopify discuss GitHub or big applications abbreviated because usually they get all the the corny case of the frameworks so if we always do that the other applications that we always are able to use two different special of the framework that would be better for the entire community also for the application are there any resources that people can go to figure out how to do this because they know now that it can be done but what what what magic do they need to do to make this happen yeah so github has a blog post a really really old blog post about that but I think it's not the github blog but it's somewhere someone from the github made that blog post and I think Shopify also has a blog post two years ago baby and I going to release my slides and also a new blog post next week excellent I'll just I'll just ask that loud the question is what do you do with the gem file and the gem file dot lock file so we have a condition right now what we are doing we have a monkey patch on bundling inside our gene file because the gen files are Ruby coding we came monkey pet side the gen file itself that's not so great so I trying to change that to use different approach like the bundle team told me that there is a option called it evolution file that you can use to share different two different gen files and I'm going to use that in your next project so we have two lock files but only one is in fire right now yeah we have to lock files but our regular gem file has a conditional in it so if the if it's github rails for whatever like it will load the gems for that and then otherwise it loads the three two no so when we haven't yeah some you hack bundler basically everything has to run with a with a environment and variable in front of it so we do rails four equals one and then it knows which gem file lock to load so it loads the four but I don't know if we hacked bundler because I I wasn't there for that I think we're running very very low on time I think we're already five minutes over yeah so I'm just we're still here you can come ask us questions but yeah thank you very much