 The talk is about debugging Ruby. These slides are up on the script. You can get a bit of a delete slide debugging Ruby and a follow along if you want. My name's Amon. I go by TMM1 on GitHub and Twitter. And like me, in writing Ruby code, you probably want to do a couple things. One is NastyBugs. Flow code Ruby is known to be flow and memory flow. And this talk is essentially about tools that you can use to fix these issues. There are tools to help you debug. And so we're going to talk about a bunch of these tools. And these are tools for Linux. Since most of us deploy on Linux, it's pretty useful to be able to debug issues that happen in production. Tools for C code, since MRI and most of the extensions we use are written in C. Tools for networks, since most of us are working on web applications that talk to web services and databases and other web applications. And of course, tools for CPU and memory usage, since ultimately that determines how far we can scale and how much it costs to actually run and maintain our Ruby application. So one quick note before I dive into these tools. A lot of these slides have a lot of fine print, essentially, details on how to get up and running, what flags to use, what command line options I use these tools with. I'm going to gloss over these. But the idea is basically what I want you guys to focus on is what are these tools useful for? When would you use them? Why would you use them? And then later on, when you actually find a reason to use them, you can come back to these slides to look at all the options that I use and I recommend copying pasted commands in to get up and running. So don't let those details bother you for now, but be sure to check them out later when you're actually trying to get up and running. So the first tool is LSO. This is something that's available on your Macs, give a Mac, and runs on Linux as well. It's essentially used to list open files. So you would run this on any PID, basically. And the cool thing about this is you can take any running process on your system, Ruby or not, and run LSO on it and get a sense of what it's doing. And this is usually the first thing I'll run because it sort of gives you a high level overview of what the process is doing, what it's all about. And so you'd run it and you would see some output like this, and immediately you start noticing some things. First of all, you'll see it's pulling in some shared libraries. And in this case, we see the JSON library, you see the Mancache library, we see the MySQL library. In addition to the shared libraries, it also lists all the open files and open sockets. And so you'll see there's connections, actually two connections to MySQL server, running on port 3306, and another connection to a Mancache server running on local list port 11211. In addition to that, there's a HTTP outgoing connection, presumably talking to some sort of web service. And there's other random stuff you'll notice from there, the current working directory of the Rails application, you'll notice that SCD out and SCD error both point to an NGINX error log, and then there's this rack multi-part file, which is presumably dealing with some sort of file upload. So that's also, again, it's pretty useful because you can run it against any process and sort of see what is it doing, get a sense for what that process is all about. The next tool is S-Trace. S-Trace is a Linux-only tool that creates its system calls and signals. So system calls are basically calls to functions that are defined in the kernel. When you make a system call, you switch out of user space and into kernel space, and basically execute a function that was defined in the kernel. The cool thing about S-Trace is, similarly to else of, you can run it against any running process by just providing a PID. And it actually prints in real time all of the system calls as they happen. So you can attach to any process and you watch it actually calling functions in the kernel. So there's two ways that I like to run S-Trace. One is in the summary mode with the dash C flag. And the way this works is you attach, you run it for a while, and then when you control C, it'll print out the summary of what happened during the time when it was running. And basically what it does is it shows you what system calls are called most often and where you're spending the most amount of time. So this is pretty typical for any sort of web application. If you're doing a lot of reads and writes, usually you're talking, you know, you're reading back data from MySQL server or web services, stuff like that. And so most of the time, 50% of the time here was spent reading from different sockets. So that's summary mode. The other mode is tracing mode. This is sort of the normal mode. And I use a bunch of flags that are listed over here, but you end up with output that looks like this. Basically streams in real time or you can tell it a file to pipe into. So if you take a closer look at this output, it basically shows you all of the details for every single system call that's going through the system for that specific process. And so you might start noticing certain things. For instance, a Rails application will be doing a lot of reads and presumably we'll be getting incoming HP requests. And so here you can see it's reading from file square 22 and it's reading what looks like an incoming HP request. And it tells you some details such as how long that we took and how many bytes ended up reading. And again, FatherScript 3.22, if you remember back to the LSO output, you could run LSO on this process and see what FatherScript 3.22 points to and see the port and the IP that it maps to as you could get the client's IP. Another thing you might notice is lots of reads and writes from MySQL database. And so again, here, FatherScript 5 corresponds to a MySQL connection and you can see we write a select statement. It's pretty fast and then we read the response from the MySQL server and you'll notice on the right, it shows that that read took about a second, almost a second and a half. And so that right there is indicative of a slow MySQL query. So I've used S trace a lot. It's actually really cool. You can run it on any process, like I said and sort of get a sense of what's going on. So one of the first things I did was on a production Ruby instance, ran S trace on a bunch of my Rails applications. And one thing I noticed was Apple like this, tons and tons of signals, SIG VT alarm. And I was wondering, what's going on here? And I started digging into it a little bit. It turns out Ruby 1.8 sort of uses the SIG VT alarm signal to manage its green threads. And basically it tells the kernel, send me the signal every 10 milliseconds. That way I know every 10 milliseconds to switch threads. The problem with this was, my application was not using the threads and I was still seeing this. And so I started digging into this and came up with a very simple patch. And the problem was basically, once you started a thread, an additional thread, even if that thread went away, it would continue getting the signal for the entirety of the process. And so I submitted the patch that was recently accepted. It took about a year to get accepted, but it is gonna be part of Ruby 1.8.8. And it basically, all it does is, as soon as the last additional thread dies, it kills off the timer. And so you're no longer getting interrupted. Another thing I noticed in S-tracing Ruby processes was, again, this is summary mode. Tons and tons of calls, here there's almost between a half million calls to a function called RT SIG PROCMAS. And so I started digging into this. And what's going on here was the default builds for Ruby on Debian and Red Hat Ubuntu all compile with a flag called enable pthread. And this actually doesn't use native pthreads, it's sort of misleading. But what it does is instead of telling the kernel to send it that SIG-BT alarm, it spawns off one additional kernel level thread to do that timer. The problem was because of a slight bug in the configured script, in addition to enabling that additional thread timer, it would also force using SIG PROCMAS every time a signal was delivered. And so this actually caused a huge slowdown, a 30% slowdown, which is why most people recommend that you don't use the default Debian package for Ruby when you're deploying it to production systems. And so, again, I submitted a patch to this, which is now part of 188. And as of the next Debian release, they actually incorporated the patch as well. So this is no longer going to be an issue. But again, something as simple as just S-tracing your process, looking at what's going on, sort of asking questions about what is this, and sort of dig into a little bit, leads to pretty major wins and a better understanding of what's going on. The next tool is TCP dump. Oh, yeah, question. Do you know if RVM enables pthreads with you? RVM does not enable pthreads by default. The only reason that Debian does this is because the only reason it's recommended to enable pthread is if you're using Ruby TK. And Debian has to sort of accommodate everybody, whereas most of us don't use Ruby TK and RE and other builds don't enable by default. Yeah, over here. Should you enable pthread on Ruby 1.9? Ruby 1.9 probably doesn't have this option because the 3D model is completely different and there are no more green threads. There's fibers that are the equivalent of green threads, but threads by default Ruby 1.9 are actual native threads. Any more questions? I'm gonna stop probably at the end of each tool to get any questions, just so it won't be easier. So the next tool is TCP dump. This is really useful for networking. And again, like I said, most of us are writing sort of web applications and applications that are talking of a lot of different network services. Again, there's a bunch of options that you can use. What it boils down to basically is an expression that you pass in and that expression dictates what sort of traffic you want to watch for. And so you can type an expression such as this one. Find the all TCP traffic that's going to port ED and that's basically outgoing HP requests. And so you'll start seeing both incoming and outgoing HP requests and you can sort of get a sense of what's going on on the system and here you'll see here's an HP request that was probably incoming for a PNG and you can see what IP it came from, what IP it's going to, et cetera. Similarly, you can do the same thing for MySQL. Start seeing MySQL queries coming in and out. We're probably the most useful way to use TCP dump is with dash W, which basically takes all the data and puts it into a file. And the cool thing about this is you can run this on your server, collect a bunch of data, you know something weird's happening, either with MySQL or Memcache or some sort of web service you're hitting, you'd be a tutorial API and you can write an expression to capture only that data right into a file and then download that file loaded into Wireshark. And Wireshark is an awesome utility. It's all a really nice GUI. The cool thing is it understands a lot of protocols and so it'll show you here's an HP request and it basically parses out all the headers from the TCP data and you can drill down into the actual protocol and it understands all of the major protocols to make it real easy to dive into problems and try to figure out what's going on. So any questions about TCP dump? Cool. Next tool is Perf Tools. Perf Tools is actually a set of tools that came out of Google and it's used in Google on their production systems and it's actually really, really nice. Here's again a bunch of details on how to get up and running. You basically have to download it and set it up and then the way it works is it provides a shared library that you can either link against or LD preload and then as long as you're linked against the library all you have to set is an environment variable called cpu profile to tell it where to dump the profile information and once you have that, you can use the pprof tool to analyze that profile and give you a bunch of data. Really cool thing about this is it provides a graphical output like that on the right and basically just by looking at that you can see where most of your time is being spent because the bigger the box is, the more expensive that operation is. So again, I ran this on a bunch of my Ruby stuff and the first thing I did again, since this is a sampling profiler it's fairly low overhead and you can actually run it in production if you want and so I ran this on a production MRI instance and just wanted to get a sense of what's going on inside MRI and I found something pretty interesting about 10% of the time was being spent in a function RV store sub bang which translates to string sub bang in Ruby land and I was able to trace this back to the time dot parse function and so 10% of CPU time in my production Rails instance was being spent essentially parsing out time being fed back from my SQL and I was curious what was going on and I pulled up the time dot parse code it looks something like this over here where it basically runs a series of about 2,000 regexes on your string because it supports a bunch of different time formats so that was pretty useful and the solution for that was to switch to one of a variety of different solutions for time parsing there's a bunch of gems, home run, third base, a bunch of other ones that basically move all that logic and see also the new MySQL 2 adapter does all this stuff and see because it knows exactly what the MySQL format stream is going to look like and so basically save 10% of CPU cycles I also ran this on a mem machine so I maintained a mem machine and for a long time there was a known issue where a mem machine and Ruby's green threads did not interact well together and so I ran first tools and 26% of the time was being spent in mem copy and it turns out you can actually follow the mem copy up and see it's being called from RB thread save and restore context and so what was going on here was a weird interaction in between how Ruby's green threads basically switch state and the mem machine was doing something weird where it was allocating a bunch of stuff on thread stacks and so was spending a lot of time copying that state in and out but once I knew what was going on it was really easy to go in and one fixed mem machine so it wasn't allocating so much stuff on thread stack but also we spent some time Joe and I patching Ruby itself so that it didn't need these mem copies in the case where you were deploying on an Intel CPU so again a simple tool and the whole idea here is when you run into a blog you need to sort of get started and a lot of these tools provide a way to get you a little bit of data so you know where to start looking and once you know what functions start looking at you can set a break point there you can sort of put prenefs in there you can do any number of things to figure out what the exact issue is once you know where to start looking so work tools was really cool but the thing I wanted was after running this a few times I was like it'd be really nice these graphs actually showed me Ruby functions instead of C functions and you know in some cases it's easy to see RV stirs up bang sort of understand that that actually corresponds to string sub bang but wouldn't be awesome for that actually send strings of bang in there so I created this project called perftools.rb which is essentially a fork of the project that adds Ruby support to it so again here's how you get started you can just install it but simple example here's a simple scenario application has two actions one is sleep heavy sort of weight heavy so it's not actually doing anything on the CPU but it takes a long amount of time it's sleeping for a quarter second the other one is CPU heavy so it's basically calculating the first 10,000 Fibonacci numbers so you can run perftools.rb on this and I'm sending each of those requests 50 each of those actions 50 requests and what I end up with is again graphs like this and so there's actually two different modes you can run perftools.rb and one is the regular mode which is the CPU profiler mode and one is the real-time mode and so on the right and the CPU profiler mode of course the compute function shows up at the top because it's doing a lot of CPU burning and in the real-time mode the sleep function shows up on top because it's not a CPU intensive but it actually takes long and so you can run it both and on the left in real-time mode it's stuff like HP requests and MySQL calls and anything that's sort of IO intensive is going to show up whereas on the right anything that's CPU intensive is going to show up so again I use this on a bunch of projects early versions of Redis.rb would use the system timer jam on every single operation and that was really expensive and it was really easy to see in here and so we were actually able to go in and get rid of that and only use that when connecting and use normal and cheat-to-be socket timeouts for the actual read and write operations it's actually also really cool this sort of graphical output that gets created for just understanding a piece of code and sort of seeing the call paths and how the functions call each other and how the code is laid out and so for instance, running this on RubyGems you'll notice RubyGems spends a lot of time querying the file system because it's trying to figure out where the gem specs are and what gem specs especially when you require a file that has to go on the file system and check all the load paths and try to figure out where that file exists and you can sort of see where these are getting called from and start looking for spots that you can sort of optimize these things Recently, I ran this on Bundler and I noticed that about 23% of time when running Bundle install was being spent in gem version spaceship was basically comparing a bunch of gem version objects starting to get out of which one to install and so I knew there was some sort of optimization that could be done there and a simple patch to RubyGems was able to improve performance overall performance of Bundle installed by about 15% and it was just some micro-optimizations in gem version but knowing that a lot of time was spent there those micro-optimizations actually made a big difference overall and then finally, this is actually really recent probably a couple weeks ago I added an object allocation mode to Perk Tool and RB and so in this mode each sample corresponds to one object that was created and in Ruby, it's really easy to create objects and objects are actually rather expensive and it's nice to be able to see what functions are creating a lot of them because the more objects you create the more work GCNs do to garbage like those objects the more memory, the more pink memory usage you end up hitting so again, this is on a Rails application running in the object allocation provider mode and you can see again, date parsing and time parsing is not only CPU intensive but also when you're doing it, Ruby creates a lot of objects because you're doing a lot of string manipulation and again, we switched to MySQL too recently and all the stuff went away because it all moved down into C and so you were no longer creating like 16% of the Ruby objects created overall over being created in date parses so that's Perk Tools on RB any questions about that? Cool, so the next tool is Elatrace Elatrace is very similar to Strace but instead of tracing system calls it traces library calls again, you can run it in summary mode or you can run it in a tracing mode so this is the example from before with the mem machine and threading and again, this is just sort of confirmed that mem copy was taking a long time and you could actually see that the mem copies were happening right after each signature alarm so the kernel would tell the process it's time to switch threads and it would do two mem copies and you can actually see the arguments mem copy where it's copying, in this case, over a meg back and forth every time it was doing a context switch and that was because the mem machine was allocating a bunch of stuff on the thread stacks and again, this is cool because you can actually see it happen in real time and you can see the arguments but the big problem with Elatrace is that it doesn't work with the re-extensions and the reason for this is because they're loaded at runtime and they're not linked at compile time so Joe DeMotto worked on a branch of Elatrace that adds libdl support so here's how you use it there's a bunch of options that he added but basically what it lets you do is trace calls to any C function inside Ruby and so you can go ahead and trace all the calls to garbage collection, for instance and so you can actually watch in real time all you're doing is passing a PID to a running process and you can start watching in real time when a garbage collection is happening and how long it's taking and so here you can see this and this was again in production in Ruby and since I was running it against every garbage collection cycle was taking about 200 milliseconds and it was running almost every five seconds and so what that means is in Ruby, in MRI at least and even in 1.9, the GC is stop the world GC and so if the garbage collection gets involved during request response cycle that's 200 milliseconds that's gonna be added to the response time similarly you can run this against other queries other, sorry, other function calls so here's the query function and leave my SQL and so you can actually watch all the mysql queries and how long they took you could do this for memcache if you use this in the past you're trying to figure out where cache collisions are happening and you could just run this for a couple of hours and then run a Ruby script on the output try to do some analysis that's Ltrace, any questions? yeah so one of the first ones you showed that mincopy was being used as a revenue by min machine how did you get mincopy up to figuring out the min machine doing it? the easiest way is with GDB so you attach to the process instead of breakpoint and then you can basically check the entire stack grace I'm gonna dive into that a little bit so any other questions? yeah how do you give the function names to those people that use the breakpoint? just looking at code I guess any questions? yeah the question was how do you get the function name in the first place so you know what to Ltrace and this was just I guess I knew my SQL was making queries so I pulled up a look at my SQL client and tried to figure out what functions to start tracing and the thing you have to do with this is actually provide the function signature so it knows how to show the arguments so you just give it an Ltrace cough that has all the function signatures you care about so GDB is like the standard C debugger and since we're all using again MRI which is written in C and a bunch of C extensions it's a pretty useful tool overall the most useful use case for this is debugging segpoints so every now and then you'll end up segfaulting and you have no idea what to do and you end up, you know, Ruby tells you that and that's not really useful Ruby 1.9 actually shows you a back trace a C level and Ruby will back trace which is far more useful but to sort of demonstrate this I wrote a simple C extension that defines a global segv function so whenever you call segv it's going to segfault and then I wrote a simple Ruby script that does a bunch of stuff before segfaulting so it's, you know, doing four dot times and a door.thdir or a hash.new and then finally segfaulting so there's two ways to catch segfaults inside GDB one is to attach to a running process and you'll notice a lot of these tools have sort of a strong emphasis on being able to attach to a running process and this is really useful because a lot of times you have problems that show up in production and it's really hard to replicate them on, you know, a different environment your Mac is a different environment than your Linux production system and it's hard to replicate them and so it's really useful to be able to go to your production box and say, you know, attach to this process right here and start seeing what's going on so the way this works you, you know, find a page of your process attach to it and it's going to drop you at a prompt and you continue or see and that basically waits until something happens and something in this case is going to be a segfault and so it's going to sit there until you get the segfault the other way to do it is to force the process to coordinate the way this works is you have to do a couple of things one, you have to tell the kernel where to put your core dumps and two, you have to inside your process set the max limit of how big your core dumps is allowed to be by default, this limit sets a zero which is why you never get a core dump if you know your process is about 300 megs usually you know, you can set it to 300 megs and as long as it's under 300 megs it'll dump out a core dump to whatever directory you want it to dump it out to so once you have a core dump or you caught it inside GDB all you have to do is type where or back trace and it basically shows you the entire C stack trace and once you have this I mean you could start looking at the final line numbers or the easiest thing to do is copy and paste this into a GitHub issue or something at least let the maintainer know sort of what's going on they're going to have enough knowledge about their code base where they can start looking at this issue and try to solve it so again here if you remember you know we called four dot times to our CH to our hash nu and you can see the C equivalence of all those inside your backtrace as you see it's due times in numeric dot C which calls RB yield and you end up in dirt dot C and you end up in hash dot C and you can see how that relates to your C code which again was useful but what I really wanted was the actual Ruby function inside my backtrace and so again I worked on a project called gdb.rb which just adds a bunch of Ruby level folks to gdb so you can do a bunch of cool stuff in here again you can attach to any process and you can start evaluating the Ruby code inside the context of that application so again you know you're running thread.current or thread.list size and you're actually getting details about that Ruby process you can get a lot of details about the threads so how many threads are running which one's currently running what state they're in you know the third one down there is waiting on a file descriptor five again I could run lsol on this process and see what file descriptor five is the first thread which is the main thread that's waiting on another thread which is the third thread down there so interesting details it also provides a little bit of introspection into object space and what types of objects exist and so here you can see there's you know about 900,000 live objects and 83% of those are nodes and so it sort of gives you a sense of how much memory you're using how many objects we have created and you can start to see if there's memory leaks in certain areas and then you can actually even dive into different types of objects so you can dive into all the strings and see you know if there's a certain string that's being repeated over and over again unnecessarily so again going back to the example from before we have the site fault and now with gdb.rb instead of just running back trace we can run Ruby threads and see the stack trace for that thread and instead of getting C level file line numbers we're getting a Ruby level file line numbers and it makes it much easier to debug so again I've used this on a bunch of projects to fix a bunch of issues the first one was Rails Warden which had a weird leak Rails Warden is basically a sort of authentication and optimization framework and what was going on here was it was injecting a rack middleware but the middleware was leaking and since middleware has references to the previous and next middleware it's basically causing a leak of all the middleware in the chain and so this was a production application again that had about 4,000 requests and so there were 4,000 instances of every single middleware and once we saw this we were like okay we know where to start looking at least we know why this process is taking up with gdb.rb let's start looking at all these middleware and try to figure out what's going on fix this issue I know that interesting thing I noticed just running this in production mongrel has this really strange piece of code in it where every single mongrel instance has one thread that all it's doing is a loop-sleep one and the only reason I came across this was again I was looking at all the threads inside a Rails instance and I saw there's this thread in mongrelconfigurator.rb that's just doing a sleep and I want to look through the code and it's just like the global thread that exists for no reason as far as I can tell the next one is God God for a long time had a lot of memory leaks and using gdb.rb Eric Lindvall was able to find at least five separate leaks to fix them all and so God is now very very stable and he's been running it for last year without any leaks and again you can dive into all the arrays and what we noticed was there were a bunch of arrays that had almost 100,000 elements and so these were arrays that were just constantly growing until we knew something was going on there and again just looking at instances there were a bunch of driver events and even watches, 43 watches even though I'm just watching one process there's 43 instances of God watch something's going on there. So that's gdb and gdb.rb any questions? Cool. Next tool is a leak house and so a lot of tools are covered so far through the standard Linux and Linux Linux and C utilities. One thing that's unique about Ruby is the way it manages memory which is very different than you do it in C. C is all about explicit memory management allocating, deallocating stuff where in Ruby land you just sort of create objects and let garbage collector deal with it. The problem with this is it's really easy to leak references or hold references such that the garbage collector thinks a certain object is still in use and it's not going to get rid of that object and so there's a variety of tools that help figure out where these objects exist where they were created, why they're sticking around. So the first one is Bleak House. Bleak House is similar to, so gdb.rb shows you what sort of object exists but Bleak House goes a step further and when it actually tells you where those objects were created which is extremely useful because there's specific parts of your code that are leaking objects, right? And so the caveat of course is that you actually have to install a patch version of Ruby and reinstall all your gems and then use that separate Ruby on your application. But once you do that you can get reports like this that basically show you for each file, line and object type combination how many objects still exist. And so you might notice there's 5,000 objects, 5,000 strings, but then you actually get the final line number and you can go to that piece of code and try to figure out why they're being created and why they're sticking around. The next tool that's sort of similar in a similar vein. Before we move on to the next slide with Bleak House so you have to patch your Ruby VM and then reinstall the gems does it have any performance load if you're not actively debugging your code? Yes, so the question was what is the performance impact of Bleak House and basically what it does is for each object that's created it saves somewhere the file and line number. So CPU-wise it's not that intensive but it increases the size of each Ruby object because it adds two fields to it. And so your memory usage will increase which might or might not be okay for production. Do you recommend running that production then? Usually it's not recommended to run production. I mean, if you really have enough memory on your box you can. I'm gonna talk about our tool that's a little bit easier to use and you can turn on and off. Question over here. What do you feel about Bleak House in RBM? The question was how do I feel about RBM in Bleak House? Like, is it playing well? Yeah, so the way this works is when you gem and solve Bleak House it'll actually download into Power Ruby. It'll probably be pretty easy to just add an RBM build for Bleak House. I think, so Evan Weaver wrote this and used it on Twitter for a while but I think it's mostly superseded by the next tool Memproff. Just because Memproff's a lot easier to use and you can turn on and off. So I'm actually not gonna talk about Memproff at Linn because there's not that much time but if you're interested in checking it out bit.ly-slash-memproff is a talk I did a while ago about Memproff. So the cool thing about this is basically it came out of, it's sort of the evolution of GDB.RB and Bleak House in that it does not require any patches. It's just a gem that you load and in addition to telling you what types of objects and where they were created it also tells you where the references are that are holding that object. So the coolest feature by far of Memproff is that you can basically snapshot any Ruby application and upload the set of all objects to memproff.com and you get this sort of GUI that you can start clicking around to navigate all of those objects. And so for instance you could click on an array and see all the elements of that array or you could click on the references for that array and see what other objects are referencing it. Actually, I think I have a few minutes so I'll run through a quick example of how to use this. So I used, we were using Rails 3 in dev mode and we noticed this was Rails 3 beta one and we noticed that over time you get slower and slower and it was actually leaking a bunch of memory. So by the end of the day my Rails application was two gigs and basically crawling. And so I decided to let's try to use Memproff and try to figure out what's going on. And so all you do is first step, load in memproff and then reboot up our application, send it 30 requests and that was enough to make it leak. And the way you use memproff is you basically run it and give it a PID and your API key and stuff and basically what it does is it attaches to that PID, dumps out all the objects and uploads that dump to memproff. It gives you a link so you end up on a page like this. And so you can start clicking around to figure out what's going on. I clicked on, do the classes by name, just clicking around and so I ended up on a page like this and this is all built on MongoDB and so at the top there you see the MongoDB query and the number of results and basically the view of all the results. And so what I noticed here was there were 30 copies of test controllers which is really interesting because I sent exactly 30 requests to the application. So I knew something was going on there so I clicked on that and again it just does another Mongo query where it's saying show me all of the classes that have the name test controller and so you see all 30 of them with their memory addresses. So I clicked on one of those to get details about it and then I clicked on references and so again here you can see all of the other objects that are referencing this test controller and so if there were no references garbage collection would go, oh there's no references let me get rid of this object as long as there's references it's gonna stick around with it. So in this list of references I noticed a hash that had length 30. Again 30 requests, 30 copies the hash with exactly 30 elements that's probably where the leak is. So you click on that you get details about it and sure enough there's all of your test controllers and so what's going on over here is it's using the controller class as the key into a hash and it tells you where this hash is being allocated so basically tell me your line to look at and it was able to pass this but basically what was going on was this was a Dev Mode thing where in Dev Mode every request it reloads all your application code in case it changed but at the same time there was a bunch of caching logic in there as an optimization to speed things up that was caching all the partials associated with every controller and the problem was basically since you were reloading all your controllers it would get inserted into this hash over and over again and the patch was really simple instead of using the actual class object we used the class's object's name as the key and so if you use the string test controller it would just overwrite that entry over and over again instead of inserting an entry. So that's Memproff and again if you're interested bit.ly slash Memproff has a lot more details about not only Memproff.com but also sort of the API's that you can use even locally to try to do better so. Any questions about that? Cool. So this is the last tool and I say the best the last if you do nothing else I would recommend you definitely check this out get up and running with this and what this is is basically it's a Rack Perp Tools profiler it's something that Ben Brinkerhoff wrote and it's basically a Rack middleware around my Perp Tools app out of your project. And the way this works is you basically just pull in this middleware and you pull into your Rails application to not know whatever it's just Rack middleware and once it's in there you can profile any web request and the easiest way to do this is on any URL you just append profile equals true and times equals whatever 10 times 100 times and it'll spit back instead of the actual response it'll spit back the GIF profile that we saw earlier. And so this is extremely useful because on per URL, per request basis you can get a profile out and it's so easy to get up and running and this is the way I usually use it down here at the bottom where like the home page or you know certain URL is taking a long time you just append profile equals true take the output, stick in a GIF open that in preview and turn out you'll see the biggest boxes in there is where you need to start looking and again you can set different modes all three modes work with this so you can do it in object allocation mode and see which functions are creating most objects or you can do it in real time mode and get a sense of what sort of IO and other real time operations are taking on this. So that's it for most of the tools any questions about that one? Yeah. Yeah. The printers are, there's text, GIF there's a couple other ones if you check out the Perf Tools RV documentation it's basically all the ones that the Perf Tools project itself exposes there's like PS there's like a PDF one but it's basically either text or graphic. So that's basically it but the main point I wanted to make is you know like when you have a problem it's really easy to sort of ask what the fuck and to be confused but if you have the right tool to go to then you can start chipping away at the problem and it's all about knowing what tools to use and so I really recommend you know at least start playing with these get a sense of what's available at least have in the back of your mind you know this use case why if there's a site for GDB is kind of gonna do the right thing and let me know where to start looking and once you have sort of stuff in your toolbox when you run these problems it's gonna be much easier to go into that tool box try to figure out what the right thing to use is figure out what the problem is once you have at least a small clue about what's going on then you can start making progress and end up fixing that issue So that's it, any questions? Yeah? So I've tried to do these Perf Tools type things in a situation where I'm doing a lot of forking a lot of them kind of fall flat on our face once you start forking because it's really hard to work across processes do you have any of those in particular that you thought worked well or any other tools for debugging forks? Sure, the question was how do you debug across forks? And Perf Tools actually does a really good job of this the Google Perf Tools code itself anytime you fork it'll start the profile over again and basically take your profile or path and append it to it and so it actually works pretty well I don't know how well it's supported in my Ruby stuff but it should be pretty seamless where it should just work and so Perf Tools is definitely the one I would check out The other way to do it is to use some of the APIs directly and sort of add some code where after before you start the profile explicitly which can get a little bit messy but it does work Any other questions? Cool, thanks a lot