 Good morning, everyone. Good to see you all here. You're enjoying Barucho? Having a good time? Excellent. Yeah. This has been a great conference. So, today I'm here. The title of my talk is Spelunking in Ruby. So, for those of you that might not know the term spelunking, it's a word that means cave exploration. And I think that this is kind of a metaphor. I just really like the word. It's a fun word to say, spelunking. It's a metaphor to me for sometimes the situation that you end up in with code. There's a lot of Rubyists that will say, you know, I write my tests. I write tests and I don't need to debug stuff because I just TDD my things and that's the end of the story. But sometimes you end up with code that maybe you didn't write yourself. Maybe you get a gem that you have to use, a library to dig into. Or you get the code from the person that was at your job before you that maybe you need to explore and figure out what's going on. Sometimes tests aren't going to give you what you need and you need to be able to debug into your system. And so what we're going to do today is we're going to peel back the layers. We're going to dig down through the caves of your Ruby application. And I'm going to show you some tools and techniques for being able to figure out what's going on and get to the root of problems that you might have in this code. So my slides and information, all of the resources from this talk are available at jsonrclarke.com slash debug if you want to follow along now or later. But to begin with as well, there's also something that I want to say that reading code is a big part of this as well. Understanding your code is critical. And actually our next speaker, Evan Phoenix gave an excellent talk at Scottish Ruby about the act of reading code. Now what I'm trying to suggest is not that these debugging things are the first thing you should reach for. A lot of times you can dig in, you can read to understand what's going on first and that's going to put you in a good place. But when you've reached the limit of that, when you need to figure out what's going on at runtime, that's when debuggers and various other techniques come into play. So we're going to cover three broad areas. We're going to talk about things that are built directly into Ruby that are just available to you all the time that you can use to figure out what's going on. We're going to take a look at gems, both at how you get into the gems that are on your system and gems that are useful for you to be able to debug. And then lastly we'll look at some other different tools that are kind of maybe outside of the scope of what you normally use in your Ruby development and see how those can help us get insight into what our applications are doing. All right. So let's start out at the surface. This is going to be pretty basic to begin with, but don't worry. We're going to go very deep as we go along. So the fundamental thing that everybody does when you're starting out and when you're trying to get a hold on a problem is the lovely put statement. Now, this gets maligned a lot, printf debugging, puts debugging. It's a terrible way to do things, but the truth is it's really easy to get your foot in the door. It's a way to figure out whether something gets called. It's a way to see where you're at and see values out of the program. Now, I've come from languages before Ruby that were compiled, and this sort of debugging was really painful there because of the compilation cycle. It took a long time to build the thing so that I could see the wrong output and then I have to do it again. Ruby actually makes that really fast and easy. A little tweak that I like to do to it is I put my initials when I do this because if you've ever accidentally committed a puts message and something that you've written, well, that's kind of embarrassing. And for me, my initials stand out. This looks like something that I'll notice when I'm seeing the diffs a lot more readily. Save yourself some trouble there. But sometimes maybe you have code that executes too frequently or that gets called from a lot of places and you want to do more than just puts every single time. That gives you too much output. Well, the nice thing about puts debugging at this point is that it's Ruby. It's just Ruby. So if I had a method like this where I want to see that output, but I only want to see it sometimes, I can just set a variable. Personally, I like to use a variable called BDUG to make sure that it doesn't conflict with anything, flip the B and the D just for fun. But this is, you know, it's nasty, it's a hack, but it's fast. And you can start to get some insight into what's going on before you move on to more sophisticated techniques. Maybe you only want to see it every so often. If this is a really hot method, and I wanted to only see it every thousandth one, well, you've got Ruby available, you can do the math, you can do this every, you know, every thousand times. Or you can even do things like this. So caller, which we'll look at in a second returns you the call stack that you're looking at. So you could look at the path and see whether this is a place where this is getting called from a particular location in your code that you care about. There's all of the opportunity that you need to be able to be specialized about exactly when you break that point in the code and when you show yourself the bit of information that you're looking for. Now, this is just the very introductory stuff. This is things that everybody's probably done before. But there are other things that Ruby provides for us. One of my favorite methods is the caller method. So here I have a sequence of methods, Bork, Dork, and Morc that call through each other. And if we call Bork and then at the end of it, we print out what the callers are, what we'll see is an array that shows us the stack trace. Now, I don't know about you, but I've come from languages before Ruby that this sort of introspection was hard. It was difficult to figure out. You could not do this sort of thing. And Ruby, you just get an array of strings that show you exactly how you got where you arrived. I like to play this little trick with it as well. I do a puts and I join the callers, specifically doing a new line and then a tab. And this makes it really easy to see where those stack traces happen in my output because I get an indentation. The first line will be all the way over on the left and the following will be set in. And so if this happens multiple times, it's easy to see it there in the output. Really basic but really useful to figure out how you got somewhere in your code when you don't know. A lot of times, though, the reason that you might be debugging and you might be digging into these various layers is that something's gone wrong. You've had errors that are occurring in your application. So imagine a scenario. You were given this Ruby script. You were told to go run it and you were told that it's going to have some output when it works. You run it and this is what you get. There's nothing there. There's no error. There's no output. Nothing. Well, you dig into the code and you find this. I'm sure none of us have ever had code that's like this, where somebody just rescues and swallows exceptions. Now, admittedly, it's not going to be this simple. This would maybe 20 layers deep. This may be buried and hidden where the exception is being raised and who's swallowing it. But it's kind of hard if somebody is trapping all of those errors for you to figure out what's going on. Well, luckily, Ruby comes to our rescue here because it has a minus D flag. So if you tell it minus D when you're executing that, it will print out every exception as it is raised, whether it gets swallowed or propagated all the way to the top or not. And so this gives you insight into what errors are occurring in your application as it's running, which is really helpful. There's also other ways that you can get a hook into what errors are going on. So if we imagine we have this method, it increments a counter and then decrements it when we're done. And this flaky method, it has problems. Sometimes it raises exceptions and we're not exactly sure what's going on. Well, if we put in our insure block, we look at the dollar bang global variable. While an exception is being raised, that will show you the exception that is there. You don't necessarily have to rescue it to get your hands on the exception as it's occurring. Now, this global variable doesn't hold on to that past the scope of while the exception is being raised up the stack trace. But if you happen to have code that's running while that error is occurring, it's really useful to be able to get your hands around it. So in that case, this would be the output that we would get. We received that exception, shows us the back trace, tells us where that error is coming from and what's happening. Multithreading and concurrency is a big topic in Ruby these days. And it actually makes error handling a lot harder. If an exception occurs in a thread, that thread will die. And by default, that thread will die quietly and just go away. So in this case, we have our main loop down below and we have a thread that's supposed to be doing some work for us, but it had an exception and it disappeared. And maybe it's hard for us to even tell whether it finished, you know, you can put logging or rescues around it, but if you don't have those things, threads can go away and you may not know why they've disappeared. Well, thankfully, while you're in debugging and testing, there's actually a setting that'll help with this. You can set a board on exception to true. This is available both on the thread class and on the instances. And when that happens, Ruby will change its behavior so that an exception raised on another thread will stop the program right there. So in this case, we would end up with output that looks like this, tells us exactly where we failed in that other thread. Now, this is not how you would want to run in production, you know, and obviously it's good to rescue your errors and trap those things, but this is very useful for debugging. If you've got a multi-threaded application that's giving you problems. So that's a lot of built-in stuff. That's a lot of what comes with Ruby directly. But gems, you know, just about any sizable application, you're gonna have to use some other gems. You're gonna bring other things in. And getting access to what those gems are on your system to be able to examine and use them is really critical. So this is one of the things that I love about Ruby. Now, I appreciate what Eric's saying about a compiler, but to me, it is really nice that my gems are just source files sitting on my system because that means I can examine all of the code that's going on in my app. Bundler, thankfully, makes this really easy if you're doing this in a Rails application or somewhere else that you've got a gem file. So you can set your editor variable. Now, there are other incorrect settings that you could use for that editor, just, you know, depending how it goes. Then you tell it bundle open, and it will open whatever editor you specified at the location of that gem as it's installed from your gem file. So for me, it looks like this. I end up right in that root directory of the gem, and then I can go navigate that source and look at what's going on. Again, coming from languages that are compiled, Java, C-Sharp, and various other technologies, the ability to look at my libraries and see them at a source level without having to go try and find their repositories online, it's hugely powerful. This is a really nice feature and they actually picked it up in RubyGems as well. So if you're running version 2.4 or later of RubyGems, it has a gem open, which behaves in exactly the same way. So this is how you can find where the gems are on your system that are actually installed and get access to them. And that's a really helpful thing. Now some of you may be looking at the things that I've shown you and okay, I've just shown you how to find the gems that are installed on your system and I've told you that it's okay to put statements into your code to figure out what's going on. This is kind of a terrible combination, right? You put something into your Rails gem and all of a sudden, you know, every time you're running Rails, it's running whatever debugging code you might have crammed in there. Well, thankfully, the RubyGems folks had thought of that state, thought of those problems and there's the gem pristine command. When you install a gem, it keeps a cache copy of what was downloaded. And when you say pristine, it will blow away the source files that will take away all that stuff that you've edited and put it right back to the state that it was to begin with. You don't have to go uninstall and reinstall your gem. You can just tell it, hey, go make this exactly like it was when I pulled it to begin with. All right. So that's how we can get into the gems that we have on our system. But there are a lot of gems that help us while we're in our system to be able to debug things further. So let's take a look at some of those helpful gems that you can use to figure out what's going on in your Ruby application. The first is kind of near and dear to my heart because I wrote it. Instantiating objects and where objects are created in a Ruby app is extremely important. Ruby is an object-oriented language. But in a large application, sometimes it's hard to know where something came from. Or in tests, very often you'll instantiate objects in uncommon locations that can make it hard to debug if the state carries across between multiple executions. So that's what the hometown gem is for. We'll take a look at how this works in a little more detail. So we require the gem and we tell it watch on a particular class. And this is saying, I want to watch all instantiations that happen for this class. In this example, since I'm watching the thread class, I'm going to spin up a new thread object and then have that go do whatever work it does. And then, lastly, I'm going to ask what is the hometown for this object? Where did this object come from? And what this returns to us is a hometown trace object. And part of that is the back trace. So that shows us exactly where in the source this object came into existence. Now, this has been really useful for me in debugging problems where an object got replaced by some other piece of code. And I don't know who replaced it, but they weren't supposed to. This points right to the spot where that object came into life and got replaced. So this is a really fun gem. I would love feedback on it because it's in the pretty early days, but I've already found a lot of use for it in my projects. But that's, you know, object creation is only one part of what you're looking at with an app. And if you need to dig deeper, pry is a great option. How many of you here have used pry? Yeah, lots. So it's extremely popular and it works really well. We're not going to go into tons and tons of detail, but if we have this code here, we've got a method. We're setting an instance variable there. We've got some locals and we say binding dot pry. Now you can pry on any object. Binding happens to be the object that represents our local scope within this method. So when we run our app, we break into a sort of debugger-like setting here. It points out the line of code that we're on and gives us a prompt where we can interact with pry and tell it what's going on. LS, it's probably sort of uses a metaphor of a file system. So when we say LS, it shows us the things that are in our current scope. And so it will show us the instance variables and the locals and the methods that are available. You can LS on another object. You can CD to move to other objects. It's well worth looking into the documentation for all the things that you can do there. But this is a really great way to, you know, with puts debugging, you have to know what you wanted to output. If you don't know what you're wanting to look at, pry lets you interactively do that. One of my favorite commands with it as well, because often I start poking around and then I forget where I'm at in the code. You can say, where am I? And it will reprint you that trace for the location where it broke out. Very useful to have. If you want to know more, the help is great. There was also a fantastic presentation at RubyConf in 2013 by Conrad Irwin, which steps through all sorts of features that pry has. You should really dig into it. But, you know, there's a little bit of a problem with the way that that pry was happening for me. And it's very often that I don't get the pry in the right spot. I put it there and I look and then I realize I wanted to be further on in my code. What I really want is an interactive debugger. I want to be able to step forward. And thankfully, there's an ecosystem around pry that allows you to do that. Here are three gems depending on your specific Ruby version that will all provide you with interactive debugging capabilities built on top of pry. Personally, I end abusing pry nav a lot because the gem that I'm on supports 187 still. But if you're on a newer Ruby, you can use those others and they're faster and have more features. So how this works, you get the typical sort of pry scenario like we had seen before, shows you where you're out in the code. If we say next, you can see that that arrow moved down the line, moves in on the code. It next is sort of step in the current context to the next thing in this scope. You can then say step. And that says step into a method. It goes down into whatever's there. And so here we've gone into step two. We can see what's happening. And when you're done, you can say continue or control D. That will tell it, okay, stop prying, go on, let the program carry its execution forward. So this is really useful when you're trying to locate where the problem is and maybe your binding point didn't quite work the first time. But that's not the only way that you can get the location wrong. For me, personally most of the time, when I put a break point in, I end up finding that the change that I was caring about actually happened before where I put my spot. Thankfully, there's a gem called pry stack explorer, which helps out with this. So we'll look at this code as an example. So we've got a parent method. The priority of a parent, especially a parent of a young child is sleep. We call into the child, the priority that the child sets as a local variable is not sleep. It's playing. And then we have a binding pry there at that point so that we can stop. When we run this, we'll see we stop in the child just like we would expect. And if we look at the priority variable, we'll see it's set to the child's priority. And that's fine. Now I want to highlight at the top there that there's that spot where it says frame number and it's frame zero of two. So pry stack explorer lets us say go up. And this steps up a level in our stack. And so now we can see how we ended up in the child method. We're in the parent context. This does not just show you the code where you were at or the location in the stack, but it's actually the full scope of where we were at. So if we say priority and ask for what that local variable is, we don't get the value of where we broke out. We get the value of when we were at that location. So this sort of lets you roll back in time to some extent to see why you got where you got and what the values were at that time. It also has a very nice show stack. The caller that I introduced you to earlier, if you run that while you're in pry, you'll see about 12 lines worth of stuff in your call stack from pry. And it makes it a lot harder to read. This properly strips those things out. It shows you just the code that you actually care about. Final call out about pry, this is my favorite thing to put in my pry rc as an alias. So it matches the regular expression that if I hit return with nothing on the same line, it will repeat the last command that I had. This makes it so that if I'm navigating and I say go next, I type next, then I hit enter and I go enter, enter, enter, enter, enter, and it carries on moving as far as I want to. This is something that I actually picked up from the Ruby debugger, a feature that it had that pry through its configuration will let us replicate, which is really fun. All right. So that's some gems. That's some things that you can install pretty easily and take care of doing a lot of debugging. But there are other ways of getting insight and digging down to lower layers of what's going on in your Ruby application that we should look at. And we're going to start out by looking at a couple of things that come from alternative Ruby spaces, just to get your mind going about the possibilities there. So the first is an example that's going to come from JRuby. So we had a lightning talk yesterday about shoes. I'm a contributor on that project. And so it runs on JRuby at this point with the base implementation. And here's a JRuby shoes app that we've got running. It's the manual for that. So once I've started that up, there is an application. It's built on the JVM. This is not specific to JRuby, called VisualVM. And when you start that, it will give you a dialogue. It's Java, so there's lots of GUIs and lots of clicking and buttons. But it shows us instances that it's got JRuby applications running. And one of those is that shoes app that I've got going. Now, once we drill into this, it provides a ton of different options for how you can figure out what this process is doing. You can do profiling and sampling. One of the things that I love best, though, is it provides a heap dump. So a lot of times, if you're having problems with a library or with an application, it can often revolve around memory. You know, you may be allocating too many objects or you may be using memory in places that you don't think that you were. And this will give you a snapshot of all of those objects that are in your JRuby process and show you what's going on. It's actually kind of fun because you can see some of JRuby's own implementation and how it maps to the underlying Java objects that are available. But this is something really cool that I don't feel like we have a lot of in some of the other Ruby spaces. There's a lot of great tooling in the JVM that you can use when you're using JRuby. And I'd like for us in the Ruby community to take more inspiration from some of those tools and not always just rely on command line things that we've got today. So this is also something that the JRuby team is working with a lot to try to improve. And I just want to give them props for the things that they've been building into JRuby. It really pushes forward the tooling story around Ruby. And I hope that we see a lot of cool innovations out of that in the future. But JRuby isn't the only other Ruby that's out there. Rubinius has a good debugging story as well. So much like Prye, which we install as another gem when we're running on CRuby, Rubinius actually has its own debugger built in. So you tell it debugger start. That's the equivalent of that binding.Prye that says, hey, stop at this location. And so in this little example, I've got my message. And I mixed up the code that should have said my message down at the bottom. So we're executing that function later on. So this will stop just like we saw in Prye. It gives us sort of a command line output at the location that we're wanting. But one of the cool bits that I like about Rubinius's debugger is its breakpoint support. So I can say breakonobject.myMessage. And that will make a breakpoint very like when we inserted that binding.Prye, but without us having to edit the code to make it happen. Even cooler at this point, this message does not even exist. This method has not been defined yet. And Rubinius recognizes that and says, hey, do you want me to put that breakpoint in when I figure out what this message is that you're wanting to deal with? And so we say yes. And then when we run the code, it says, oh, hey, I just got this method defined. I'm going to put the breakpoint in that you asked me to. And then it will break at that location in the code. This is a really nice mix to go along with the fact that Rubinius provides a minus X debug that you can pass when you're starting. And this will stop immediately on starting the loading of the app. So you could stop at this location without having modified your source, make breakpoints at the spots that you want in your app. And you can do it by file and line number, not just by method, without having to actually modify your source code, which, admittedly, is something that kind of bothers me at times about some of the privacy debugging and some of those techniques that I've got to remember to roll those changes back so I don't commit it. All right. So that's a lot of Ruby level stuff. But you know, Ruby is not the only thing. Ruby, CRuby, and Rubinius are implemented in C and C++. And sometimes you run into problems that maybe you need to know what's happening at the layer below you. This can often happen when you're dealing with threading and concurrency issues. So we're going to give an example here of a classic deadlock. And we're going to dig into it with an application called GDB. It's a new debugger. It's a C debugger. It's on most Linux systems and Unix-based systems. And it's a tool that you probably have heard of and you may never have used, but it actually can come in handy for some of these scenarios, even as a Rubyist. So in our application, we're going to spin up a couple of mutexes. We're going to have two locks. I'm going to make one thread that's just going to sit there and run. This actually is just to avoid Ruby's deadlock detection. Because in fact, if you don't have that, the rest of the code that I show, the application is smart enough to sense that there's a deadlock and just lets us know, which is not a very good demo. So with those mutexes in place, we're going to spin up one thread. It's going to lock on the first mutex, sleep for a moment, do some work, and then lock on the second mutex. And we're going to have a second thread that takes the second lock, sleeps, and then tries to take the first lock. Now hopefully you don't write code like this yourself in your application. Deadlocks are more subtle when they actually happen in practice, but this is the basis of what goes on. We'll then join on those threads. And what we'll see is that we never see that done message. Those two threads are in conflict. They each have a lock that the other wants, and they're waiting for the other to let it go before they try and grab their lock. So how do we see that this is happening if it's occurred in our code? So we've run our Ruby application. It's done its initial output, but we're not seeing our done message. Our threads are not getting through. How do we figure out where they're stopped? So let's fire up GDB. Takes quite a bit of privileges, so you probably need to sue to it. It's going to show you a bunch of stuff, and then drop you at a GDB prompt here. At this point, we say attach to the process ID of the process that we had just started. And this is going to spew a lot of text at you, a lot of different things. But don't get too worried. This is a big deal with using these sorts of tools that you may or may not understand fully is don't get overwhelmed when it gives you a lot of output. Pause, think, and look for the pieces that you do recognize. So in this output, it tells us something about reading symbols, debugging symbols, and I've had cases where it ends up giving me a bunch of question marks for method names and locations. Well, oftentimes that will tell me what things it wasn't able to load, where those debug symbols were missing. This is also telling us some things about our threads. We're dealing with a concurrency issue. It shows us the threads that we have working in our application. So that's useful information. And then lastly, it drops us at a prompt and tells us where we're located, what our last stack frame is. All right, so where do we go from here? We're attached to this process. What do we want to find out? Well, we want to see what our threads are actually doing under the covers. And so you can say thread apply all, and that says run this command, this last command, to all of our threads, backtrace. And this is gonna give us the C level backtrace of all of the threads that are currently in our application. Now it gives us this first thread, thread five, doesn't look like anything that I recognize. But if we look through the list of some of the other threads, if you let your eyes kind of wash over that, you may notice this line, rb mutex lock. Now, if you've ever looked at the MRI source code, most of the C methods in there, the convention is that they start with rb underscore. So this is part of C Ruby's implementation that's telling us something that's going on. And actually that's a pretty sensible name, like you can probably guess what that's doing. Some of them don't map us directly to the Ruby code, but we'll take a look at a moment at how you can resolve that. So thread four has this mutex call. We can see in another thread, in thread three, that it has the same thing. The key bit to figuring this out, if you don't recognize where that's coming from, is it tells us the source location. Tells us exactly where to find this function. And if we go over into the C Ruby source, we'll see rb mutex lock. And in fact, it gives us an extra hint. The documentation in this form tells us that that is mutex lock. The casing's a little different, but that's how it binds it. So you can see what the Ruby methods are that map directly to this C implementation to track what's going on. Now, if my Ruby process had just stopped, and I didn't know that it was deadlocked on the fabricated case that I made, looking into this would tell me that there is locking going on. And you can find out what those threads are paused on, what they're doing while they're sitting there. And then backtrace that to the Ruby implementation of what's going on. It's really powerful when you're in a situation that the Ruby debugging level of things just isn't telling you what you need to know. So this is kind of headed into a little dangerous territory. And I'll say, if you're spinning GDB up and attaching to a process, I really hope that it's probably a process that you're okay with dying because it is easy to get things to seg fault. It's easy to not have that application carry on. So this is sort of a thing of last resort. You don't want to do this just willy-nilly. So having said that, there's a lot of interesting things that you can do once you're in there. For instance, we can call CRuby functions directly from GDB. So we say call void RB backtrace. That's a method that's there in CRuby. And then on standard out, wherever our application is running, it's gonna give us a backtrace of the current context that it thinks that it's running in. Pretty helpful if you didn't know where you were at paused in your Ruby code to figure out. But this might get lost in the wash of all the output that's going on from my application. So I actually wanna capture this stuff. I wanna do something a little more sophisticated. So let's go and let's monkey around with our file descriptors a little. I'm gonna redirect standard out and standard error and put them to my own file. So we'll say we're gonna call the C method, close on file descriptors one and two, standard out, standard error, and then reopen them to some arbitrary file that I've chosen. With this method from our GDB and net file, kind of a configuration file for it, then I can say, hey, redirect all of the standard out for this process over to gdb.log. And then when we run that backtrace again, and it's good to call a flush to make sure that it's all written out, that output goes into my file instead of going to wherever my application used to be writing out to. This can be useful for being able to capture some of that data. And at New Relic, actually, we've built some application, a script that will do some of this in an automated way and capture that information for us to be able to debug gnarly issues like this. Now last, and certainly most dangerous of all the things that I've run into trying to do this, and your mileage may vary depending on Ruby version, this particular little gem tries to run a certain function in MRI. And it's the eval function. So you can pass this arbitrary Ruby code while you're attached to it in gdb, and it'll try to run it. This is pretty crazy, but occasionally we've gotten some good signal out of it. If it's a process that's already stuck, it's already dead, and you're just trying to figure out what's going on, this is a tool that you can use to try and dig in there and get a little more information out. Like I said, it may give you what you need, it may give you that kernel of information that you're seeking to be able to understand your problem, or it may segfault, you know. But it's a fun thing to do and a good way to try to dig in. So while we're down at this level, there's another tool that's less invasive that can give you some of the insight that we were seeing, and that's called Strace. So if we run our same debugging, our same deadlock scenario here, it's running on process ID 3621, and if we say Strace pointed at that and the minus S is to kind of limit some of the output, what we get is we get an output of all of the system calls that are happening in that process. Now, a system call is anytime that the user LAN code, so your Ruby code and the applications running at that level, need to call to the kernel, the operating system, to have it do something. So all of your file reading and writing, all of the memory management, a lot of those things are system calls that get made, and you'll see them in this Strace output. It makes a lot of output. There's a lot going on in that communication. But in this particular case, we can see, I mean, it's not mutex, but futex as a system call in there. So if you were watching this process and looking at what's going on, you might be able to glean some of these calls and what's happening. Now, all of these system calls are well documented in the Linux source and other Unix sources, so you can say, man, for that particular function, and here it tells us, oh, that's fast user space locking. So this is actually the system call that mutex is implemented in terms of. So if we hadn't wanted to attach GDB, hadn't wanted to go to that level, we could take a look at just the trace of what calls are happening, and that might give us a clue about what's happening under the surface. So there's a great talk by James Gallick on exactly this topic of using STrace, that he goes into a lot more detail and tells some stories about debugging on platforms and languages that he doesn't even know and figuring out problems with STrace and how he steps through debugging them. I highly recommend it if you're interested to get into that level of things. There is one sad aspect to this for the many of us that are running on a Mac. This is a Linux-only utility, and so you can spin up a VM pretty easily to do it. It's worth your time. It's an interesting exercise. There is Detress, which is a similar tool that does work on Mac OS X. So I haven't used that much. I've done most of my work with STrace, but it's worth taking a look at. So that's pretty deep. That's pretty far down there. Let's take one step back up. Sometimes I wish that I knew what was going on at the Ruby level, what all of the things were like. What I want is I want STrace for Ruby. I want to see what's going on, all the method calls that are happening in my Ruby process. And thanks to Aman Gupta, we actually have something that'll let us do that. It's called RBTrace. You put it required into your process, and here I just have a very simple thing that sleeps in prints every so often, just so I continue to get some output. If I run that, find out my process ID, and then in another shell, I go over and I say, hey, RBTrace, go watch this process and give me the fire hose. Give me all of the things that are going on. What I get is I get an output, nicely nested of all of the method calls as they're occurring in Ruby in my application. This is incredibly awesome to be able to get that level of insight to see all of the messages that are flowing through your Ruby. And I'm very grateful for this. I've just discovered it fairly recently, but it's a really awesome tool, and you should try it out. Unfortunately, there are some stability issues with the fire hose on OS 10. They're documented in the readme that you may or may not get all of the calls. I've found that I get some, so you get enough information potentially, but that may be a caveat for you. But there are other methods that RBTrace allows you to be more specific. You can ask it to only show you methods that are a certain time, that are slow enough. You can ask it to only watch certain methods, only watch GC activity, and it has built-in tracer sets, which basically are a collection of methods that it knows to try to trace together. So RBTrace is really awesome. I think you should take a look. It's kind of my new favorite tool, and I'm looking forward to finding ways that I can use it in my job. So that's a lot of information. That's a lot of different tools, but I hope that you will come away from this feeling equipped to dig into that next layer below where you've been at. Maybe you've just been writing Rails applications and you've never looked at a gem, or maybe you are a gem author, but you've never looked at the next layer below and looked at how things are actually implemented and see Ruby and looked at those problems. There are tools at every layer of the stack that Ruby provides for us that will let us get insight, figure out what's going on, and solve the problems in the deep dark caves of our Ruby. Thank you.