 All right. Hello everybody. Yes, thank you very much. My name is Derek Parker. I am once again going to be talking about Delve. This is true. So, this time the predominant theme of this talk will be deterministic debugging with Delve. This is a talk that I'm particularly excited about as I think that this style of debugging is not only not very well known but as a result of that underutilized. I think a lot of people think of debugging in the traditional sense of you're debugging a live process. A lot of different things can happen. A lot of things can happen between debugging sessions and between runs of the process and things like that. So, how do we eliminate a lot of that? And especially if you have a bug that say only explained only exposes itself maybe one out of 100 runs, how do you sanely debug that? That's the kind of things that I'm going to be talking about during this talk. Just to introduce myself a little bit more, again, my name is Derek Parker. I am a software engineer at Red Hat. I work on go tooling upstream and I work on Delve as well and some of the go packages that we ship through REL. I am the original author of Delve and have been working on that for a long time among a few other things. OK, so I want to give kind of a rundown of what we're going to go through in this talk. So, as mentioned, I've started to steal the idea for the state of go and I've spoken here for a few years so I like to kind of give an update on the state of Delve. What's happened, what new features and what things have changed since last year? Because in a year a lot of things can change, a lot of new features can be completed and I think it gives kind of an idea of the pace of the project and the types of things that we're working on and focused on. Also allows a little bit of feedback from just users in general, everybody here about maybe what they would like to see in the future. After that I'll go into what deterministic debugging is and explain how it works. First of all, just explain the concept for those who may not be familiar with it and then I'll dig into a little bit of how it works under the hood. So actually how you can get rid of all of these deterministic or undeterministic things that happen in every process run and have a repeatable debugging session. So how that actually works from a relatively low level. Following that I'll talk about how Delve leverages this to be able to debug go programs. And then finally I'll do a live demo just kind of showing off the concept a little bit, showing it in action and giving a sense of just how easy it is to do this yourself in your day to day debugging and programming workflow so that hopefully this is something that you'll reach for or at least know to reach for if you come into a problem where this style of debugging might come in handy. Okay, so first of all let's just jump right into the state of Delve. Let's talk about what's happened in the last year. Some of the new features, some of the new changes and then we'll briefly talk about some things that we're going to work on going forward. So the project is now six years old, which is crazy to me. It's, you know, that grew up so fast, right? I'm very excited to have been working on it and that so many people have found it to be useful. It's very exciting. We've had 175 commits over the last year, not a ton but for a more esoteric project as a debugger. That's really, really good and we've had a ton of new contributors, which is great. We love first-time contributors and try to encourage that as much as possible and folks building in some really huge features, which I'll talk about next. So what's changed since last FOSDOM? Well, there's been a couple of go releases, right? So we've released some new versions of Delve that support that, so specifically 113 and 114, which will be released. Another big, big, big change that's happened over the last year is ARM64 support. So before this, we only really supported x8664. We've just gotten a great patch in recently to add kind of initial ARM64 support. There's still some more work to be done there, but that's still a huge addition to the project. We obviously aim to support every architecture and operating system that go supports natively, but, you know, it just takes some time, so contributions are always welcome. Another big thing is free BSD support. Again, we want to include as many operating systems and everything as we possibly can. So that was another big, big addition. We already support, you know, various Linuxes, Windows, and OSX, so this just adds one more operating system to the repertoire. We've added the ability to script Delve via a language called Starlark, which was originally created for the build system Basil. It's a Python-esque dialect, and that adds a lot of powerful scripting features into Delve, so you can define new commands within Delve and other things just by writing this Starlark language, and those commands will be available to you in your debug session. We've also added support for debugging Go plugins, so for anybody using that feature, that's now supported. Windows mini-dump support, which is basically Windows core dumps, and improved support for debugging Pi's position-independent executables. Some other notable just changes. Some internal refactoring, just ongoing code maintenance cleanup, I think that's an important thing. Several performance improvements, especially in breakpoint handling and a lot of other things. Improved handling of optimized binaries. I think this is an important one, and this took a lot of coordination with the Go team. If you have a binary that's actually running in production or something like that, or you build it just normally with Go build, it's going to be optimized, so we want to be able to debug those, ideally as well as you can debug binaries that you've built with optimizations explicitly turned off. We've improved function call support, and just in general, we've included a bunch of other improvements and bug fixes. Now let's go on to part two, the real kind of me and subject of this talk. First off, I'm up here speaking about it, but I would like to thank my co-maintenor who happens to be here for doing all of the hard work of actually plumbing up Delve the record replay backend that we used to achieve this. He did a lot of the hard work. I just reviewed it, and now I'm up here bragging about it, so I just wanted to give credit where credit students say thank you. Okay, now let's talk a little bit about what is deterministic debugging, and let's also talk about what's that weird little character down at the bottom of the screen. So my son is four and a half years old, watches a lot of cartoons. This is a new Toy Story cartoon with a lot of his sentences asking what is something. So I thought it would be funny to include him in the slide saying what is deterministic debugging. It's more for me than it is for you. Anyway, so just to kind of gauge the room a little bit, how many people have heard of deterministic debugging? Very few. That is perfect. That is actually exactly what I wanted to see, and it kind of proves my point that I think this is a relatively unknown and underutilized way to debug your programs. So I want to talk a little bit about what it is just to give some context, and then as I mentioned, I'll go into a pretty deep dive of how it works and the different kinds of things that is done to actually achieve this. So one more quick question. How many people have seen the movie Groundhog Day? A good amount of people. Okay, so for those who haven't seen it, it's a movie where Bill Murray goes to this town, Positani, Pennsylvania, where every year, actually today on February 2nd, it turns out, Groundhog comes out of the ground and if it sees its shadow or doesn't, it has some sort of mystical bearing on how long the winter is going to last. But the connection here is that every day in this movie that he wakes up, it's Groundhog Day over and over and over again. He's repeating the same day over and over and over again every single day, every time he wakes up for a really long time. And no matter what he does, no matter what he changes in his environment, when he starts the day over again the next day, he goes through the same exact series of events, the same exact people talk to him and say the same exact things over and over and over again, and this happens forever. Well, at least for a really long time. And the connection here is that that's essentially what deterministic debugging is. That's what it's trying to achieve. So the point of it is that every single time you start this session debugging what we're going to get to know as a recording, everything is exactly the same. And this includes memory layout, signal delivery, you know, a lot of the different variables that may change and may slightly change how a bug is reproduced are always exactly the same. So as I mentioned before, if your bug only shows up maybe once in 100 runs, if you capture it just that one time, you can replay it indefinitely to try to figure out where the bug is. So how is this possible? As we know, every time you run a program, there's a lot of things that are different. Memory layout, threads, where variables are stored in memory, what happens as a result of certain sys calls when signals are delivered, etc. There's a lot of things that are different. How can we eliminate all of this non-determinism and faithfully reproduce the execution of a process and as a result faithfully reproduce the events leading up to the bug that you're trying to fix? So the solution turns out to be record and replay. And what this actually means is we essentially record the execution of a process, record all the different types of the result of any kind of non-deterministic operation and are able to replay that back. So anytime one of those operations is hit again, instead of actually going through and doing the operation or making the actual sys call or anything like that, the pre-recorded output is just replayed back to the program. Now, this turns out to be pretty difficult but also very, very powerful. Now, there's a lot of different types to do this. Some are kind of better than others. I'll go through a little bit of existing implementations. So there's VM recording, which isn't actually very widely supported. VMWare actually even dropped it from their virtual machine offering. There's a couple things, but they're really clunky, really heavyweight and really typically you only want to record a process, not the entire execution of the kernel and operating system and everything else. So it tends to be bloated and not very useful. There's also some user space recording, UndoDB, replay engine and Mozilla RR. We'll talk about RR a lot more in the end of this talk. There's pros and cons to each, so some of them do code instrumentation, which I personally am not a huge fan of. I think leaving the code as unchanged as possible is kind of in the spirit of debugging. You're not trying to change a person's code, you're just trying to show them what happened so that they can find the issue. So let's go through some of the pros and cons real quick of some of the existing implementations. As I mentioned with VM recording, you have to debug within a virtual machine that supports it. That adds a lot of bloat and not a lot of virtual machines actually support this. So it's really not portable and doesn't tend to work very well with your workflow, especially if you're not using a virtual machine or you're on bare metal or just kind of writing code on your laptop, you're probably not running it within a virtual machine. It produces large traces, a lot of bloat. It's not widely supported, as I mentioned, and so let's move on to user space recording. It doesn't, some problems with that, it doesn't record kernel execution. I'll explain later why this actually turns out to not be that big of a deal. As I mentioned, some require code instrumentation. The backend that we use does not. So we get rid of that. And some require kernel extensions, which again destroys portability. So there's a lot of cons. Sounds terrible. Why am I even talking about it? What's the solution? So what we use is Mozilla RR, and I'll talk about how it works a little bit. But it solves a lot of those problems by not doing any kind of code instrumentation. It's pretty lightweight and relatively performant. There are some drawbacks that I'll discuss a little bit, especially with certain Go programs. But it's still, I think, the best solution that we have available right now. And it's a start, right? I think, as kind of shown by the polling in the room, not a lot of people know about or are using this technology. So I think as awareness grows, more people can work on it, improve it, and we'll get better solutions in the future. But right now, RR is the preferred solution. So eliminating sources of non-determinism. How can we do that? And what are some of the sources of non-determinism? So here's a couple that we can control, right? Certain CPU instructions. So most CPU instructions are deterministic. You feed them, you know, the input or whatever. They do an operation. They put the results of that operation in memory or register or whatever. And it's going to be the same every single time you call it. But there are some that aren't. So like RD-RAND instructions to generate random numbers. Hopefully that's not deterministic. So the way to get around this, the way that RR does it is it actually, it turns out it's not used very much. That actual instruction, most people prefer DevRandom. So it can just be patched out of anything that uses it. There's also like timestamp counter instructions. Those can actually be trapped and recorded. And CPU ID. That's mostly deterministic if you're running in the same hardware. But it does return which core the process is running on. That can be different each time. So that's also trapped and recorded on newer kernels and Intel CPUs. Thread scheduling. So this is important, especially for Go. So what RR does, and one of the kind of performance hits that it takes, is it runs everything single threaded. So it handles thread scheduling. It does do preemption and things like that for via signals and everything. But it runs your program single threaded. So for large Go programs and programs that are really, really highly parallel, this could incur some slowdown. But unfortunately, that's a penalty and a price that we have to pay for right now. So system calls. The result of a system call could potentially be different. We can trap whenever a system call is executed and record inputs and outputs. So we can just replay that back after, when we're replaying the recording. Memory layout. Again, we want to ensure that variables are at the same address as memory because potentially that leads to a certain bug or exhibits certain symptoms. So we want to, again, we want to recreate the process as faithfully as we can as it was executing exactly when it was first run. Shared memory. This is a big one actually for non-determinism. But for RR, again, it runs single threaded. So you don't have to worry about different threads or whatever within your process messing with shared memory. There are some things where the X server and Pulse audio and things like that communicate back and forth using shared memory, but there's ways of disabling that. Signal handling. So RR is also able to handle this. We can replay signal execution and delivery by using hardware performance counters. So basically we can count and know where we are in the program and what's happening when we're at that specific spot. So when a signal is delivered at a certain point in the execution of the process, we can record when that happens and replay it. So some sources that we can't control just really quick. We don't control anything that actually happens outside of user space, but as I mentioned, it turns out we don't really need to because we just record the result of that anyways. So hardware failures. I mean, if that's happening, you're not going to have a lot of luck anyways. So I'm kind of getting a little bit low on time, so I'll kind of go through some of this pretty quickly to show a bit of a demo. So some pros and cons. Pros relatively low overhead except for single threaded. You can faithfully reproduce bugs every single time. You can record a trace and replay it anywhere. So you can record a trace on your server and replay it locally for easier debugging. You can execute programs in reverse, so you can go to where a bug exhibits itself and kind of work backwards instead of trying to work forward towards the bug and replaying that every time, which I think turns out to be pretty powerful. You can replay from the beginning, execute the same sequence of events. It's always recorded, always the same. Cons, there's a performance hit for highly parallel programs. It only works on Linux and only works with performance counters enabled. So let's talk about how to use it with Delve. First, use Linux. Right now, it's only supported on Linux. Unfortunately, OSX is not supported. Install RR, Mozilla RR. That's not installed for you automatically when you installed Delve. And there's three options for using the RR back end with Delve, which I'll talk about quickly now in a little bit of time still for a demo. So option one is a little bit more manual. So the first thing that you're going to want to do is build your program. Again, ideally, you're building it with optimizations disabled. That's what those flags are. You use RR and pass it the path to your binary. It's going to record it and save a trace somewhere. And then you can use Delve to replay that trace once it's completed. Option two is a lot easier. There's one command that just kind of does this whole process for you. So you can use the Delve debug sub command, pass it the back end RR, and then it'll compile the program, run the trace, record it, everything, and then open it up so that you can start debugging. And then last, you can use the scripting feature that I mentioned earlier. So this is actually included in the documentation of the HUB, so you don't have to memorize this command. But essentially what it does is it introduces a command called flaky where it'll re-run the process over and over and over and over again until it hits a bug, so it exits with a bad error code. And then once it does that, it'll stop running the process over and over again, and you'll be debugging that trace. So if you don't want to do this manually, make the computer do it, right? And once it's done and once it stops, you'll be recording the bug. So let me do a quick little demo just to show how some of this stuff works. I'll get really quick. So as mentioned, you can just debug with back end RR. It's recorded. You can see the output of the run of the program, and now we're actually debugging the trace. So if we set a breakpoint at main, continue, we can see some of the things that it's doing, right? So it's saying it's asking what date it is, printing that out, and it's also opening dev random, which should be different every time, right? So if we next a little bit, we can see, we read from dev random, and if we print the result of that, so like the bytes that we have, you can see kind of roughly just the output that we get, right? So just remember the first two, 180, 183. So let's set a breakpoint here on RR.test.go24. So we'll restart. We're restarting all the way from the beginning again. Continue, continue again to our second breakpoint, and if we print the value of that variable again, we see it's exactly the same. So even complete sources of randomness will be replayed faithfully every single time, and just to show that it's not a trick, you can see when I run it for real, every single time the output is actually different. So we are recording and replaying faithfully, and that's time. Thank you very much.