 Hello. My name is Derek Parker. Today I'm going to be talking about deterministic debugging specifically with a tool called DELV. Now, before I start getting into a lot of these concepts and kind of really getting into some of the kind of nitty details and things like that, who has heard of or is aware of or knows about the concept of like deterministic debugging or some sort of record and replay type thing? Okay. So one, two-ish. That's perfect. That's absolutely perfect. One of the reasons why I chose to give this talk is kind of specifically for that reason. I feel like this is a debugging technique that is very underutilized, underdeveloped, and underappreciated, really. I think it's very powerful for even a generic class of bugs, but especially powerful for certain bugs which may be hard to reproduce. That's kind of the sweet spot of this style of debugging. But there's also other benefits that kind of layer on top of it. And I'm looking forward to presenting this to everybody, and hopefully everybody will walk away with an appreciation of this style of debugging. Now, it is deterministic debugging with DELV. If you haven't heard of DELV, I'll explain it a little bit more later, but it's specifically a debugger for the Go programming language. However, not everything in this talk is going to be specifically about Go. A lot of the concepts and even some of the tooling that I discuss are relevant to any programming language, C, C++, Rust. You can apply these concepts to anything, really. But I'm going to specifically talk about it in the context of Go for reasons which I'll explain a little bit later. Now it's going to stop working. There we go. Okay. So as I mentioned, my name is Derek Parker. I am a senior software engineer at Red Hat. You can find me on Twitter at DirkDerek. At Red Hat, I work on the Go language, so I work on stuff in the Go compiler, upstream work. I do work for the Go tool chain for REL specifically. And I also work on DELV, which is a project that I created. We now are going to be shipping that as part of the Go tool set package that we ship in REL. And so I maintain it as part of my job at Red Hat. As with our previous speaker, I'm also previously CoreOS, although I went to Red Hat on purpose, not as part of the acquisition. And as mentioned, I'm the creator of DELV, the Go debugger. I maintain it now with one other co-maintainer and the community at large. So I want to give everybody an idea of kind of the pace of this talk, everything that we're going to go through and all of the major points that we're going to hit. So the first thing that we're going to talk about is what deterministic debugging is, which I think is very important, since it's kind of a concept that is not very well known and that's going to set the stage for a lot of the rest of the talk. Second, we're going to talk a little bit about how it works under the hood. I don't know about you all, but I love to know how things work under the hood. I just like to get those kind of details. So in any talk that I do, I try to do a deep dive into whatever subject I'm presenting. So we'll get into a little bit of a deep dive of how all of this works, which hopefully everybody will find as interesting as I do. Following that, we'll talk a little bit about how DELV leverages all of these concepts for debugging Go programs and some of the pros and cons of that and how you can do it in your daily workflow for finding kind of hard to reproduce issues or just having a more sane debugging experience. And then at the very end, I'll do a little bit of a live demo. Nothing crazy, but just showing off some of the concepts, kind of going through some of the commands, how to use them, just more practical and showing off examples of how everything works. So first off, what is DELV? And second off, what is that weird little thing in the corner? So I have a five-month-old son, so I've been watching a lot of Toy Story. This is a new character in the Toy Story movies, and he always asks, what is things? So I thought it'd be funny to put in the slide. The context is lost on anybody. It's all right. It's more for me. It doesn't matter. So let's talk a little bit about what DELV is. Another kind of poll of the room, who knows about or has used DELV. Cool. So a decent amount of people. So for those of you who don't know, as I mentioned a few times already, DELV is a debugger for the Go programming language. There's plenty of debuggers out there. Why another one? So DELV does certain things that are Go-specific. So it's able to do specific coordination with the Go runtime for doing things such as making function calls in a Go process as it's being debugged. It also knows about the concept of Go routines as a first-class citizen. So as you're stepping through the execution of your program, you want to make sure not only that you're saying on the same thread, you want to say on the same Go routine, which is a concept that's held within the Go runtime, not exposed by the operating system or kernel. And so DELV specifically knows how to figure out what Go routine you're on and follow it, even as it's kind of multiplex and moved across threads. So that's a very important concept. It also knows about Go types and things like that. Supports all major operating systems but for right now only works on X86, 64 and ARM64. We're working on adding more arch support in the coming year. So yeah, I kind of explained this already. So I also want to give a shout out first to my co-maintainer for DELV, Alessandro. He actually did a lot of the plumbing to get some of this stuff to work that I'm going to talk about today. I just get to come up and brag about it. But he did a lot of the, you know, I just reviewed a lot of the pull requests and things like that. But my co-maintainer did a lot of the plumbing hooking all of this stuff up. So I wanted to give credit where credit is due. Okay. So what is deterministic debugging? This is another reference that might be lost on people but hopefully not. Has anybody seen the movie Groundhog Day? Yeah, a couple people. So for the people who haven't, it's a movie where the actor Bill Murray, he goes to this town, punks its honey, and it's on Groundhog Day. Groundhog Day is a tradition where a groundhog comes out of the ground. If he sees a shadow, I think it's more winner or something I can never remember. Anyways, in the movie he wakes up and every day is Groundhog Day for like years. Like this just happens for a really, really long time. So that's kind of what deterministic debugging is. It's replaying the same execution over and over and over and over again and nothing changes. Every time you replay it, it's exactly the same. You're executing the same exact sequences of instructions. Memory layout is the same. Thread IDs are the same. We try to remove all kinds of non-determinism so that every single run is the same. So essentially you're debugging the same exact sequence of events over and over and over again. It's predictable. You're always going to hit the bug. So if you have a bug that maybe only hits one out of a hundred times or one out of whatever, that one time that it actually hits, you can record that and replay it endlessly so that you can figure out where the bug is without having to wait another 99 runs until you hit that bug again. So that's the main benefit of this. It's hard to find bugs. You're able to record them once, replay them indefinitely, and you don't have to waste a lot of time trying to reproduce them. So it's like Bill Murray waking up in his Groundhog Day every single day. So how is this possible? How can you actually record a program and replay it? Every time you run a program, it's something different. The addresses and variables are slightly different. If you do get time of day, it's going to give you the actual time of day. If you call for a random number, it's going to get you a new random number, hopefully. How do you take out all of these variables which may or may not be leading to your bug so that you can reproduce it every single time? So the solution here is called record and replay. So essentially what we're doing is we're running a process, executing it, and recording certain things about the execution of that process, saving that information, and then replaying that back once we replay the process again. So the goal is to take out or record as many sources of nondeterminism as we possibly can so that when that process is played back and, say, a syscall happens or something like that, we can just replay the result of that as opposed to actually having to make the syscall or actually having to do certain things. Basically making your program more of like a black box. So we're just kind of replaying inputs into it as opposed to it actually making some of these syscalls and things like that which could be different on every single run. So I'll go into a little bit more detail of how this actually works under the hood, but basically we can accomplish this using a combination of things like ptrace, hardware performance counters for measuring where we are in the program so that we can do things like sending signals at the right place and scheduling threads, and even doing a little bit of second ppf filtering to mostly for performance. So there's a few different ways that this has been tried kind of over the years. This is like a relatively new-ish technology, but people have been working at it for a while. So you have kind of some of the heavyweight solutions such as like VM recording. So recording everything that happens within a virtual machine. So VM workstation actually used to do this, but it doesn't anymore. They took out support for it. It's clunky. It records more than you want. Likely you only want to record maybe one process or a group of processes that kind of form from that initial process tree. So the traces ended up being huge and it's just overall clunky. You don't really want to play the state of the whole virtual machine, the kernel, everything like that. Another attempt at this is user space reporting. So there's products out there such as UndoDB, TotalView, ReplayEngine, and Mozilla RR. I'll go into it a little bit more, but what we end up focusing more on is Mozilla RR. So some of these other tools and technologies do kind of undesirable things such as instrumenting code and things like that which I've never really been a fan of in the grand scheme of debugging as I feel like instrumenting the code kind of fundamentally changes it. You don't really want to change the code. You just want to debug it, right? So I think anything that's intrusive like that, I've never really been kind of a big fan of. So nothing is perfect. There's always problems with some of the implementations that exist. So I'll kind of go through some of them. I touched on them a little bit already. So with VM recording, you have to debug within a virtual machine that actually supports the feature. There's not really a lot that exists and I think the ones that you have to pay for. But like I said, even VMware Workstation, they took the feature out. So there's not really much that supports it anyways. It produces large traces. This can be a problem for if you need to store a lot of these things or move them around from computer to computer and debug locally or something like that. It can be very large and unwieldy. Yeah, and not widely supported, as I mentioned, it's kind of been taken out of some products. So user space recording, you're not recording kernel execution. That doesn't actually end up being a problem, as I'll explain later. We don't really need to record kernel execution just what the result of what we asked the kernel to do was. Some require code instrumentation. Again, something that I'm not a big fan of. I don't like fundamentally changing the code. And some require kernel extensions which kind of destroys portability. You want to be able to just kind of take this piece of software, put it on a server, put it wherever, and just kind of be able to let it do its thing without having to worry about if you have certain things installed. So there's a lot of downsides. Sounds terrible. Why am I even talking about this? What's the solution? So the solution for now is Mozilla RR. Mozilla RR gets around a lot of these problems. So it tends to be very lightweight. It doesn't do any kind of code instrumentation. It doesn't run your program within a VM and record that. It's just kind of like a lightweight program that you can throw around different places and point it at a program and tell it to record it. So how do we actually eliminate sources of non-determinism? During the execution of a program, there's a lot of things that can happen that are different with each run, right? The result of a sys call, getting the time of day, things like that. How do we, even thread scheduling and when interrupts are fired and things like that, how do we kind of get rid of all of this non-determinism so that we can replay the process execution back faithfully? So let's go through kind of some of the sources of non-determinism and what we can do about them. So certain CPU instructions, right? It turns out a lot of the CPU instructions are deterministic, right? You give them an input, it's going to do something on that input and then put the result of that someplace. Every time it's going to be the same computation. But there are a few CPU instructions that are different every single time. RD-rand, generates random numbers, hopefully going to be different every time, right? We hope. So in the case of Mozilla RR, they kind of end up patching, they find that it wasn't used that much, like on Linux, DevRandom is used, so libstandard C++ uses it, so they kind of patch it out of programs that use it. RDTSC, reading from a timestamp counter, obviously that's going to be different every time we do it. So that instruction can actually be trapped at the CPU level. We can record the result of that and replay it every time it gets trapped again. CPU ID, so this is mostly deterministic, but it does return the index of the running core. So on newer kernels, this can actually be trapped, but RR can kind of use some like CPU affinity to pin to a certain core, so we can get rid of this non-determinism as well so that you can run it on the same core. Thread scheduling. So as your program is running, different threads are running, right? And they can be scheduled at different times by the kernel. So how do we get around that? So RR runs only one thread at a time. So this could be a downside for large, highly parallel programs, which go programs kind of do tend to be. So in practice you could see a lot of slowdown during recording and replay, but we're doing this for debugging anyways, and it's still a relatively new feature, so ideally this will get better in the future. So we run a single thread at a time, and RR kind of does the thread scheduling. So all of the threads are managed by ptrace. When a thread enters a syscall, we know if it's a blocking syscall or not, and we need to run another thread, and so on and so forth. So we can kind of control thread scheduling and play that back in a deterministic fashion. So syscalls, right? System calls can return file descriptors, map memory into the process, things like that. They return data to user space using registers and memory and things like that. So how do we record everything that's happening in syscalls? And how do we do it in a performant way, too? So again, under the hood, RR is using ptrace. So each system call is trapped. Some are filtered by a second BPF filter, which kind of goes into this interception library that is injected where kind of common syscalls can just be recorded all without having to do like a ptrace context switch, so it improves performance a little bit. But for the rest ones, we can just use ptrace stop when we hit a syscall, record the values of it, and then play it back. This even holds true with exacting programs. So RR can play back recordings where a process execs and forks another process. It will record that other process as well and play back that as part of the subtree. So it's pretty powerful. So memory layout. RR, when replaying, will reproduce, recreate the memory layout that was there initially, meaning all variables are going to have the same memory address and so on and so forth. So it removes that kind of nondeterminism as well. Shared memory. Again, RR does single thread at a time, so within the same process, there's not too much raciness between shared memory, so it kind of eliminates that as well. So it's not the best for detecting race conditions and things like that, but it avoids sharing memory issues between threads. There are issues where some applications will share memory with maybe like a device driver or like X or something like that or pulse audio, but RR specifically disables things like that. Signal handling. So signals can be delivered to your program at any time, your process. How do we replay that? So under the hood we're using performance counters to track when that signal was delivered combined with certain register values like the PC value so that we can replay that and send that back at the same time when we're replaying the execution. So let's talk about some sources of nondeterminism that we can't control. So those were all things that we could control. Now we're talking about things that we can't control. So code executed outside user space in the kernel. Turns out, as I mentioned, that doesn't really matter. We just record the result of whatever it does. Hardware failures. If your hardware fails, there's really nothing that we can do about it. We can't make it fail again when we're replaying it. So that's just, you know, something that won't be able to be reproduced. So let's kind of go over some of the pros and cons of this style of debugging. So the pros, it's pretty low overhead. It's relatively performant. The largest hit you're going to get is from running everything within a single, kind of as like single threaded and especially for go programs that you can incur a decent amount of slow down. But I feel the benefits outweigh the negatives here for being able to kind of consistently reproduce a bug. Another yeah, then that's the next pro. You'll always be able to faithfully reproduce your bug. For things that happen intermittently, this is extremely important. But even for bugs that you can reproduce fairly easily, the ability to kind of, one of the things that you can do with these recordings that you can't do with just a typical process is do reverse execution. So instead of having to exit a debug session and start all over again or anything like that, you can get to the source of the bug and work your way back, or you can get to where the bug is exhibited and try to work your way back to find a solution instead of always having to try to work your way forward to get to try to piece together what went wrong. You can start at the moment that things went wrong and kind of work your way backwards if you want to, which could be a little bit easier for being able to find your bug. You can record a trace and replay it anywhere. So there are some limitations on where you can record. Some VMs don't support it like virtual box. You need access to the hardware performance counters for this specific style of recording and replaying. But being able to record it maybe on a server or something like that and then debug it locally on your laptop can be pretty powerful. And then you don't have to have access to that program every time. That trace is always going to be there. You can always look at it and run through it again and debug it. As I mentioned, you can execute programs in reverse, which tends to be pretty powerful. And you can replay from the beginning and execute the same sequence of events. So you can just kind of restart the process over and over again, and it will always be the same. And I'll show a little bit of that during the demo. Okay. So some of the cons that I had mentioned, performance hit for highly parallel programs. This can be a downside for go, but not always. I mean, this is only really if you're doing a lot like actually in parallel. So in practice, there is some slowdown, but you can definitely get away with it. Another con maybe depending on your view of the world is that it only works on Linux. So there's currently no hard set plans to port this over to OS X or Windows or anything like that. It does use specific things of the Linux kernel, the proc file system and things like that to do what it does. So it's a little bit less portable that way. And also as I mentioned, it only works with like VMs or Windows or hardware that supports performance counters. This is needed for a lot of the replay and recording certain events that happen. Great. So that's kind of an overall explanation of what it is and some of the details of how it does certain things under the hood. If you're interested about, that certainly wasn't like an exhaustive representation of what it does under the hood, but if you're interested in learning more about like how technology like this is implemented, RR did deliver a white paper, which kind of goes over a lot of the design and a lot of the low level functionality that it does, why it does what it does and how it does all of this recording and stuff like that. So if you're interested in learning more, I would seek that out. It's pretty easy to find just off of the actual website. So let's talk a little bit about usage with Delve. So how do we use this with Delve and how does Delve leverage it? So it turns out Delve, we made it so that it can use certain different back ends, right? So we've written our own native back end for your typical debugging flow that's completely written in go. We wrote it from scratch. But we can also plug it in using the GDB serial protocol to RR. And we also talked to, say for example, LDB on Darwin OSX, because they make a lot of changes, kernel changes and things like that that can be difficult to keep up with. But that's neither here nor there. So Delve can talk to different back ends and RR happens to be one of them. So what happens is you get the power of RR and being able to record and replay a program coupled with Delve running on top of it. So you get all of Delve's knowledge of go internals, go execution model, knowledge of types and things like that on top of this record replace system. So it ends up being really powerful for go. There we go. Okay. So how do we use it with Delve? First, be using Linux. So you have to be running on a Linux machine or on a Linux server or something like that. As I mentioned, this is all pretty specific to Linux. The first thing you're going to want to do is actually install Mozilla RR. So if you just go to the GitHub, their releases page, they have RPMs, dabs, tar balls, and you can also build it from source. They have documentation on how to do that. It's fairly easy and straightforward. And then once you have it installed, we have kind of three different options for using the RR back end through Delve. I'll go through each of them in a little bit of detail and then I'll do a little bit of demoing. So option one requires a little bit more work, a little bit more setup work. So it's a little bit more manual and I'll run through it in the demo and everything like that. But it can be one of the more flexible options for doing this. So the first thing that you're going to want to do is run, go build, and that flag at the end basically disables optimizations, certain optimizations, some variables, certain inlining, and things like that. Go has gotten a lot better with providing debug information for optimized variables, but it's still not perfect. So the best thing to do is just kind of cut that all out. You have to actually force go to produce unoptimized binaries. It optimizes by default. So produce unoptimized binaries for the sake of debugging. You can do this on just regular standard go binaries and as I said, with modern go compiler, the debug and dwarf information that they put into the resulting binaries has gotten a lot, a lot better. So you actually could have a really, really good experience. But if you do have control over it, running unoptimized can produce a slightly better result. So after you've built your binary, just run RR and the path to your binary, what it's going to do is it's going to execute your program, and during the execution of your program, it's going to record it. Once your program terminates, it'll save that recording to a trace directory somewhere in your computer. It's configurable. By default, it throws it, I forget exactly where, but there's a default directory, but you can control the output of it. Once that's done, the trace is there, and then you can use delve to replay that trace. So it'll load up the trace and start a debug session. From there, you can treat it as like a normal debug session. You can start setting break points. You can continue everything like that. But then once you hit break points, you can do things like reverse execution. So you can rewind from one break point to another break point. Or you can restart without actually having to restart the entire process. You're really just restarting the recording. So it's like hitting play from beginning on Netflix or something like that. It's really, really easy, and you're still replaying the same exact sequence of events every single time. So that's option one. Option two is a little bit easier, but could be frustrating for intermittent bugs. So one of the main entry points into delve is the debug command. So basically what that does is it will compile your program for you, turning off optimizations, and then it will exact that program for you, and then begin debugging it and land you in a debug session immediately. So it's pretty much like one command does everything that you would want to do to start a debug session. And it sets everything up in the most ideal situation so that you're going to have the best debugging experience. The problem with this could potentially be, as I said, if it's an intermittent bug, you're going to have to run this a lot. And if it doesn't fail, you're going to have to exit the debug session, try to run it again, and do this over and over again until it fails. Once it finally fails, then you can start figuring out where the problem is. Also, with this, by default, we delete the trace directory. So after you end the session, the trace directory is going to be gone unless you copy it out during execution. So I don't believe we currently have an option to save that, which could be a good thing to add. But as of right now, it's going to clean everything up afterwards. So if you want to replay over and over again after this session, you're going to have to kind of manually copy it out. So option number three, which solves some of the pain points with that approach is using some scripting within Dell. So you can script the debugger, excuse me, using a language called Starlark, which is a Python ask language that was originally created for the build system basal. So it's kind of been adopted, made it to slightly more of a general-purpose language, but it's kind of a sandbox-safe language, really good for scripting things like this, like the debugger. So you don't have to memorize all this. I can show where this is, but we have an example of this in the documentation. So you can write just a little script, a little command. So basically what this does is it creates a new command within Dell called flaky. And it starts the process, and essentially it continues to run it over and over and over and over and over and over and over again automatically until the process fails. Once it fails, it stops and puts you into a debug session. And you're going to be debugging that recording. So it rerecords the program until it fails. Once it actually fails, it lands you into a debug session debugging that recording and kind of replaying that. So as I mentioned, I can show some of the documentation on that. I can show it now, actually. But I'd like to kind of switch over to a little bit of the demo portion before questions just to kind of show some of this stuff in practice. Is that big enough for everybody? Okay, cool. So in here we just have like a small little go program. It just says, it just prints some stuff, gets the date, and prints that out. So if we run that, you can see every single time it's printing out slightly different information. You know, like the seconds are going up. Every time it changes. So the first method that I showed you, we can do just by running go build. We have the resulting binary there. So we can do rr on that. So it prints here where it saved the trace directory to. So then we can just do replay with that trace directory. And then we can start issuing commands. So let's set a breakpoint at main. Continue. And you can see the address that it was set at. And we can say go routines, see where they're at, the addresses, the threads. So we have thread ending in 948. And if we restart and continue back to that breakpoint, hit threads, we have the same exact threads, thread IDs and addresses and everything like that that we had last time. So everything is just kind of being replayed exactly as it was. So if we start stepping through the program a little bit with next, we can print like the result of date, which it kind of prints it like expanded out. But you can see the wall time there. We can print the address of it. And then if we restart, maybe I can make this a little bit smaller so that you can see. If we restart and then continue next, next, next a little bit, prints the address of it, you're going to see the address is exactly the same. So EF0, yeah, EF0. So the address is the same. The memory layout, everything is exactly the same every time we run this. So that's kind of the first way to run it. So if we just remove that, I also mentioned the back end flag. So instead of doing all of that, you can just issue one command. It basically did everything that I just did but in one single step. So now we're debugging a recording. And we can essentially go through the whole process and do the whole thing all over again. But we have to issue less commands. And I won't actually run through the script but I don't know if I actually have Wi-Fi. But the script, if you go on our GitHub and go to documentation, CLI, Starlark at the very end, so it gives you a breakdown of the whole scripting API but it also gives some example scripts to kind of get you started. And at the very end is the script that I showed of rerunning a flaky program over and over and over and over again until it hits the bug and then being able to debug that recording. So, yeah. Anyways, that's pretty much it. I was trying to get my thank you slide up big but it doesn't really matter. That's it. And I think we have a little bit more time for questions and stuff. So, yeah, if anybody has any questions or anything like that. So the question was if there's a syscall to like read from a file or read from like a network socket or something like that, is that also recorded as well? Yeah. Yes, all of that is recorded. So file reads, anything like that. I probably should have maybe put some more interesting stuff like that into the demo. But yeah, file reads, anything like that is going to be, is going to be recorded and replayed back exactly the same. So you're saying, so the question was if the program calls like date twice, right, and then you try to call it manually? Yeah. Oh, so like, like doing a function call through the debugger, right? Yeah. So doing a function call through the debugger, you won't get a recorded output. There's no pre-recorded output for that. You'll get the actual time of day. But you know, the two date calls will be different. But they'll be different correctly if that makes any sense. So the earlier date call will be the earlier date call, the later date call will be the later date call. But they'll be the same as they were when the original process ran, if that makes sense. So you like, you change the value of a variable or something like that. Yeah. So it's up to the debugger up until that point to faithfully replay what happened, right? But you can still modify the contents of memory, right, if you want to. You can make it nondeterministic if you want to or change things if you want to. In the end, the address of that variable is still going to be the same. Just the contents of it are going to be different. So that's still possible. Yeah. I guess it's sort of two questions. I get it for command lines, but I think OpenShift will deny ptrace and all that stuff quite tall. So I know there's debug containers on Kubernetes, so there's the general debugger use in Kubernetes. The question would also be, I guess my question is, do you have specific war stories of like above that you found this really useful for as opposed to print up debugging or like none recorded debugging? Sure. So I, so as far as OpenShift and things like that, I believe there's a way to like inject certain debuggers and stuff like that into the containers and maybe remove some of the second stuff that's blocking ptrace or whatever. Not 100% sure on that, but there might be a way. As far as my own personal war stories, I mostly spend a lot of my time working on a debugger now, so I don't tend to debug the debugger too much, but I mean off the top of my head, I can see, you know, I know that there's things that earlier in my professional career that if I would have known about this, this could have been valuable, especially for different processes running on servers and things like that, crashes happening intermittently, being able to record that and kind of take it back to my laptop and figure out what was going on. I know that there's been several times in my career where I would have found that to be beneficial, but I think the problem is kind of raising awareness that something like this is even possible. If nobody knows that it's even possible to record a process and replay it, you'd be more hard pressed to find a use case for it if you don't even know that that's a tool that you can reach for. And if you run into a situation where this might be useful, at least now you have it in your back pocket. So if you're running multi- process, you actually have distinct process trees running multiple threads, so each process is going to run single-threaded but Mozilla can follow several different processes and trace them. So if they're running, I believe what will happen is every process will be running single-threaded, but there will still be multiplexing or parallelism on actual the processes that it's running, if that makes sense. So if you're trying to debug interactions between distinct processes and some of that results on maybe like a race condition or something like that, that involves some sort of parallelism, this could still be useful for you. And sorry, just to repeat the question, he's asking, is it useful for recording and replaying multiple processes if the bug happens to be kind of like a parallelism sinking issue maybe with locks or something like that. Anybody else? All right, thank you. So your question was how can you move in reverse during the debug session? So there's a few different ways. So with Delve specifically right now, we have a rewind command which will, if you're at a breakpoint, it'll rewind you into like the last, whatever breakpoint happens in reverse execution, right? So if you have a breakpoint on line 10 and you have another breakpoint on line 5 or whatever in the same function and you reverse, it'll go back to line 5 when it hits that breakpoint. Another feature that we have currently that I'm currently reviewing right now and we have a pull request up for right now is doing even finer grain stuff like reverse next or reverse step where you can reverse step in and out of functions and things like that or like reverse step instruction, right? So a little bit more fine-grained execution in reverse but it's all definitely possible and yeah, that finer grain stuff will be available in Delve pretty soon but right now you can basically rewind to the last breakpoint so yeah. Cool and if anybody has any more questions I'll be around for a little while so we can take it offline or whatever and thank you.