 Alright, hello everybody. I'm going to go ahead and get started. Thanks everybody for showing up and being here relatively early, especially given Delirium last night. So, yeah, my name is Derek Parker. I am an engineer at Red Hat. Previously an engineer at CoreOS, and now it's the same thing. So, I'm going to be talking about advanced go debugging with Delve. And one disclaimer, despite the advanced word in the talk, this talk will be approachable for everybody. I plan to go through some more advanced things of usage of Delve and some advanced debugging and some of the advanced features that we've added, and also some deep dies into go and things like that, but it'll all be very approachable and there will be a lot of material for everybody from you know, newish go developers and people who are new to this kind of debugging to people who have been doing it for a very long time. So, just to give a little preview outline of how this talk will go, there's essentially three main things that I want to cover, and then I'll try to leave some time at the end for going through some actual examples on the command line, which will be interesting to do one handed, but maybe I can just add a microphone into my beard or something. So, the first thing that I'm going to cover is what makes go different. So, what about go necessitates new tooling, new debug techniques, things like that, and a lot of, I'll take a deep dive into some of the interesting features and a lot of the things that make go different fundamentally from other languages, but I'll also go into how that ties into debugging and how that can tie into just in general knowledge of what your program is doing and how you can figure out when things are going wrong. After that, I'll talk about what makes delve different, how it handles some of these differences in go, and the things that it does to make your go debugging experience a lot better and a lot more successful and a lot less painful, most importantly. Then I'll talk about the state of debugging and go, kind of piggybacking off of the state of go talk that we just watched. There's not going to be any emojis in my slides, so I apologize for that in advance. But there's a lot of interesting things for debug support that are coming up in the 110 release and most importantly in future releases, 111, there's slated to be a lot of really interesting important changes, so I'll cover that. As I mentioned at the end, I'll try to save some time to go through some examples on the command line and run through a few things to give everybody an idea of how to use it and what kind of things you can do. Okay, so one of the things that I tend to like to do just to kick off these sorts of talks is to get a feel for the room. So how many people before today have heard of Delve? Awesome, quite a few. Some who have not, so that's great. This will hopefully be a new tool to use in your daily development toolbox. The people who have heard of it or whatever, how many people have actually used it? Okay, cool, quite a few people. Also very exciting. For the people who haven't, also this will be a good introduction and talk into some, even some advanced techniques in that, so again, hopefully another tool for your development toolbox. So moving forward, I want to spend some time talking about what makes Go different and why new tooling had to be created in the first place and just understand fundamentally what are some of the interesting things about Go and what are specifically from a debugging standpoint. What makes it difficult? Why is the language so different? What interesting problems can you run into with Go that you might not run into with other languages when it comes to debugging? The reason why I think this is important is because I think it's important to understand what's happening in the programming language and the process and things like that that you're running, especially when it comes to debugging. The more you understand fundamentally, the more you'll be able to go forward and fix having that knowledge. It just makes it a lot easier to dive into problems and solve some of these tough issues. So here's a list of some of the things that are, that make Go different, make Go stand out and affect the debuggability of it. So first and foremost one of the most interesting things is the different execution model and that ties into the next point a little bit with the runtime and scheduler and I'll dig into all of that stuff in pretty good detail here shortly. But fundamentally Go, the execution model is different than any other programming language and this is one of the things that's exciting about the language and brought many people to it. But for traditional tooling and traditional debugging techniques, it can be confusing and it can be difficult and break a lot of existing tooling. So what do I mean by different execution model? So in a traditional process that's running on your system from the perspective of like a debugger or something like that, the smallest unit of execution is the thread, the operating system thread. As long as you're staying on the same thread, you're generally following the same execution path, you're staying within the same context and frame and everything and life is good, you're figuring out your problem and you're on your way to solving it. With Go, it's a little bit different. So fundamentally the smallest unit of execution is the Go routine and this is an important distinction for many reasons which I'll dig into later in this talk but when it comes to debugging what you want to do is you want to step through obviously and step through your code and see what's happening. So when you're stepping through your code it makes it very difficult to figure out what's happening if the code that's actually executing moves out from underneath you and this can happen with Go because of the execution model. So Go routines are multiplexed onto different operating system threads. If the tools that you're using are not aware of Go routines, which Go routines are running, which Go routines are running actively on threads and things like that, it's very possible for things like context switching to happen where the code that's actually executing is moved out from underneath you and you end up at the same spot in the source code but you're on a different Go routine, different stack, different memory, looking at different variables things like that, which if you're not aware of that can make debugging really difficult and confusing and I'll dig into all of that a lot more later. So the second part is the run time and the scheduler there's some interesting things with the run time when it comes to everything from function calls to a lot more and the scheduler kind of ties into the different execution model which I'll dig into in the next few slides. Another difference is that can be a little bit tricky is Go's memory layout. So for example over the years the stack setup has changed within Go so it started out contiguous, moved to segmented stacks and then went back to contiguous stacks. There's also the Go routines stack, there's the system stack for Sego and other things. So stitching a lot of those things together can be difficult and can make debugging difficult potentially for traditional tools. The garbage collector as well not a huge issue when it comes to normal debugging and for some advanced debugging features that's something that is definitely of consideration. The type system, Delve knows Go really really well so it's able to interpret some of the different types, interfaces and things like that and display them a lot better. So moving on I want to spend some time digging into the scheduler and what it does and then we'll look at how it impacts debugging. Just to introduce a little bit, for those who aren't familiar with the actual scheduler, the scheduler is the piece of the Go runtime that manages operating system threads, moving Go routines around, things like that. So the components that make up the scheduler internally are known as the P, the M and the G and they're explained a little bit right there. So P is processor M is thread which is actually a machine in the internal parlance and G is Go routine. So like I said P represents a CPU processor core. When you're manipulating Go Max prox internally in the runtime this is what you're manipulating. Each P processor value maintains a queue of runnable Go routines and that will be important a little bit later. So the M is the OS thread. The thread needs a processor context to actually execute and we'll start executing the Go routines that are attached to the processor queue and just pick them off and start executing. And then finally we have Go routine which is a concept that I'm sure everybody is familiar with. So why this is important for debugging and specifically with Delve is it's kind of important to be aware of like context switching and through that how that happens via work stealing within the scheduler. So one of the fundamental problems of debugging Go programs is the fact that as I mentioned earlier as you're stepping through your code that's running on a Go routine it's possible for context switching to happen. Now what that is is the Go routine will be paused put back onto a Go routine queue somewhere whether it's on the queue of that processor or somewhere else the global queue and it's potential there's potential for that Go routine to be stolen by another processor and thread inserted and executed there. So if the tooling doesn't understand that fundamentally you're going to thrash around and you're going to end up in different places in your actual program looking at different memory and variables and that can be really confusing. So context switching as I mentioned is when a Go routine goes from running to idle and potentially switches to another thread another execution context and this can happen for many different reasons. So blocking syscall, blocking IO it can happen during a function call. There's a stack growth prologue where if the Go routine stack needs to be grown the actual code for that will run on the system stack and another Go routine can be run in its place on that thread. Runtime go-sked is an internal function that you can use to cause this intentionally and then creating a new Go routine can cause it and I'm sure there's a few other things but those are some of the most important I feel. So the next thing I kind of want to talk about with relation to work stealing or context switching is the work stealing algorithm within the runtime. So I stole this diagram actually from Yana a little bit from her talk on the scheduler at .Go I believe but I use different colors and I made it myself so I feel pretty happy about that. So this explains a little bit about what the scheduler looks like and kind of what the layout is. So you have processors, there's threads running on those processors that use like the active thread that has the context of that processor. There's the active Go routine that's running there's local queues for every processor context and then there's a global queue. So when it comes to that when you have a processor that doesn't have any Go routines in its queue it's possible that it can steal from the queue of another processor context and that's when you get context switching and that's when you get Go routines running on different threads and things like that. So to tie this back in the most important thing is that Delve knows Go so it knows about all of these details it knows about all of these internals and can make your debugging experience a lot saner. It knows about the scheduler, it knows about Go types, it knows about Go calling conventions and the run time and a lot of other things that just make your life a lot easier when it comes to debugging your Go programs. Okay, so now we'll switch a little bit to something different and get into some of the next areas of the talk. So let's talk a little bit about debugging Go programs. That's why everybody's here. So I want to start off a little bit with talking about just internal Go debug support and what that looks like. So initially debug support within Go wasn't all that great even Delve had a hard time doing certain things even though it was a Go specific debugger. So that is getting a lot better. So there's a lot of things from within Go from the compiler linker and run time that are going to improve your overall debug experience within the next few releases. So 110 has a lot of really big things and 111 is slated to have a lot as well. So one of the big things is improved dwarf information and overall debug support. So we've been able to talk with a lot of the Go core folks to improve dwarf information generation which for those who aren't familiar dwarf is the standard format used to describe debug information within binaries. So there's a lot of things that were wrong or incorrect or not implemented initially in some of the earlier versions of Go but as new versions have come out and we've complained louder and louder these have been fixed and these are some things that we've also been able to dig in and fix ourselves. So for example some of the things that have improved is better variable scope information even relatively small things like the ability to detect function arguments from function return values. It's a pretty useful feature and has only recently been implemented properly from the compiler and linker. Better support for cons, improved dwarf information for optimized binaries which is really important for debugging binaries that Delve didn't create or debugging binaries that did not disable optimizations. Possibility of calling Go functions from within Delve. There's still a lot of discussion around this but one thing is there is discussion and so we're hoping to be able to implement that pretty soon. And there's a lot more so if you're interested about this you can check out the debugging tag on the Go repo. Cool so now let's move into actually talking about Delve and how you can use it to debug your programs. So one of the fundamental things that I did in mind when I started this project and as the project has gone on is simplicity. So the Go tool chain is simple the tools that you use around the Go ecosystem should also be simple as well. And I think this is especially important for debugging because if you're reaching for a debugger things already aren't going your way and so the tooling shouldn't frustrate you anymore. It should enable you to solve your problem. So let's go into some of the usage of Delve. So one of the main things that you'll be using is DLV debug. So what this does is it will compile your program. So if you run it from within the same directory that you have your main package it will compile your program with optimizations disabled run it attached to it and start a debug session. There's some interesting things that you can do with this. So you can also pass a package path that works kind of similar to Go build but one of the interesting things that I want to mention here that can be useful for some advanced debugging techniques is one of the things we've implemented fairly recently is the ability to use different back ends with Delve. So for example you can use our native back end that we've written. You can use GDB as a back end. On OSX we use LDB as a back end automatically because of some changes in the mock Darwin kernel that broke things but most importantly and most interestingly is the ability to use the Mozilla RR debugger back end. So folks who maybe aren't familiar with that Mozilla RR debugger is a record replay debugger. So what it does is it executes your program, records all of the execution, the memory layout, everything and allows you to replay that. So if you're debugging like a Heisen bug where it's very difficult to reproduce or anything like that, if you can trigger that during our recorded execution you can play that back deterministically. So you can always trigger that bug, it'll always be triggered the same way and it can make debugging a lot easier and a lot saner for some tough bugs. It also allows you to do interesting things like reverse execution. So you can play your you can run the process in reverse to a break point and things like that which can be really interesting and really powerful features. The other command is build a test binary with optimizations, disabled, runs and attaches to it. Works pretty much the same as like go test. You can also pass flags to the underlying go test if you want just by doing like dlv test dash dash and then whatever flags you want to pass. Attach so you can attach to a running process with better optimized debug support from go coming up. This is going to be a lot better experience but for the time being if you're running an optimized binary and you attach to it, it could be a little bit wonky but for the most part it should be alright. Just something to note. Trace is another thing that I think is pretty cool. If you're familiar with Strace or something like that you can use dlv to trace your programs and with the improved support for being able to tell arguments from function return values this feature is going to become a lot more interesting and a lot more useful within the next release of go and some of the next release of dlv. Another interesting thing is dlv core. With dlv you can debug core dumps. To enable core dumps there's a few things. Just make sure you have the proper U limit set. Run your binary with go trace back equals crash and then whenever it panics or crashes or anything like that it'll produce a core dump in that directory and you can use dlv to debug that. So you get all the power of dlv and what it knows about go to do postmortem debugging which I think is really interesting and can be really powerful. Also replay. If you have an existing recording that you've made with rr you can use dlv to debug it and run it. This is interesting if you want to keep the same recording around and not just use the back end flag to do the recording and replay for you every single time. With that I'm pretty much done. I don't know that I have actual time for any terminal examples or anything like that but I'll be around alessandro co-maintenor of delve will be around as well so we can run through examples or talk about it in a little bit more detail. That's it. Thank you everybody. We have some time for Q&A so if you have questions for Derek please just raise your hand and shout really loud. Yes. So the question was are there any fundamental difference between platforms so switching from different operating systems and things like that. And no, so the debugging experience should be consistent regardless of what operating system you're using. Yes. So the question was how far are we as far as function call support within delve. And right now there's been a lot of discussion between the core team on delve and some of the folks in the core team on go at Google. And so we're in discussions on how we can actually achieve this. There's a lot of really complicated things that go into actually implementing this feature because of some of the things that I mentioned. But we're actively working on it. We're actively discussing it. We're bringing in folks from the go like core team and the runtime team and things like that to come up with the best solution so that we can provide this feature going forward. So hopefully relatively soon within the next few releases of delve we'll have that support. Do you remember when I said that if you're in between the camera and my face you're on the video? That is still valid. So if you're in between the camera and our faces you're on the video, live stream to the world. Just so you know. Thank you. If there's any more questions I think we still have some time. Any more questions? Cool. Thank you.