 Hello, my name is John Baldwin, and I'm here to talk about Writing custom commands in previous these ddb kernel debugger So I'm going to start off by talking a little bit about what ddb is kind of like that I guess I didn't give an introduction to it Then I'm going to talk a little bit about what the execution context is like in ddb because if you're writing a custom command You're writing a function that's going to run inside of ddb and there's some kind of special rules Around what it's like to execute inside of ddb compared to the rest of the kernel Then I have I have a couple of Demos we're going to go through kind of building up and the first one is just kind of very simple commands using ddb's built-in Kind of command line syntax that it parses for you Then we'll talk a little bit about how you can write commands that can parse arbitrary syntax You can work directly with ddb's kind of parser or lexer to kind of accept more syntaxes than what it defines And then lastly I'll give an example where you can define kind of namespaces of new commands What ddb refers to is command tables and for example One of the drivers I work on we define a new table that we put all our custom commands over just to kind of have them neatly organized So first of all, what is ddb? ddb is an interactive kernel debugger that's included in free bsd and some of the other bsds It runs on the system console So either a serial or a video console that's something you kind of just directly interact with the way you would interact with gdb inside a terminal When ddb is active it interrupts system execution There is for example with the kernel variant of gdb There's a way that you can analyze the system, but it stays running But ddb is always going to pause system execution and freeze all the CPUs from doing anything while you're using it Kind of like when you attach gdb to a userland process and you're able to examine what's going on So in that sense it is somewhat invasive because you have to pause the system while you're messing with it Best of my knowledge ddb was developed in mock originally because that's what all the licenses say And it looks like it first appeared kind of in the bsds as part of the 386 bsd port but the Detailed history of 386 bsd is somewhat Unclear so it's as far as I can tell it showed up at some point during that time line And that's how it eventually made its way into free BSD As a debugger it does provide some run control So you're able to do things like single-step you can set breakpoints you can even set hardware watch points Which could be kind of nifty and Then it does provide the ability to examine things, but it's ability to examine Data structures in particular and memory is somewhat simplistic more or less you can do hex dumps But you can't for example pretty print structures It doesn't understand type information in the way that gdb does And so that makes it somewhat limited that you you can look at a hex dump or a dump of memory And then try to intuit the structure layout yourself and kind of walk pointer chases chasing by hand and so forth and that's That's doable, but that can be rather painful and tedious as well And it does give you a pretty simple way to Look directly at a system after a panic and that's probably the most time that ddb gets used honestly is right after a panic you want to kind of Gets a quick snapshot in some cases the stack trace alone if you're doing development Kind of tells you where you're going wrong and it's quicker to go ahead and get that and look at that Then it does to get a crash dump and then wait for it to reboot and fire up gdb on the crash jump and kind of look at all the details And that's kind of the most common use case for it But then the last thing about it that we're permanently here to talk about today is it does it is extensible You can define new commands And in fact you can even write commands and inside a kernel module that you can add dynamically at runtime and even unload Sure move them so you could even in theory iterate on commands Like on a running system by like writing a set of commands go to the kernel module loading it That didn't quite do what I expected unloading it Modify it some more and kind of iteratively develop your custom commands that way inside a kernel module So now let's talk a little bit about What it's like to run inside of ddb and so what you need to be aware of when you're writing the code that runs inside a custom function So it is part of the kernel so it kind of has all the Rules you might have in the kernel such as you don't have a lot of stack So you know don't be making assumptions that you can have large local variables But it has some additional constraints besides just that for example, you really cannot block or sleep the scheduler is not running in ddb You're you kind of you're borrowing the current thread that happened to enter the debugger But you're not really able to do any real scheduling work The system is really paused and so you need to do very simple things that don't involve trying to do complex scheduler activities if a fault occurs inside of your command while it's running ddb will kind of recover and go back to its main parsing loop And the way that this is implemented is that ddb calls set jump before it calls your command function And if anything goes wrong, and we fault we long jump back out So one of the implications of this is there is no notion of cleanups Or you can imagine like how pthread has pthread cleanup push or pop or in c++ You have objects that will have their destructors run if you throw an exception that crosses stack boundary I ain't none of that in ddb. So anything that you might have done that had side effects like oh well Like you there's no way to kind of gracefully recover from that Another thing is that ddb uses a kind of slightly lower level version of the console than is the normal used in the kernel in particular the kernel console will also log stuff to syslog So it ends up in viral log messages or wherever else you route kind of your kernel console and ddb doesn't do that generally speaking Because that involves a little more synchronization work So as a result of these rules There's kind of some guidelines or best practices to follow when you're working on a command So the first one is you really should try hard to avoid side effects and that kind of goes back to the long jump thing There's no good way to recover if things go sideways while you're trying to walk a trail of pointers So you kind of need to be careful and try to avoid doing things that are really going to change state Unless that's what your intention of your command is And you also should really not use locks because again you could if you Lock something and then you fault you're going to leak the lock because you're not going to have a way to recover and unlock it So you like that's also Disturbing state you should avoid that if you really have to do this then you should use try locks because you can't block one example, I guess of a command that breaks many of these rules you need to try locks is a Many years ago. I added a command to kill a process And so use it your own risk, but it will try lock the process and kill it Which in very very limited cases means you can recover by breaking into the kernel and kill nine in some process It somehow has hosed your entire system And in general aside from things like that You should avoid complicated API if they because they get you tangled up on all sorts of these things if you're calling Complex API's likely they're going to require locks. They're going to be making side effects to the system state that can't be unwound Correctly or safely if things go sideways And usually the most common use case for custom commands in ddb are effectively to be pretty printers Most of them are working around the fact that you can't print a struct nicely like you can in gdb when you have debug information so they basically are kind of Hand-built custom routines to print pretty print a structure or details about a structure So you have routines to print things about stretch proc for processes or things about Objects live inside the virtual memory system like VM object or the list of v nodes that are locked or things like that one of the other use cases though and one that is kind of Particularly useful for ddb compared to things like gdb, and in particular you cannot do on a crash dump You can only do live is you can also Pretty print information about hardware state So and I've used this on a occasion so For example at previously I have added when I was working on x86 interrupt code I wrote a bunch of little commands that would pretty print state about the pics Both the local a pic and kind of how we had allocated our Cuckoo cookie cookies and which IDT vectors inside of each local a pic We had routed them to so that I could dump that table and kind of look at it to find mismatches Or also information about the pics themselves So for some reason you have a bug in interrupt routing, which we don't usually have nowadays anymore Many years ago. We used to have you could maybe see an interrupt was asserted in the pic When you are missing an interrupt for a device and you could find which interrupt was being asserted versus the one that the device Thinks it should be getting so you could kind of debug routing issues So one of the most commands I wrote recently that was committed to the tree was actually very similar to this I had some colleagues at Cambridge that were Working with a soft core that ran on it for risk five around an FPGA and the soft core has a on-arm hardcore That's kind of a management core for the soft core and there are some devices So for example the serial console for the risk five core was kind of managed by the arm core And there was interrupts propagated from the soft core to the hard core and they had a problem where the interrupt for the serial console Was not working and we were trying to narrow down where this problem would be So I added a command to dump the gig for the arm that prints out a line per pin So you can see which pins are masked or not and if a pin was active that was mass That would kind of indicate that perhaps there was a problem with the routing or the fdt. They had published and it turns out It wasn't that it turns out and their logic They there was a think of bust that was three pins wide that carried the interrupt number and somewhere in their logic I got truncated to one pin and that's how they lost the interrupt and once they fixed that it was all fine But we could use that command to kind of narrow down where to look was it a problem with the fdt Was it a problem in the logic? So that's a kind of a use case of something that you can do with ddb that doesn't it's not quite as easy to do With with gdb and it's not quite and it's definitely something you can't do post mortem because the hardware state has gone post mortem So one of the other guidelines I guess is that there is a custom API that you should use instead of printf for outputting things from ddb Which is so the next thing I'm going to dive into is kind of what is that API? It's not very thick actually it turns out it's quite thin Mostly consists of the fact that there was a different printf that that you you must you should use db printf instead of printf and that uses that lower-level console I mentioned earlier, so it Is much more simple talks directly to the console driver, and that's it and avoids any kind of synchronization that way So it's a lot like printf takes the same arguments It does direct console access one little thing that that db printf does have is it has a built-in notion of a pager Which the other half of it so when you are printing stuff out to ddb It will kind of trap new lines and notice and keep kind of how many new lines It's done and based on that drives some very simple logic around a pager to avoid spamming the system console as much of content And one of the options you have Something has built-in logic to handle things like single spacing or a page at a time But it also has a way to quit an abort a command and when that happens it sets this global variable Db pager quit so if you were writing a command that loops to be a good citizen You should check this command and break out of any leap you have if this if this variable has been set And I have an example of that later on So I've kind of mentioned that the way these commands work is they are They work by writing a C function and when you invoke a command from ddb It's going to call the C function and that's where your logic all has to live Ddb supports kind of a general syntax out of the box that most commands use which is up here It's also in the man page which is you have the command name You can have an optional modifier that's prefixed by forward slash You have an address most of the time and then you can have an optional count after your address and the address is optional Although most of the time you get it and then what's your actual command function looks like is that you're going to call function with This signature so you take an adder arg That basically corresponds to your address I have adder account and this modify thing And so each one of these pretty much maps to what you see in the syntax So adder holds the address to operate on if it's there if there's an explicit address It holds that if there's not an explicit address DDB maintains kind of an implicit address of whatever the last command operated on similar to kind of gdb does and that's what it Will be Have adder tells you if an address was given so you can choose Whether or not you want to acquire Address as always and kind of treat it as a syntax error if one is not provided And then the modifier is just this string of optional modifiers without the leading slash and finally the count is there if it was specified If the count was not specified It's given as negative one which is how you can tell if it's present or not and then to kind of declare these functions We have some helper macros that wrap a little bit of this logic so it's slightly some syntactic sugar to make it slightly easier to define new commands and They do a couple of things they define a helper structure and It does say linker said it's actually more accurate to say that nowadays We define a couple of sys in its to dynamically register your command with ddb when your module is loaded or when the kernels boot it up And to unload it if you don't load the module And we also declare the actual function for you and expect you to kind of put the function body directly after the macro And you'll kind of see what this looks like an example that I have coming up So the first one is just db command if I have an example here And it takes two arguments The first one is going to be the name of your command that you will type in kind of when you're using the ddb interpreter And then the name of the c function that you're going to define so in this case we're defining a command named foo and The function that we're going to find is called db foo command for example You usually don't really care or need to know what the function is called But if you want to disassemble it in the debugger or find the symbol for some reason because you're debugging weird toolchain crazy Not seen us then you know what it is We also have a couple other Rappers for this that work with kind of some built-in command tables that ddb has we have a db show command that Defines commands that are to live under a show kind of prefix So in this case this example defines a show bar command. It's going to call us a function And we also have a show all That holds things that and the intention is that show commands generally speaking are going to show information about a single object So for example show proc takes a pit or an address and it's going to tell you information about a single process Whereas show all procs which has a shorter alias of ps is going to loop over all the processes in the system and print information about each one and then The function name pattern that I'm using here Which is to kind of have a db underscore prefix and underscore cmd suffix That's kind of common convention mostly common not always followed convention But it's not required There's nothing that kind of and there's nothing any implicit macros that we use for example that really assumes that's there So here's my first example And these all my examples are toys because I'm trying to demonstrate like the parts that I care about not complicated logic in the C part So this defines a new little command called double that If you give it a number, it's going to print out twice the number. Otherwise, it's going to tell you that you didn't give it anything And so we can show that Now I have it at the end, but I've written I've have a little get repository with all these little commands in them If you want to look at them later So first I'm going to start at the end Not that one and while that's booting we'll get ready to log into it now Build my little kernel module here that has my new commands in it And then I'll load it in this case. I'm not going to panic my system because that would be true to sleep painful So there's a syskittle we have you have a couple of syskittle helpers for the debugger that you can force the debugger either You can force a panic if you want to test that or force page faults or things mostly useful for Debugging the code that unwinds across the stack across a page fault But we also have a helper one that will just enter the debugger nicely for us and when we're done It will come back out Okay, so now I'm in the debugger and the command we defined was double So first I'm going to show what happens if you just give the command and then if you remember it winds it you and says You didn't give an address so that works correctly And then if I give it a nice little number it's going to print out twice the number So that's all that takes if you want to have a simple simple command. It takes some kind of number or a symbol address This You know you have adder and you can just do whatever you want with adder You can cast it as a pointer for example if you need to in your command. So that's a simple command It just has Parsing an address So the next thing and I'll get them out first The next thing I'll talk about are if you have a command that wants to do something slightly fancier Maybe you don't want just one address you want to support more than one or you want to do something It's not an address you want to handle some other kind of arbitrary stream So there's two flags that you can set kind of in the metadata that's associated with a command to say that You want to do you want to have some level of additional control over the command line parsing as a command function? The first one you can set is CS underscore more and this command this flag means that you take an address You like you kind of follow the normal syntax But you can take additional arguments after the the original syntax over kind of as an option and Then the second flag you can set is CS own which means that you're going as a command function You're going to completely own all the parsing yourself You don't want DDB to parse anything for you. You want to do it all by yourself And you can there are kind of new or variances and these are in head And it's a little more painful in older branches, but in head We have extended versions of all the db command macros. I showed you before That are variants with underscore flags that take the flags as an additional third argument And if you're going to do this in your function one of the kind of Contracts that you take with DDB is that after you've finished parsing the command line You're supposed to call a helper function called db skip to el that just that kind of flushes the lecture state to kind of discard all The tokens up to end of line so then when were there when we were Going to parse things as a couple of functions that you can use to interact with DDB sparser The first one is mostly useful if you're using the CS more mode when you want to parse additional words is db expression And that's going to kind of start chewing up and eating tokens from the current pending command line to evaluate an Expression and this will do full expression syntax. It will handle the different operators that DDB supports whether different Priorities that handle symbol res resolution and things like that If for say if it runs directly into end of line at the very beginning So there's no more stuff that happens that returns false and if it successfully Parses an expression it returns true and the expression that it parsed is returned in the value your argument points to however If there's an error in your like as it runs into in syntax error that prints a little message and then Triggers and an aborts the command effectively doing the long jump back out to the main loop So this is one of those little gotchas that if you're using db expression You have to know that like if you follow the general rule of not having side effects That's helpful, but it's kind of be aware that this thing will just magically long jump out from under you If the user gives bad input there's nothing you as a consumer can do about that Another routine that you can use if you're in particular if you're not doing CS more and you really want to do the more low level Parsing is db read token and db read token works directly with the lexer that db uses and it returns a T Foo constant the constants all follow this pattern of a lowercase t and an all uppercase and they're They're an enum. That's not an enum. They're defined as just pound of fines And there's several values of these most isn't related to operators that the expression handler uses that you don't have to care about But the two that you probably really care about our T ident Which parses a C identifier in particular? It's not just an arbitrary string. It has to start with an alphabet or a Letter and then only contain valid characters in C Identifiers and if if this is returned in the actual string was saved in a global variable db underscore token a score string that you can then access inside your function and The other one that's kind of meaningful is t number which means that we parsed some valid number That could be octal or decimal or hex and we've saved the value of that number in db toke number And then one final function that you probably won't use in most cases is db unread token in particular Sometimes if you're writing something very complex like an expression parser you may need to use this where if you're expecting some set of tokens But you have the ability to continue to be valid if you don't get an expected token Then you may want to use this to kind of push the token You just got back that you didn't expect back on to kind of db's lexer stack And then allow that your your parent code that you return to to deal with that new token in some way So then another thing you need to do when you're inside these functions is you need to be able to handle errors Now that you're parsing Your own syntax and you defined kind of your own grammar for what your commands do the simplest way to do this also a lot of the Commands the tree don't currently use this as a function called db error and db error just takes a simple C string That prints it out if it's not null if it's no it doesn't print out anything to the console And then it completely flushes the lexer state and triggers a long jump back out to the main loop So again, then if you have any side effects that you need to undo You need to do them before you call db error because you don't get a chance to continue after it The other alternative if for some reason you don't like the long jump as you can call db flush lex Specifically which completely flushes the lexer state and throws away everything it's bending So now let's look at some examples of using These two options So the first one I have is an example that's using cs more and I think it's a sum command So some is going to take and like an arbitrary list of numbers you have to give it at least one and it's going to add all of them up So we're just calling wild db expression in a loop and waiting for that loop to end Once we have finished kind of dealing with parsing all the words that we've got well I guess we should say first Notice that I have to figure out do I handle not having a word at all in this case? I'm going to require a word and then my first word is already implicit So I have to pull my first word from adder I don't like the first words already done for me and then I use a wild loop with db expression to handle additional words that I'm going to Parse and then once I'm done parsing I do the db skip to el to kind of get db db's lexer into a good state to handle the next line of input and Then finally I can actually do what my command is trying to do which is print out the sum of the numbers Then I have one more. I'll show and then we'll actually go run these Come to the second one. It's the fonts a little smaller as the code gets bigger This one is actually using CS own So we're doing our own parsing entirely in this case This is a command. It's kind of a toy command show soft C That takes a name of a new bus device kind of a device infribusties kernel and as a string And then we'll look up that the point that the kind of device T for that object by name and then print out the soft C pointer Um Pretty not the soft C is not necessarily amusing, but several of the commands I've written actually take a device name and kind of use This pattern So first I'm going to read a token and if I didn't get a C identifier because device names are more or less Follow the same conventions of C identifiers Then I complain because I don't know what you gave me and you gave me something It doesn't look like a device name Then I'm going to try to look it up to see if it's also if it's valid in the system And then I'm also going to go ahead and take this opportunity to flush the lexer at this point And then if it failed to find the device I'm going to complain again to say I didn't find whatever device name you gave me It's a valid like lexical name, but it's not a device in the system And then once that's happy I will print out the pointer I'll call device gets get soft seal on the device and print out the result So we can show The examples of these I guess I actually hadn't exited the bugger So if you remember, I think the first one is some So again, if I do some without anything it winds at me If I give it just one word, I didn't notice what that is, but if we add more it parses each one as an expression Something I didn't try beforehand. It's presumably you could try to use C's expression or the parser that expression parser that DDB supports. We'll see how fancy that gets. Ah, I did the right thing yes so It'll just parse and eat up arbitrary words and then allow you to sum them the other example I gave was show soft C I think so let's see My laptop so if I just do show soft C I get the one error, which is I didn't say anything if I give you something. It's like a number Then that's not valid But if I give you a valid device name like I should have ACP on this laptop that's it's made in this Millennium Then there's a pointer to the soft C structure for a CPI zero if I wanted to find that for some reason So the last topic well any before I move on I guess any questions I can always go back to the code later if y'all are curious All right, so the last main topic I have is you notice in that case My second command was a show command so it was under the show table And I kind of alluded to the fact that it can be convenient Maybe to define your own kind of tables your own namespaces of commands, especially like I said As a device driver you may want to kind of have your own set of commands that live under your kind of name for your device so You can do this DDB command tables for example like show it's actually a command in DDB It just has a special handler that knows that it takes a instead of kind of parsing an address and doing something It's associated with another table of commands that it kind of parses the next word index inside of So they use a special function DB show table. That's the handler for the function They also have a global variable that they defaulted I think why variable of type Structdb command table, which is really just a list head because the command tables are stored as a simple link double link list of objects They're not currently in frequency quite as well abstracted You kind of have to look at some of the internal macros and how they use and abuse some of the internals to get it to work so maybe if other people really care about this at some point we can make the Makros a little nicer and easier to use but today they just kind of like a little grotty But I have some examples to show what it looks like One of the things you have to do is just like something like syscuddles You have to kind of plug into the tree somewhere So if when you define a new table it has to be a child of some other table that already exists and The three tables that we already have in the system is the DB command table is the top level Table of top level commands, and then we have the tables for show and show all So here's an example Which I've defined for my little demo module I've defined a new kind of demo top level command That's going to be my table and these sub commands are going to live on inside of under under demo as it were So this is the grotty part. I have to define this DB command table object Which is really a list head and I need to initialize it And so in my case I'm going to use a static initialization versus trying to use a syscuddles to do it Then some more grotting this I have to use this internal Macro DB underscore set That actually kind of creates the sys in its to register in the data structure to register the command for demo And in this case The command is named demo it's going to use the demo command table as its function and demo table is The name of the global variable that holds the things Or holds the list of commuter commands then for each command that I defined I also I don't have quite like a I could perhaps probably if I were doing this for reels in something production I would define some local like DB demo command that wrapped this underscore DB funk Similar to how we have DB command and DB show command but for having fewer lines in the slide I didn't do that this time So in particular the first token is kind of similar to syscuddles is Underscore and then the name of my table the name of my command within the table The C function so for example the the kind of wrapper functions that we currently have kind of only take these two arguments Then the table I belong to This would be flagged such as CS more CS owned if I took any And then this should always be null what this argument actually is is when you're a table It's the pointer to the table so I have all that blob to basically to find a single command in one It's going to print out one and another command in two. It's going to print out two So now we can show that So if I just run demo just like if I were to run show It's going to tell me that you're this is a command that takes subcommands and prints out the list of subcommands that were required And so in this case we have one and two I run one of those it's going to run the little toy function that I had right and I lied I have one more example So the last example I have is something that's going to show you kind of interacting with the pager and I Basically took charge in from INETD I made a charge in inside of ddb because I needed an excuse to have a lot of content a lot of output But I didn't want that will not fit on a slide in a way that is readable so I've alighted most of the code So this is kind of but in particular. I want you to see the main loop I mean at some point in the main loop you print out a new line And then every time you print out a new line We're going to check to see where did we get a request to quit just how we can cope with quitting if the user asks us to So that's that's the only part of this slide that really matters is seeing that we added these two lines in the right place You go back to our demo Think I named it charge in For in charge in it's going to give me the nice output I believe if you had to enter in ddb it does one more line So for if you have a lot of output and you want one more at a time you can do enter You get space you get what ddb considered as to be a screen full which is The defaults to 24 lines. There's actually a special variable in ddb dollar lines It's how many lines it thinks it is that if you want to change it you can But the real thing I'm trying to show here is if I hit Q Then that returns it sets that db page or quit and my helper function notices and behaves and is a good citizen And falls back out without printing forever and an endless loop Okay So in conclusion Most of the commands that you would probably want to write if you were working with ddb and actively developing something Most of them are pretty printers and they take kind of a pointer to a structure as that address argument And they work with that in some cases you may want to take some other token That's not an address in which case you probably have to do something like CS own to deal with that There are lots of examples in a tree if you just grep for db underscore command or you grep for db printf You will find ones that do all sorts of various things and that's probably the best way to find other patterns and examples beyond The little toys that I showed today is to go find some examples in a tree If you want to see the toys you can find them. There's a really tiny github that has them available there And then questions Yes Yes, so I guess I should Yeah, so for the stream Christoph asked can you define? Modules and a kernel module and so for this talk I actually did so and so I can continue the example to show that son of you remember Over here. I had actually Compiled my customers. It's actually a little module that I have so if I actually continue Out of here I'm back to learning back to running on the system. I can even unload the module It doesn't go boom Go back into the debugger Now my commands are gone so that was what I said earlier you could actually iterate so if we wanted to Modify on one of these commands in some way that I had we could do it on the fly load it Like you could replace an existing command you had named It doesn't care because they just go away on unload and you can we add them on the unload So definitely you can do that Tom is there Tom's question is is there a reason I shouldn't use this on Simple kernel drivers, so I will I'll give you a larger answer to your question I use ddb Rarely my preferred debugging experience is to use gdb Just because the flexibility that I don't have to write a pretty printer if I need to examine something I don't have so when given the opportunity I I use this when I have to I guess My probably ideal environment for working on a device driver is using pass through into a VM To pass through the hardware and then using kind of a some kind of debug stub inside the hypervisor if it's functional enough Or using like gdb over serial over a momentum connection into the VM if I have to to debug that way because that gives me a richer debugging experience for device driver Typically than it would if I were having to kind of make do with this But there are some cases where you can't do that so the example I gave earlier of My colleagues who are building this little risk five core and they need to run a very tiny little memory file system Currently image thing on it and they don't really have a good way to try to connect to nested remote gdb inside the kernel then Like or in particular wasn't even for the risk five core in this case. It was for the arm hardcore In that case having using a kernel that's resident inside the debugger itself was just a simpler approach to deal with So I I guess to answer your question you could It's one of the tools that's available to you And I guess one of my points here is like if you this is an option you can use and if you want to use it This is how you can use it But it doesn't necessarily mean that it's the best tool for any given instance of a job Yes so I Mean it depends on how dangerous on the edge you want to live I mean so you can do some things I mentioned so I guess I should repeat the question was is it safe to do things like Network IO or is it just kind of console access? I mentioned earlier that a you probably shouldn't do that But it's true that there are things you could maybe get by with if you if all the stars are aligned And you've made them all line versus things that will just not work at all If you wanted to use our network stack out of the box It does too many things involving context, which is a so forth that that wouldn't work But that does not mean you couldn't do something else that was built to kind of run in this execution environment So for example one of the things that was contributed back from I salon to a previous team the last couple of years as we have the ability to kind of have a Debugged network stack in the way that we have a low-level console that you can use in the debugger And this is used to allow you to actually do gdb over a network connection kind of that's kind of pre-configured And you've arranged Kind of what your ip4 address that you're going to use and the writing details have been kind of set in the kernel ahead of time Or you can actually even set them from ddb itself if you get a panic And also to support sending dumps across the network instead of writing them to the local disk, but this does require Explicit support in the driver that you're going to use to kind of have a separate path of routines that kind of can deal with Not by the only pulling and not trying to expect interrupts or do with context Which is the so forth it's not not dissimilar actually from how we have a custom path ish in storage controllers to deal with crash dumps So they know to do pulling and not kind of interrupts and so forth So you could do some complex things if you want to But in many cases you may have to do a lot of work to get there because you may have to duplicate things to provide alternate services That depend on polling for example Any other questions? Oh Go ahead, Bjorn Yeah, so I Well while writing this talk because I'm actually the guilty person who added DB pager quit like maybe 20 years ago support your own was questioned for the stream was that Hidden his experience. There are some commands that don't actually honor the pager correctly So it occurred to me during my talk that now that I have a better view of what would be perhaps more consistent design It may be worth revisiting the pager quit to just be another long jump It's a really forceful way of not requiring cooperation And maybe I'll actually go back and do that and just remove the global variable and just say you get long jump Please don't do side effects Because there's so many other places have already do that now it turns out anyway Kirk so most of my strips, which are somewhat hackish I really revolve around GDP and not GDP to be honest and I do have This might be awkward Let me figure out how to get a safari tab open on the right display. All right. Well, you do the right thing No, you'll not that's okay So I do have a set of scripts that I have developed over several years That work with GDP and I have a GitHub repository that I keep these things in along with some other stuff So can you see it? So under my go GitHub place? There's a K Debug that I have repository and it has a couple of different things inside of it a Few detray scripts that may or may not be interesting, but then most of the kind of fun bits I guess are some Gdb stuff written in old Gdb's internal scripting language and not in Python sadly Maybe someday I will retrofit some of them into Python because that would be a good use of time and a better thing set of things to use but they didn't work with Gdb 6 1 on Fribiosee 4 6 which is when I started using these things in fact you can see The two main scripts are actually called Gdb 4 and Gdb 6 because I this was when I was at Yahoo and when I started these and They pull in they define commands specific to OS versions and Gdb 6 actually has a bunch of version if deaths in it to handle things From 6 through 14 and then it we I have some architecture specific ones Like I have helper scripts that will walk page tables and kind of print out the PTEs if you're trying to debug stuff and all sorts of arcane weird crap, but they're They're documented by reading That's about the best way that I could put it so it's It's accessible for someone who wants to use it, but I don't really have it easily consumable by People who aren't already kind of hacking the kernel a lot Say what? Yes, it definitely is that and I believe there are some people who who have used them Maybe it's the people in this room who have used at least some of them in some form or fashion, so I Think Mark has some nice scripts that he might have done in Python that might honestly fully duplicate all the work I think they do some of the work nicer. He has a nice thing to deal with V net because V image makes Debugging less exciting or no less pleasant I guess is the word I would use and he has some stuff that will actually like take a V net pointer in the name of a symbol That's a V net symbol and kind of resolve it for you. Nice and cleanly and with some Python helpers But that's not ddb. That's all GDP and I must say that might I spend most of my time Using GDP for debugging stuff ddb, I use in certain constrained environments where it that's the best tool to use Any other questions? Okay, well, thank you very much