 Hi, I'm Tom Trumme. I'm going to talk today about LODM and for us in debugging. I work in Mozilla. I like to start when I talk by saying thank you Mozilla for backing Rust and also sending me to this company. I also wanted to tell you a little bit about myself. I used to work at Red Hat on GDB here. I worked on GDB for a number of years. I moved to Mozilla and worked on a lot of things. I have a long term of interest in debugging. I also really like Rust. I think Rust is what the program of the world needs. A nice thing about Mozilla is that it gives me the opportunity to combine these two things that I really like about Mozilla. A little overview. What we're going to learn about today is going to learn a little bit about Rust. I mean a little bit, it's like one slide. It'll probably take me about 45 seconds at this rate of speech. I'm going to tell you about today's state of debugging in Rust. I'm going to tell you about my plan for how to fix debugging in Rust. My theory of debugging is that a great language deserves a great debug. So that's what I call Fuji. I'll tell you about the work in progress, things I'm working on right now. And maybe how you can help. And then I'll end with some commentary. And in yesterday's talk I said commentary, what it really means is that I'm going to complain about other people's work. I'm going to tone down with complainable today. I put a little weird yesterday. Okay, so who knows Rust? Okay, so Rust, the system language for Mozilla. I feel like it fits in kind of the same general niche as C and C++. It has an emphasis on safety. Now, if you write Rust, when I think about programming languages, there's kind of two views. I mean there's a lot of ways to do it. But there's two that are interesting for the point of this talk. One is like the compiler. When you write Rust, you're dealing with the compiler's view of the language. There's all these things in Rust that are different from C and C++. There's match statements. And all kinds of statements can yield a value. And there's lifetimes and there's the bounce checker. Almost all that's invisible to the debugger. The debugger doesn't care. From the debugger's point of view, what's interesting about a language is its type system. And then sometimes there are some auxiliary things that are interesting. For instance, in Rust it would be macros. Macros are something that maybe the debugger has to know. But all that other stuff is just not visible at all. And then there's a few things. I touched on this a little with my statement about macros. There's a few things in debugging Rust that I still don't really know what we should do. One is exactly how to treat macros. And another one is the kind of general problem in some of these... I don't know if I want to use word modern. But in Rust, values have moved semantics by default. You see a lot of this in C++ as well now, right? Do people use rvalue references? And how do values go? That presents a challenge to debuggers. The question is, when you're debugging, moving the value out, you're sort of destroying some state that the compiler has put into your program that's important. Another way to look at it is if you're debugging C++ and you create temporary objects, when do you call it a struct language? Should it call it? Otherwise, knock those things. Print this variable and then you trash a program state So, I don't really know how to solve that problem. Maybe you have some ideas. So, the standard things today. A while ago, a year or two, I wrote a GB support. I did the bulk of it. A guy named Visha works at Mozilla. He did some of it. Actually, he did the hard part. So, GB support works pretty well. Rust, so that's part of the standard things today. The other thing that is peculiar about today is Rust has its own fork of LLVM. Now, it's sort of complicated. The Rust compiler can be built against multiple versions of LLVM. There's one sort of blessed one, which is the one that the official builds are built against. When I talk a little bit about what it takes to fix a debugging bug in the Rust compiler, this will come back to you by us. You'll see. Even with GB, there's a GB support for Rust debugging. And GB understands a lot of stuff about Rust types and that kind of thing. It has a Rust expression first. There are still things that are missing. So, for instance, if you're debugging a Rust program, there are some kinds of unsized types that aren't represented well. So, there's no way to expect them. Unless you know how the Rust compiler happens to live, you can do that. I've done that. It's terrible. There's also, like, traits or not in debugging a pro at all. You can't do anything that requires you to know anything about a trait. So, for instance, that move problem, you know, the trait is copy. I mean, if the type has the copy trait, you can just do that. However, there's no way to know what the type is. And there are many other things like that, like if you want to add operator overloading to GB, what you can't actually do, which types implement those traits. Rust debugging pro currently is very and sort of consciously C++ like. Now, that doesn't mean a whole lot, necessarily, because, you know, Dwarf is just a Dwarf, and there's sort of, if you read Dwarf, usually there's just kind of one obvious way to represent it. But what happens is, the reason it's a little more C++ like than you might think is so that LLDB will have some mode of understanding what's going on. And this leads to, like, a logic. I want to, you know, improve the debugging for information for things. I want to add, like, debugging for traits, or I want to do this or that, but you can't do it because LLDB won't understand it. So, and because LLDB thinks it's all C++. So we're stuck. So. So part one of the plan is to unblock this log jam by writing an LLDB log. And that's in the works. I'll talk about some details about that in a few minutes. Then once LLDB is working, the idea is to improve all the tools in lockset. So add a debugging feature to the Rust compiler, which might involve adding a debugging feature to LLDM, and then add support for that to GB, and support for LLDB. And then just sort of repeat. While we're at it, there's some bugs. Like, there are a few bugs that I can just fix without, you know, needing to have LLDB working. So I'm working on one of those as well. And then as we go through this list of all the things in Rust that don't quite work yet with the bugger, we'll find that we need new things from the work. And so my plan is to submit a bunch of, there's already at least two, I think, in LLDM and in the Rust compiler that I haven't even filed with before a lot of that. And then eventually what I want to get to is a new rule for Rust, which is you can't add a feature without adding the bugger. I think the way that I would want that to work is that the compiler folks will do the compiler side and someone, probably me, will do the bugger side. But the idea would be to explain what the feature you have to plan for the bug information at that moment and not. We've already seen things where, like for instance, last year there was a big patch series to add better compile time optimization of Rust email layout. You know the compressed emails and cases where you don't have to read out some data and that broke a lot of the debugging foam, debugging of that, those types that was already there. Which was already a hack, but now it's a hack that can't even possibly work. So remember back here I talked about, like it's like I talked about maybe the difficulties of Rust having its own LLDM. So I'm going to take you through the workflow that I do. For instance, I did this email problem I just mentioned. I've been working on a patch series to fix that. So the way this patch series looks is, well, to add email support, it turns out you have to patch LLDM to add this thing called a work type called DW type variant, which is like for variant types. It's the thing that was introduced for Pascal. But we can reuse it for Rust. I don't have a problem with that. I think it's good to reuse things even if maybe they're not a perfect fit. You know, like my view is, you have Rust, you have the expression of Rust in the debugging foam, and then you have what the debuggers do. I don't think that the tags all have to say Rust or be an exact fit, as long as that process is documented. Like somewhere we say, well, this compiler should do this. So to add variants, LLDM doesn't ever generate variant parts in the debugging foam. So first we have to patch LLDM. Well, which LLDM do we patch? Another part of my philosophy is you want to test everything end to end and write all the patches before you try to submit them. And the reason is you might miss some stuff. I don't trust my ability to sit down and just think it through in a pure plot sort of way. So we start by patching Rust to LLDM. Okay, great. Takes a little while. Then you patch the Rust compiler to use this new stuff you added to LLDM. So now you can compile a Rust program and read the Dwarf and it looks right. So then patch GV, because GV and LLDV also never dealt with variant part. They ignored by GVTter and also LLDVs. So patch GV. And the reason I patch GV is, you know, the LLDV plug-in is not done and I don't want that to be C++ in this C++ feature. It doesn't make any sense. So I patch GV. Now it all works, you know. And what, you know, this particular patch is kind of depressing because what it all works means is it works exactly the same as it did a year ago before the compiler came to sign in. So it does nothing currently. So that's all great. It's time to submit our patches. So what do you have to do? Well, first you have to port your Rust patches to LLDM Trump. Rust LLDM doesn't really take the forward place. They don't encourage, say, development of LLDM off on some fork. It's more like a stability thing with a few Rust tweets. So they'll accept a patch, but they wanted to go into LLDM Trump first to preserve their ability to upgrade the LLDM version and that makes perfect sense. Of course, LLDM changes all the time, you know. It's a huge project. As far as I can tell, it's like just anything goes, we'll just change it all. You have to forward port your patches from Rust LLDM to LLDM Trump. And then you have to go through patch review. So this patch went through four rounds of patch review, which isn't too bad, honestly. Once in GDV, I saw 17 rounds. That was about how many rounds too many it's got. Anyway, patch review pointed out that way back at the beginning, this read the dwarf standard and did everything wrong. This is a story I told yesterday about this area. It turns out that everyone else, except for this one LLDM reviewer, everyone else who ever read that part of the dwarf standard also interpreted it the same incorrect way that I did. I don't know, something's going on there. Anyway, I did it all right. I fixed up at LLDM, then it lands. But now, I get to go back to the beginning and start on it. I have to back port here I forward ported, now I fixed a bunch of problems, now I back ported and then I fixed up the rest compiler, and then because I did the dwarf problem I had to fix up GNP. And I think if I was being truly meticulous, like in this case, I think it's, I have a pretty good handle on it. I feel like it works. The chain was not too big. But if it was the Harrier problem right here, before landing, I would go back and do it. This one I cheated a little. That's what the workflow looks like. How much work does it take? I feel like, I feel like, well, you know, I do multiple things in parallel. So it's hard to say, but the last time is maybe two months. This is one where I really thought, oh, I can just whip this out and... I couldn't do that. So, we'll talk about this, we'll revisit this stuff a little bit later later. Okay, so now, I wanted to talk a little bit about LDB. So I'm writing the Rust plugin for LDB. It's here. You can go see it. I would say it's about halfway done. So I wrote, you know, LDB in parallel. Okay. There's this thing, Rust ASD contacts, and there's Rust language. There are two classes you have to implement if you're... or like two files, modules, that you have to implement if you're writing a plugin. Those are pretty much done. I wrote each language in kind of responsible for some of its own work interpretation. Yeah, like, LDB does all the mechanics of reading the work, but it sort of hands them off to your language to make sense of what is this type, what does it mean to you and then you your code. Figures out what it means, and talks to your ASD contacts. This is sort of a nice design. GDB does it in basically the opposite way. But both designs have interesting properties about them. This one, the interesting property is you don't have to know anything about any other language except your code. You control everything, the representations, and you look at the dies and decide what they mean. In GDB, there's like one thing that decides everything, and so in GDB sometimes you'll be touching something and you'll say, well, I could do this here, but I don't know how to break the data code. But on the other hand, the downslided LDB approach is that every person who writes one of these languages has to be a super expert on work because you have to view our responsibility, that those decisions are centralized. So in GDB if the new work feature comes along, it's added once, or in LLDB maybe it has to be added every language separately or maybe some of them will lag in some. So it's an interesting set of trade-offs, I don't know, anyway. So, all the door-freeing stuff is done, it seems to work alright, you know, I have some tests in the tree, I wouldn't actually use it in real life. Expression parsing is done. An expression parsing is not a name for it here, but basically LLDB also will if you type in some expression and print this thing it will hand that expression to your language plug-in to parse and understand. So I haven't done that yet. I've talked with other people in the rest community, how should we do it? Now in GDB there's a very straightforward way to do it which is you write a parser. That's how everything in GDB works and it has one nice quality which is GDB's light self-contained but it has some other downsides in terms of adherence to language standards maintenance issues like GDB has module of two parsers that hasn't been touched in 15 years because it doesn't work because I don't even know module of two. So one idea that we had is in Rust, one obvious idea in LLDB context is just reuse the compiler. Compiler guys are like, no, no, don't do that. So I guess the compiler has this module of create that's the Rust word for module the compiler has to create the syntax create which is maybe not the best one to reuse that's what I understand. I've never looked at that but there's this other one called sin which is a Rust it's like a Rust language parser written in Rust and you can just drop it and use it. So our idea was well let's just reuse the sin Rust program, you know. Maybe we can write a library have LLDB deal up in the library just call into it to do the parsing or maybe we can write a the current thing I'm looking at is maybe I can write it as an external process and talk to it with like pipes type in the data and get back something LLDB has this somewhere in like the utilities is this thing that do like the YAML parsing write into C++ objects or something like that that would sound cheap, you know like we'll put that together things last word. So but then you know so we can do that, I'll probably do that but then how are we going to shift this thing? Like if it's a library it doesn't really matter it's a library it's an executable if we wanted to have it in tree that would be like a Rust file or dependency for LLDB which I think would be we could ship it with Rustup you know about Rustup Rustup is like a Rust tool chain manager. You use it to install your tool chain so one idea would be another artifact so you have Rustup to get the LLDB plugin and create that would work but then that makes testing in LLDB great so I don't know I don't feel like we have an adequate solution but we'll come back to that too in the discussion part at the end because maybe we don't use the solution to that problem there might be a different problem okay so now I'm going to talk a little bit about work we need to work extensions we already have some we cheated on this thing for how we emit virtual tables I was talking about this a lot yesterday C++ debuggers or C++ compilers don't emit stuff about virtual tables into the work at all it's just that we not represented in the bug info on the side channel the debugger also knows about the ADI and we're all going around the ADI to find these things in Rust that didn't work Rust doesn't have a stable ADI not yet maybe not ever I don't really have any idea but it doesn't have one so our idea was we'll write information about v-tables bug info so doing that nicely required work extension I was also talking about this yesterday that dwarf extension is currently documented in a comment and an LLBM yeah trying to find that which is a great comment and we're going to need some extensions for aspects of the Rust type system that don't quite match up to other existing like so for instance Rust has two kinds of structure it's nice to differentiate them somehow and the dwarf way is to differentiate them explicitly have a different hack so I'm going to have to file some dwarf bugs I was talking about this yesterday as well but it bears repeating I think you know dwarf is the logical place to collaborate with the debugger the centerpiece the compiler socket the debugger socket and because I'm working on LLBM and also GDB and LLDB it's like a cross tool so like with this enown thing discovering during review to talk to Ada people about their issue because Ada can do that and talk to LLDB people or LLBM people about the interpretation of the standard and the best place to really do this is the dwarf standard however dwarf standard is currently running in a very closed way Pierre Marie submitted a bug you know against the dwarf standard for this thing that we were talking about and it took three weeks for the bug to even appear on the website and once the bug appears there's no way to comment on it like I can't go and say this bug came from here and I want to do this and I think this is more logical it's just not possible there's no way to do that maybe I could join the committee so I want to I want to somehow get everyone to agree I don't know how to do it but I think it could be more open if you're on the dwarf community or if someone you work with is you should give them a little push okay so now you've all been waiting for the commentary part you know this workflow sucks um it's terrible to do all this stuff like two times it should be very complicated it's just not good I don't really know what to do about it you know one idea would be maybe rust could somehow track LLVM a little more closely so that the backwards are less good you know but I think that's complicated as well right, like because of LLVM's sort of rapid rate insurance I think it's pretty difficult to do one version two another I don't actually know the details about that but I feel like they're still stuck on port out whatever it is you know for a reason there's something blocking the upgrade and I know that when people try the upgrades there's been regressions and stuff so there's really like a tail right like they do the they do the initial upgrade and do a lot of testing and then there's a bunch of bugs that have to be fixed first they can really like connect to that's my wish list better workflow for me another thing is like with LLVM so LLVM has a plugin architecture you can write these plugins that are separate of shared libraries that get loaded by LLVM except you can't do that for the language there's no way to write a language as a plugin the reason is there's kind of two reasons one is like there are these work header files that don't get installed but also I think there's a stability issue where LLVM only wants you to depend on a certain set of APIs and this would require a much bigger API service and they don't want to do it but if we could do it externally we totally would that would be the easiest deployment situation for us even exactly parallel we're deploying the situation for how rust is LLVM you get some version of LLVM and you can build rust against it well it would be the same you get some version of LLVM but we can't do that so that's what got us into that deployment problem I was talking about how do we deploy the parser how do we deploy LLVM and recently there's been some talk on the LLVM maybe we don't even want to land new languages until they're proven in some way which I don't know what you know it was a little vague for me so I don't really know what the way is but what that said to me is maybe instead of figuring out how to deploy my plugin or how to deploy my little helper thing to do rust blended parsing I'm just going to have to have a work of LLVM for some period of time who knows how long and figure out how to deploy that I'm not too happy about that I don't really know what to do about it other than you know I think for me this is a little confusing one but I don't know then just generally when I work on LLVM you know I think you can use more application there's a little bit there is a web page about how to write a new language plugin and there aren't many comments on the source so I ended up reading through the other language plugins and trying to figure out what they're doing and write mine based on theirs but that's like that's a dangerous way to go and the reason is you don't really know am I copying a good one or a crazy one maybe the quality varies a lot of course you can do the flagship like go see what C++ does but it's so complicated because it's like the most fully featured one so it has a lot of depth to it there's a lot of things in there where you think I don't even know if this is something I can worry about or some weird corner case so it's very nebulous it would be nice to have just a little more of a god like one idea would be to have a sample language or a skeleton so do this or do that that would be the ideal now of course I say that I'm probably not going to do it even though maybe now I'm well situated having less than classic okay that's actually the end so are there any questions we're talking about you so the question is this is all Dwarf and it's Unix centric and what about Windows 2 so LLVM can generate PDBs Microsoft tried to help out there's this thing on GitHub which is their documentation for their debug format documentation consists of a code drop which I would describe as classic Microsoft code so there are people skilled in reading that I'm not one of them so yes I would like to I think it would be good if we could these things that I changed in LLVM if they were hooked up properly to PDB I don't know how to do that and I don't even really know if you have any questions anything about Rust philosophy whether it's one of you as a foundation I think you would want to go and suggest one of my things Rust philosophy of compiling and linking currently Rust is currently Rust doesn't really have great support for making sure objects it's pretty much just static linking I don't I don't know that I could answer that question like I don't know what philosophy is or why it's not we're also speaking about both LLVM and PDB so we'll execute for debugging right so the question is what do people use for debugging Rust do they use LLVM or PDB people use both just if you use LLVM right now you have to and there are some things that don't work yeah I want to fix both in GDB that work is further ahead primarily because I knew GDB well enough to just write it for an LLVM you know I had to learn more but in fact actually in LLVM there is those lines somewhere in there where it's like if you're reading the work and you see Rust language tag just route it over to the SQL so even if we like you know land all this stuff separately or somehow get a plugin or whatever there's still one patch that's going to have to go in any other questions yeah so right okay yeah yeah yeah yeah right so the question is why aren't we using the Rust compiler because Clippy which is this sort of winter type thing for Rust does use it and we pointed out that like the Rust compiler does have like type of information where the Cyn create is just a parser doesn't really know very much yeah you know I just talked to the I talked to some of the compiler guys I talked to Nico and he just said shouldn't put words in his mouth because it's my understanding of what he said and I could just be following but my impression is his opinion was that the Cyn text library just wasn't really reusable and they were hoping to do massive changes to it or something and it's better not to depend on it and then for the type thing you know so there's a you know a certain weirdness to doing these things in the buggers the buggers have access to the types but they have access to like some transmuted form of the types right so you could go either way you could use the whole compiler but then require the source to be around and sort the more the more they actually serialize their IR in these files maybe that's doable I'm looking to do something a little at least in the short term to like get things off around something a little simpler and so Cyn for me it's like it could handle the parsing of like rust expressions and dealing with what do the names mean and what are the types something the debugger already knows were so I thought it was a decent it may not be the best one like I'm not actually sure but one thing I was thinking is just to get something working so we can like I was saying earlier remove that log jam and start moving things forward so we can revisit any decision with something better and change it and we'll have the freedom to do that can you evaluate complex rust expressions with a debugger that can you say I'm sorry I couldn't quite hear you can you evaluate complex rust expressions complex rust expressions in the debugger that's the goal so like in GDB you can use like rust is like this expression language you know the difference between statements and expressions is a little not sharp right like match technically an expression GDB doesn't implement match you know it implements sort of the C like subset of the expression but yeah like GDB understands rust syntax pretty well you know you can use slices you can you know you examine two both using dot zero one thing if I ask your question well so yes type infers can be a problem and so can alright my time's up but let me finish type infers can be a problem and so can you know like kind of talking about the little semantics or like operator overloading can't be done to portray something represented so it's like a bootstrapping problem we have to do something to unbreak the log jam so we can make the debugger feel better so that we can make the expression parsing better so that we can advance very well