 Hi, I'm Marc Wielhardt and this isn't Frank Eichler. Sorry, he couldn't be there. But well, I will try. It's a bit of a pity because Frank actually did all the work. So that's not complete to Frank did most of the server work and the client work was done by Aaron, Aaron, Mary, who is also not here. But well, so if the presentation is lame, that's on me. The code is actually pretty good. Okay. We all three work for Red Hat, but the goal was very explicitly not to make something just for well or for Dora. So, okay. So the summary is we have a debug info, the demon. We realized too late it should have been dwarfed, dwarfed demon, of course, but well, now we're stuck with the name. And it's essentially just a file server for debugging artifacts. Why would you need debugging artifacts? Because maybe you have bugs in your code. Some people do create lots of bugs in their code. So that's not a problem. You debug your program. You install all your debugging stuff. You use a debugger. Not going to talk about that. A debugger. Yes, what is a debugger? A debugger is anything to introspect your program basically and think that uses this kind of debug info that we serve. So, right. It is about native code. If you write in some scripting language, you're already happy because you always have your source code with you. Yay. Okay. It is relevant for go or rust. There are some improvements that could be made there, but they're not really relevant. It should kind of work out of the bugs for go or rust too. And well, I will show them off some of the things that work with it. So, no. So, for the debugging info, basically, compilers these days are pretty good. Always use minus g. And you do use gc, which has for tracking assignments, which makes things really, really nice. And don't disable them. The kernel disables for for tracking assignments, which makes debugging kernel codes really annoying. Apparently, kernel hackers don't debug. There was once a bug in gc that generated something bad. And since then, the kernel has disabled this. Don't do that. Just use a fixed compiler. Sorry. There are various tools that can compress your debug info, which is kind of useful because it's really big. So, some examples of that. There are some subsets that you could use, which are already useful, like convert showed, but we want more. That's basically it. And there's the source code, which is also pretty big and which is kind of interesting because you don't want the source code, the pristine source code, but the source code that was actually used to build your program, which is sometimes different because your build might post a pre-process your sources or generate them on the fly. So, in general, you use minus g and you're happy, except if you don't or if make strips it, well, then we really can't help you. But you might be using someone else, their belts or, well, distro packages, or you might be debugging remotely or in a container or from outside a container and where is your debug info? So, the idea is in all those depends. We want to provide you with one way of getting at your debug info. Right. So, why doesn't everybody ship all their debug stuff? Because basically, the debug info is your reverse compiler that goes from binary to source and describing that is, oh, big. It's five to 15 times the size of the script executable. So, yeah. Especially if you would like to have it for a complete distro, it's really big. So, no wonder that people try to move it away and it might be hard to get it. So, most distributions at least provide you with some way to find it. So, Fedora does it pretty nicely, I think, but I work on Fedora. So, as you can see, it just says, well, just DNF debug info install, that's back it. And it even gets all the dependencies. So, Finn might rely on Tlipc, also pulls in the Tlipc debug info. So, it just works, except you have to be rude for it to install it. And it's pretty big. So, you'll have to have this ways. And, well, for Ubuntu, I'm not that familiar with it, but that also makes it immediately harder for myself. There is a difference between debug sims and debug d-dabs. It seems about Debian and Ubuntu now produce debug packages for everything, but basically, you have the same limitations. And then you have other distributions which don't create any debug information. So, what do you do while you rebuild your program with the way you hope it's the same program that you debug then? So, the NixOS kind of always has an interesting approach, which is a Dwarf file system fuse thing, which is actually pretty nice. This is kind of what we want, although we wanted something slightly simpler and something that doesn't do, that is easy to set up for both distributions and your local builds. Right. So, it's pretty useless that you have to have privileges to install debug info. It would be nice if there was a mechanism that combines the packages with your local builds and maybe UFA, a team that publishes some of the code. And we always want access to the actual source code used. So, the debug info d is a simple HTTP web server. It has three, it has two, it has three. I missed one. It has three get requests, one for the executable. That seems weird that you might want the executable, but you might have gotten a core file from somebody and you want to debug that. So, you want to get the executable, you want to get a debug info and you want source code. And basically, that's the whole HSP server. They're all indexed by built ID. You can run it on any port and the servers can federate to each other. So, the built ID is kind of the thing that ties it all together. A built ID, well, x-string basically has over your whole executable. It's unique in the way that if you have a reproducible built, it generates the same, it should generate the same built ID and you use it to uniquely identify a built but you cannot use it for verification. It's not a security thing. And it's also if you, a built ID is generated by the linker for every loadable module, program, library, kernel module, kernel. And what did I want to say? Oh, yes. And when you then split that loadable module like in the main executable and debug info, you copy over the built ID. That also makes it harder to verify because the built ID is calculated over the whole, so you would have to merge them. So, the debug info is fairly simple. The version that was released, you can point it at your build directory or at a collection of RPMs. In current Git, you can also point it at a collection of depth files and it uses lib archive so it can basically use any archiving format that you use. It creates a database and from that it serves the files you want. The client is even simpler. If you have support like a GDB, well, it's not integrated yet, but I will demo it anyway. You just set an environment variable. I have all my debug stuff. My source is there. Fits them. We have a little example program, debug info defined. Give me the source or debug info or executable for this hex code with the source. That's it. Oh, right. Let me just show it. Oh, right. I can't see anything on this screen. Sorry. So, yeah, I said unprivileged, but I'm using Docker files. You script your whole demo and then you can't run the demo because you lost your password. Yeah, let's first show. So we don't have any debug info installed. We do have a few those debug info installed. Well, just to show that I'm not. Yeah. So we start a server with lots of verbosity. Come on. Yeah. So lots of verbosity, look in, create a database here. Scan it every couple of seconds. Index all the RPMs and the files that you can find in this directory. And basically it indexes them and you're done. We don't really need that window anymore except to see that it actually gets the files there because this is all running locally. The demo is actually too smooth. It just works and you should see it downloading, but it's immediately there. So, I should have shown you. Okay. So in the demo directory, I put the, I happen to put the debug info RPMs for Bess and Helip C and, well, everything I need for the demo. So, this window is too small. Support is built in in the L-Futile tools. So even though we don't have any debug info installed, it gets them from the client and we have a backtrace with source files and symbols. So this was for an RPM-based thing. Maybe we want to build something locally. We make a binary and we strip the thing that we are going to install. So, in this scenario, you have your built directory. You give your binaries stripped to your coworkers to run a different machine, which we now create. Yes. So, we use Spotman and it, right. So, on this machine, which we just created, there is the stripped binary. So, we run GDB on it and there's nothing. This is frustrating. Yeah. So, yeah, yeah. So now we do it again, but we give it access to the debug in for the server that we are running. So, it uses, okay, it's cheating. It actually uses the host network, but it's as if it is on a different machine. And it actually finds the symbols. You should have seen it download, but it was so fast that it immediately gets them from the cache. And if I do a start, yay. I can debug my program. And this would be the scenario where you have a team which presents their own built environment to the server. It also works for system tab, for example, give me all the, oh, you can't see what it, right, give me all the functions in glipc that started via printf. And it was too fast again. Actually, it got the glipc debug files from the previous run because now they're in its case. Okay. The demo actually worked. Was it too smooth? Probably it was. Yeah, it should have crashed. I should have made the network slower. But what you can see, actually, do I have it here? No, I don't have it. Oh, right. Oh, I could do this. Yeah. Okay. The server has HTML page. Actually, these are statistics from Prometheus. And people that do metrics like this type of presenting statistics, well, you can see that the little file server, HB server, knows about a lot of file names and 2,202 built IDs. And it actually worked, really. Okay. How do I get back to my presentation? Sorry. Where's my presentation? Sorry. Oh, wait. And now? Yeah, that's good. Thank you. Okay. So what did we see? What does it do? You put all your debugging profiles in directories. You can tell it to scan RPMs, DAPs, your full build. What? Anything. Yeah, well, it does need a bit of information to cross-reference them, especially for the source files. That's not always easy. Sometimes on Fedora, source files are put in a separate packet from the actual debugging file. So that is somewhat tricky. So you can basically let the server running. It scans what you have. If it is a built directory, it also will clean up, but it's not relevant anymore if you put new RPMs there, it will index them. And, well, it's basically a little SQL, SQLite database. And it also supports DWZ, which makes the indexing also a bit tricky because DWZ compresses the debugging file by putting common elements in a separate file. But essentially, a DWZ file is also something with the belt ID, and you can just request it. I feel a question coming up there. Okay. Well, basically, SQLite is nice. That's a message from this slide. Okay. Yeah, it indexes everything, and you can really nicely partition what you do because you can make servers query other servers. The debugging for the server is also a client. So you can say if you don't have the built ID go one server up, and you could do the same insights, your company or your programing group where you have one server that questions all the others. Yeah. So the client guesses everything, which also means the servers guess everything. And there is a kind of a sample client in Elfutiles, the debugging for the client, but it's not much work to create your own clients. The sample client just uses lib curl and has a least recently used cache of files. Well, basically, to implement a client, you have to be able to make HTTP requests, and that's it. So this was the minute slide. Thank you. This was actually the slide I was describing with the client. Sorry. Let me go back what I should have said here. No, this is good. I already said most of this, but the scanning can take long, mostly because the RPMs are compressed, the decompressing reading takes most of the time. The database can become a bit large, but that's mainly because there is so much data, especially if you try to index all the RPMs in the Fedora CoG build system. That is really big. But in these, it's a bearishly parallel problem. You can just partition it. You can have multiple directories, multiple servers. You can even merge databases. So like I said, there is an implementation of the server and of the client in Elfutiles that was released in last month, I believe, in 0178. There is now support for depth files and for more parallel parsing of your directory trees in the current Git directory for the next release. There's a prototype GDB client on a branch. It works perfectly. Tom, are you going to review it? Yay. There are several clients in the way. I have a little list and there are already some public servers if you want to try it out. Right. So this is actually already too old because this slide was made before the weekend, but there are two experimental servers now. One for system depth that Frank is running and one for SUSE. Frank is trying to get the most common binaries that you use with system depth for all the different distros there. The open SUSE one tries to index all of the tumbleweed is the last one for open SUSE. They're both experimental, so if they don't work, sorry, but hopefully others do that. We had a talk with Alex Larson, I believe, who does the flatbacks and he was also looking at making a server available that serves the debug info for flatbacks. He actually wanted to write his own server, but the server is so simple that you can also do that. For the clients, of course, the L-futile tools use it by default. System depth uses it because L-futile provides it towards the same thing. Some binutile tools can use it that's merged in Git. The DDV patch is proposed and Tom is going to merge it. And there are some in progress. I believe NO check was also recently done by reloading that page. There's also some help wanted and we volunteered Aaron for most of the work, but I'm sure he wants help. Yeah, so for the client support would be nice either directly or using the L-futiles simple client. Okay, I don't know what that is. You should think about security for the server a bit more. Of course, the CDBugin file needs always to be improved, but that's... If you can help with that, that would be nice. If you could run a server for your own stuff, that would also be nice. Let us know then we put you on the page and we get lots of people debugging your stuff because it's so easy. I think we hit the right spot by making it file-based, built at the index. But we might want to see if we can improve the protocol either by making things, making it smaller. For example, I just want to die 3, 4. This thing or just the file table, so I don't have to have the whole debug file. Maybe the other way around, for example, if you try to trace all the modules in the kernel, then you currently have to fetch 2000 debug info files separately and maybe you can batch that. But I think the current setup is so simple that it's good that we started with this. Do I have time for questions? Okay, but there are no questions. Ah, what was wrong with the next approach? There's nothing wrong with it, but it is a much more complicated client. It's a Fuse file system which is nice. I think what we have now is easier to set up and easier to federate and I think that's something that is good. I think you could maybe take that Fuse file system approach and build it on top of this. Yes? Yeah, so yes, it has to have access to your build track. But you could maybe have your build track. But you're right, that is one of the things that's also why we have separate indexing of RPMs and whole file freeze. You could make it smaller. The interesting thing about having it on the host is that you are sure to also have all the header files as installed on the host. So, in a way, you have to have it on the host to have the complete picture. Okay, I think I stopped before being dragged off the stage.