 Okay, so let's get started and like here's our next talk. Stanislav, back to you. Hello, thank you very much. I hope you can hear me. I'm really terrible in using microphones, so I hope it works now. If not, please give me a huge shout-screen, whatever. Thank you very much for coming. My name is Stanislav. I work as a as a kernel manager in FredHats and this talk will be a little bit about very basic introduction to the debugging information, which is very boring and the very exciting part is about how you can use gzip to make your stuff smaller. So I wanted to make sure your expectations are set correctly about the interesting stuff in the talk and you won't be disappointed by the end. So what the debugging information are, through the history there has been several different standards, but at today the devarv debugging information is kind of standard on all Unix systems. So it's devarv because it's the counterpart of Elf and Elf is the executable linking format, so the devarv is as joke. They have this wonderful logo with a real devarv in the middle of it. Usually people just care about debuggers, so debuggers complain if they don't have debugging information, so just compile with the dash g option, it will generate the debugging information, that's enough to then the debugger complains a bit less and you can use it for actual debugging. Currently version 4 was released, version 5 is almost ready and I guess will be released very soon. What it really is, is that because it's counterpart to Elf, it's really built on top of Elf. So Elf is an interesting format which allows you to ship your binary data on executable files and it provides two views on the data in the way Elf is stored in the file and in the way how you can actually execute it. So what the devarv is doing is that they are introducing a couple of more sections or program segments in your Elf binary and these sections contain the real debugging information. There are about, I don't know, eight of them, usually about five or six, but maybe the most important sections are the debug info which contains the real type of your debugging information, the debug abref which are the abbreviations which is the real type schema but it really goes very closely with the debug info, so I would just say that these two should be considered as together like one thing, it's more of a space optimization. And another interesting section is the debug line which contains kind of virtual machine which can be used to identify which type has been defined at which planes in your source code. So then you have, need to have your actual source code next to it so you can find the proper place where it was defined. It's not used only for debugging but also for some tracing tools like system type or the things built up of K-Probes which is the internal kernel stuff how you can actually trace the kernel itself on Linux. So this is just a very brief example of using Read Elf which is a program which can be used to read your Elf binaries to dump the Elf sections and you can see that there are a bunch of various debug sections which you need to look at if you care about the debugging stuff. So what it really is inside the debug info and debug abref sections is that it's a type graph, there are basic nodes in the graph, they point to each other as their reference and etc. The real graph is formed by a link list of compilation units. So a compilation unit or a translation unit or for C programmers an object file that's the basic unit which you compile. So you know the C source code, right? You compile each file separately and then at the end you just link them all together and if you have a very nice linker it will do something to optimization. So the debug information follows the same process. There's one compilation unit for each object file and then a stupid linker just puts them all together in one linked list and it's done, it's very easy. For each compilation unit you have a type graph of the dice which are the debugging information entries. So this is a real graph which tells you which type is pointing to which type and where you can very easily follow how all the types are interconnected. It's oriented graph because you can really follow the edges how they are referencing to each other. Each die also has attributes like the name of the type or the size of the type or this kind of stuff which is really like whatever you would need to know. So writing a linker which can link your debug information is a really easy task. This is an example of how a specific die, the debug information entry, looks like. So we can see that there is a type of the die as a structure type because it's the beginning of some C structure. It has about five attributes, it has a name, size, variables defined and it has some sibling in the graph. So how it really looks like is something like this. You have a linker list of compilation units and each compilation unit has some random graph of types which are somehow connected together. And now you see the obvious problem, right? A lot of types can be actually stored several times in the debug information entry. So I think that just for fun I look at the VM Linux which is the base of our kernel we ship and I think that a definition of a task struct which is the structure which represents one task is stored there I think 250 times. So it's really not very efficient. If you can just just merge the same types together it really becomes smaller. This is really obvious so everybody knows that and people look at it and even the DWARF standard committee look at it and it's part of the DWARF 3 standard. So what you have in a DWARF 3 standard is that you can create a virtual compilation unit which is called partial unit and then you can link this partial unit into all other units which actually need this data in the form of imported unit. So it's great. It's like wonderful. We can finally move all the same types into a separated virtual partial unit and then link it in all the compilation units which need them so we can unify all the analytical types. I think it's just my private opinion that the DWARF committee thought that a compiler would do this at a time of of linking so it's kind of link optimization. I think in reality I think it didn't happen but what Jakub Jalinek did in about 2012 is he wrote the DWZ tool which is DWARF compression tool which is exactly this is exactly what it does. So it looks at all your compilation units it identifies the same types it moves them to the partial unit and it links the partial unit from all of them. So you can just very easily run DWZ on your binaries and they should become smaller. Very trivial example I have a program which has a debug info of 428 hexadecimal bytes and a brief section of 1D8 hexadecimal bytes. After I run DWZ it becomes about one third smaller so it works. If you look at it at the DWARF level you will see that there is a partial unit which contains my structure type and it's later linked to all the compilation units which which need it. So this is how it looks like after you do the unification. The compilation units which need the data are linked to to the dice which are in the partial unit. The partial unit is really virtually copy pasted into every compilation unit which needs it. So if you use this on something bigger like VM Linux for example you will see that the debugging form in VM Linux after you compile it is about 140 megabytes. There are two megabytes of ABRAF section after you use the tool it becomes 41 megabytes about one third of total size. Wonderful. I noted Fedora is doing it for all its all the packages it ships. I'm not sure if it is done for the kernel but but anyway this is the way how you can make it about one third of the original size. It's really I would recommend it very much. Now let's step back and think completely different. What else we could do instead of working in the boundaries given by the DWARF standard? Maybe a stupid idea. Let's just strip the debugging form sections out of the binary. That's what I'm using EU Strip for this. So the first line EU Strip I'm stripping all the debugging information sections out of VM Linux in a separate file which has 74 megabytes. Then IG is a bit and it has 30 megabytes which is really not great because these are random binary data and Gzip is not very good in decreasing the size of random binary data. So good try not the best. What we could do else? So the DWZ is a tool which works with your debugging information entries which it needs to be very cautious about what it does because when you are debugging any slight change in something might actually make the debugging much harder. You need to make sure that you know whatever you look at is absolutely correct until you can see all the stuff. But there's another completely separate tool which is its goal is to allow an ABI comparison of the library. So you have two libraries you wonder how much the functions and types they work with differ so you can use a tool called LibAbiGale for example there are others for the comparison of the binary data. What is interesting is that how it internally works is that it reads all the debugging information sections into memory in all these graphs for all the compilation units but then it merges everything together and is more aggressive than what DWZ is doing because for the ABI comparison you don't care about little tiny differences as long as the binary representation is the same you can merge the things together. I will go a little bit deeper into this later but let's say it's more aggressive so you get a less number of nodes after this unification is done by LibAbiGale for example or any other ABI any other decent ABI comparison tool. The LibAbiGale can also damp its output in the form of XML file for example it's I think really very inefficient in the way how it starts data but whatever it works and it somehow gives us a file which contains the layout of your binary structures. So this is how it all looks like ABI DW is one of the tools which is built on top of LibAbiGale and is shipped as part of the LibAbiGale so you can tell it to actually load all the types all of them which are even unreachable files from the export interface and just dump it in a file and in the file you will see an XML element which contains what originally was in one of the debug information entries so there is a class it's a C structure but because LibAbiGale is written in C++ they don't work with the term structure they used a term class so we call it a class called foo which is 64 bits in size and there are some other attributes of of this type. So when I'm saying that it's more aggressive what does it really mean I have this is the most trivial example of what I can show you. Let's say there's one header file which contains one structure but it declares the inner structure so for the structural foo2 we have just a declaration we can't say anything about its size we don't know what it is so we have a declaration of foo2 and a definition of structure foo then we have one file which defines the the internal structure so we have a full definition of structure foo2 and then we have another file which just doesn't care about the internal type so it is fine with the mere declaration and it just works so you think about how type graph would look in this case we have two compilation units we have the same debug information entries in them for the same type of the same size anything what is in there is the same but they differ just in one thing and that is that the internal type in one case is just declared in the other case it's fully defined would you merge it together or not so on the abi level they still can be exactly the same so live up against merging them together while dwz isn't because you know the types are different in one case you don't have any information while in the other you have all of them so so it might be different dwz doesn't merge them i'm trying to show it here that at the end you see two different types in in this very example while live abigail does it it merged them together so at the end you will you will see the foo two only ones in the resulting file and i'm being too quick how are you um anyway how it really works is that if you slip abigail on vm linux for example it will down it will build a graph it will all unify it it um so at the end of the first line i'm just dumping the output in a file this file has 36 megabytes it's already less than what we have seen with the debug information the xml text file is smaller than the binary representation which was designed with the size in mind and then because it's it's text you can very easily run it through gzip so at the end we have 2.6 megabytes of data which contain which still contain all the debugging information entries for all the vm linux file the vm linux file has in my case i think it was about 28 megabytes or something like it so we are able to store complete debugging information for vm linux in about 13 percent of its original size so now that's that's interesting um it's really this doesn't exist it's just an idea which you know it's it's good to know um what we could do with this um currently if you need debugging information entries for example for the live tracing of your kernel the only way to get them is that you install the debugging information files on your system which have like 130 megabytes or something like that so it's so it's pretty huge and then anytime you need to actually trace a specific point in your kernel you need to have some some userland program which will look at a debugging information entries it will find up the real position in your kernel and it will then instrument the specific place in your kernel so this is how example system type works right it's it's it's a userland compiler which will which will look at the debugging information files it will find what what types it needs to trace it will look at the offsets and then it builds the kernel module which it puts pushes in the kernel and and this kernel module has hard-vired offsets it needs to look at this also like for example the ebpf the extended barclay pocket filter has recently been extended that it can actually be used to enable the kprobes in kernel but but you need the very same same information you also need to start in userland look at the the debugging information entries and build your specific program with the right offsets and that's how you push it in a kernel so what i'm thinking about is like what would happen if we would you know just build a kernel vmlinux binary link to it directly this this raw data would just 13 percent of the size so our our kernel binary will be bigger by 13 percent then when the kernel boots it could unpack the data so it will ate 36 megabytes of our kernel memory it's it's bad but for this price we have all debugging information in the kernel always correct so we can then tell the kernel to to trace itself we don't need to care about offsets anymore and layout and arguments of functions because the kernel knows it would be at the end it should be much more stable and really handy okay we have plenty of time that's pretty much it it was really open very open minded about like you know what are you thinking about at night so so this is what kind of i'm thinking for the last three years um any do you have some interest any questions comments why would i the question is how i would compress it and ahead of time right so the question the question is why i would decompress it at the boot time instead of right in time when i need the data um the answer is that this really doesn't work well enough because what you would also like to have is a quick way how to search your your debugging information which barf isn't really good at all so what you really need to do is for example i need to see definition of struct foo it's one of the 10 000 types i have in my kernel what you have to do with barf is to just walk all the list and find it this is not very good i would really love to have for example a hash table next to it so so you could look at a hash table find it and go to it but but you can't do this if it's all compressed so that's why what what i was thinking about is let's first the index is so we have something very quick and very stable for lookup of anything we need yes but you could do that by sure um the comment is that we can do this by access to a device for example or whatever at the time when it's really needed sure we can right absolutely thank you very much the comment is that in with bar four in fedora we are shipping a non-standard extension which already contains the index for all the types but in bar five this becomes part of the standard so with bar five which is really going to be released very soon there is a way how to have a fast lookup index for your var failure thank you thank you um i said dwz is trying to be very cautious and very safe and the comment was that it also whatever he does it does strictly in line with the standard and it's really cautious and really tends to be on the safe side of of things so it doesn't produce a big compilation units for example right thanks one question there but uh seriously using xml as a replacement is not searching for the thank you very much uh it actually is a question why not to use something better than xml something better for searching for example ascula database or something like i'm totally with you this is not my idea i hate xml i hate it but that's what lip abigael generates and for the purpose of lip abigael it it works it works well enough and i i just use it because is there it works it it was easy for me as a you know what i wanted to show it works but definitely if i would like to produce something which would be smaller i wouldn't start with xml thank you the question is if we could recreate the dwarf file which would be smaller from the results i well whatever we would create at the end based on the xml file would be different from the source because it's already after the unification so we have already merged the types which are in there so if we if we would be able to produce a wallet dwarf i don't know it's very likely but it would be very different var from the original valve and uh as our colleague here said it you might so so we destroyed the compilation units we just merged it all together so if you if you would create the dwarf from the from the final result it would be just one huge compilation unit so for example gdb could have very hard times opening just one huge compilation unit right one question in the back so there are two questions the first one can we build a one common debugging information across several separate binary files