 So don't worry, it means nothing, you just miss the agenda page and there is nothing else. Okay, the first thing is key dump. So the key dump to go is very simple. It's just dumps the kernel VM core when it have a panic. And so it works by design, it will reserve a chunk of memory and the preload a key dump kernel in that area of memory. And also a key dump you need to run my S into it. So the memory is really excited controlled by the correct kernel size, the correct kernel parameter and it will be reserved already on both. And after the kernel is panicked, it will keep like jumping to the key dump kernel and the key dump kernel will just ring. So in it, in the preloaded key dump, you need to run my S and that's used to the user space. The user space will just read the info from the program core, which is the interface provided by the key dump core so that you can read all the info from the data kernel and also the user space will set up the dump package and the dump the VM core onto it. So, but it design into it pretty robust but with a very, with a memory overhead because this part of memory is never used until the panic actually happened. So there is a complex that pre-reserved memory size should be large enough to contain the kernel that you need to run my S and the user space. And it also should be small because you don't want to waste the memory that it's actually never used. And the overall usage of that key dump memory is greatly dependent on the user space and the driver. So the next part is for the Fedora and real because the user space design here is quite, maybe could be quite different from other distributions. So you need to run my S and the user space part in the Fedora and the real it designed to support many different kind of dump targets like the Roblox device, dump to our LVM partition, dump to an FS partition, dump over Fibro channel or Fibro channel over internet. And as I say, everything you can find with basically the Fedora and real can dump the kernel onto it. So to do that, it heavily use the existing implementation stacks. So it's heavily use the DRACAD because they have a very similar goal. The DRACAD is to generate the, so you need to run my FS for an ordinary boot. The ordinary boot will just bring on the boot target and switch you to it. As for the key dump, it will boot and set up the dump target and dumped onto it. So the progress is very familiar. So it just use the DRACAD. The good thing is that this kind of implementation make it very powerful and then avoid a lot of labor heavy works and support a lot of targets without too much work. But the problem is that your space will be heavy because the many of these implementation stacks are supposed to be running in an ordinary environment, not key dumps. So they will cause a lot of memory and the binary side is very large. So the first thing we do is to shrink the user space and you need to run FS sets. For example, the typically you need to run FS sets and compress for key dump will be like 80 megabits. And it will be even larger when it's actually being extracted in the key dump kernel because of some fragmentation, fragmentation issues. And for example, for in real age, a typical reservation value of the key dump kernel is 168 megabits. So almost half, more than half of the memory is used just to contain the unit around the FS that's a very big memory consumer. So the first thing that will be used and improve the key dump is a strict house only mode. The drag card usually it will include a lot of written down drivers in the unit around FS in case of any hardware or environment change because you do not want your computer to stop booting just because you change the graphic card or change the network card. But for the key dump, it's going to, you need to run the FS to be as small as possible. So we use a new house only strict house only mode. In that mode, drag card will only bring in the hardware and only bring the modules that are strictly needed for setting up the certain dump target. So this one initially added in this project request and there was a lot of other fix and improvements. So now basically the FS multi-initial machines and other drivers are optimized with this mode. So now if you use the drag card house only mode strict the drag card will not build a much minimized unit around the FS and everything is much smaller. But that's not enough. The next thing is a scratch that you need to run with content. Remember that just said that the unit around FS will got uncompressed as a whole into the RAM once the current boot. So instead of uncompressed everything uncompressed the file in raw format it's keep it compressed as a scratch image and add an overlay, read write overlay onto it then switch root onto it. We just replaced the unit program which is systemd by default with a custom script that which one will just set up the scratch image and also a root read write overlay on the scratch image then switch root onto into that image that the image contains systemd and to systemd the whole workflow seems never touched because everything look the same. It's just the unit around FS is now a scratch image is sort of a RAM FS. The idea is simple but it comes to be very long journey because many part and many corner cases it's really hard to cover. And so it takes a lot of time to make this work perfectly but now with all the changes now it's possible to build a very small but the functional track that you need to run with as to with this command track dash dash dash automatically strict and add a scratch module. So you need to run with FS will be much smaller but the functionality is the same. And also this command line is a whole command line used by K-Dump to generate the unit around FS if you are interested you can look have a look. And also we repackage the systemd because systemd is very large and it's still growing. We have talked with some we read the issue some time ago. So the systemd usually in the early version use only five megabits in total for the binary file but now it's 20 megabits. So basically in K-Dump we do not want all these fancy new features. So we just turn off all the features and repackage it and use that systemd instead of the host to install one and that one also reduce the memory by a lot. So with all these user space changes the quad kernel value had to be cut by almost a half in the Fedora. And also part of the changes it landed in real. So the quad kernel usage in real is also much smaller with these changes. So this is the part for the user space and you need to run by fast changes. And the next thing is for the kernel and some features new tools. So we for kernel usually it'd be pretty good but sometimes some drivers and you're using a lot of memories and then we want to have a better look of where the most memory goes. So previously Dracad have a RD domain debug which is very simple. It's just print out the main info and the slab info. With RD debug level is two it will use the abstract to trace some memory locations but it's only traced for a very short period and they use a batch to do some collection info jobs. So that's why it's very, it's quite slow and not very really helpful. So the problem is with tracing the memory usage of the KDOM kernel is the first for the kernel modules. They may only consume very little memory upon load but it will consume a lot of memory actually being used like you if up turn on an IC and the IC will only allow this memory when the interface actually on. And also memory will get allocated and freed. So the peak usage is very important but it's not needed to cache it. And also user space info is already matters and then you want to know which user space progress allocated the most memories and also the memory is already itself is pretty precious in KDOM. So the tracing tools should not cost you a lot of memories or else the KDOM will ring out of memory and you cannot finish the tracing. And also the tracing tool we expect to be out of box smaller as much as possible and do not have a lot of library dependencies because we want to integrate the whole workflow into KDOM. So to make this thing, to make it possible which I tried to make a new tool, name it MemTrack and put it here, this GitHub link. So it's basically it supports binary proof it just started set up the tracing and then collecting the info from the ring buffer and it will maintain an integrated tree of the page allocating stacks in the user space. And the memory footprint of the tool is quite small small enough for KDOM. It managed to reduce a lot of redundant stack trust info and typically it only used a four megabit of memory in the whole KDOM ring that is good enough for now. And also it can record the total memory locations the peak memory allocation of each stacks and also the stack trust of each historical peak allocations and during the whole KDOM ring. So we will not miss any detail of where the pages goes. And it's for different result output format so you can tune the output format very flexibly. And also support the access, proof and the page owner. So the early allocation is also possible to be trusted with this tool. And the tool is very, what's very good about it is very tiny only rely on lefty. So the binary is small and able to embed it into the KDOM environment is very doable. So to make it even easier to use like any the UI you can have a look, this is a TOI. It also shows how it organizes memory allocation how the memory is allocated internally. So this is how it manage the memory allocation tree. And also it can collect the memory allocation by the modules. Like in this picture I just said to do some demo so a lot is scattered back that module and it costs a lot of memories and also a lot is some other modules to just show it. So when used with KDOM how it works so it's very easy to just start a mem-stract with these parameters give it a true report format and let it just track the top 80% of the memories and being used. And when the KDOM is finished and before the KDOM kernel reboot we just kill it with six inch and it will print the following info. So you can see in that output, so in that KDOM range the scratch FS use the most memories and the peak allocation is much higher. Also the next thing is the work out console and some other things. So this is for a normal KDOM ring in a VM in a machine. And also you see I expected to two report here. One is a module summary when it's module top. So this way the module top will tell you where the memory goes by print out the sectors. So it's very clearly in this, to insert KDOM ring, those things just allocated for the page cache. So it's fine. Or if we give it a more memory pressure the page cache will probably get free. So there is no issue with this one. And we have used it to fix some practical real life problems. So for example, the safety before we used to have the issue that safety before some machine with this graph with this network card will have OM drink it up. So I just increase the crash kernel value and start a mem stack with it. So the mem stack just pointed out where the memory goes very clearly. The allocated happen with the system DUD just try to load the module. And the module allocated about 20,000 pages and the stack trees went through the safety before a debug VM call add KDOM. So just to read the code a bit of a funnel that it's trying to add a lot of things to the VM call for debug and just disable that part and this everything doesn't work normally. And also we have fixed other many other issues like when the IP command tried to start an interface some network card, the ring buffer location function allocated a lot of pages doing through the page set issue. And this one is also easy to track with the manufacturer. I lost the log. So I just posted a typical one here. And also here's another example like for NFS monocle it shows which where the memory is being consumed by NFS monocle. I stripped a lot of the lines because it's very long. So that's why we would be easier to be showing the slider. You can see the amount to which most of them also spend on the page cache reading the binaries. And the NFS4 module in the kernel only takes about 20 pages and which is very acceptable. And also it can trace the user space memory usage and also the user space peak usage and the final allocation of the pages. So like in this report it generated with the main stack data report text summary and sort by peak. So the process of using most of the memories on its peak usage will be shown in the top. So the init which is the system D consumed about 600 pages and the journal D which also the system D login program consumed up to 600 pages. So in the hope report there are like seven pages consumed by user space. And the user stack is also available just to change the report format. So we can analyze where this map page is being consumed in the kernel space. And also I just said it's a spot page owner. So some early allocation is also traceable with the tool. So if you use the page owner at the back end you can see it's only the kernel boot. So you need to call it because it's using a lot of memories. So generic file write which just I think is writing to the init around as file system is consumed a lot of pages. And also the P store, the P store init which is a init call consumed a lot of memories. And P store is not needed for key down so we can consider disable it later for data. I have a fundamental more info about the tool as this data page. And we have integrated with a DRACAT you just use DRACAT a memory equal four or equal five which equals to passing this arguments to the memstrak and the QH after the pivot hook of the key down of the DRACAT both progress. And there's something managing to end down about the tools like a more detailed users with tracing. Now all the stuck traces are limited to either kernel space and do more debug with the memstrak so we can better understand the memory usage of the key down and optimize it even more. And also integrate it with the page owner backend and the perf because now you can use the only one. So you have to either trace the allocation or the green time allocation but it's doable to integrate the tool. The next thing is some next steps and the working progress items. So the first thing is still we want to reduce the memory usage even more because now even after the compiler you need to run this it will cost about 30 megabits. A squash is a most mature solution yet but there must be other fields based compression system that looks promising. We are also trying to off. And also the kernel memory usage we are trying to optimize that too because now the neural space is much smaller. We want the next thing we'll be working on the kernel. And the next thing that's been worked on is the kernel X auto because first it's very hard for anyone to guess the value. And the kernel memory usage very variable depends on your space and depends on your hardware. Like the memory encryption after you have that's when you build you may have to reserve extra usage. And sometimes you it's familiar firmware will use a lot of memories. You don't want to change the quad kernel value manually every time is not very smart thing to do. So having a quad kernel auto could be a good beginning of this whole automation progress. Yeah, I think that's most of the things of a lecture chart day. So I think it's fine for QA. There is a question in the chat by Alexander. Does memstack work for Debian or it's real specific? Yeah, it works with Debian. And also I see some people have to package it for our clinic. So I think it works, is destroyed independent. It works anywhere.