 Okay, so welcome to our talk. Next generation resource management. I'm David Edmondson and I don't know how to go to the next slide. There we go. I'm David Edmondson. I work for Blue Systems. And I'm Henri. I work for a software consulting company based in Paris called Enioka. I'm a long-time KDE user and also a new developer. So we'll start with the problem. The job of a desktop environment is to get you a user to your applications. You need to launch them. We need to show them. And in general, you need to be in control of them. So if we look at how you manage your running processes and your applications, what used to happen a few years ago, five years ago, you'd open your Firefox, you'd go to dig and MySpace, you'd open your instant messenger, Capetta, and you'd run PS and you would see Firefox and you would see Competta. And it's fairly self-explanatory. You could teach up to computer science students. Now, the situation is very different. You open Firefox, you run Discord in a flatpack because that's cool now. And Discord in a flatpack is 13 processes. This list here is truncated because I couldn't put ListView in a PDF. It's insane. It's basically unreadable to me, particularly if you don't have a parent-tree relationship, which you don't always have, and they're just random names doing random things. So understanding is difficult. But also, anything that tries to aggregate resource stats like how much CPU time has this application used while the application has been open or how much battery has been used by this application, it's not less just because the application has split it up. We need to track this properly. In numbers are so meaningless, you could write it on the side of a bus. It doesn't mean anything. So we need some metadata. The other problem we have is we want to make something that's fair. As I mentioned before, Discord in the flatpack is 13 processes. Krita, Advanced Graphic Application is one. Krita is obviously going to be using your CPU because it's an intense graphical application doing clever things. Discord is going to be using your CPU because it's written in electron. So how are you going to divide this fairly? A scheduler has even less information is just going to see 14 opaque processes. No idea what they are or what they really correspond to. We need some metadata and not just any metadata. Metadata propagates all the way down through your system level so it's available to schedulers and so forth. Another problem we have, one of Plasmus jobs is to try and map everything that's going on on your system together. So you open something in your start menu, we have a desktop file with an icon and a name. And then later a window appears and it tells us what a name it thinks it is and we hope that they have some sort of resemblance in common and we can match it up. And very often applications just write any garbage in their window properties. It's difficult to match. We're full of hacks and heuristics to just guess and we don't like guessing. Also because everything is multi-process, if you look at your task manager right now, you'll see Firefox, you'll see a little audio icon which is very clever. But that little audio icon isn't actually in the same process as the process which owns a Firefox window. We're having to grab this two bits of information, guess and smash them together. And we're not always right on here. It's a complete arbitrary guessing game. We need some metadata and the smarter people among you might have spotted a theme going on so far. So the interesting and the good news is that it's essentially a solve problem. So some of you have probably heard of control groups also called C groups. They are a kernel feature that's been introduced in 2007. Next slide. So it's essentially a partition over the process space that also has a hierarchical structure, meaning it's a way to categorize processes inside of groups that have a tree structure. The C groups have originally been designed for servers and they have also seen a lot of usage in container runtimes. So once you've got these groups, you can attach controllers to them. But what exactly are those controllers? So there are three main ones, the CPU one that affects CPU usage, the memory one that affects memory, the IO which affects disk IO. And they do three basic functions on these resources. One of them is limiting the overall usage of a resource. It can also influence scheduling. And also it does accounting, meaning you can actually know at a application level how much your application is using CPU. And that also translates to things like power and better usage. So another feature that you can get out of C groups is a finer control of the out of memory policy. Regarding the IO controller, it's currently difficult to use for desktop because it was initially meant for servers with a pretty well-defined storage configuration. So this one doesn't quite work out of the box yet. So the way that C groups can influence scheduling is through weights. These are defined relative to siblings inside of the tree. So the CPU weight strategy is more interesting and more powerful than what NICE does, which is the old API for defining priorities for CPU usage. So it's more powerful because you can not only have weights that make the priority go down but also go up. And also, of course, it's more powerful because it's at a control group level instead of a simple process level. And also it has this recursive nature, which makes it much more powerful. So, technically, the C group API can be controlled through a simple file system interface that is inside of CSS. But soon enough, you're going to need a central demon to manage the creation and attaching processes to the C groups. And the good news is that system D has actually been supporting C groups since forever. And actually, every system D unit that system D launches is inside of its own C group. It has been for a long time. Also, system D is capable of delegating management of the control group tree to your user through its system D instance, which is basically just a second system D that is running as your user. And conveniently, system D also has a D bus interface, which you can use from user space, which means you can control resources from any application that is running as your user. Also, system D solves the problem of configuration through several levels, either at the system or a distribution level. And also, you can define configuration at the user level. So, this is what a typical system D C group tree looks like. And you can try this at home right now using this simple system D dash CGLS command. So, here you can see what are called slices and services. And also, you can see the system D user instance, which is the user at 1000 dot service. So, C groups have been redesigned completely in 2015 with the new C groups V2 API. And it's solving a lot of architectural issues that they had with the previous API. And what you need to know about it is that currently, all modern distributions actually expose the new API through what is called the hybrid hierarchy. But inside, they are still using the C group V1 API, just exposing the new hierarchy. So, in practice, this means that system D is not always capable of doing what is always capable of doing the grouping at a user level. But currently, most distributions, except Fedora, can't actually apply controllers at the user level. And also, a previous slide. So, C group V2 has a constraint where the inner nodes of the tree can't contain processes. This is to make the counting simpler for the kernel. And the new API also introduced once, basically once a C group has other C groups as children, you can't attach processes directly to it. And these inner nodes are called slices in the system D. And they can also have drop-ins, meaning you can also define configuration ahead of time for whole categories of applications. So the question is, how do you actually leverage system D to manage C groups for when you're launching a new app that you haven't defined ahead of time? And actually, system D allows you to define units at runtime, meaning you... And there are two ways to do this. One is the simpler one called scopes, in which you launch the process as usual. And then you ask system D to target to a C group. And the second version is services, which are basically like your typical demon services. Only they are defined at runtime. And in that mode, system D is doing the actual launching and basically manages the entire lifetime of the application. Yeah, thank you. So what have we done so far? A current rollout. So Plasma 5.19 has every application spawning inside its own C group. And most of this is a tiny piece of code in Keo, in the method that David Ford talks about this morning, application launch job, command line launch job. But we needed a lot of effort to go through this KDE code and make sure we were all using a unified way of launching new applications. And that's something that had become quite fragmented over the last 20 years. So it was worth unifying anyway. And we found dozens of edge cases. I'm sure we might still find a few more where we had to identify these and fix them up. So the goal is to do everything safely. We don't want to have any regressions because we're introducing a fairly sizeable new change into how we run things on Linux. So the deployment strategy is we start using a C groups. And because we're deploying things as scopes, we're launching exactly the same way as before. We're just tagging it afterwards saying this thing we've just launched is part of this group. And here's some metadata about that group with the name of the .desktop file. So we started doing that. We can check it's all working. And only then, when we know everything's working correctly, we can start making more use of this metadata. So what does that mean? Well, the Linux tar scheduler starts using it already. As soon as we've got it in a C group, the tar scheduler can make use of it. We've got some patches on the view for your task manager and that sound icon and various parts that we mentioned earlier. But also lib cases guard and cases guard can start surfacing this information to your user. So we look at cases guard now and this is a rewrite that Arjen is doing and there's a good place to introduce this new feature. We present the exact same information. It can find all of the same information as before. It's just drilled down in a more convenient way. You've got your application name as you would expect to see it in your task manager and inside, you've still got that list of processes, but the aggregation is more reliable than just the guesswork that we could rely on before. And KDE, even though KDE is great, Plasma doing on its own isn't really going to get out far. But what we found out was we started going down this path and then we started talking to Nome and they had very similar problems. They really wanted to focus on the umkilling, out-of-memory killing and making sure that was prioritized, weighted with different metadata attached. So they started wanting to do the same thing and it all came together and we started unifying on a way of tagging .desk files, their application name to your C group names and we came up with a set of different slices that programs and services are going to be running in. So we met up at FOSDEM, met some assistant folks, some external folks, known folks obviously, and had a meeting on this and there's work going on on the free desktop mailing list and so forth. We linked it to the end of where all this upstream work is happening. So it's going to be universal. And I've talked about how it solves problems, but then we can go to the next step and start adding features on top. I said we've got fair resource distribution, well now we can make unfair resource distribution. So what do I mean by that? We can statically assign some weights to various services and programs. There's going to be three default slices which will be set up. The session which will contain Quinn and Plasmashel and things that you don't want to be killed, things that you don't want to freeze or start up, particularly on Wayland where Quinn is doing some very important jobs. The application slices will be competing for CPU and memory only within this one slice. And then the background services where we can just dump everything which still needs to run, but maybe doesn't need to have as much priority as some of the other things that are happening. And then we can apply limits which can take us further and they can be done by application, the distro or the user. So we can have it so we limit file indexes to only ever take 10% of the CPU status. We can set some various open memory stats and we can stop fault bombs from happening. And we can also change resource allocation based on runtime information. So for instance, we can prioritize the currently focused application inside Quinn or any window manager. So working towards that goal I started a small library that just simply wraps the system D-Bus API to expose resource control for all your currently running apps. And this is done in a way that is friendly to KDE applications and actually it's also fully asynchronous and it can be used easily from QML. So while you're welcome, of course to have a look at it and just start hacking on it if you're interested. And well, the library also includes a small demo app that is doing that, meaning it's talking to your window manager to determine the currently running a currently focused application and it just gives it a high CPU weight. So you can also watch a demo of this at the link at the end of the presentation. This is a screenshot of it. So one of the things that we are working on is trying to move to system D services rather than scopes. That way the entire process lifetime is managed by system D. And for our efforts we get improved login. We also get some extra configuration that can be done with scopes. Also it fixes a bug with system D for older versions before 238, where you basically could not group applications in scopes. So we're going to fix that. So we actually already have an implementation for this already. It's currently hidden behind a flag. But we did run into some issues with system D limitations, which would make the experience not as good as it was before. So we also, more work is needed to get the crash annoying to work nicely with Dr. Konke. We've talked about getting applications into your own scopes, your own managed and services to manage correctly. But one of the other tasks we've been doing is managing the rest of plasma. So all of plasma is core parts of the system. So we're going to have to manage the rest of plasma. So all of plasma is core parts and back on services. And we could just tag them in the relevancy group, but it's something we wanted anyway of having entire plasma session managed by system D units. So we get all of this resource management that we've talked about. We get the good logging where you can actually find out the logs from that one process in a really queryable way. But also it provides that familiar management. It's something that was requested by the Munich folk back when they were doing a distribution, because they already have to learn all of this stuff, their system, and then they get to plasma. They want to do some various tweaks, mods, modifications throughout of our services. And we don't want them to have to learn another new layer on top when we can just provide this fairly similar seamless experience. The goal is to do a seamless work. It's going to come in optional, hidden, and then get some more users. And we've made a lot of effort to make sure we get every single feature that used to work working exactly the same as before. So desktop files in your auto start folders. Thanks for some work by Benjamin Berger from Nome. They get converted into system D units magically. And all of the details attention to details with environment variables and such should just work. That's the goal. So we've got some further reading. I'm being told time is up. So I'm going to copy this into your notes and get some Q&A. Thank you so much. I hope there's some applause on the chat. And yeah, you actually have a few nice questions. So let's have a look at what we have there. Thanks for the talk again. So somebody's asking for slide number 20. I'll leave that open while Excellent. So yeah, lots more questions. It's interesting that all this requires changing the code paths via which things are launched. Is there no way to enforce mandatory grouping regardless of launch method? Or is it that groups are managed entirely from user space? Because we want to do everything without any sort of privilege destination. The person mentioning as possibly knows it is actually a way to automatically flip things and see groups, but it doesn't quite work in a way that's suitable for a desktop. I want to focus on these other questions, but we can sort of go back to that. It doesn't. It's just not quite because you've already got extra arguments and such and starts by these environments. It's more complicated. Yeah, chat during Academy. Okay. Why isn't this fancy view on my plasma master built? If you use cases again, I'll add a link in a shared note in the chat and there's a repository. If you look in the plasma mailing list for an email from iron about plasma cases guard, you'll see this and you can just compile it with your KD source build and start running this. Alright, and you're informing the new UI isn't called cases guard. Nice. Yes, plasma dash system monitor. Alright, so from your blog, so David, could you elaborate on how us users can help with this? So if you run system D dash CG LS, which is mentioned on one of these slides, you will see a tree and what you should see is app.slice and you should see app and then a name which corresponds to your desktop file and then things you would expect to see in here. If you see a bunch of stuff inside session doscope, things aren't working and that could mean some things which are pending. We've got a couple of the background services aren't quite done yet. Debuss dot services upstream, but that's going to move into background slice when they merge a patch. But if something looks horribly wrong, it's worth looking at. Alright, so is it going to be built on top of system D minus minus user, user service units? Yeah, it's built on top of that. Alright, so and plans to integrate with OOMD. I mean, generally our goal is to put as much metadata into a system as possible and then that's available to anyone using a system. If the metadata is there, whatever service you want to use should have it. And just to expand on that out of memory stuff is something that's absolutely amazing is in the metadata you can tag how much memory you expect to see a group to use. So if I have plasma shell using 400 megabytes and I have kViteD using 400 megabytes, us users probably can identify which one's wrong because one just echoes some text to another place. Warm is an entire UI. We know what's wrong. How is a system meant to know what's wrong? And just an arbitrary number isn't enough. Is it metadata we have allowed to put all sorts of interesting things in there? Cool. Alright, we are running really short on time now. We can maybe try to do a bit more foreground booster. Is that something the Wayland compositor could do by default? And what about evil apps boosting themselves using Windows API? It should work with Wayland. We've got to stand alone just so it is just for convenience, for your rollout and you have no need for it to be in-quin. And technically any evil app that isn't sandboxed could, but hopefully we're moving more towards sandboxing. Alright, so the next one's question. I'm just going to ask the question ten just to interrupt. There's a question about BSD. Obviously what we're doing is adding things on top. And in terms of all the secret stuff, we're adding stuff on top, we're not taking anything away. So that exists. And in regards to startup, we've done so much refactoring on startup to get to this point that we've completely sped it up for everybody. I mean I've been refactoring KSM server for about three years doing a tiny bit, have a release, tiny bit, have a release to make sure we didn't break anything. And I think everybody has benefited from that even if you're not going to use this. Sorry, I've been talking to you if you wanted to add anything. That's fine. Alright, I think then we are wrapping up. There's one last question maybe. Plasma mobile and phones, do you see advantages there? Yes, definitely. That's great. One of the interests of doing this foreground booster stuff is mainly for mobile. Alright, thank you so much.