 I want to talk about some changes that have been happening in Fedora and the slides are online. And I'll start with a disclaimer that I'll be talking in a way that is from the point of view of system D because I work on system D and often I will use GNOME as an example, which is because I use GNOME myself and I know it relatively well. But the idea is to let different desktops and distribution collaborate. So please treat this as examples, not as the only way to do stuff. And well, I want to talk about changes that connect different components in the distribution. The high level and the low level, very high level things like, well, the GNOME terminal that is in the title and everything in between and then very low level things like the kernel, the bootloader, the system manager. And some people say that different things should do separate jobs and shouldn't care too much about each other. And I disagree. And in this talk I want to show examples of things like this that are happening right now and explain why they're done like this and why I think it's a good thing and why in the long run I expect that we'll have even more of such integrations. And some of those things are kind of complicated and the implementation at the bottom is often quite complex. But I think it's easier to start from the bottom because the stuff that is at the bottom of the stack of components, the low level components, they usually have a clear goal. And since I'm not talking about the implementation just about the purpose, then it's easier to start at the bottom and go up. So without further ado, let me start with something that is called boot blessing. So boot blessing is a name that is used in system D. Some others call it a bit different. But in general, the idea is that the bootloader boots a kernel and assuming that the kernel loaded from this client and the boot started, the bootloader cannot know if the kernel booted successfully and if the boot was actually useful for the user. So we need to create a feedback loop and we let the user space, that is part of the boot, decide if the boot was a good boot and feed this information back to the bootloader. And then the bootloader can take some action based on this. And we have two implementations. Since most people use Grub, there's an implementation in Grub and a competing implementation maybe in system D for the bootloader as the boot. So let me talk about the Grub implementation first. It's quite simple. It has a counter that when we start booting, we bump the counter that says that we tried this boot and we don't know yet what the result was, boot indeterminate. And then there is a user service. So a service that runs as part of the user session that has a very simple job. It consists of two parts. There is a timer that waits two minutes and after two minutes have passed, the timer starts the service that does the actual blessing. And this blessing consists of calling a Grub binary to set a Grub variable. So this is conceptually very simple. And it actually works. This implementation was approved as part of two changes. I think it was Fedora 29 hidden Grub menu change. And then in Fedora 30 or 31 flicker free boot. And the effect of all this is not very big because Grub just hides the boot menu by default and shows it if the last boot wasn't successful. The competing implementation in system D is more complicated. So system D is known for complicating things too much or at least a lot, and this is no different in this case. It also has a service to bless the boot as successful. The difference is that the service works as the boot loader, not for Grub. And it also essentially sets a variable. The implementation is different, but the effect is very similar. But that's pretty much all that is similar because the service is enabled by a generator. So here we have this first point of flexibility that the service is only active if the boot was used to boot the machine. That may be a minor point. The complexity starts in how this service is activated. Instead of just saying that the service should be part of the transaction statically, we have a level of indirection. We have something called boot complete target. And this target works as a synchronization point between services that do the blessing. So our example is the system D bless boot service which runs after the target. If Grub was converted to the scheme, then we would have something like Grub boot success service after this target. On the other hand, before this target we can put arbitrary services that do checks on the boot and tell us if the boot was successful. This is all nice except that system D actually doesn't provide any useful service to run before boot complete target and do a useful check. There is a system D boot check no failure service that does what this name says, but that's not enough to figure out if the boot was successful. So let me run through two failure modes. So the first one will be where kernel crashes in early boot. And both implementations do the right thing here. We mark the boot early with some appropriate counter and we never get to the phase where we mark it as successful. In case of system D because we never complete the transaction successfully. In case of Grub because the timer doesn't have enough time to run and we never start the service. But let's consider a different failure mode. The user logs in but the mouse and the keyboard is broken. This can happen. This is nasty. Or maybe the screen is garbled or something else that causes the user not to be able to use the machine successfully. Well, two minutes pass and the Grub timer fires and the Grub entries mark is good. The system D service does the same thing. It has the same effect. Everything finishes successfully. All services are started. No errors are reported. It's just that the session is not useful for the user. And so we have this feedback loop where we talk to the bootloader from somewhere in the middle of the stack of things that we start to have a graphical user session. But I would argue that it's actually too low. What I would like to see is that GNOME session itself would be smart enough to figure out that the user is, I don't know, not opening any windows. So maybe even though the session started, it shouldn't be marked as successful. So we want to have feedback from the very highest levels back to the very lowest levels. I also want to mention that the system D implementation is more complicated in one more way. It does boot counting. So we install a new kernel and we decide on some number, three or five, something like that, when the kernel is newly installed and we create the boot entering the bootloader for it. And then every time we boot this kernel, we decrease the counter by one. And once we mark the boot entry as successful, then we stop decreasing it. We set another flag. This allows us to create an automatic fallback so that if the machine has a new kernel installed, the kernel is booted a few times without success, we stop booting this kernel. We try to boot, for example, the previous kernel. Then ideally, the machine would start successfully with the order kernel. Maybe a board would kick in, figure out that something was crashing in this newest kernel version, upload it to Fedora servers, and then Fedora developers would be able to fix this and then yet another kernel update that was installed would now work correctly. And the user could have this auto magic experience where the bug is handled automatically and they don't even need to do anything except maybe press the reset button a few times. But for this to work, we would need to have everything working much more reliably. What we can do right now is that we can show the menu if the boot fails, but doing something as complex as automatically falling back to an older kernel, we are not there yet. But it's within, I mean, we pretty much know how to do it. It's just a question of integrating the right bits. The second topic I want to talk about is system D units for the user session. The units started by the user manager for any given user, and not units started by the system manager for the system itself. And what has been somewhat quietly happening is that GNOME has been slowly converting to this mode where everything that GNOME starts started as separate user units. The way that this happens in the low-level implementation in GLIP, GNOME shall use GLIP to, I mean, has always used GLIP to start processes. And now instead of starting the process directly, it does a debuts call to the system D user instance and tells it about a job that needs to run. And this lets system D user instance manage the processes. So that's, by itself it doesn't do much. It's a, because the processes are still started and they run pretty much the same. System D is not actually doing too much monitoring. On the other hand, it's a big change because it's a change in philosophy. The low-level bits in GLIP need to be changed. There's, I put some major requests in the slides. But the important part is that system D has this generic infrastructure to manage groups of processes using C groups. And in the long run, this will give us a lot of power in how we manage services that are part of the user session. I wanted to show an example. So that's a bit unclear. But on Fedora 32, pretty much everything that is started by the user session is already in individual C groups managed by system D. For example, my Firefox with all its processes using the CPU is in one control group. I also have a flatback instance running Telegram and so on. So why do we want this to happen? So three items. Obviously system D is very good at keeping track of processes. This is why we have it. System D also allows dependencies to be declared between jobs. So finally, instead of trying to start different processes as part of the graphical session and hoping that we got the order right, we can declare real dependencies. We can declare failure modes. We can do more stuff in parallel. And let's say that we want to start the music player after Pulse Audio has started. And by after, I mean not just that the Pulse Audio binary has been spawned but that the Pulse Audio binary has reported to the manager that it is ready to handle requests. And this is what system D does. And so far, this was partially duplicated in the graphical environments, but it is better to do it in system D. It actually makes the graphical session a bit simpler. I also want to mention, as an aside, the system D XDG, so the system D cross desktop group Autostart Generator. It's a little generator that converts desktop files in the Autostart directory into system D user session units. And it's a pretty cool thing. So we'll have this mode where we will be able to start things as part of the graphical session Autostart and actually use dependencies for them and error handling and stuff. But the third reason is that we can assign resources to control groups. And I want to talk about this because this has been a really big subject in Fedora recently. So a short reminder how the kernel manages resources for control groups. In the C groups V2 world, we have high level controllers. CPU, memory, IO are the three important ones. And one of the changes from before is that those controllers try to provide a simplified, even if not simple, interface to the user space and hide all the complexity underneath. So before there were many more controllers and there were many more knobs. Now we have a much simplified view of this. There are weights and limits. So weights are for stateless resources like for example CPU. And we can specify that we want to divide the CPU at any given time with some weight between different tasks. And there are limits which specify how much of a given resource can be used, which is mostly useful for stuff like memory where we say this, I don't know, the graphical session should not use more than two gigabytes of memory or whatever. And we have limits which limit from the top and we have protections which kind of do the opposite. If a unit is below its protection level for a given resource, this resource will be taken away from other units before it is taken from this protected unit. And the two obvious examples of this is the, well, okay, sorry. A good example of protection is this change that was merged for Federa 33, which is the micro resource Dmon which will set the protections for the active user session. So we have all those settings for all this kernel magic and system D exposes it for its interface. So the graphical session can talk to system D using D-Bus and D-Bus can talk to the kernel set the appropriate kernel knobs. And I want to emphasize that this is kind of similar to this previous case. The kernel is doing the low level job in this case. It's very good at managing resource allocations. But the kernel doesn't know it and cannot know what processes are important to the user, how to divide the resources. So we need to have high level components that run as part of the user space and that feed information back to the lower layers. And one example is the U resource D or micro resource D. It sets the protections for memory, CPU and IO for the active session. So it watches which session in the sense of the login session is active and reserves 250 megabytes of memory for the desktop. And the kernel cannot ever do this because it doesn't have a concept that would allow it to tie the display manager and the GNOME shell and various parts that form the session. This large group of processes is together important. And so this information needs to come from the higher layers. Another change that is actually already happening in Feather R32 is the introduction of early UMD. And for various reasons, some people are very unhappy with this because they think that it should be enough to have the kernel UMKiller. But the kernel UMKiller, first of all, it doesn't know. I mean, it has some heuristics, but it cannot ever know which job at any given moment is the most important for the user. And when we have sessions where people login and then log out and leave some jobs behind. This creates this dynamic state which is too complicated for the kernel to manage. First of all, and second, the UMKiller in the kernel, its job is to protect the kernel and to make sure that stuff is happening. So if we try to open too many Firefox tabs and the machine is swapping heavily from the point of view of the kernel, work is being done all the time, so everything is fine. But of course for the user who can see how the cursor slowly jumps around on the screen, this is not very useful. So we need to have some component in the user space which helps the kernel do the job. And this is a common theme, right? And in all those cases that are flexible enough to handle the user cases that we want, we have this interaction of high level components with the middle layer. Because the high level components, first of all, they shouldn't talk directly to, we don't want our GNOME shell to talk to the bootloader directly. So we want to have some intermediate helper, which is usually system D or various helper services. Also, the users, the top level components, they don't have permissions. But in this middle, this intermediate layer in the middle, it needs to be flexible enough to accommodate various use cases. On the other hand, the implementation itself needs to be done at the low level. For example, in the case of the bootloader and boot blessing, it's the bootloader itself that must do the counting because the boot sequence can be aborted at any point in time. So we must set the appropriate counters as early as possible. And similarly for the resource allocation stuff, the accounting needs to be done in the kernel so that it works as a, it's always in place that the machine never hangs even if there's a sudden spike. Any kind of user demon can always be overwhelmed. And also, the kernel is the place where the counting can be done efficiently enough. And the kernel is the place which can react quickly enough. And some people say that this is a violation of Unix principles. And in a way it is. So, in a way it's not. So it is not in the sense that we do have things that are doing one thing and, well, they're doing it well, or at least better than before. If we let the system de user session. Manage jobs and we remove this this handling from from the graphical environment. We simplify the graphical environment and let it will actually split split how things are done. On the other hand, the Linux environment as a collection of little processes that that that don't really care about each other and only do one thing and don't communicate. Well, this is this is going in exactly the opposite direction. And I'm saying that if we want, if you want to build solutions that are featureful enough, we will need to go even further in this direction. And I want to underline the fact that in some way. All of this was possible before I mean even 15 years ago we could write a script that would set some. The user would click an icon on the desktop and this would launch some script and the script would do some job. But those were ad hoc solutions. If we want to have systems that are reliable that are flexible and support many, many, many use cases. We need more and more flexibility and more and more, more features in the in this middle plumbing layer that provides that connects the high levels which set the policy and the, the, the low level implementations that that do the actual job. And Well, that's that's what I wanted to say about what was what has been happening up to now. And as the last few what will happen in this area in the future. So we when we look at the way that the user session is managed is undergoing a similar transformation as that which happened for system services with the introduction of system D. So we will have this this drone out process where we paralyze things where we figure out what are the dependencies between different services that run as part of the user session. And then we figure out what resources are needed to to to to keep various services running and and the introduction of the early umd and micro resource D. It's probably just the first step and we will most likely introduce a system the umd in system D. So there will be a demon which will whose job will be to kill stuff when there is not enough memory itself will probably take over part of the job done by micro resource D and possibly set protections. I think that's that's those are the the login D is probably the right place to do this kind of thing. One thing that will be different in umd is that umd corresponds to the live information from the kernel about pressure on resources. So instead of just looking at how much is used we can look at how much the over allocation of resources is slowing things down. I don't want to talk about the details. I just want to say that instead of having a policy that we kill things when the system is running out of in out of memory for some definition of that we we can do things like no kill a background processing task which we know for a policy set by the user that is unimportant when there is any kind of slow down. Or we would say that tasks like the user session the graphical user session should be protected at all costs. But we have even more fancy solutions possible. We can freeze tasks using the the the freezer when there is memory pressure. Or maybe freeze tasks when the user logs in. I mean the future is open. And there are really interesting things happening. The fact that Fedora 33 has so many changes in so many different directions. It's I don't think it's an accident that I think we are at a point in time where we can do really interesting things. And the good part is that this is done through cross desktop cooperation and cross distro cooperation from the beginning. We shouldn't have the situation where different desktops implement the same bits in slightly different ways. And I don't know. I think that this will allow us to to build a more variable and more feature for graphical workstations and other kinds of systems. And I know that's that's pretty much what I had to say. They forget anything important. I know probably not. I don't know if there are any questions then I wasn't watching the chat too much. Sorry. So please please ask questions in the chat if there's still a few minutes. Are there any plans to support other bootloaders so maybe I didn't make this clear enough that the boot blessing framework is flexible enough to to support easily support any kind of bootloader if implementation for this bootloader is provided. So this part is easy. On the other hand, SystemD itself will not provide support for other bootloaders. There is this implementation for SD boot and I mean, this is already too much for SystemD. I guess if it shouldn't be part of SystemD actually. So yes and no. Okay. And so if there are no more questions, I will finish before Mosquito killed me.