 So I work on system D. I'm actually a developer and I also make the end system in Fedora and I work on, that's not good, I also work on like creative technologies and one of those creative technologies is the bootloader and the EETRDs that I use afterwards and I'm talking about the project to simplify the way that we construct EETRDs and well to make the boot process simpler. So what is the EETRD? It's a small file system that the bootloader passes to the kernel and the purpose of the EETRD is to, well it has one main reason to exist, it's so that the kernel can mount really complicated storage. So the kernel itself can handle a root file system that is simple, it can handle for example a single partition but if we have encrypted data or radar arrays or LVM and stuff like that it is too complicated for the kernel to mount on its own so it will unpack a very simple system into memory, start the system and then the system starts quickly, mounts the real file system and switches to the real file system and I want to mention that in this talk I'll be using the terms EETRD and EETRMFS and interchangeably and EETRD is an older technology that involves an actual file system image and EETRMFS did away with the image part and we just have a bunch of files that we unpack into a temporary file system in memory and this is our EETRMFS in EETRD. So how does it work right now? We use Drakwood and Drakwood is a complicated beast implemented in a bash and so we have bash scripts managing other bash scripts and also non-bash scripts and Drakwood has functionality that is used to construct the EETRD image but it also has functionality that is used in the EETRD image when it boots. So there are two parts and this can be even split further. So if Drakwood modules are parts that provide some function, for example we would have a module for LVM and we could have a module for networking and stuff like that and there are dependencies between modules so one module requires another one or conflicts with some other module and Drakwood also has so Drakwood provides mechanisms to the dependency resolution. It has helpers to install different kinds of things. A very important one is the ability to install an executable so we tell Drakwood to install an executable and it will pull in the executable and all the libraries that this executable links to so that the binary can actually execute properly but it also has helpers for new dev roles and kernel modules and different stuff and so so Drakwood will execute those scripts they will produce the image and Drakwood will zip up the image put it in the right place and then after we reboot Drakwood also drives the execution it so controls what happens in the EETRD and some of the things that are involved are normal programs and some of the things are scripts and helpers specifically Drakwood. Now this is pretty complex and it kind of makes sense if we look at the evolution of EETRDs over time so initially when when EETRDs were started to be used computers were much slower and memory was slow this was slow CPUs were slow so like unpacking of a large archive to have a large EETRD image would be very painful so people created EETRDs that were very customized so like for example it could be a it could use a busy box and it could have some custom written scripts to do some some very specific things but people wanted more functionality and over time there was a natural pressure to take programs from the host system and add them info to the EETRD so like for example you want to support LVM you don't want to implement the whole thing again so you're pulling the the binaries from the from the system but still the environment was very special and at some point one of the programs that Drakwood started pulling in into the EETRD was system D and if you let system D run in any environment so on a on a real hardware or on a visual machine or in an EETRD it will set up the system always in pretty much the same way so it will like mount slash proc and slash sys and slash dev and slash fpts and set limits and environment variables and the host name and so if you are running for programming underneath system D in the EETRD and in the real host system you are running in pretty much the same environment and so so I would call this almost normal but we still have some custom bits and and I'm well the idea is to remove those remaining custom bits and just let system D handle all of the runtime environment in the EETRD and just use normal programs and normal services that we would use on the host so I want to draw a parallel to to a transition from XOR to Wayland in three aspects so the system that that we have for creating in EETRDs it evolve all time and it addresses problems that that that that were valid 10 years ago but but and computing environment has changed the CPUs are much faster and we don't really want to optimize the size of the EETRD that much we don't need to do that and also if we are using binaries that are compiled is of scripts there isn't really that much flexibility in what you can put in the EETRD anyway so and in some ways like with X that X was addressing problems that were valid problems but the solutions were really complicated and at some point people said well okay let's try with a with a completely new approach much simpler and the push to this for this change was done from the inside so it was the X maintainers X developers who who introduced the new system and something similar is happening here I'm working with the talk but I'm working on this project with Lukash Nekrian and Mikhail Sekretash who are and and Lena Pottering and other folks who help maintain Drakut in in Fedora and Rail and who are Drakut developers and the third aspect that is kind of the same is that the new system and the old system have to coexist for years because well we provide something new but there is a long tail functionality that is very hard to implement and we have to be prepared to use to let people switch back and forth between the two when it makes sense for them of course here the problem with the interdice is way way smaller but I think it's kind of similar so so why is the interdice so special if if we look at this from the point of the view of the kernel actually the interdice is the kernel doesn't care it's just a file system and it's well it's a temporary file system in memory so it's gone after your reboot but you start the programs and do things in exactly the same way and the times before system D I mean for example in the interdice you might not have slash proc mounted or something like that so things were much different but with system D the environment is very similar to the to the real one and in fact we have to use a special file to tell programs that are running in the interdice so we have slash etc in the release instead of slash etc always release and there are like cosmetic differences so for example the kernel will invoke slash in it instead of slash has been in it for some reason and I want to highlight the the fact that there is nothing special that we would want to do in there indeed that we wouldn't want to do in the host system at some point so like if we have ability to to mount a degraded rate array in the interdice environment then we certainly want to have the ability to to mount this degraded rate array in the real system and the same applies for pretty much any other thing and it also goes the other way so you might think that we for example we don't need bluetooth in the interdice environment but let's say that you have a bluetooth keyboard and you need to do some debugging or some type in some password or whatever then of course you want to use your keyboard so you need bluetooth you there was a wonderful talk earlier today about accessibility and if you are using a screen there you also want to use the screen there in the interdice so you want the sound system to be working and so and so on so there is this actual tendency to want to be able to just take any program that you including the whole stack of dependencies that you use in the real system and plug it into the interdice and not not all not at all times but at least optionally yes and I want to highlight the fact that in the past maybe there things are done with scripts but now we do them with with demons or with these compiled programs and we just don't want to implement the things the second time and also we shouldn't try to do packaging twice right if we have a some some program and it gains a new file the maintenance of the package will take care of this but then if the program is used in the interdice then we probably have to also adjust configuration of charcode and it just means duplicate work duplicate packaging work so to simplify things we have those various aspects of functionality of of charcode and we can replace them easily by other stuff that we have so the modules and and dependencies between them we can if we build the in-card from rpms then rpm has a full-blown dependency system and we just use that we have to create the interdice but if we are using rpms and we get this for free so basically we tell dnf to take a set of packages installed to a temporary directory and they create a cpio archive so any tramfs is a compressed cpio archive so there's some additional tweaks to this so you probably wouldn't in practice want to do it this way but it's entirely feasible we have the the runtime in the interdice so the event queue and this is what system d does and it does it quite well we are already using system d in the system d in the edrd so it's more about removing things than adding anything new and all the other functionality well here is this this is more complex the some functionality is only implemented as for example as as rapid modules and it might be necessary to either take those programs and make them into separate packages or or implement them since this is more complicated than in general the idea is that we will use normal programs the same ones that you can install in the whole system in the interd um so why um right it's about simplifying things it's about not remembering everything stuff again and rpm is very good at installing packages and packages are doing i mean a lot of packaging work is figuring out dependencies between between rpm's and and specifying them in a in the appropriate way so we can just reuse that uh a big difference is that's kind of like becoming moving towards more modern approaches to to software development is that um right now we uh when we are building in rd we take files from the host installation and copy them over into the interd so if there is some local modification it gets carried over if there is corruption to one of the um files it is also carried over and the edrd can be slightly different on every system and if we just take rpm's and install rpm's well rpm's have check sums um we produce an imaginable image and the image is reproducible and it looks always the same on every system if we use the same recipe to build it so this is more like our current approaches to packaging we want stuff to be predictable and reproducible and well yes the idea is to to get rid of bash from the from the boot sequence so this parallels the transition to system d with some people like the idea but it was painful to some extent yes things need to be to be something's needed to be implemented and it will also be true here but in the end we get the system that is simpler um and we don't if I mean nobody wants to maintain uh debug track with issues because they have the environment is very special and it's just much easier to to say yes this uh this program has this issue and this is just try it out on a real system debug it there and if it dissolves there it should also be solved in the rd um and the last two aspects are a kind of social uh so uh if right now because the environment is so special it's not clear who should own bugs in the entire d so this is like the the for example if we have something goes wrong with lvm is it lvm maintainers is it the maintainers of track with is it system d uh there's too many parts involved and by simplifying the system we kind of also simplify the handling of bugs and we don't do it twice so you know in the normal community spirit of open source things are immediately reused in other places and uh well the the project to do this is currently implemented as a something called mks and ksi in rd so mksi make operating system image is a program that was developed to test other programs so for example system d during development and it takes a list of packages and quickly builds an operating system image from those packages it supports many different distributions but in particular dnf and then you can take this this this this image and install your program that you want to test on top of that of the image and and for example put a virtual machine with with your install program but for for the purposes of interdiscipline we don't care about the second part we just care about this first part so we take a list of packages and install an image and some some small processing on it and mksi interd the project is actually a set of configuration files for mksi that tell it what packages to install to successfully build any interd that can be used to for example build a machine with a vm and and loss um and uh i looked at some alternatives so qe is a system to build operating system images developed by open susa uh i mean say cross community project and i was built as something similar from red hat um and it seemed easier to to adapt mksi to do the necessary tweaks that are necessary for interdies but i think the main idea is to use rpms and and and dnf and maybe in the future mksi interd will be replaced by i don't know but it's built in here early this is entirely feasible so what's what's the general plan so in the first phase and we are quite early in this phase interd images are generated locally so there is a script you invoke and you get a uh cpr archive and we put it in the right place and well you have the interd so just like with track and once we have that working um and we recover three important cases we want to move to stage two where we centralize the building of interdies right because if the images are producible and every build gives you the same result doing it over and over is uh on every machine it doesn't really make sense and um uh right now if you install a kernel uh the slowest part is actually generation of the interd uh so if we centralize this we will save some some speed on on the end save some time on the end using machines uh and once we have this generation at some point we want to add signing so the kernels are signed the bootloaders are signed and we can also sign a set of interd images and the the end goal uh is that you if you're using secure boot you can uh check the signature because right now with secure boot we check signatures on the bootloader and on the kernel but not on the interd which turns the kernel what to do so it's a pretty big hole uh and in the usual scheme so um well we want to extend this tool to interd these two um and uh so how does it work um well right now if you install a kernel uh the kernel has a a scriptlet to uh that calls kernel install uh and uh kernel install has a set of plugins to different things like uh in particular one of them is to generate the interd so uh uh using output and we have a a new kernel install plugin that will generate an mksi interd image and well it does it by by calling mksi with the right configuration for this for the specific kernel version and uh kernel install puts in puts the the interd in the right place so basically uh if you install the this is not not all of this is not packaged yet but probably with the next version of system d you just be able to upgrade the kernel and if you put the right configuration file in the right place you will get an mksi interd image and it should boot your machine if you like um so the images that we generate um they're not the same as as not as optimized as traffic images they're maybe twice as big and if we unpack them it turns well the contents are about twice as big it's uh you know an extra 100 megabytes is uh noticeable but it's uh feasible for for to work with this and it's not too painful um we have been doing uh we have been doing optimization work on system d for example to reduce the dependency tree of libraries but in fact it uh affects jacquard and mksi interd images in a very similar way so so the duration will probably not change but if we look inside of those images and compare that change um the big difference is that mksi interd images have lots of kernel modules and um um what the general idea is that instead of uh hand picking which uh modules should be installed we just install everything from the kernel core package and the module not the kernel itself uh and at some point i hope that the kernel will split out a um kernel modules in interd sub package or something like that uh because the kernel maintainers are better positioned to to set to to know which modules should be should be part of the mkd and i know if the kernel adds or removes modules in different kernel versions uh this should be done at the level of the packaging of the kernel uh and not externally i think so uh well there's this big difference but i hope it will be uh changed by kernel packaging at some point and if we look at slash bin and slash sbin uh so the the actual executables there's maybe two times difference um and again this is because jacquard hand picks uh which executables to install and here i'm just installing a set of uh rpm packages that are necessary and um so so there's there's a bit more stuff but it's not enough to matter what takes a lot of space is the dependencies so the libraries and it turns out that already we is kind of the same in both cases because um you have lots of programs but they share most of the dependencies so if you install at least some of the programs you end up with pretty much the same set of libraries and um then there are some some more complicated differences like uh i don't know three megabytes of licenses and maybe we could remove them but if at some point we want to distribute the interd as a separate thing then we according to um to the rules we'll probably have to include the licenses they compress pretty well so maybe it doesn't matter that much um then we have some nifo files and then we ask to yourself does it matter if the interd is able to display local time properly um probably it should uh is it worth five megabytes i don't know that's a tough question and and so on so on um and one big difference is that charcut interd's do not include the hardware database so when you put the machine and you for example you have a keyboard which requires some keys to the remap because they are labeled improperly um you will not get the stream mapping in the interd i think that this is not a good idea i mean it's it's the hardware database is pretty large and a large part of it is um just descriptions of hardware that you will not use in the interd but some of the stuff is really necessary and uh if we add like more user-facing functionality into the interd then most likely we want to have like the full hardware database description and the full udev rules so that things behave the same as in the real system so maybe we'll just end up with those slightly bigger interd images maybe it's worth to pay this price um and so i kind of lost over this problem but the reason why charcut builds images um locally is that you want to do modifications and variants and so on um and the partial answer to this is a system called system dc6 so system extensions it's something that works on the host system and it also works in the interd it's a system where you give a part of an operating system image to system d and it mounts it on top of the the real file system using overlay fs so uh this sounds kind of complicated and so let's say that you you want to have um you have a system without sshd and you want to add sshd so you take your system and you create a and slash a system a partition with slash user that contains the contents of the sshd package and maybe any other packages that sshd requires uh recursively that are not part of the of your base image and uh you create this um this this this this image on its own it's not usable right because it requires other stuff that is not included so it's it's pretty small and the only way to use it is to mount it on top of a compatible base and and so the idea here is that we'll build system sys x for the interd to i don't know add um like rescue mode or uh or bluetooth or sshd or sound or sound with a screen reader and so on and then people will be able to mix and mash to get the appropriate functionality and we will not have to build a hundred different variants of the interd because if you want to do it centrally you kind of have to cover most functionality with some um pre-built set of things and um the important thing about sys x is that they support um embedded signatures so uh a normal way that the sys x is delivered is a um that you have a disk image with a gpt table and it has three partitions one partition contains the file system with the with the data that you that they do the actual contents and then you have a variety partition that specifies hashes for the data partition and then you have a third even smaller partition that contains a signature for the uh variety data so system d will um before it uses the extension it will verify the signature and it's correct and it matches the the entirety data and then the kernel verifies that the data partition matches the variety data and and it's a bit complex but it's efficient and flexible so uh before I move to the implementation so let me reiterate so so the idea is that with in the end uh with secure boot you have this approach where uh the the bootloader well the the firmware verifies the bootloader and the bootloader verifies the kernel through a signature and the int id for a signature and then we we put the int id and the int id uh wants to add extensions and it will verify the signatures on the extensions and then mount them so it's all everything is verified before being run so uh we have uh added support to build uh sys x with mksi 2 you always build an extension on top of an existing system so uh we kind of repetitive process that is done for interdies we create a temporary uh directory putting the the packages for the base layer this is pretty quick because it's just it's really a 100 megabytes or so and then create an overlay fs it starts with the interd as the lower layer and then we put the an empty upper layer initially empty and into this upper layer we inject packages whatever for example sshd and the dependencies and then we discard the lower layer and the upper layer because becomes our data partition for the sys x and then maybe we add verity and the signature if you want um and well i mean this this mksi does this kind of automatically a specific list of packages and you get the extension so uh what works right now um you can create interdies um if you install the kernels templates you can also get the interd generating automatically when you install a new version of the kernel um you can build extensions you can load them and like basic laptop use cases are covered um we have been using this on our laptops for three months and it seems to work without too many issues um you can put virtual machines and lvm locks this is this all works um emergency mode also works hibernation and resume um french checksum style has been uh working on adding automated testing for this uh and one of the things that we quickly found out is that ice fuzzy does not work there's some uh supporting track with to to figure out the kernel parameters for ice fuzzy and we don't have an implementation for this that is separate from track with um and uh i won't go over the list but there's like a hundred different things that uh might work or might not work we don't really know until they have been properly tested uh but in principle everything that works on the host should also work on the should work in the interd so it's more of a question of um checking things and figuring figuring out if there are any bugs rather than creating anything from scratch so um to summarize um so what are the main ideas build any interd from system packages uh let the um normal package installation mechanisms that do the complicated work and in the interd itself let system d um take care of the whole uh runtime um so basically remove things um and do everything in the interd the same as on the host and well use the uh various system d functionality for extensions and um signature verification and and uh if that works we want to move to the to the next phase where we kind of centralize the building of interd images and at some point also allow people to to uh prove this into secure boot and those are links to documentation also for for uh dm variety and overlay fs and i think i've time for some questions so should i start with the first question um jaco is used in all kinds of distributions also many outside of the rp universe how would they deal with this if their packages are not managed by rp well it's a beautiful question i'm sorry sorry hoping just kick me out i'm yeah so i'm i'm saying the first question um mksi is distribution independent uh it supports dnf uh uh debjan ubuntu uh gen two alma linux uh rocky linux and arch east and maybe photon so if this works for rpms it should also work for all those other distributions uh because right now we have the situation where uh some parts of the stack are shared between distributions so in part the system d and all the units and the higher level programs and the the interd generation is one of the few remaining things which are significantly different between distributions and i think that with this approach we can unify and if we convince other distributions to use it but we will see okay so next question is should we start moving this approach and how could we can prepare for this so um that's i don't have a very good answer if you if you go to the mksi interd project on github we have a a list of to-do items um and in particular it means you know testing with uh different parts of the functionality and kind of improving the the coverage and figuring out what works uh so we have to finish this phase first and then once it actually all works then we can kind of move with the um further phases so i think that you cannot really prepare in any way for this except that if you want to get involved with with the testing and in development then it wouldn't be very welcome okay next question is from Timoti how do we support local configuration changes and have those signed verified yes so um in general we try to handle local configuration for the kernel command line um and the the idea is that um well basically you somehow verify your kernel command line and this is the the interd is is generic and supports all kernel command lines there are there are some mechanisms to kind of similar to the sysx staff for additional data for example um local secrets and and the the general idea is that you store the secrets uh in a file in that is that the bootloader passes to the to the interd uh and those secrets are um encrypted and they are encrypted for the tpm so to protect them from an offline attack um and but this is for for for secrets more for secrets and for configuration the configuration is usually done for the kernel command line but um maybe this will not be enough and then we will need other mechanisms to to inject more configuration to the interd it's an it's an open question to some extent okay thank you for your answer a question from a chat so the eventual goal would be to make the init rd another package installed along and along other packages yes yes the the the interd images would be built and you it would build the kernel in colgi and then you would this would fire off a build of init rds and then there would be like another package um and then the package well like right now the kernel package uh it has a kernel in user live and then there is some some hook to take the kernel and copy the boot partition and this the same would happen here okay another question from you if um mc init rd is used by default then is it possible to drop system md udef trigger dot service which triggered at events for all devices possibly already processed by udfd previously so the thing is that we will we will not have all packages installed in the interd so the it's possible that you have more roles in the host system uh so we want to offer this situation where we have different roles but like special roles in the interd but you always have to be prepared for this situation where you have more programs installed so i think we will need to do the trigger anyway because we need to handle this case but maybe you can get rid of it i don't know okay thank you and the last question i mean instead of re-triggering an event you events is it possible to export udf database from init rd file system to switched file system um so i think that right now we um when you have stops uh it um and we want to restart udf and the version of udf that is running in the host system might be a different version so it will want to rebuild the database anyway um maybe it would be possible but um i think that udf um well wasn't really i mean it's not prepared for this um we probably could do it but if we have to re-trigger anyway for for additional roles uh then um this optimization wouldn't really give us that much um so uh like with track would we have the situation where we uh completely dropped the the database in the entirety and this part is removed it kind of lives until we restart udf um so this is and we are already handling things a bit different uh and maybe we could even make udf more continuous like with system d right system d serializes the state and passes it to the to the registered instance but this would require some refactoring of udf uh i don't know it's uh hasn't been uh we haven't really discussed this upstream okay thank you for your answer and thank you everyone for your questions it seems that this topic interested a lot of people uh zbishek will be in our virtual virtual venue also work adventure so you have a link in a chat so feel free to hop in there and discuss this topic more with zbishek and thank thank you zbishek for your uh for your um presentation it was great have a nice day and everyone uh enjoy your lunch lunch break and there's a lightning talks in room one and also a session to a little bit of move your body around before lunch uh in main stage