 Good morning, everyone. Thank you for everyone for coming and to get up this early, and I'm delighted to be here. It's my second conference. I was here two years ago and yeah, I'm today the talk is Linux boot and booting fast and My name is Paul Menzel and I work at the Max Planck Institute for Molecular Genetics here in Berlin By profession I'm actually a mathematician but I was always interested in free software and I'm currently active in the community of the of core boot the free Alternative to proprietary Biosys or you you EFI firmware and yeah, I work at the Max Planck Institute All right for everyone who wants to edit typos in this presentation right away you can clone this and Yeah, then It's available online okay, so Why this talk at first the warning? I'm more into x86 although I want to change it so a lot of things are on the concern to x86 or on the apply to x86 And why this talk? I was always wondering why Systems boots are so slow. Yeah, you see it in all Every day like your phones you when you reboot them it takes forever your TV if you turn it on even Yeah, and your system of course and you see the buyers Screen like for a very long time and there's no focus on that and that's Kind of strange because those systems are fast and for example if I want for my parents for example to have like a Multimedia system. Yeah, I want them to just turn it on and then To be readily available so that they can have a better user experience and can start right away And that's unfortunately in my experience not possible with proprietary firmware because the vendors and probably the customers and don't put so much focus on it and I had other bad experience like a very long time ago You all knew when you USB devices came Into the market you wanted to boot from those right and the firmware didn't support it So you couldn't boot from those although you had like USB connector and it was just a firmware issue and had nothing to do with the hardware and because the vendor didn't think You deserve it You couldn't get it at least not there's a lot of time So it's a good idea to have free software in the firmware also and the second thing for example currently I have an AMD Ryzen system and The driver in there doesn't support MST displays multi-stream transport displays which have two panels and So when they tell you yeah go into the firmware menu for example, you don't see anything Right, so and I cannot do anything about it besides complaining and the vendor well probably already moved to the next product Yeah, also to be more close to the hardware right x86 The promise is more more or less for the systems in contrast to arm for example where the device tree situation Implementation is going to fix this x86 gives you a common interface I'd say so you that's why you can boot more or less all x86 kernels on any device right in contrast to arm systems for example And that's also interesting to actually see that the x86 Model is more or less emulated for all current devices although they more or less move to To a system on a ships right and there's no source South North Bridge and so on but they kind of still emulate all these old devices so it's quite interesting to actually see how the hardware is set up and To all of you who have Seen like Ron Minnick for example who who is very active in this community and all the other guys They're all very smart and it's really cool to meet them. Although. That's probably true for all For a lot of free software projects All right, let's start with Linux boot So a Lot of them I already talked about but the main Model is we do not trust firmware and we actually do not like it right to update it. You have to have this flesh from ship Updating as the pain most of the time So it would be great to have actually all of these things in the Linux kernel And if there is a problem you can actually just reboot right and update it You don't have to flash anything you reboot and then the problem is fixed firmware is everywhere right all Devices are more or less now little computers right even in the CPU there are components which run Which run separately which have like small control processors or parts in them and For example the management engine yeah on Intel systems You have like a separate device which runs its own OS and it was in the news quite a lot in hard drives as firmware in monitor as firmware you can update or Have to update to to fix bugs right so it's unfortunately everywhere and most of the time although probably everyone here uses Linux The system is actually not free and the problem with this is that firmware has high privileges and for example you if I because it boots first right it can set up all registers and do a lot of things to your system and and Mock to use to the operating system that certain things are What it expects but in the background for example locks your key events or something like this and the operating system And in the Windows world if you have like an anti-virus program or malware checker It's it's totally Obfuscated it doesn't know that this these things happening and one of those things is for example system management mode it's like a way that's a firmware can set up handlers for certain traps and then the operating does something but the firmware decides okay We got the CPU instructions And we know this is a CPU bug So we have we cannot run it and have to rewrite it so the system goes into the system management mode and the operating system doesn't know about it and That's not only a problem security device and there were a lot of Vulnerabilities in this regards. It's also a problem for high computing systems. And that's the reasons Siemens for example in their Devices and their CNC devices which have like 32 axis or 31 They use core boot and because they want to disable the system management mode right away because even two milliseconds delay would Make their product or the the product they work on would make it bad because then the axis And they cut stuff and they would destroy each other All right. Yeah, I already said it's slow. It's often buggy. It's a pain to update. It's often proprietary and Even if it's free software It's unfamiliar code base if we put Linux in the firmware and do a lot of stuff in the Linux kernel That's the code most of us are familiar with or not familiar with but there there's a big community Who is familiar with it who works on it? so in this regard, I think it's good to To To share this code base right and to to use it and for administrators who are familiar with Linux It would be great if also in the firmware there would be Linux because they could use all the tools and That's the next thing right Do you want to boot from pixie for example over wireless? Right is it possible with you if I probably not because there's no driver in there Or you don't have like the command line tools whatever so if you actually had control over the firmware you could boot over Wi-Fi With pixie or do your own boot protocol? Yeah, and as I said, but this also applies to Linux boot firmware is normally a Pain to update because those flash chips are small so you want to have the phone a small firmware image Okay. Yeah that Linux do it It's a solution for the problems. I mentioned. It's a familiar code base. That's also a big reason at Facebook I heard why they want to move to a Linux boot and Google because they have a lot of Engineers who are familiar with Linux developers who are familiar with Linux so They can improve and work faster on implementing stuff and improve stuff It's well tested. Um, let's see if I Say it. Um, there's a cool Metric for this and there are a lot of developers for example in in Edk which is the And Implementation of you you if I and which a lot of vendors use For making you you if I firmware GitHub shows 100 I show GitHub because they do these statistics. So you see 163 contributors, right? So If you do it with all Linux Then There's an infinity sign yes, so the Community is much much bigger. They can work on more stuff. They are more eyes I mean, it's it's not a hundred percent proof, right? But the gut feeling says it's it's good to have more developers And because you get more features and you have more eyes who have spot bugs or can review code And That's the next thing for example with you if I You more or less have an operating system in your firmware, right? If you look at it It hasn't has an IP stack for example for network It has all those drivers and so on and more or less they we have to re-implement everything So why not just use Linux? Do you have do you have support for Braille devices in in your firmware, right? for to to have people So people can use it with disabilities or their tools they want those Braille keyboards Probably not a Wi-Fi is not there you would get it with Linux and if there's enough space So now most of you say well, but I want to boot my operating system and There is a message for this. That's a kexec system call more or less or the kexec program And you can Boot and other Linux kernel for example with kexec and I also think free BSD I'm not totally sure with if it's possible with Windows. I don't think so but It surely could be implemented Yeah in the inner drama as you could put all your tools you are familiar with and Yeah, you just fix issues If the firmware small and most of the stuff is in the Linux kernel on your hard drive for example or which you load over the network You just fix issues there and just reboot Okay. Yeah, so the implementation max makes the firmware small as possible that also improves boot time and the attack vector Move as much as possible into the Linux kernel for easier maintenance and administration and this is called the the boot kernel and the Linux boot ecosystem they Kind of mimics the names of you the UEFI spec there and You use Linux as boot loader with kexec in the user space That was actually already a reality in 2001 and Ron Minick who started Linux BIOS Had a lot of problems with the cluster for example Once there was a BIOS bug for example, which When you rebooted the system and of course at the cluster there is no No keyboard attached. It told you press F1 to continue, right? And for a lot of nodes, that's And there was no keyboard attached So somebody had to go to all the clusters and connect the keyboard and press F1 So it could boot and they could fix the BIOS This is actually a link so for everyone who is interested there were several papers about Linux BIOS and it's so interesting that 18 years ago or 17 years ago all these problems, which are still present today were actually Present back then and it has more or less. Nothing has changed about it. Yeah And yeah, there are some clusters here which used Linux BIOS Yeah, so these are links when you look at the presentation you can click on those Okay, so maybe for this and More or less there was no real interest and probably it was then also a heart without vendor support to actually support new devices so The Linux networks for example who supported Linux BIOS based clusters they more or less went out of business and the big manufacturers they didn't have any interest and so they had the new hardware and everyone moved there for example and Intel and AMD they also Didn't publish documentation anymore, right? So actually the hardware you own without schematics and without the data manual you cannot do It's hard to to write your own firmware Okay, so what's the present so? Another problem was back then that Linux didn't support the PC couldn't enumerate PCI Subsystems well enough. So and the flash chips got too small. It was only like 256 kilobytes So and the Linux kernel got bigger and bigger So Linux BIOS wasn't a solution anymore and the solution was to move the Linux kernel out of it Just have a small firmware and the so-called payload which could be grub or Filos a file loader and this would only have a small drivers and load the links kind of from disco over them over the network So and now we come to the Linux boot project again the present is thanks There's one advantage of you if I because we now have an operate a big operating system in our firmware The the the size got bigger and bigger Yeah, so we actually can now put the Linux kernel into the flash ROM image on the flash ROM ship again Common sizes are 60 megabytes for example Sometimes it could even put the distribution kernel in it one first has to say that IBM also Saw this and since power 8 they use something called PT boot And which is more or less also like Linux boot and it's I believe on build route They built it in a ram of s and also puts a kernel into the flash ROM ship and which is executed quite quite early and So they actually do this already since power 8 which is I think five or six years old So IBM moved there, which is good to know And also a lot of more things for Linux boot will be talked about by Chris Koch tomorrow at 6 30 he actually works at Google on this stuff directly So he will also talk a little bit more on the user space, which is go-based you wrote for example and Also focus a little bit more on the Linux boot UEFI site So what is Linux boot? So we have an image with Linux as a boot kernel and in in it ram fs or in it are the user space environment which Works as a bootloader Yeah, we on x86. It's now possible again to use it due to the increased a flash ROM ship sizes and the idea is To also to to Okay, sorry, let's back up the biggest problem on x86 for example to support a new platform is to write the ship set code And there the biggest problem is the RAM initialization for example all the undocumented register rights which We don't have access to the documentation so In the UEFI case for example, there is a so-called dexie phase I'm sorry the PEI phase and more or less this means after this phase the RAM is initialized and most drivers have run which are dexie drivers and The idea is now to strip down this UEFI image and extend Linux this way that it is Works more or less as a new UEFI driver and then the UEFI firmware Will load Linux after the RAM is initialized so you cut down a lot of these components Which are shipped with a UEFI and then put Linux in there, but it's still not totally free software, right? This would be also possible with you boot and Of course with Corbo to there the goal is to get rid of the RAM stage which runs after RAM has initialized and Because the Linux PCI code has improved a lot put Linux in there, too So these are the interfaces Linux boot tries to build upon or to use and On the UEFI case, I guess Chris will talk more about it tomorrow Okay There are several projects who more or less our frameworks or make it easy for you to have a user space to So that you can boot actually other systems and that's Hats which is found which was started by Tremel Hudson. He also Was Ron Minick started the Linux boot stuff, which was named nerve Non-extensible firmware In contrast to extensible firmware because we want to have a small by Ron Minick and Tremel Which is now called Linux boot So heads more or less uses of some kind of bash grips around KX like and I try to show a demo in a few seconds You root it's go-based The idea is to because goes so fast to just put the go binary in there and all the sources and to If you execute a program, it's compiled in in just in time and go so fast to compile stuff the advantages are if you like a go Expert that you Then also can extend these scripts very easily and it's good to have these sources, right? And if there's enough room why not put the sources in there? So that's what Google I think focuses on a lot As I already said, you could actually also use a petite boot stuff from IBM and Of course, yeah use build route use open embedded They have a lot of experience to make small images for their embedded devices So why not leverages leverage their Frameworks to also make an inner drum FS for Linux boot Okay demo time We tried it a little bit The graphics might not be so good So we have a seven-year-old as rock e3 50 and one system here It's a It's not a very powerful system There's a socketed flash right here Socket is good for development because you can just unplug it and if you put a pushpin on here glue it on It's you can remove it quite easily so we have this and Yeah, there's an old SSD H a HCI Device connected. So I just Try to connect this So the problem is because we want it still fast We probably will have problems That you don't see anything which is more or less a goal, right? but That's the disadvantage of core boot for example to present it as at like fairs or Conferences, right because actually your goal is that nothing can be seen. So Yeah, let's try it. I I started now Okay Yeah, and now Corbett runs the ROM stage Did Daniel do we already see it on it? Okay? Okay, so Yeah, there it will come so heads is on here. Oh, and And that's better. So let's now the heads user space, right? If you want it more fast, you can of course already boot the thing you want to right? but now it gives you a boot menu because you That's used by default and this better for all users, right which use it for the first time so What I wanted to show you I mean, let's try two times so the default boot now There are scripts which look on the hard drive and pass the group configuration fire and to get the parameters for Kexec and Now it Did Kexec and yeah, we all come to booting fast already This will be shown later, it's that is now the operating system, right? So Let's see because of the time. I maybe Show it right now Yeah Okay, so Is So if you cannot see it so the boot kernel actually it run in 900 860 milliseconds, right? We come to this later and the system D This is some deep Limit where it says all services have started finished after one point three eight seconds so the total time is two point two seconds after I pressed why in the heads demo and Because this is core boot and we have we we can capture timestamps and Let's see if I can show you those Yeah That's a problem on this system. I I don't know how It's a exterminator. I'm sorry. I'm if I wanted to show the timestamps I oh That's a I just reboot and show you the timestamps in the in our user space because we have the tools in there one second Yeah, so The problem is even I have to be honest even if the if you had a fast monitor connected The X server actually takes some time. So the system D timestamp you saw there is actually not It's actually not Valid for the user, right? It's more less five seconds Okay, so let's go into the recovery mode So we now have our shell here, right? It's it's now an inner drum fs. I could disconnect the hard drive and this would still work and Yeah, we have LS and I hope we have CB mem. Yeah, CB mem is a core boot memory. It's a small part which core boot Reserves from memory and puts console messages in there or timestamps for example Okay demo didn't work so it would Would be around one second for this or 1.3 seconds But console messages can be seen Yeah, so that's a core boot messages. We see Yeah Nothing for this talk. Okay. That was the Linux boot demo. It worked. I'm and Yeah, I probably won't show This again because of time reasons, but you got the idea, right? I counted the seconds and it was more less like five six seconds, but it was no genome also to be To to be honest With genome I would probably have taken a little bit more time. Oh nice Okay, now to booting fast. So that was the You wash now that was a good start and the demo for for booting fast. So It's possible, right and it's strange the system is seven years old and the systems got faster and faster and My laptop it's still boot slower than this device. So why don't customers want this? For booting fast, we have to of course first optimize the firmware because that's today the problem I mean Leonard is here and all the system defaults it improved a lot right the boot time for your s, but you actually It actually doesn't really matter because your firmware is still too slow right on the current MSI system I have it takes 11 seconds here takes at least six seconds. So Even if you could boot your operating system in one second, it would still be a bad experience In my opinion suspend to rum is is bad and it was actually just a workaround for actually this problem that The boot was so slow, right if you actually could reboot your system in two seconds or so and put everything And and suspend to disk would work, right? People would probably not need suspend to rum even those who want to have their sessions saved So and with suspend to rum came a lot of problems I mean, I believe every one of us had problems with suspend to rum on That that's well not ten years ago. It worked always right on auto support. Okay Yeah, but before with Yeah, but when you used Nvidia Okay, I think he said was Nvidia devices you you had problems. Yeah Yeah, and that's also it consumes energy, right? So I mean for for TV devices multimedia devices it's There were studies if there would be a power off switch on all devices You could save like to you could shut down two power plants, right? so Yeah, so my opinion suspend to rum is as bad and it's just a workaround for a slow boot time and Yeah, so but customers don't request it I guess more focus is on the fastest hardware and not on fast boot time or free firmware Chromebooks actually put focus on it because that's a use model and They actually have boot time requirements that they want their Chrome OS to have started from pushing the power button to The browser in less than 10 seconds and that's why they also use a core boot Due to the boot time, but also to have verified boots. They have their own security requirements and Even on servers the fast boot time would be awesome There are a lot of firmware who initialize the run modules in serial that means although you can do it in parallel and That means all of you who have like a Dell server or whatever it takes like six minutes to reboot it What if you could do it to reboot it in less than a minute? That's Less than a TCP timeout for example standard or 60 seconds, right? so you could current updates would be much easier and Also one thing again to suspend to run it increase the complexity a lot all these problems with suspend to run all these vendors They now have to also test suspend to run So why not put all these effort which goes into fixing and testing suspend to run to just improve the firmware and make booting fast Okay There were now let's go back to the operating system There were efforts to boot it in like less than five seconds This was ten years ago by Aryan Vandevan and Coco out from Intel and there is a LW Linux weekly news article the link is in the On the slide or there is a link here and they used an EPC and After the firmware they measured it and they had to fix a xorg and so on and they more or less managed to have an autolot lock in in less than five seconds or in five seconds, but they had to do a lot of Work to get there, but it's still not the standard today Okay, yeah now due to time reasons Jump over the demo, but as I said, it's it's five to six seconds on this system Was more or less Without patching anything Besides the Linux kernel So what to do to get a fast system in the firmware? Yeah, you use core boot or Linux boot for example, and then you get one second in the firmware So you have four seconds for the rest This device is actually quite slow with one second if using in this case an Intel based system Which is not totally free because it uses the firmware support package from Intel So there's a binary blob to initialize their memory You actually get a boot time from 380 milliseconds, which is half of the the AMD system I have here because AMD open source their platform initialization code and it was more less kind of Fasty glued into core boot, so there are certain things which could be optimized and which is the reason why it takes one second Also the option ROM on AMD has to be run Which is a downside compared to Intel because also graphics cards have firmware and AMD devices have atom bios and It requires the option ROM to be there which is in with Intel It's possible to to initialize hardware more or less natively So that's a different Okay, in the operating system. We should need to optimize the Linux kernel the in a drama fs and the user space so The distribution kernel is too big first, which takes too long to load and secondly There are too much there's too much stuff enabled which you Probably don't need on your laptop for example or on your desktop system and normally you don't switch drives and put it And want to boot different systems right at 90 percent or 95 percent of the time you have a fixed system with a fixed configuration So you actually don't need this flexibility which a default kernel gives you So that's why you should if you want to have a fast system build it yourself There's a Linux command in it called debug which Shows you the initialization time of all the modules which also system the boot chart Shows you to to see how long certain modules took with Cape probes certain devices are also now Profiled and instrumented and From Intel there is a PM minus graph project which has a script called boot graph It's a Python script which traces certain modules with with With F trace and so you actually get how long each function Takes and it's interesting how many m-sleeps are in the Linux kernel, which are over a hundred milliseconds. Yeah So but unfortunately, I think there is not much focus on this because during review or so I would always request if there's a sleep over five milliseconds You actually should document why in the data sheet this is needed or in the specification Especially in graphics drivers. This is very bad Let's come to in a drama for us if you have an SSD you can Get I'm sorry. It's off the chart. It's 53 milliseconds which With LZ for to decompress the in a drama fs and put it in and load it With the standard G zip it was on this laptop system over 300 milliseconds. Yeah, so all of you you have an SSD adept on Debian update printed update in a drama fs to to In they've been unstable. It's supported by default and use LZ for as compression Make it small and put only the necessary modules in this but if you have a static system the flexibility of in a drama fs is actually not needed. So even to Shave off these 53 milliseconds, which is quite a lot which is like 1% if you have like Five seconds or 300 now. Well, which is still quite a lot 50 milliseconds Build your kernel yourself put all the motive modules in there, which you need and then you don't need an inner drama fs Okay in user space Probably all of you have heard from system d analyze and it has several commands like system d analyze critical chain or blame Which shows you which services take a long time There are some For whatever reason these numbers are not always true. I was told so there is a system d boot chart which which you put in the on the as your in it Binary on the Linux command line and which Then collects all these data differently at least a little bit differently and gives you a nice chart I will show it to you the time is almost over, but I will show you this graph in a second Yeah, then Yeah, there are tools as trace perf to to to profile what took so long and Of course the services you don't need you should deactivate it on a standard system in my opinion My experience for example modem manager is used But I never use modem manager because I only use wireless LAN or on a desktop system I have a network cable so I never need these UMTS devices for example to use as a modem Yeah, you should reorder some services those dependencies if I have a local system and don't need my users don't I Don't need NFS or network users, right? The dependency that system D lock in D is on C started after the network is up It's actually slowing this down so one should look into this and find solution for to To support systems which don't need it so it's too general in my opinion system D journal has a flushing which takes long in the beginning and Currently, I'm investigating how to make you deaf faster Yeah, ACP is reality So there's also cool cool code steep graph to debug the corner in this regard and What to do? I just say it really quickly Actually put pressures on vendors complain support vendors who care about free software in the free firmware who? care about fast boot times Power is now also available as workstations from Talos their resellers for older Corbett devices There are newer systems. You have part have parts which are binary reblocks, so if you are Okay, and want to have recent systems and there's prism who actually sells laptops with With Corbett on it For servers Facebook Okay, yeah Yeah Chromebooks Dell supports updates from a user space and yeah how to improve the situation I'm looking forward to talk to all of you in the next on Sunday To hopefully find solution that all users can profit from it So thanks for your attention. Sorry for taking a little bit too long. Oh, no, I started five minutes late. Oh Anyway, there's thanks a lot Paul for this interesting talk So I'm afraid we don't have time for questions anymore But of course we are here for two days so you can of course ask Paul the questions in the hallway track Yeah, so we have You