 Hi, I'm Colin Watson, this is a recruitment drive so I'm glad to see quite so many people here, that's very useful. Some high over the years I've found myself as the, oh it's a little bit of feedback. Okay, is that any better? Oh nice. Some high over the years I've found myself as the de facto grub maintainer in Debian and I'm also a grub developer and I've come out over the years. And I'm here to persuade you all that this fun thing to do and that you can help because I need more people. So, yay. Grub has, it's moved on a long way from its beginnings. It's called the grand unified bootloader which is a bit of an aspirational name and certainly was an aspirational name back in, back in 1995. And back then most of us just used it over Lilo which is the really traditional x86 loader because you didn't have to remember to reinstall your bootloader when you installed a new kernel which is really annoying. But even so, quite a few Debian developers have hacked on Grub over the years and nowadays it's a very powerful bootloader. It's been ported to many architectures, it's actually quite rewarding to hack on. So I'll be giving you a tour of its history and of its designs suggesting some particular areas where we could really do with help. Now the Grub project which, this is Grub 1, it was originally just called Grub and we now call it Grub Legacy. But it was started in 1995 by Eric Boleyn. He was initially trying to build something to boot the GNU Hurt. And among the loaders of the day, it was quite unusual that you could edit its menus and fly and that sort of thing, it had an Emacs or Bash style interface to that. Other loaders usually just let you append kernel command line options and that was about all you got. Grub even then had a reasonably capable file system interface as well. So it was bad, it was by and large good enough. So many people adopted it as it was. And of course it was originally designed for the Hurt to start with. So it started out with a focus on the new boot method they designed which is called multi-boot. And the history file tells us that they were determined not to add to the large number of mutually incompatible PC boot methods. Grub did soon become a little bit more generic and supported the methods for booting Linux and such like. And now I'm being a bit unfair, multi-boot has been genuinely useful to people doing academic experiments with kernels different from scratch or other custom payloads because they don't have to do all of that work from scratch. Anyway, I wouldn't cover multi-boot further. This is a rough layout of Grub legacy back in the day. You had stage one. This was a tiny thing that fitted in the 512 bytes master boot record. It just knew how to read the first sector of the next stage along, stage 1.5, from a fixed location and jumped it. That was all it could do, it was really stupid. There was a separate stage 1.5 for each separate file system type that Grub understood. And that had enough fuzz from code to read stage 2 from an ordinary file system. So it's a usual bootstrap your way, gradually up the stack thing you have to do. The rest of the real meat of the loader lived in stage 2 and that included all the command line commands, configuration, file parsing and the code to actually boot a payload like a Linux kernel image, which is probably what we wanted to do with it in the first place. So far so good but there were quite a few problems with the design as it was initially put together. The fuzz from abstraction was pretty ad hoc. It was achieved by building a separate stage 1.5 blob for each file system that Grub knew about and shipping it in the package. The terminal abstraction was sort of reasonable as well but other than that there wasn't much in the way of internal expressive power going on. So it was very difficult to extend it safely. It's hard to work out how Grub legacy could ever have been taught to handle LVM for instance because you would then have to have a stage 1.5 that knew about each of the fuzz systems on LVM and so on and you end up in a complementary explosion. So it wasn't very elegant from that point of view. There were lots of PC BIOS assumptions as well. It may have been ground and unified but it wasn't really particularly portable at that point. Federer did manage to get it to work with UEFI, actually probably Red Hat then managed to get it to work with UEFI but it wasn't a total straight forward exercise and it was very hacky. So as time went on upstream maintenance kind of fell by the wayside. It was such a pain to work on and even though there wasn't really anything newer that was usable, nobody was really interested in it upstream so the distributions did what they had to do and this meant that we have ended up with Grub legacy packages in Debian and Ubuntu and Federer and SUSE and all the rest that are actually completely different products really. And I don't just mean have a different patch set in the usual way that you get, they have completely different installation and configuration tools that don't exist in the multiple branches. So today it's basically impossible if somebody comes to us upstream to support Grub 0.97 because it just isn't enough of a thing by itself. You have to figure out what distribution people are using and tell them what to do for that. So we end up punting a lot of that to the distributions to support. So there was a Grub 2 rewrite started in about 2002. It's a very rough timeline of it. As you can see it was a pretty long project, pretty slow project. I hope I pronounced his name correctly. He was the lead Grub maintainer at the time and he started work on what he called the preliminary universal programming architecture for GNU Grub or Pupa. So it's GNU, there's a bad pun in the acronym. A Pupa is the next stage along from a larva or Grub in an insect life cycle. So yeah, I'd say it was probably roughly usable by about 2007. So there was a long period when it was very experimental. And my pretty biased viewpoint is that distributions started adopting it properly and starting to present it to users probably in about 2009. A snowball effect kind of kicked in from there and it became much better quite quickly until we finally got 2.0 out in 2012. So I also noted in here the point where the first non-X86 architecture port showed up. That was the PRPC port for New World Max in 2004. Because that's a very unusual milestone in something as architecture specific as bootloaders have historically tended to be. By now if you look at some core bits of Grub 2.0 you can kind of see the echoes of bits of Grub legacy in there. There are a few common lines of code but it was a pretty sweeping rewrite and there isn't really much left. And this is a very large project by most standards. The leadman tenorship has changed hands three or four times along the way. It took a lot of forward thinking for people to get involved early on. So the design that we have now is around a small kernel that offers core features like initialization, virtual file system layer, module loader, almost everything's in modules. Sound familiar? There's an insmode command to loader module. It's considerable inspiration taken from full fledged operating systems here. You mostly don't have to worry about much of this by hand. Grub install, which is the main installation tool, works out the bare minimum set of modules you need to read everything else from boot Grub and builds those into what we call a core image. That's the rough equivalent of what stage 1.5 was in Grub legacy but this time it's built dynamically rather than all being shipped statically in the package. And also at run time the loader will load some modules into itself automatically. So for example if you request a command name that isn't currently loaded then it will go off and load it for you automatically. But this architecture makes it a lot easier to construct images that fit into a constrained space in a much more flexible way. We make very heavy use of abstraction layers. It's rather like the way, not quite identical, but rather like the way that a Unix-like system presents block devices and file systems. And those are generally composable. You can quite easily boot from XFS on LVM on RAID, whatever. There's even a loopback module so you can treat a file as if it were a disk, much like we have in Linux. So this makes it quite easy to do strange things like you can embed an entire Debian system in a file on a Windows system, which we actually did with Ubuntu for a while. And it's quite scary, but it largely works. This design also lets us easily build user-spec tools from very near to the same code, which is very important and I'll come on to that later. And as you can see from the random stats at the bottom, there's very little assembly in this. It's almost all portable C. Even some of the, almost all of the C is obviously common. The assembly is, of course, more architecture specific, but for PC BIOS, we're only talking about four and a half thousand lines, which is considerable, but not too much of a mint in this burden. So here's a brief summary of what architectures we have now. We have a number of X86 platforms. The newest of those is that you can now use Grubb under Zen power virtualization with a bit of effort. The newest architectures are ARM and ARM64, which we added last year. In practice, only some ARM32 devices actually work with this, because they require the platform's U-boot to be built with the U-boot API so that it can act as enough of a firmware for Grubb, and a lot of them don't actually have that turned on. But it's at least a start. I believe that in theory, all of Jesse's release architectures other than S390X have at least some level of Grubb support, which is pretty awesome. Now, in Grubb Legacy, the hardest and also the most common problems we had to debug were problems with reading files from disk in one way or another. So people had problems loading the kernel and init.rd, because their file system was in some strange shape. And if you wanted to debug this, you were usually stuck with trying to set up something that roughly matched in an emulator, and either using just crude printf debugging, or in extreme cases, trying to attach GDB to the emulated machine over QMU. Some people in the audience may be familiar with this routine. It's not very much fun. So in Grubb 2, as I said earlier, we have much of the same code built into utilities called Grubb Probe and Grubb FS Test, and these use the operating system's block device as a back end. So you can ask Grubb to use its own partition table and file system tools or parsing code from right there in user space. This makes it very much easier to attack lots of common problems. You can use GDB, you can use Valgrind, you can use the debugging tools of your choice. Tom Marble was asking me earlier about F2FS, and putting that together in Grubb 2 can be done almost entirely in user space. So you can put together a new module, build a version of Grubb Probe that's linked against it, and iterate until you get something that works, rather than having to constantly reboot, even having to reboot an emulated machine is a lot of work. So a similar trick also gave us this spin-off benefit of Grubb-Mont, that's a Fuse file system implementation that uses Grubb's file system code. So that gives you a guaranteed true read-only-mont using the exact same view that the bootloader will have when it tries to look at your system. This lets you avoid the caveats that apply to things like journaling file systems in Linux, which it turns out you sometimes can't safely read-only-mont, or sometimes the kernel will simply refuse, and it's also useful for things like OS Prober. If somebody felt the urge to make that work as a hard translator, that would probably be a pretty good fit too. Now you can't do everything in user space, at boot time, Grubb gives you a pretty nice bash-style interactive shell with runtime controllable debugging levels, so you can set the debug variable to various things, and that will give you different amounts of spew on your console. So you can often try things out and fly and figure out what's going on. You can do quick miniature image builds using the GrubbMakeRescue command that gives you an ISO image containing the version of Grubb that you're using that you can then boot in an emulator and try out. And a useful trick is to boot your real-life system with such an image using QMU-snapshot so that it won't write anything back, and then you can see how the loader that you're working with would boot your laptop without actually risking the possibility of breaking anything. It's easy to pull out bits of configuration files, run them, write new commands, there's a hello world command which is about 30 lines, so it's quite easy to put new things together. Now getting onto things that are a little bit more involved and are broken, one of the things that was part of Debian's GrubbLegacy changes was the Debian specific update Grubb script. And this worked by trying to guess which parts of bootgrubmenu.list were user-modified by a way of magic comments and updating everything else to match the current state of the system. This was, it was sort of okay, but it caused a lot of complaints. Sometimes it was just because people didn't read the comment text that said which parts of the file were safe to edit, but just generally mixing user-editable and automatically updated content in the same file usually turns out to be a bad idea, and we've seen that in various other places in Debian as well. So for Grubb2 we brought this, well my predecessors brought this system upstream as Grubb make config and made it generate the whole configuration file from a small amount of user-edited configuration in Etc default Grubb and a bunch of scripts in Etc Grubb.d. And you can still of course write your own Grubb.config directly if you like, it's about the same length as it would be in Grubb1, but for a general purpose system we normally prefer the auto-generated approach. So far so good, just about everything's customizable if you try hard enough. The problem is you have to try quite hard in several cases. They all require, anything involved requires editing quite complex shell scripts. If you have to, if those are changed then in the package letter you're going to have quite a difficult merge resolution to do, and some changes require things like moving conf files around to different positions in the order which isn't going to be handled well by Debian config, sorry, conf file resolution in future. So all of this probably needs somebody sadly to sit down and design a third iteration of the system that uses perhaps a templating language or something, but that really hasn't been at all started yet. Okay, so now onto some things that are still genuinely quite hard. The PC BIOS architecture has well secreted over 30 or 40 years of gradual development and it's basically pretty awful for modern purposes. The usual partition table format is called MBR for master boot record, or sometimes the MS-DOS partition table. It doesn't offer any formal space for keeping the boot loader, or for keeping boot code in. You can cram a trivia loader into 446 bytes. That basically gives you the Debian MBR package, which jumps to a loader somewhere else. And there are a few approaches for where you can keep the rest of the real meat of the loader. You can embed it in a file system somewhere at an offset that you know won't change. For instance, you put it in a file and trust the file system is not going to move that around. The file systems sometimes do move things around. You might run FSCK, or there are some file systems that will do their own tail packing in RyzerFS as one of these that will sometimes move blocks around for you on the fly. That doesn't play very well with a boot loader that has hard coded addresses in the MBR. You can, a few file systems support an embedding area at the start, I think, ButterFS does and one or two others. This is good, but it's not really widespread enough that we can build the whole installation strategy around it. Or you can do what Grubb generally does now on MBR. You can sketch the edge of the specification that sort of doesn't really exist anywhere coherent and you can use the boot, what's sometimes called the boot track or other names. That's the unallocated space between the MBR and the first partition. This makes some people itchy, it's a bit nasty. It means we have to do some alarming things to avoid conflicts with other software that put things in the boot track. Fortunately, almost all of that software is evil. There are a few bits of Windows software, well, some of them are backup utilities, which I'm not quite sure why they put things there, but they probably have a reason. There are a few things on Windows that like to put a note in a sector in the boot track to say that they have been installed so that you cannot simply uninstall them to work around their 30 day license trial terms. We need to know to wipe that bit of the boot track as well and that's all terribly nasty. Grubb actually now has error correction on some of its own code in the core image so that it can, Grubb can literally cope with a couple of sectors of the core image being overwritten with completely random data. It's horrifying that we need this, but yes, that surprisingly works. Fortunately, this is becoming less of a problem. The alignment requirements on modern disks mean that for decent performance you want to have your first partition start at one megabyte or sometimes even more rather than the traditional 63 sectors. We now usually have plenty of space on NBR, but not quite always. Things are also unpredictable when you have multiple disks. You can't tell from Linux which disk the system was actually booted from. You can, there are some things on some BIOS versions that let you make better guesses, but it's not, unfortunately not universal. You also can't tell what disk is going to be booted from next time. It's not necessarily the same. Sometimes it depends on which order the devices are enumerated by on the PCI bus, even from the BIOS level. The least bad answer is usually to install to all of the fixed disks on your system, but then you annoy some people who have complicated multi-boot setups, so you can't really win. The good partition table format or GPT is much better. It's probably the best thing to have come out of the UAFI spec. This gives you allocated space for, excuse me, let me not do that. Right, I'll not do that again. This gives you allocated space for bootloader code in the AFI system partition in a partition type that we were able to safely allocate for ourselves, because now partition types are a good, they're ginormous. We were able to allocate one for ourselves even when using that on top of BIOS, which you can do. There are different interpretations of the spec, unfortunately, and this causes some problems, particularly the bits that, well, when you install a system with GPT, you're supposed to put what's called a protective MBR in place. That means that anything that tries to parse the disk as MBR knows that it should stay away and not try to scribble over the partition table. The interpretation of that part of the spec is particularly bad. Apple have at least used to have a completely incompatible both ways version. You were required instead of a single giant partition to do the best job you could of representing the GPT partitions in MBR. That's incompatible both ways. I don't know if that's still the case. If you're trying to use GPT on BIOS, some PC class systems require you to set the active flag on the protective MBR partition, which the GPT spec explicitly forbids for some reason. So you're stuck both ways around on a few of those. Now, UFE is where it's at now for PC firmware, apparently. Even if your new machine looks like it's BIOS, it's almost certainly a legacy layer implemented internally on top of UEFI. The economics for firmware manufacturers are much more favorable this way, instead of having to maintain their own, I don't know how many, 100,000 lines of codebests that have accreted over the years, they can now fork Intel's Tiano Core as their starting point and just do their driver layer on top of that. This seems to have attracted essentially all of the BIOS manufacturers. We should generally expect even that legacy BIOS layer to go away soon. The direction that we're hearing from firmware people is all about that, and we need to cope with that. Now, Grubb's core support for UEFI is basically fine. It has more or less the set of drivers you'd expect, including relatively arcane things like serial support. Now, of course, the big thing recently has been what's called Secure Boot. That's a mechanism by which the firmware makes sure to only ever execute signed code in a pre-boot context. Now, it's obviously possible for this to go very wrong from the point of view of user freedom. The FSF calls Secure Boot restricted boot instead for that reason. But a lot of systems now come with Secure Boot enabled by default. We have to figure out how to work with it in the same way that we've had to figure out how to work with all kinds of devices over the years. Just as a hardware enablement matter. But the important thing is to work out how to do that in ways that don't end up impinging on our user's freedom. And that's been quite a difficult slog. The communities managed to figure out schemes for this that don't stop you modifying the operating system on your own computer, which is the important thing. We have this working on Ubuntu. We talked about this at DevCon last year. To get it working in Debian, we need DAQ to be able to sign bootloader objects that we submit to it with the Debian key. I failed at pushing that forward, so that would be a great project for somebody to take up if I've missed it and somebody else is already working on that. Brilliant. I should also mention that there are some outstanding problems with the layout of the EFI system partition, which is UE's place for putting things like bootloader code. The spec prescribes how you're supposed to behave on fixed disks versus removable disks. We follow the spec, but unfortunately some systems don't, and essentially require the removable layout. And Steve McIntyre is here at the back there. We've gone back and forward a bit on that. We need some way to select the removable layout and have that actually persist, and ideally detect that this is a problem in the first place, so that we don't have to ask an incomprehensible question in the installer. Now, more cheerily, a number of non-X86 arches are in a pretty good state. They get fairly limited testing at the moment. I know that some people use them, because occasionally I get bugs when they break, but generally they work OK. Poverty and Spark seem fine. We should probably consider switching the default bootloaders there at some point. If you're a porter for those architectures, please get in touch with me. Some MIPS type architectures work well, too. Ryan Lorty lent me a Yiluang laptop a little while back, so that I could port the Debian installer to it. And I found that Grubb was basically fine. So I was able to build the installer on top of that, and it was a tenth of the work that it might otherwise have been. There's been a pretty pleasing trend towards having new arches include a Grubb port very early on. Arm64, also, POPC64 has had Grubb working from very early in its life. And this gives them a pretty full-featured loader with not a lot of effort, so it actually works out quite well for the ports now, I think. The most recent Arm64 port I went back and looked at it was about 2,000 lines of patches to do the initial enablement. Now, that got to take advantage of the Arm ports that already existed, and also of UFE code. So that helped a lot, but still, I think it indicates a pretty porting-friendly design. OK, so if I haven't persuaded you already, why do we default to Grubb on Debian x86? There are certainly other loaders. There's the venerable Lilo, which still largely works fine. Cislinux and friends are actively maintained by some very smart people who, including folks who maintain the Linux boot protocol, and there's Xlinux is one of that family, which you can use in the same kind of way that you might use Lilo. There's Yaboot and POPC, and so on and so forth. You can even boot a kernel and an interim fest directly from the UFE environment if you want to. So some of those are pretty good bootloaders. They're almost all smaller and simpler than Grubb, and that explains that. That appeals to a lot of people. And I do get the simplicity argument. But I tend to argue that the result of that is actually moving a good deal of complexity elsewhere. So the installer, if you have a bootloader that can only handle some setups, the installer needs to know what those setups are. It needs to forbid you from doing anything that you won't be able to boot in your partitioner. You end up with slash boot partitions scattered everywhere. Having a bootloader that you can trust to handle almost anything means that you don't have to think too hard when you're moving things around. Sorry, I'll come to you in a sec, Ben, thanks. You can easily do things like having the bootloader notice when the last boot fails and behave differently. All that sort of thing is useful for a general purpose destroyer to be able to assume. What some bootloaders on MIPS couldn't load in the RAMFS, they could only load the kernel one. So for that reason, the kernel configurations we used on MIPS have had to be restricted so that they don't assume there's a new RAMFS. And you can't boot off file systems other than X2, X3, and X4. That's KOLO, is it, or Sibyl? I don't know. One of those. You have recently made it switched over, but that has been a restriction to it, including Weezy. Thanks, that's useful ammunition. I think that we do not yet have grub working on all of the MIPS big-endian architectures. It's working on little-endian, but I think there are still some problems in big-endian. So we maybe can't save the world just yet, but it might not be too far off. But thanks. And plus, Debian runs on lots of architectures. Generally, we try as an architectural thing, we try to run the same software across different architectures that it all works kind of the same way. We have the same interfaces, the same tools are available, and so on. And it makes a lot of sense to have that be the case with the bootloader, too. The installer would certainly be simplified by being able to assume that grub mount is available on all architectures, for instance. OK, so the important bit of this, we've had a lot of people involved over the years, too many to name. I've almost certainly missed some. Robert Milan and Felix Zilke and Hordi Malak have done excellent jobs in Debian a few years ago, and Vladimir Serbinenko does a great job as the upstream antenna. But as far as Debian maintenance goes, it's mostly just me at the moment, and has been for the last couple of years, and I can't do it all. We'd probably have had grub 2 in 2.00, rather than 1.99 in Weezy if we had a bit more redundancy there. I was kind of demotivated by the secure boot stuff at the time, so ended up putting it off until it was too late. And that's exactly the kind of thing where having more people involved is helpful. There are lots of bugs. Every boot problem is critical for them, quite understandably. So keeping on top of the release critical list is absolutely murder. But there are a few specific problems I'd like to highlight that are more substantial and could really do with somebody sitting down and thinking quite hard. I won't go into too much detail because I've just assigned from the video team that I'm low on time, but you can ask me about any of them if you're interested. I mentioned earlier that the system for generating the config file has a bad case of second system effects and really needs a third system to make it easier to customize things. All problems can be solved by another rewrite. On NBR systems, we often have robustness of problems with grub not installing its boot code to the place where your system actually boots from, particularly when multiple disks are involved. This tends to manifest as module loading failures, because what happens is that over time the core image that you're actually using that you didn't know about becomes incompatible with new grub modules because the API changes over time and eventually becomes incompatible and can't boot anything. And I think that that needs some kind of overhaul somewhere, although I'm not quite sure where. There are the remaining bits of UE, which I mentioned earlier. There's getting signing sorted out in Debian, getting a thing called mock manager, which is part of a separate project called shim, nicely integrated to give users control over the whole signing process, which will let us require signed kernels under secure boot, and that's to make Matthew Garrett want to kill me a little bit less. And there's the issues with the API system partition layout. There's some work to be done with upstream Zen. Excellent, both in the audience. I've discussed this with Ian Jackson in the past, to lay out, effectively, a new boot protocol for a PV grub two so that we can have Zen guests install a bootloader in a consistent place in the fast system. In fact, Ian Campbell recently sent me something which might be useful for this when I have time to read it. And hopefully, this will eventually let us kill off all of the old strange things that live in Zen to do this. And finally, we should take much better advantage of grub's ports and actually switch over to them by default on many more architectures than we have downstairs. So, well, I've just got onto questions, so accents. One of the things that I found. There's a microphone coming up. One of the things I found difficult in getting groups with grub one and grub two is that the documentation hasn't always been as good. Lilo was exceptional in that it came with an extremely good document which described its principles of operation and how to get it to do, well, almost anything you might want it to. Is that also something you're looking for help with? Yes. About a year or two ago, I went through and tried to start systematically documenting all of the grub commands and overhauling some other things and got about halfway through that before I ran out of time. But one of the problems has been with grub two that the documentation got out of date during the rewrite and thus nobody has an incentive to do it properly for their incremental changes. So I think getting that into ship would make it much easier to keep it in ship. It's better than it was in 2010. It's not where it should be. So, yes. Anybody else? Anything from IRC as well? Can you stand up? Very good. Tom. So this is sort of a related question to your comment earlier about how BIOSs are wildly inconsistent about how they choose which disks they're going to boot from. Is there anything that grub could do to help us with best practices for failover configurations? So for example, let's say I have a server and I want to install two disks that are mirrored and use LVM or something else. One of the challenges is how do I configure grub on both disks to be configured the right way, despite all of the weirdness with BIOSs, but have a reasonably easy to maintain setup where if one disk actually goes completely offline that the system could automatically reboot? Right. One of the things that I tried to do the first time I systematically attacked this in Debian Grub, the thing I tried to do was to arrange that we did by Uyrid, I think, install to disks, possibly as Bypath, I forget. So we encourage people to install to all of the disks on the system. I think this, sorry, also the grub PC post-ins notices when the disk has gone away since it last checked and is supposed to offer you the chance to update this. But you're right, it's a little bit fiddly to manage, particularly if you're in a system that is large enough that statistically disks are failing quite often, then it's a problem. And I think what I'd probably like to do is make sure that we install to the disks that are associated with a red device or something so that you could just bootstrap off the red management. I think some of that works, but isn't hooked up very well in Debian. But there are several things to do there. And I think there's also a bug in DI where it doesn't install to all of the fixed disks on your system, but only one. Yeah, there are several things to fix there. There's a question over there. I think my question could be summed up as, how tall do you have to be to ride this ride? It seems like getting started, it seems like introducing regressions would be a really scary thing for someone who wanted to get started with the project. And I'm just wondering, what kind of facilities do you have before we actually upload a new version to test across a broad range of architectures? Right, well, one of the points of this talk was to try to emphasize that you don't actually have to be as tall as people think you do. But you're right about the need for regression testing. One thing that really helps is that the different parts of grub are much more isolated now than they used to be. So you can quite easily change one module, test changes to that, without having to worry about strange leakage all over the place. But there are things that that doesn't cover. There is some boot testing arrangements in the build system, so there are some make targets that go off and do boot checks of real systems against the current version of the code. And that sort of thing helps a lot. I would like to have grub hooked up better to some kind of auto package test arrangement so that it does emulated boots across various types of systems on ci.debian.net. I'd love help with that. Steve? Now, of course, UEFI, there's a whole slew of things. I mean, as Tom was asking, of course, about NBR and reliable fallback, of course. I'm not aware of any UEFI implementations that actually do fallback for the system partition either. So that's a utter train wreck. I suspect maybe you're meant to use hardware read or something. Clearly, that's what people are expected to use, and it's a mess. We should definitely get together and talk about the removal media path. Again, there isn't a good answer, but let's try and make it as not crap as we can. Yeah, do you know of any way to detect that system, even if it's something like a blacklist of broken DMID code? I think that's all we can do. I mean, we had a system, I mean, I think Leif and I spoke to you about this. As an example for other people who may not have come across this, there are firmware implementations out there which are straight after install may not actually recognize that you've put grub into the path where you're meant to put it. Once you've booted off a rescue system, however, and done some not particularly well understood sequence of things, including booting off a removal media path, we boot a couple more times, shuffle things around, like spin twice, clockwise, suddenly they'll then will recognize that you've put them in there. God forbid you ever want to update it again. It's painful. There are some exciting problems on Apple Macs that are a shadow of the same class of problem. Yeah, and of course, secure boot. Kibbe asked even about this a few days ago, of course, we've been talking about this for like two years. Yeah, I know. Yeah, I know. I feel rubbish as well that we haven't really got any further with it. If somebody really, really wants to help get involved in that, oh, absolutely, there's a whole load of people who would love to see you help. Not only are there a whole load of people who'd like to see you help, but there's actually roughly a road map for what needs to be done. You don't have to play as a trail. Just take the names off the road map and put your name there. It's fine. We don't really care. We just want to see it done. And the things that need to be done are, as I say, the next step, I think, is in DAC. And then somebody needs to chase up getting shim into Debian with the right kind of signatures on it and so on. But none of it is actually fundamentally hard. It just requires time to push all of that forward. Exactly. So like I say, the equipment drive. Actually, we're out of time. OK. Thank you all very much. If people in the back of the room could vacate at least temporarily so other people can get out, that'd be cool. Thank you. Yes. Do you have one for what? What connections? I'm in the store for a minute. So there is a Grubzen bin package which is unstable. Who's going to buy it? The pot is missing. It is conventions for all of us. It's coming in the way. 3-2-3-8.