 Okay. So, my name is Alan Jude, and I'm going to speak about Geleboot, which is booting from a fully encrypted disk on FreeBSD. So, I've been a FreeBSD server admin for about 13 years, and over the last couple of years, I've become a FreeBSD doc committer, and then a source committer, and then a member of the FreeBSD core team. I've also co-authored two books, FreeBSD Mastery ZFS, and FreeBSD Mastery Advanced ZFS with Michael W. Lucas. At my day job, I run a video streaming company called ScaleEngine, and I host the BSD Now podcast, and until recently hosted the TextNet podcast. So, I do a lot of work with ZFS, in particular, helped build the ZFS bits in the FreeBSD installer, so that you can more easily create a ZFS-based system, and then I did work to integrate boot environments, the Solaris concept of having basically a clone of your root file system, before you make an upgrade or a change, so that if it doesn't work, you can boot back to your base system how it was before, but your other file systems like your home directory, and bar log, etc., are not disturbed. So, when you roll back the system image, you don't roll back what was in your home directory. So, once we had that, I integrated into the boot loader, so that at boot time, you could select this alternate boot environment, because the most likely case when you would want to use a different boot environment, is when the current one doesn't work. So, that worked. The problem was if you used FreeBSD's disk encryption system called Geli, which is a block-level disk encryption, which you take your raw disk, you put Geli on top of it, and anything you write to it that gets encrypted, and then you can put a ZFS pool on top of that. The problem with that is that ZFS boot environments depend on the kernel and the root file system being the same file system. But if you want to boot from an encrypted disk, traditionally what you've done is had a small slash boot partition somewhere that had your kernel, the loader, and your modules that was not encrypted, and then the rest of your file systems are encrypted. That way, the boot loader could load the kernel, load the module that provides the disk encryption, and then mount the root file system. But having that two separate file system layout, it makes boot environments not work because you need to have your kernel be in sync with your root file system. So, I devised what's called Geli boot, which is a system where the lower-level bootstrap that loads before the boot loader could actually understand the disk encryption to be able to load the boot loader from the encrypted disk. So, especially when I started this, I was very novice C programmer. I had written one other tool before and that was, I like to use Shellscript and I wanted to use this new library previously had called libucl, which is a config file parser that also supports JSON. I wanted to use it for my Shellscript, so I wrote a little C program that would allow me to use the functions of this library from my Shellscript. That was all the experience that I really had. So, I attempted to build this, thinking it can't be that hard, right? It's only the boot loader. So, I implemented a very minimalistic version of Geli, which is the previous disk encryption subsystem, for the two different boot codes that are. So, there's GPT boot, which allows you to boot UFS and GPT-ZFS boot, which can boot ZFS. So, I spent a lot of time just understanding what was already happening in the existing boot code before I tried to modify it, and working with that, and I had to learn a lot about C because I really didn't know what I was doing, and it didn't help that all the existing boot code is terrible. It's all clutch that's built up over the years, and as soon as it starts working, everybody stops wanting to play with it. It's like, it works now, nobody's touch it. Then, support for GPT, the more advanced partitioning layout was kind of grafted on, and then it's like, nope, don't touch it. So, I had to navigate through quite a few obstacles, and just to figure out what was what, because there are four different sets of boot code, and most of the code is copy-pasted, but there are subtle differences. There are enough differences that you can't just diff the files to see what's happening, but not so few differences that you can't, a lot of the code looks the same. So, to give a little background, before we dig into what actually I wrote the code for, I thought we'd cover a little bit on how a computer actually boots. This is being like an i386 x86 type computer. So, when you power up your computer and after the BIOS does its post and props all the drives and that type of thing, it reads, the BIOS reads the first 512 bytes of your hard drive, which is your master boot record, and that consists of a 446 byte bootstrap program written in assembly, and then the remaining 64 bytes or so there give you a partition table where you can have up to four partitions. That's MBR. So, that bootstrap is then executed, and that bootstrap in the source tree, you'll find it as boot0.s, so it's called boot0. What that does is it examines that partition table, and in the case of MBR, finds which of those four partitions has the active flag set, and then inside that partition, you will find the volume boot record or VBR. So, it reads the first 512 bytes of whichever partition is selected as active and loads that into memory and executes that program. That one is called boot1 in the source code. That 512 byte of assembly program then reads boot2, which in the case of UFS is the first 15 sectors of the UFS partition, which the file system just purposely doesn't ever use so that you can write the boot code in there. What the boot2 contains is just a minimal enough bootstrap of a read-write version of UFS. So, all the code you need to be able to read and write files from UFS fits in 15 512 byte sectors. Now, that's not the whole file system, it doesn't have all the features, you can't do snapshots or anything like that, but it's enough that you can say, hey, in slash boot, give me the iNode number for kernel and then let me read the blocks from that iNode. So, once you've loaded boot2 and now have the ability to actually read a UFS file system, it loads the file slash boot slash loader, which brings up the beastie menu, which then after you wait for the timeout or choose an option, then the bootloader actually reads the kernel using its UFS driver which is more advanced and then the system boots. So, if you want to do that with ZFS, it actually turns out to be a bit evil. So, to boot ZFS from an NBR form at a disk, the problem is that the first 15 sectors of a ZFS partition are used by the ZFS label, so you can't just stick the boot code there, and the boot code is much bigger because ZFS is kind of more complicated than UFS. So, again, when you're booting off NBR, you have that boot zero, which is the first 446 bytes of the drive, which reads boot one, but boot one is different in ZFS. What it does is seeks into an offset after the ZFS labels in the ZFS on-disk format. They purposely left this region of three and a half megabytes of unused space to hold things like boot loaders. So, in there, you have this code that can read ZFS, and it's originally was 64 kilobytes. It's bigger than that now. But since it's 3.5 megabytes of space, it's fine. But in the code, it's actually padded out to a specific size so that the boot one assembly code can just say, copy this 64 or 128 kilobytes of data off the disk, put it in memory, and run it. So, that's the ZFS boot, which is basically a boot two. That one has enough understanding of ZFS to read only, so it can import the pool, understand read Z in mirrors and compression, and all that stuff to be able to let you boot ZFS. Then what it does is it actually reads slash boot slash loader and calls it with an argument of the globally unique ID for that Z pool. So, when the loader starts up with its more advanced understanding of ZFS, it knows which pool you actually want to boot off of. Because when it runs, it's going to find all the ZFS pools. Maybe there's only one, maybe there's a bunch, and it needs to know which one you actually booted off of. Once it reads the loader from ZFS, the loader presents the menu and then can load the kernel and you can continue. But most of us have switched over to GPT or grid partition tables. Biggest advantage is that you can have 128 or more partitions instead of four, which is handy. More importantly, you can have disks that are larger than two terabytes, which is pretty run-of-the-mill nowadays. But in order to make sure that Windows 98 and Windows XP don't offer to reformat the drive when they don't see a master boot record on the beginning of the drive, a GPT actually has as the first sector a protective MBR. So, it's a fake MBR that looks like it covers the whole disk, and it just stops legacy operating systems from deciding that that disk is empty and I can just reformat it. But again, the first 446 bytes of that are a little assembly program that boots the operating system. So, in the case of FreeBSD, that PMBR instead of looking for the active partition and seeking to a certain location, instead what it does is it actually understands GPT, looks to the partition table, and finds the first partition that has the type of FreeBSD-boot, which is basically just one specific grid that was decided to go years ago. Then it loads of all that into memory, either the full size of the partition or 545 kilobytes whichever is less. The main reason for that limitation is that, in this 16-bit real mode, you can only use the first 640 kilobytes of memory. So, we don't want to load more than that or we'll run out of memory. So, on a UFS-based system, this will be the GPT-boot code or in a ZFS case, that previous reboot partition will contain GPT-ZFS boot. That then contains what's called GPT-loader, which is actually a boot one. Again, that's a little 512-byte assembly program that just loads the next part. Then the boot two is GPT-boot or GPT-ZFS boot, which is again, a little program that understands UFS or ZFS and enough to actually read the loader or the kernel off that file system and start the operating system. What's interesting here is the GPT-loader has to relocate itself in memory to the location in memory where the BIOS is going to execute the operating system. So, to do this, it copies itself in memory, but to make sure that it doesn't overwrite itself, it copies it backwards, which becomes important later on. So, I now understood how the computer started up, and where all the different pieces were, and I wanted to now be able to boot from a file system that was completely encrypted, and so normally, I couldn't lead the loader file at this point. So, the first thing I did was find GPT-ZFS boot, and just copy that whole directory of source code to GPT-Gellyboot. My idea was instead of having GPT-boot for UFS and GPT-ZFS boot for ZFS and then making encrypted version of each of those, I'd make one big bootloader that could boot anything. That's still a good idea, but that's not what I end up doing. The first question was, if a system has both a UFS and a ZFS partition, and your bootloader supports both, which one should it boot from? There wasn't really an easy way to answer that question, so I decided that's for somebody else to figure it out. So, instead, I started by implementing Gelly in both of them separately, but doing it in line rather than having doubling the number of bootloaders that were available because it was already too many. Yeah. So, the ZFS boot is MBR only, and it's a fixed size, so I decided not to touch that because, I wouldn't be able to fit all the code in that fixed size and I didn't know what it would take to expand it at that time. The first thing I learned is that working with the boot code is difficult because there are no debugging facilities. If something goes wrong, most likely the machine will just hang with no output. You can't really attach GDP to it or something. Your only option is lots of printfs and hoping you can read text that's going by really, really fast. So, in order to actually get somewhere, the first thing I had to do was how to tell if this partition is encrypted or if it's not. So, the way free BSD's Geom classes work is the very last sector of a partition will contain some metadata that says what kind of class it is. So, in this case, the first couple of bytes of the last sector will say Geom, Eli, and then that means I know that it's a deli encrypted partition. If it doesn't have that, then it's not and I should treat it differently. What was interesting was trying to actually find where a partition starts and stops. So, in the boot code, there's some code that understands GPT and NBR, and it populates this struct disk. But depending how it started, it may or may not have a start offset set for where the partition starts. Then, so you have the partition table which has a start and a length for each partition, but then the disk objects you're working with may or may not have these start offset set. So, you have to just make it relative to what it says, but it's not always what you think it is. It turns out, depending which boot code you're working with, the numbers will be different and that makes everything more complicated. So, it turns out that the ZFS boot code actually made my job easier. So, what the ZFS boot code does is, instead of reading the disk directly, it actually uses a callback. So, it calls another function to do the reading. So, there's a function in ZFS and it takes as a parameter the function it should call to read from the disk. So, what I was able to do was just replace the regular read function with my encrypted read function that would then call the regular read, do the decryption, and then return the result to ZFS. So, that's what I did to move that and then I actually decided that I like that approach a lot. So, I changed the UFS code to basically do the same thing, so that I could implement it the same in both UFS and ZFS and try to make it easier to understand. So, after I figured out if the partition is encrypted or not, the next thing I need to do is actually decrypt it so I can start reading from it. So, by looking at the Gehli metadata, which is in the very last sector of the partition, I get the basic information I need, like what algorithm it's encrypted with, how big of a key it uses, and an encrypted copy of the master key. Once I have that, I can then try to decrypt the master key by asking the user for the passphrase. If the user provides the correct passphrase, then an HMAC will match the signature in the Gehli metadata, and I know that they entered the right password. If not, I can prompt them again in event and then eventually just give up on them. So, at this point, the problem was I needed to have some crypto. I need to be able to decrypt that master key, and the bootloader doesn't have any crypto by default. For UFS, the bootloader was 14 kilobytes. There wasn't any extra scope in it. When I started, the ZFS one was 47 kilobytes, and most of that was the checksum algorithms and the compression algorithms that ZFS needed to be able to read the disk. So, I looked at how Gehli does it, but it uses the kernel's crypto API to support offloading to crypto acceleration cards and stuff like that. That was way too big and complicated to try to put in the bootloader. So, I used Google and I just looked for like tiny basic C AES implementation, and I found this thing called tiny AESC on GitHub, and it was BSD licensed and it allowed me to get started. But as I started working with it, I realized it only does AES CBC 128, not 256, and it doesn't do AES XTS. So, while it would get me started, it wouldn't actually solve the problem. So, I purposely created some encrypted disk using the one algorithm I had support for at this point, and fiddle with it enough until I got it to actually work. So, by stealing some functions from Gehli that read the metadata and do the decryption, and just replacing the calls into the kernel crypto system with the basic C implementation I had here, then I could decrypt and validate the master key, and then using that, I could calculate the HMAC and make sure that the master key was correct, and then I could use HMAC that Gehli uses to calculate the sector key and the initialization factor for the actual encryption, so I can decrypt random sectors. Then, I looked at what other bits of code I was going to need, and it turns out that Gehli uses MD5 for the signature on its metadata. So, this is something across all the GM stuff is that the metadata that's written in that last 512 bytes on the disk has an MD5 hash just as a sanity check. It's not really meant to be cryptographic or anything. Inside Gehli itself, it uses SHA-256 for all generating all the unpredictable IVs for the disk encryption, and then it uses SHA-512 for all the HMACs for verifying that the master key is right, and if you actually have the data authentications or not. So, now I needed all three of those hashing algorithms to fit in the boot loader. So, I first tried to just include them, like you're supposed to be able to do, is just grab the headers for each of them and go, but it turns out doesn't work like that. So, elsewhere in the boot code, when they need some code, they actually used include on the .c files, just bringing that stuff in line and try to make a small boot loader. But the problem is that there was a bunch of conflicting defines in AS-256 and AS-512s, where defining the same function name, but they actually do different things or have different size values. So, then the approach I took was, there's this LibStand 32, which is a 32-bit version of LibStand that the boot code uses. So, I just hacked up the make file, so it would pull in the existing chunks of code into this library and expose all the symbols, and then I could use them from the boot loader. Eventually, I replaced this by creating my own library, LibGallyboot that brought in all the dependencies I needed for that, instead of loading LibStand, which is also used by other things, and was causing all the binaries to grow instead of only mine. So, then I had to prompt the user for the password to be able to decrypt the disk. When I was first writing this, I just hard-coded test pass as the password in the boot code and created encrypted disk that were with that password and eventually got it to work. But then I needed to support actual passphrases. So, then I was like, it shouldn't be that hard to ask for a password, right? So, I went into the common console code and got the getster function, which allows you to get text from the console. I changed it, so instead of echoing back the characters you type, it would echo back a star and it should be just all well and good. But it turns out the loader reuses some of that code, but it actually uses completely different functions to read from the console because it has to support serial. So, I had to write a completely different implementation of the same thing. But then during code review later on, we found that actually both of those functions, the one in the console and the one in the bootloader contain the same bug that was found in NetBSD like eight years ago and fixed. One of the break or something, they didn't use curly braces on one of the if statements and they had two things in the if and it looked like it was working because of the indentation but it wasn't. So, this would cause it to not stop when you reach the maximum number of characters and you just keep overwriting memory and doing crazy things, which is not what you want to do, especially with a password where you've assumed it's only going to be this long and you're going to overwrite that memory with zeros later and then not overwrite anything in excess. So, instead I actually found the version from NetBSD and get s that was actually correct from our LibStand and wrote my own pwget s and put that in my little library. Future work for some days go back and fix all the other ones. Actually, here's the code that's wrong. So, if you haven't overflowed the buffer, then this should be inside this if instead of not. So, if you haven't overflowed the buffer, keep counting but otherwise put the character there anyway. So, it doesn't do what it's supposed to do. So, at this point I thought I had enough working so I could actually take it for a test drive. So, my expectation was that Boutu would start, taste the partition, determine that it was encrypted, then read the master key, which is encrypted in that last sector of the drive, decrypt it with the passphrase I just provided, and then stand ready to determine the sector key and decrypt individual sectors of the drive as I tried to read them. So, I booted it up in VirtualBox and VirtualBox got some kind of triple fault and it just crashed. We're like, what's a triple fault? So, I wrote it out to a USB stick and tried it on my laptop and it just rebooted in a loop. I'm like, this is not going to work and I have no idea why. So, it turns out by adding all this, I'd actually cause the boot code to go beyond 64 kilobytes, which at the time I didn't know was a magic number. So, now the boot code was like 70 something kilobytes. It turns out the GPT loader, that little 512 byte assembly program that grabs GPT ZFS boot copies it to the right place of memory and lets the BIOS execute it, only copies the first 64 kilobytes, because that'll be enough for anyone, right? Because when I started the project, the UFS one was less than 16 kilobytes and even the ZFS one was 42 kilobytes. So, 64 was actually probably enough. But once you started adding the encryption and hashing algorithms, all of a sudden that was not going to be enough. But because this is 16-bit real mode in the OS, you can't actually copy more than 64k at once. So, it wasn't just a matter of changing the assembly code to copy more data. I tried that, it didn't work. I tried that and the assembler actually laughed at me, and then put it back to 64 kilobytes. Yeah. So, I was over 90 kilobytes and this wasn't going to work. So, I was like, okay, put that aside, let's try UFS. It's small enough, I think I can fit it all. So, I stuck with just UFS, even though this whole project was only originally for ZFS, and I didn't think I cared about UFS, but I was like, UFS can't be that hard. So, I decided to use UFS and then basically did all the same, reused all the same code to decrypt the master key and everything. Luckily, when I compiled it, it was under 64k. So, I put it out and booted it up and the decryption actually worked. I'm kind of glossing over like a month of it not actually decrypting correctly and printing out Hex stumps to the screen and trying to figure out what's not working. But anyway, so now GPT boot decrypts the file system and it was actually able to read slash boot slash loader and run it. So, I had actually decrypted something and run it. I was very happy. So, the loader starts and immediately failed because it tried to read the disk expecting UFS and got gibberish because it was encrypted and so, it was like, I can't do anything with that. So, it had all worked except it didn't actually go anywhere now. So, I could load the loader but then the loader would just laugh at me. And of course, the boot loader isn't much better, although slightly better than the boot code, which is the bootstrap that loads the boot loader. So, there's no kernel, there's no libc, there's no malloc implementation, there's no panic, there's nothing you can do. There's even complications with that. When you're trying to use code that's meant for user land, it wants to include string.h. But when you're building the boot loader, you have libstand which has its own string implementation and you can't have both at the same time. So, that complicated things. So, in the boot code, the actual like UFS and ZFS implementations, there is a malloc function, it just doesn't do what you're used to it doing. So, in the case of the ZFS one, there's a three megabyte stack variable set aside. And when you call malloc, it just increases the counter to what offset in that variable you are, giving you back a pointer to some memory you can use. But there's no free, right? So, you have three megabytes of memory and you can allocate, but if you free, you don't get anything back. And so, you can only have ever used three megabytes, not you can only use three megabytes at once. So, this means you have to try to avoid doing a bunch of small allocations that are just temporary. So, the next step was to teach the loader how to actually speak Gehly so that it could load the kernel so that the operating system could start. So, I had to figure out where in the loader it actually reads from the disk so I could do this same hack of, oh, I see you read some data from the disk, let me decrypt that for you before you try to use it. So again, I had to do, taste the disk and determine if it's Gehly, read the master key, prompt for the password, do all that stuff. So, in the loader, it has this array called file systems and it has all these different file systems defined, UFS, ZFS, NFS, et cetera. So, my first thought was, oh, I could just add, you know, Gehly underscore UFS is a new file system and wrap all those functions. But when I looked a little bit further, it turned out all the different file systems that I actually expected to read from the disk used libI386 to actually do the BIOS calls that read from the disk. So, I intercepted the data there instead because it was less work. Probably not the best place to actually put it, but it was less work. So, ideally we'd implement it as part of probably the read cache that Thomas Soome did for Alumos and then port it back to FreeBSD, but that's future work. So, after I basically redid all the work I just did in the loader instead of the bootstrap, which is relatively different code, I was stuck here. So, I had successfully actually booted a Gehly encrypted disk with a UFS file system, but I only had support for AESCBC128, which nobody wants to use. There was no support for ZFS. I had this 64 kilobyte binary size limit, which meant no ZFS and that if I added much more complexity to it, I was going to have no UFS either. So, while I had done a lot of work and actually accomplished all these things, I was actually nowhere. So, in order to go any further, I needed to get rid of this 64 kilobyte limit. So, I tried some naive things. I tried to compile with optimized for size, but that didn't really help. I tried optimizing it more or switching to O2, but that just made it bigger. I tried increasing the number of blocks. So, in the GPT loader assembly, it's like copy this many 512 byte blocks. It's like 80 because that's five hundred or 64 kilobytes. I tried increasing it, but the assembler is like that number is too big. Sorry, I'm putting it back to the maximum. The fact that it still compiled and put the number back to a small number instead of just failing kind of amused me. So, then I reached out to various other people at the FreeBSD project to see if I could get their help. The first person I asked was my documentation mentor, the person who was mentoring me for my documentation commit bit. He looked at it and thought, well, this shouldn't be too hard to convert to 64 bit, but it turns out that doesn't actually work like that, or to make it copy two blocks of 64k instead. But he didn't really have time to write the assembly for me, so I tried someone else. John Mark Gurney seemed receptive to the idea at first, but once he understood that it was the boot code and that it was complicated and 16 bit real mode, he quickly suggested I ask someone else. So, then I asked the original author of GPT-LDR, John Baldwin, and he suggested that you shouldn't make the boot code bigger, you should solve this some other way. Like having a partition that had only the loader on it rather than the entire slash boot and hacking around it that way. Well, that's not too terrible, it's just a lot messier than the way I was hoping to do it. Then Peter Greenhan is like, oh yeah, sure I can help you with that. Then he's like, actually here, let me teach you how to use QMU and GDV to debug the assembly instructions as they go by. I'm like, if I knew how to do assembly, I wouldn't have asked you. Then at VBSDCon, the BearSign conference would have been 2015, my friend Dylan was there so I asked him and he started trying to write new assembly and sent me a couple of bits of code but they didn't work, and he spent the rest of the conference with a pad of paper drawing the memory stack, trying to figure out how to do it. But after the conference was over, he had to go back to his day job and didn't have time to help me with it. Then at UroBSDCon 2015 in Stockholm, Sweden we were having the Dev Summit and a one-point call in personal approach means I heard you're having trouble with some 16-bit assembly. I'm like, yes. He's like, ah, that's 16-bit assembly. I know. It turns out that was the only assembly he had ever done, but what he had done when he was in school and still remember how to do it. So that night he wrote up a draft patch for the assembly code and sent it to me, but it just crashed the BTX client, which is one of the little pieces of the bootloader. So his first patch didn't work, and then he sent another one and that still didn't work. Basically, he sent me a new patch each night of the conference until it was over, but none of them worked. So we all went home and I kind of figured I was out of luck still. But then a couple of days later on IRC, Colin poked me and sent me a new patch. This one still only copies 64 kilobytes, but it does it by copying two 32 kilobyte chunks. So try this on a regular size bootloader that we know would work and see if it works. So we tried that and it did work. So we were moving somewhere. So then a couple of days later, he sent me a later draft that copies four 32 kilobyte chunks and it actually worked. Now, if you ever need more, you can just increase this number. So it's future-proof. So now that I had solved that and whacked all that together, I could now boot ZFS from AES-128 encrypted volume. But that's not the disk encryption people want to use. You want AES-XDS because it's faster and designed for disk encryption or at least AES-CBC-256. So now that I didn't have the space constraint anymore, I could ditch the tiny AES implementation and get a real one. So I actually stole the canonical one from the kernel and started just by including the C-files into the GPT-ZFS-boot-C. Play with it some more and then I needed an AES-XDS implementation, which uses this code but has all the functions that do the tweaking for the initialization factor. So I stole that from OpenCrypto, which is a framework originally created on OpenBSD, I think, that provides all the different crypto functions and offload versions of them for crypto cards and so on. The problem with that was all that code, it was in one giant file, xform.c has all the crypto transforms. So it's got Cast, Blowfish, DES, AES, AES-XDS, AES-CBC, AES-GCF, all the different ones. That was a lot of code that I didn't need. The other problem was it used kernel malloc implementation, which on FreeBSD has extra fields because you define what type of memory it is you're allocating and which bucket you're allocating it from. So I made some changes to the implementation, which are basically I copy and pasted just AES-XDS from OpenCrypto into a new file and use that and modify it so it wouldn't actually malloc any memory, it would use stack variables. So once I had that, I had a working UFS and ZFS that actually did the disk encryption people might want to use. But it worked now, but it wasn't something anybody was going to let me commit to FreeBSD because it was a mess, a very ugly mess. So at this point, the code was all ugly, it's full of debug printfs that are printing out bits of pieces and things so you can tell. I'm just pounding including a bunch of C files out of the kernel, it worked but it was ugly and I couldn't just include some of the stuff from OpenCrypto because the file was too big and it had, so X4.c had in addition to every crypto algorithm had every hashing algorithm including all the Mac variants and the deflate algorithm. It's a really big file. So I asked the FreeBSD security officer, what I should do about this and they said, it probably makes sense to actually break up OpenCrypto into the individual components. I'm like, they'll let me do that. So they did, so I actually used SVN copy to copy X4.c to X4 underscore each of those different algorithms and then deleted the XS code. I'm not sure if that was the best way to do it, but in the reviews, in the diff I created out of it, it was clear that I didn't add any new code. I just copy and pasted that same file and deleted everything except for the bit I wanted for each file. So while it made a bigger diff to review, it showed that I didn't actually change even accidentally any of the implementations. So I'm not sure if it was the best way to do it for the code review, but it was for me, the easiest way to show that I didn't accidentally break any of the crypto. Because that was what I was most worried about. So at this point, most of the GeliCode had survived not being that modified. I just copy and pasted it. I hadn't had to go through and remove Malix or anything. So I looked at just being able to use the code from the kernel by linking it into the lib GeliBoot instead of copying and pasting it. There were a couple of places, the struct GeliSoft-C, I had switched to the simpler metadata struct. So I had to undo that. So the soft-C has a whole page of variables in the struct. It's everything you need for running the disk encryption in the live kernel. Whereas the metadata is just what data is actually stored on the disk. In my implementation, I only needed a subset of fields but I had to switch back to the other one in order to use the unmodified code. The other thing was the crypto and the HMAC were all in one file. So I split those into two separate files so that I could include only the bits that I needed. In particular, I didn't want the crypto bits from the kernel version of Geli because it used the kernels crypto framework. I wanted to use just the straight stuff I had, which doesn't support acceleration or multiple threads. But in the case of the bootloader, that's all I wanted to have. Then the other problem was that a couple of places, there was kernel-only stuff, so I had to add some if-def kernels to Geli so that the code would compile in userland. So I had to move a couple of things around, but it was only minor stuff. So once all that was working, I tried booting off a ZFS mirror, which resulted in me having to type in the password six times. Once for each disk in the bootstrap, and then once for each disk in the bootloader, and then once for each disk when the kernel actually starts it before mount root. I was like, that's a lot. It's like, what if I have a RAID Z of six disks? That's going to take all day. So previously, Colin Percival had done some work with Devanteschi and Chris Moore to create a system for this. So Colin originally created a password caching system in Geli so that when you decrypt the first disk, it caches the password until it's done mounting all disks, and it attempts that password on the next disk. If it's wrong, it doesn't count against your three attempts and it reprompts you. But if it's the right passphrase, you only have to type it in once for all your disks. So I was like, I will shamelessly steal that concept and applied it to the bootstrap and the bootloader. So now, when you boot with two disks or three disks, you only have to type the password in three times. But Colin had also worked out a way to pass the passphrase from the bootloader to the kernel. Because Chris Moore wanted to replace grub in PCBSD, and Devanteschi had wanted to move the prompt for your password into the bootloader because when it's done by the kernel late-arriving USB devices could overwrite the prompt with de-message stuff, and then you don't realize that the reason your machine is just sitting there and not booting is because it's waiting for you to type in the password. So I re-used that mechanism to pass the password from the loader to the kernel. So now you're down to two copies of the password being entered to boot. So then I actually looked at how ZFS has the answer for this. How do I pass data from the bootstrap to the bootloader? So ZFS have been doing this by passing the Z-pool ID number so that it knows which Z-pool you're trying to boot from. The way it does this because it has to pass a couple of bits of data is it passes a struct, and the very first member of that struct is a size, which is the size of that struct. This way, if the code is out of sync and you have say a newer bootcode but an older loader or the other way around, the loader and every time before it accesses any member of the struct, it checks if the offset of that member of the struct is greater than the size. If it is, then it knows that its definition of the struct is older, and so that anything that's going to be off the end, it won't try to read. So this way, we have forwards and backwards compatibility. If I add a new field to the struct, the loader is going to make sure that its definition of the struct is new enough, or the struct being passed by the bootstrap is big enough. So if the bootstrap is older and passes a struct that doesn't have say our new password member, then the offset of that member is going to be smaller than the size that the bootloader passed, and it won't go read uninitialized memory, thinking it's passphrase, it'll know that bootstrap is older and just didn't give us a password. So I hacked onto that and basically added a galley password field to the end of the struct that gets passed and then gets zeroed out. So I implemented the exact same thing for UFS. So now we can pass parameters and we can always grow the list of parameters and it's backwards and forwards compatible. So now you only have to type in your password in the bootstrap and it passes it to the loader and then the loader passes it to the kernel and the system boots. So that's how it works. So the things that aren't finished are that currently it only supports passwords, where you type in a password. Galley has always supported keys where you can have like a USB key or like a four kilobyte file of randomness that you can throw into the mix for your encryption. The bootloader currently doesn't support that at all. I'd like to add that but it's going to be a little complicated. Because where do you store those keys? Based on a bunch of conversations I had, I think the easiest way is actually to create a new partition type like freebsd-galley key and we can just say have a USB stick that'll have this specially defined partition. It'll be whatever size you want and it'll just contain raw key data rather than a file system or something. I don't know how well USB keys show up in the bootloader in this case and in particular what happens if you try to attach one too late. If you have to reboot to attach your key every time you want to decrypt your drive, it'll be kind of annoying. Galley itself supports blowfish and camellia ciphers for disencryption as well. But I don't think there's that much interest in them because with AES and I, using AES XTS is the only way to get no performance degradation from using encryption. So I think it's fine to not bother implementing those algorithms. Galley also supports sector authentication where it actually has an HMAC of each sector. So your sectors have a little bit of extra data. Turns out that's not used very much and especially in our case if you're using ZFS, you already have a checksum so don't bother with it. Then the big limitation currently is it doesn't support the EUFI booting. Although Eric McCorkle has a review out for a version that implements it there completely differently, but hopefully that will go through and then we can have it for everything. Then once that's done, can we actually improve my code to make it closer to his code? Then there's a couple of other things I'd like to do some day. In Galley, I'd like to consider replacing all the SHA-256 bits with SHA-512 truncated to 256 bits, because on 64-bit AMD64 processors anyway, a SHA-512 is about 50 percent faster. Now I can't make the hash longer because all the metadata only has room for 256 bits of data, but to steal a trick from ZFS, I could use a SHA-512 and just truncate it. Then there's various other cleanups that I'd like to go fix all those get-ster bugs and so on and get back to creating just one unified bootloader that can do UFS and ZFS instead of having to have two separate ones for that. Also, after playing with Galley so much, I realized how nice and elegant it is, and it might be cool to create a fuse or other implementation for it so that you could use Galley and other operating systems. Especially, half the point of ZFS is that it's a file system that's portable and you can take it to another computer and just plug it in and it works. If you can have a Galley encrypted USB stick with ZFS on it, it'd be great to be able to mount that under Linux or Lumos or something. I'd love to hear your ideas of what else to do. Also, spend the time resuming the entire scheme? I talked a little bit about that yesterday at the previous DDEV summit with Baptiste. It's like, suspend resume hasn't been great on previous DDEV. It works on my laptop but not newer ones, but suspend to disk would be great, and it can't be that hard, right? Just force a crash dump and then have the loader just load that back into memory and turn on again, right? It'll be easy. So we have all that working very nicely, take our code. Okay. I'll have to look at that. We solved all that. Does that be nice to have? Yes. Here's the advanced ZFS book that Michael and I wrote. You can get that at ZFSbook.com. I do a podcast, BSDNOW.TV every Wednesday. We interview developers about things and talk about what's changed in free BSD and open BSD and so on. That's good. That's it.