 Okay, so we'll continue with our show. So, the next talk is titled NixOS Assimilate. To introduce our long-time NixOS contributor, I'll first start with how to pronounce his nickname. If you are in Germany speaking, you will pronounce it as Schlig. If you are English speaking, the pronunciation correct is 8. And it's actually a person that does not need to be introduced because you can just read its pull request and commits and you will know everything you need to know about it. So, no further ado. Yeah, so the NixOS assimilate issue was open like about a year ago since a year ago and so I decided to finally crunch it together but didn't manage to fully crunch it together right now but it's getting there next week and it's actually a system which is comprised of actually three parts and two major parts. One is the partitioner, which is Nix part and the other one is a device-mapper module which is... Well, I'll get to the second one in a minute. So, this is the boring part. It's just a partitioner. Well, it's using Belivit or Belivit. It's a Python library for doing partitioning in a tree-like manner. So, you have a device tree and you build everything or every storage device on top of the device tree. So far, how this... It actually was in NixOps or it is actually used in NixOps and so far we only had the way to do manual partitioning or if you want to get NixOS on a machine where you don't have physical access to you usually need to either keep the existing partitioning layout or shrink one partition and create another one and other things like that. And for that, I initially introduced NixParch to NixOps and this was actually a quick and dirty hack. It was by monkey-patching Belivit at that time from getting rid of all the dependencies on Anaconda which is Belivit is basically a library which is used for the RedHeads installer which is called Anaconda and they separated the partitioning stuff into the Belivit library and so the code... If you want to see it, I can show it to you. It's monkey-patching but... So we're heading towards NixParch 1.0 which is basically... This is the old format which is using... This is what the Hetzner deployment is actually using for partitioning. This is the kickstart format which is quite compressed and we actually want something more... Let's say NixOS modular which also has one major downside which is basically it's not so compressed. So this is the new partitioning or the new storage attribute or the new storage attribute that will be introduced more or less in this way to Nix packages. My plan is actually to have it side-by-side by file systems because I don't actually know what to do with file systems whether to keep it or actually use just the storage definition to statelessly create the partition and file system layout. The problem here actually is that you see... I think it's... Yeah, VDA and VDB and that's the problem where you actually either have to specify device IDs or introduce labels or some other ways to manage the state or to actually... If you, for example, add another disk, the NixOS system is unable to mount the disks because it will just say, well, VDA or SDA could be in any order. So, yeah. This is actually a small demo of the Nix part implementation. Bigger... Bigger part? Larger. That's actually in QEmo. Maybe it doesn't support it, yeah. Ah, okay. Anyway, is it now readable? So, this is actually a... May I have... This is actually a quite simple storage configuration with a battery FS and the existing file system layout is actually unpartitioned and so we call the Nix part... Nix part itself is called by... It's using Nix instantiate to actually evaluate the expression, which is... and dumps it into XML, which is more or less Nix instantiate minus minus or dash dash XML and uses that for creating its device tree. So, we should have better FS and... Oh, well. That's a buggy version anyway, but it should work. Yeah. So, we can mount... Anyway, I think that's the reason why it's more or less boring because it's... Well, it's a partition. What do you expect? So... Yeah, shut down. So... Now it's getting quite interesting because... As I mentioned already, it's the question whether to keep file systems as a low-level attribute or not because actually a file system itself is more or less creating mount units by creating an FS tab and then using the converter for the system d mount units. And this actually has a few drawbacks. For example, if you have network file systems or multiple volumes and things like that, you probably want to have proper mount units for them and also... If you want, for example, to change your partitioning layout, you have to have the original input, like, yeah, I have a lux container and an LVM in it and whatever you want to box into that whole chain. And... Yeah, so basically... I would go for a stateless approach right now. It's... Right now, how we do it in the Hetzner target, it's like do the partitioning and afterwards spit out a NICS expression, which is... Which is... Which contains all the file system and swap and whatnot attributes. So... This is... Now we're getting to something that's... Well, a bit more complicated. So having a partitioner is quite boring, but the interesting thing is... Well... We can assimilate basically every OS more or less. So... Which is basically... And now we're getting to the interesting part because I don't have any nice and graphical slides here. Next to assimilate is basically a device mapper module that or should be a device mapper module because it's working correctly. Which basically works in three steps. And... Yeah, let's get to the steps. And this is... The first step is actually marking everything that's... So you have the old partitioning layout, you mount everything, read only, read all data you want to still keep. So it's marking every block that's basically used and that has the advantage that we can figure out which one or which part of the disk is not used or you don't want to keep. So for example, if you just... Which is quite common. If you just want to keep the next door, you will want to go from, let's say, Windows to NixOS or Debian to NixOS and change the partitioning layout. It's basically... You want to keep just the next door. And so far there have been other approaches towards this like pulling up an image and trying to route from a particular image and I would consider them hacks like that. And even the Hetzner one is a hack because we rely on the rescue system and we can't simply say, well, we're already on the system, let's assimilate this one and it's doing a K-exec or reboot into a kernel with a special in-it RD and the device-mapper module and we are getting to those three phases. So once we have marked everything, we get to phase two, which is we are writing the new partition table in the unused portions of the disk. So the ones you obviously don't want to keep. So in theory, I haven't thoroughly tested this. This should work with our file systems. Like if you... they shouldn't normally do something, well, let's read at a random location on the disk, I hope. And the right phase does have another table which marks everything or which tracks everything that has been written and also the offsets or the block offsets of where it usually should be. This is something a device-mapper already has. You can basically remap every block on the disk to every virtual address. And the last one is the remap part which is simply doing everything in reverse like write everything to the locations where they should be, which is not so... Well, we have to consider the order because if we write... if we have a marked block from the second phase which then corresponds to the new physical location, we actually will overwrite ourselves with ourselves, which could be quite interesting in the end. So actually, those three phases should get us to the point where we can seamlessly or more or less seamlessly, if we have something like a rollback which is... if we have something like rollbacks, it could be actually safe. I mean, theoretically, my goal was to, hey, does somebody have a laptop you want to migrate to another partitioning scheme? Let's test it. Well... Huh, beg your pardon? So, yeah, so the problem here is that if we want to have a rollback, we can't do this on a disk that's more than half of... that's allocated more than half of its size. So, we could do some workarounds, like de-duplicate stuff and compress stuff, but that would be something for future optimizations. But right now, my initial goal is... my short-term goal is to make this actually destructive in terms of just keep the next door and we don't care about rollbacks at least in the first stage. This is particularly useful for hosting providers. So, usually, like Hetzner and many more, you get a server somewhere, physical server or maybe a VM. Well, there could be a drawback, for example, if you can't change the kernel, but if you get a physical machine, usually for most of these hosts, you don't get hardware access, so you can't run in there with a USB stick and just plug it in and install NixOS on it, and that's it. But they usually come with pre-installed whatever. And most of them have rescue systems, but not all of them, and so this is an actual solution for having more or having basically a generic backend for NixOps, which is... you just point it to some host, you just need to have SSH access, and that's everything you need to have and you just say, well, I need this partitioning layout like shown before and this NixOS configuration, which is the same configuration or at least that's the goal. So you just have your storage definition in your NixOS config and you're basically doing the same as you would have NixOS installed with a config file where it would do partitioning, just you're doing NixOS assimilate instead. So NixOS install is for live booted USB sticks or for a live image and NixOS assimilate is for live systems, basically. Yeah, one advantage is because I thought about doing this by streaming the cloches via network, but this is quite unreliable and also the partitioning remapper has the advantage that we basically just need the RAM to start Nixpart and that's it. And maybe if you have more RAM then you have more disk cache which also gives a bit of performance improvement but it won't be a deal breaker for everything. So yeah, and what I said already, NixOS install and the config that would be everything for future NixOS installs. So yeah, encryption is one thing which isn't such a big problem. You just need to do, for example, if you want to randomize the data first and do your Lux containers or whatever or DM crypt on top of it and then you could, because we already know the unallocated sectors we can just write random data in it and we're done. And yeah, and if you have something like a spare drive or anything we could use that as well to speed things up because if we have enough space on there we can basically skip the second step and just copy everything to the right location. Yeah, that's basically it. And yeah, I will push the code. I guess, yeah, starting next week for the new implementation and also try to get the kernel module running but no promises here. Please don't clap. Okay, we have quite some time for questions so please. Okay, so on something like a Hetzner environment what is the advantage of this compared to starting from a rescue system? The advantage is, well for Hetzner you basically have no big advantage because you have the advantage that you have a better partitioner which is the next part but it doesn't solve anything new for the Hetzner target. It's actually more useful for adding new targets. Okay, and then maybe a small second question. Do you think it would make sense to get the next part into the mainline next source? That's the goal basically. It's already the branch on next packages which has these changes for the storage attributes but they are still subject to change and maybe if people disagree, oh no, we don't want it that way. So I have a question and a note. I'll start with a note. OpenStack has a module called irony for doing bare metal appointments including doing pixie boot and IPMI for being able to get access into the bare metal for doing these kind of partitioning things. The question I have, I'm a bit confused still how you determine the reachability of a sector to be able to separate the live sectors from the unused sectors. By having a device map or module which I use instead of the real physical disk device and via the device map or module I'm able to get all accesses from and basically mounting the volume and doing a read of all files I want to keep and at the same time at the module level I get the file extents. Any more questions? It's not completely clear to me how this is going to be used after conversion because file systems usually have something like reference counts and checksums and so on and when you just preserve some blocks of a file system and clear out the other blocks how is this going to work? Yeah, well, that's what I was already mentioning if there would be a read which is random or I don't know any file system which does random reads but that would be a problem but otherwise if you're doing a complete read of all the files you want to keep at least in theory it should be deterministic and that sense that everything that the file system driver is going to read will be the same as if we do it with a second read operation on the same volume. Basically we don't read, write, mount the volume. Okay, then we perhaps have to go step back but it's not completely clear to me what you're trying to achieve so let me try to paraphrase it here reading a portion of an existing file system then you're going to clear out the rest and create a new file system with only that portion that is going to be retained, right? Yeah, you're copying the new file systems and copying everything over to the new file system and then switch back everything over to the old one or remand everything on top of the old one that's more clear. I'll try to do it more or less let's say this is a disk which has these allocated slots and we are going to basically whether the new file system has more slots in it let's say we have a lot for files we are not interested in. So in phase two we are basically writing stuff in there and so we which are the blocks that usually need to come are in a totally different order and in the third phase we basically do this is the remap step so we have, yeah so there shouldn't be anything left from the first step that's, of course, without the option to roll back it's much more clear. I guess this is a talk maybe for after coffee or for lunch, right? But one more question. So I think I kind of understood what you meant which is like you mark all the blocks to access all the files that you want to keep Yeah, all the raw data. So for example, if I need something in slash, I will mark slash, I will mark Nick, I will mark store, I will mark the path within the store Well, you just read the file and the file system underneath it will say, well, I need to read this B3 and this block and whatever and that's where the module comes in because at that level we are intercepting what is going to be read. When you rewrite in the second step the files to the non-marked blocks will you take care that parts belong together like big files could be more than a gigabyte and go to the same coherence block or is it completely worst? Because you write everything together in the end. Okay, but you don't you don't change the order of blocks so when you write everything in one line in the end it will come out as if one file is still in one can you be sure that one file is always in the same consistency? If nothing Okay, we'll do a bit longer break I guess we can now have time for some extra coffee We started at 12.15 Let me just double verify 12.15 so we have 20 minutes, a bit more 25 minutes of break Thank you