 Thank you for coming to my talk on Pivotroot for BSD. I just heard of you this morning that this is my second talk in a BSD conference only. The first one was seven years ago and this happens to be the second part of that talk. In between I was working on Ramdisk file systems, small systems, did a number of tutorials and also investigated NECRAF on free BSD. I've worked a lot on free BSD and I have a business that also manages free BSD systems for my customers. And recently about a year and a half ago, I then became a member of the Net BSD Foundation because I actually started moving over and looked at some of their code and it was very... I learned a lot from it and then I figured, okay, let's do something more in that area. So it's actually quite nice to compare and I urge everybody to try to do this. And I've also given a tutorial in comparing the two in Japan but I noticed that the interest was not that high. It was a bit strange. The Net BSD people wanted to see the Net BSD stuff and the free BSD people wanted to see the free BSD stuff but the comparison was not what the people were expecting. So here I'll be showing you the route which I've implemented in Net BSD but I plan to also port it to free BSD. So it's a very simple structure of my presentation. First I'll tell you what problem it solves, what it is actually. And then I will present, I'll show you how it solves it and then I will talk about the implementation. So I've noticed... I've been talking about Pivot Route for ages because it was the end of the other talk I gave and people keep asking why you're going to do this, why you're going to do this every now and then and I said, well, there's not that much interest. But I noticed also that there was also misconceptions so I'm going to tell you what it isn't first just so you understand the difference. And then I'll show you how it runs and then of course I'll show you on the command line. So most people say, oh well it's just truth, isn't it? But we have that. Well it is something like truth but it's not just for one process, it's for the whole system. So truth obviously puts a process into a different route and from there on all the sub-processes are in that route. But it's not that. Initrute gets closer to it. Initrute is a mechanism where the first process of the Unix when it boots goes into a different route file system than you would have in the beginning. This actually is a workaround for not having Pivot Route because when you boot into a system with a RAM disk, for example, you then have Init already running and it's stuck there. So Initrute is a little just control or chem if it's in free BSD and that would then tell Initrute into this other file system which will be there at the time you're doing it and from then on all the other process of course on a PID1 will be in that new route. So you've actually pivoted into the other route that you've prepared and you can say in a way that's to pivot. But the problem is you're doing this at boot time and that's it because once you start Init, that's where you are. Now people sometimes say, well, it's also jails, it's also changing everything. The route becomes something new and you have a new network stack and some user in a jail doesn't realize that this is an upper route, super route above. So well, it's not that either. So the question is what is it? And I think the best way to understand what it is is to look at how you would look at the file system structure and see what you want to do. So obviously we have the root node where we hang off our file systems and we always have the root file system at least on small systems. We only have that. And that file system will then have somewhere down the line a directory which for these presentation I'll always call new root. It's a directory on the root file system. And on there you will then mount another file system. So to genose it covered the directory, the node, and this is actually a bit of the complication we'll see later. It covers that and you have the new root. Now the new root obviously should have very much the same infrastructure they have on the other root. So if you want to make that the new root, everything is where you expect it to be. Slash bin, slash sbin, var and everything else. Then in that file system you will have a directory called put old. Just a directory nothing mounted there. And now you want to pivot. So what you do when you pivot? You actually just simply move that and you're done, right? So not very hard, you would think. There's the only thing though, now root is in the middle. It's not on the top anymore so you have to relabel them. So now you have the end state where the new root has become the root of your running system. And the old root is on put old where you then can go and clean it up. I mean you usually want to get rid of the old root. That's why you're pivoting away from it. And you still have it under your new root to do things to it before you can unmount it. So we look at it once more quickly. And this is I think where the name comes from because it's quite an old name. And I'll show you on the command line how that looks like. So on an MPSD system, we usually have WD0, WD1 devices. And on a small system where you only have root and everything's in the root, this would look something like that. And here's a copy of root on the second disk. And I happen to have it mounted on new root. So we're talking about the system call, but obviously we have to have a user space program to call it. So this is a user space program which for all practice purposes, we can call pivot root. We can call it something else. I'll come to that later. And we say, well, my new root is on the mount point new root. And the old one will be on put old. I'm using these names because it's traditionally called like that in the system call I'll get to. When you do this, you get what you expect. It's flipped. WD1 is now root. It's on the mount list on top, because you want root to be first on the list in point. And WD0 is on put old, reachable from root. So you can still do things like clean it up. So the question is, does it run? It didn't run for a long time, but it does now. So I was going to show you quickly. And I'll be showing another demo later on, more sophisticated. I just wanted to show you how it runs. OK, if I just screen. Oh, I see. I'm running a virtual box here with two disks, same size, two adapters so I can go to the outside. I can lock in from the inside. It's running an FBSD head. And here you can see it's running head. And it's a kernel loadable module. So I have to first load the module into subpoena. I can type when I talk. I mean, big deal. I can type and then I just pick up the microphone and talk. So I loaded the pivot root module, because it's a kernel loadable module. So if I forget to do that, I just call pivot root. And it's not in libc yet, so then the user land program but since I loaded the module, then the code is there and the root can run. So actually, here I have WD0, which is root, and WD1, which is the root. I just call it like that. And I have a PROCFS mount, because that's pretty standard. So we're going to see something happen here. Yes, something else I actually do when I was debugging this. I would put a WD0 empty file there on WD0, obviously. On WD1, I would have one, because when you switch, you want to make sure, am I on the right one? They're not totally identical. OK, so we go. So put all of that directory. And now we have the situation. So you see PROCFS followed along. That's obvious. I mean, it was on the root before, and it went along. Just a mount now, PROCFS. I'll talk about un-mounting the other things. But my other demonstration will be running on a slightly different setup. So I'm going to now reboot the VM, and we'll look at it later. I'm going to hold it because I'm going to boot into a RAM disk later on. OK, so back to the presentation. So yes. So actually, maybe this is more an important question. I mean, first of all, it has to run. But the second thing is, why do we want this, actually? We've been able to live without it for 10 years. Piviru comes from Linux. This could be a reason why we don't have it in freebies, even because it's Linux. But also, there was no need for it really for a long time. So why have it? It just doesn't make sense. Seven years ago, I discovered that this was called Piviru, when I was doing my talk about RAM disks and so on. But there was the in-it-root solution that came out, and it was good enough to move into a different root when you were booting up. And so there was no pressure. But when you work with embedded systems where you want to upgrade firmware, or very often, your root file system comes in the way. So it would be handy to have it. And large boxes take a long time to boot. But you might want to fill up, upgrade the full user space, but you know the kernel won't update. So you just keep the kernel the same, but everything else can change. So what is actually Linux in-it-RD do? Probably most of you know who ever worked with Linux, but it's the two-stage boot process. They go into an initial RAM disk, and that's why it's called in-it-RD. And then that in-it-RD looks for the root file system. It may be loading more drivers and doing other things to get to the root file system. And once that's prepared, it mounts it, and then it pivits into it. This is, for most Linux users and even admins, totally transparent. They don't even know that this Piviru thing, Magic happens there. But then their system is finally booted. So at the end, you can free, of course, the initial memory, the in-it-RD memory, which we want to do as well. So on NetBSD or previous D, people now would like to also encrypt their root file system with CGD or Jelly or DVD on previous D. And there you need to do exactly this type of mechanism. You have to boot into something which has enough BSD infrastructure to decrypt the root file system, and then you can go mount it. So you generally do this with a RAM disk, and then you use the in-it-root mechanism. So right now, we, of course, do this with Piviru. Where does this actually idea come from? It's really old. I mean, that's how all the OS installs themselves. They come up, and they go into a Rambus. If you're here on, you have no writable media, and you take the memory to do that. And then from there, you prepare the media you want to install your OS on with manipulating the disk, and then later on using Truth to get it all down, and then you reboot into it. Now, in this case, you wouldn't have to reboot. You could just pivot root into it at the end, and the system would be running. Now, obviously, sooner or later, you will be rebooting, but you don't need to. And it might be good for a live file system, for example, where you're just never going to reboot because you don't need it at the end anyway. So this idea is not revolutionary. It's just an incremental change. The change, I think, is that you can do this any time, not only at boot time. And this is actually the firmware upgrade situation, where you're doing it at the end of the lifetime of an appliance where it gets the new image. It will actually do the pivot root to do it. So how would you do an upgrade of a firmware on an appliance? You would have a running system. You would take its full root files, which is maybe some read-only root files, as to which you would then put into memory on RAM disk. You would then pivot into that. So you're still running the same exact system. And this frees up the old root, which you then can do whatever the manipulation you need. And then you can do a second pivot root back into the new state. So obviously, this works perfectly unless you're upgrading your kernel, which then you have to first boot to take advantage of. There's also a situation where you might just have your hardest for some reason, software, or hardware not doing what you're expecting. You need a new root. And you don't want to boot your system because you might lose evidence for some reason. You want to keep your system running, have the old root there to maybe fix it, or investigate because once you boot, you lose all that information. So it'd be nice to go to a different root, which you know is safe, an NFS server, work from there, and investigate what's actually wrong before the system totally locks up. This is a picture from Windows Support. And they just say, well, you know, if you have a disk for a reboot, don't know how that's going to solve the problem. But there's a check disk, I think, on Windows 7, which is a bit more sophisticated. So they're selling that. It's a nice, cute photo here. So that's the description of pivot root. Hope it's clear that this difference between in-it-root, show-root, and pivot-root. And now the question is, how is this implemented? So I thought it was actually pretty easy to start, but I stopped working on it. I had a version in previously, which kind of worked, but it always panicked at the end. And I was never able to remove the old root. And every time I touched old root, the system would go haywire. And I lost interest because it just couldn't solve it. And I left it for a while from actually a few years. And then when I started looking at NEPISD code, I said, let's try again. And so you have to ask yourself a few questions. How do you want to do this? Earlier, I just put it into VFS mount and made it a different way and hacked it a bit to see could I get it to work. But then when I was doing it with NEPISD, I wanted to also become more familiar with kernel loadable modules. So I said, well, can I make a new system call? Because I want to make a system call which is actually Linux compatible. And I want to put it in a kernel module. So it's also a learning process for me. But later on, actually it occurred to me, and I also saw some code from Scott Long that had written something like that while it never really worked a few years back. And it was indicated into mount system call, which I think is quite attractive. Because you don't need a libc bump or anything. It's a different type of file system, you pivot. And it's all part of the mount system call. There's a drawback to that, too, that I'll get to that later. And you have a user land stuff. It's either the mount call or the mount program or the pivot root program. And that program, of course, could help you do other things before or after you invoke the system call. And the final question I had is, what do we do on FreeBSD jails? Because a FreeBSD jail, it cannot do the pivot root, per se, because it would turn the world inside out and everybody else would be underneath its own root. But you could imagine that the jail itself can do its own pivot root in its own little world. And actually, this is an attractive thing that I'll talk about later on. So what does the user land program look like? This is just the important part of it. It's really straightforward, totally straightforward. But one of the things that I think you have to do right away is to make sure that the input is validated. So you make sure that you can call relative paths or you call real paths so you get a nice string. This is important because these strings will eventually go into the mount list. And if you can do real path in user space, that's good. And obviously, that doesn't stop somebody from using your system call raw. But if they were asking for trouble, that's what they get. Here, if you use the user space, this will be guaranteed to have clean path names because real path takes out all redundant slashes and dots and anything else. And then you can do some more possibility checks here. This is visibly underneath each other and so on. And then you call the system call. So all the work, obviously, is done here. One more thing that you might want to do at the end is signal the process number one. And this is now I'm coming to the additional difficulty that you have when you do pivot route. It all seems so simple now, and I want to make it look this simple because it is quite simple. And the usage of it should be simple. But there is the problem that processes like one in it or other processes. They still have file descriptors on the old route. That doesn't really change. You'll see in the system coordination that we can do some things for the process, but we can't do everything. So maybe in the user land program, this is just about as we could do it this way, we could tell in it, OK, reopen your file descriptors on the new route. Kill hub. Yeah. Well, signal on C, kill minus one. Just something that a hub signal. Hub means paying up, reopen your file descriptors. So now about the system call itself, the other part, the hard part. When you're doing the pivot route, as to what you have to contend with the virtual file system for every structure. And this is actually the part which seems easy, but then it gets very complicated because it's all over the place and you can read the manual many times. For me, it seems straightforward, but it's not so easy. So first of all, the root V node, it's a global variable. So you're going to be changing that at some point. That global variable is in the kernel and you change it at the right point. You have V nodes to look at. I'll tell you what the V node is later, but they are connected to the directories that you're working with. There would be new route and the old put old. And of course, route mount point, the new route, and later on when you put the old route on the put old. So these are the mount points you're contending with. And all these structures have locks and references, meaning that when you use them, you raise a reference by one or if you drop the usage, you release the reference. And this has to match up otherwise, eventually there'll be a problem later on. You have the name of the local, which is the way that the path name gets translated into a V node. There's a cache there, so you have to make sure that cache is handled correctly. And then you have locks on the mount list itself, because it's one big mount list, and each mount point in that list again has locks. And actually, once you get all this straight, you still have the problem that the running process themselves, they all have a current working directory and route directory, and since you're shifting route, on them, they have to be told about this. So this is actually what we need to do. So how will we start the implementation? We first, I did it several times, but I thought this was the best way to do it. We first get the V node of the old route. That's where we're going to put the route later on. It's actually below in the route, but this has to exist, and if it doesn't exist, we don't even go on. Then we get to MVP, the new V node. This is all without locks. And we, oh, there is a lock. There's a lock in the beginning to go into the system code. There's a reason for it, I'll tell you later. So we only have run instance of the system call running, because otherwise we could create hairy race conditions. But then we get the route new V node, which is above the put old. And we check to see, does it all line up? Meaning that the put old has to be on the new route file system. New route has to be the route of a mounted file system. And it has to, well, if it's not under it, then we just exit in valid. Now, if the process that was calling this was actually not under new route, we should move there before, because otherwise we'll be blocking ourselves. So you silently move to the new route so that you're in the route later on when you've done the call. This is all preparation. And then once that has all kind of been set up without any locks, you go into the mount list and you start the first part. So you move it to the, then NMP is the mount point, and you move it to the top of the list so that when somebody does mount or DFA gets the list in the right order, it's also necessary because other places expect route to be first. And you then, for each of the mount points, adjust the stat FS, which you saw happen with PROC. And then this is where you do the main work. And you don't adjust route keynote right away because you have an interesting routine in NEPIS, you know, so in pre-BST to signal all the processes that the route file system, that their mount point has changed. This is used by the mount system call to tell a process, listen, you may be underneath a file system which now is being mounted over you, so they have to move over on top. And the interesting thing is that routine actually changes the route keynote when you call it from the top, from the route keynote itself at the end. Now, there's different cases in pre-BST slash DEFFS, the device DEFFS, so you actually would have that moved down too, and that would be very bad because anything that would happen right after the pivot route would not find slash DEFFS and would probably fail. So you actually do it the favor and inside the system point also move DEFFS back again to the new route node, you hook it up there. And in pre-BST, in NEPIS you don't need to do this because there's no DEFFS, but in NEPIS you have to update the current route variables. And this is not so easy because they're actually, well, it's obviously doable, but they're constants, they're read only, so probably have to delete them and remake them for the new route. That part I haven't done yet because I discovered it just recently that I want those to be correct. If somebody looks at this control variables, they better be correct. This could be misleading otherwise. So, I'm not gonna go into the code further because once I commit it and I will, we can then all look at it in detail. But what do you have to work with when you're doing DEFFS? You use the name I, and it's infamous name I. Everybody wants to make it better and do its thing, but it's complicated. But the nice thing is in NEPISD we have the name I simple kernel and the name I simple user, which are actually, they do most of the work that name I does for the most common cases. So kernel means that you take paths that are from kernel space and translate into V nodes. And if you have user space data, when the system call gets started, of course, it's user space data. You get a V node and you copy the user space data into kernel space and then get the V node of new route and put all. So once you have that, either through the end in it name I or through that interface, you then can reference them with, you change references by calling Vref or you release reference by counting down with V relay. So these are inverse operations. You can lock a V node. If you need to change these, you have to lock it. You can decide to lock it immediately when you're doing the name I call. So it's inside the name I call with a flag you lock it. Or you can lock it explicitly with V in lock. And obviously when you're filling with the V node you have to lock it. But I saw that it was much better to actually lock towards after all the validation because you don't want to accept, you don't want to lock against yourself. If somebody gives you two V nodes on the same file system, you could lock against your own route file system, which then of course makes the kernel hang. So you do it unlock when you do the validation to make sure that the V nodes are underneath each other and then you get the locks to do the work. And then the mountless locks are our mutexes. This is for the whole list once and this is for each mount point. And this actually is the magic sauce. When you call this after you've set up all the right things, you call mount check tiers with root V node. And it realizes that you've set this up that root V node is actually mounted on new route. And at the end of telling all the processes that this is your current working directory and the root directory has changed, it will then actually change root V node to new route for you. And this part actually is in VFS mount C already and it has the correct locks. Works very nice because in a way you are mounting root once more somewhere else. So when you're doing this, how do you debug all this? I had to, I just couldn't write down a code to make it work obviously. So I had to do debugging and on NEPST, I found some nice routines that do this for you. So you get output like this and you stare at it and stare at it and try to find out, well, the reference is there. Important, they go up, they go down and if you do it the wrong way, you then have either hang or panic later on, maybe much later on, that's the interesting thing. Then you need to look at the flags because you will be changing these. And if you're mounted, you're holding the V node as well. And you may have locks so you see that there's a lock and this is a V node which then you can match up with another V node you're watching. So by looking at this debug, as you're doing the debug you see what's happening and you see what you need to do to make it really work. And mount points very similar because obviously they interact with each other. Mount points cover V nodes and then you see which V node is being covered by the other. You can see the flag if it's through files or something else or where is it mounted. And then also the status information which is down below you see what the paths are which of course you're changing as you go through the mount list. So these two routines are actually already there you just have to print them out in the right place and you obviously have to look at the different V nodes and they're obviously pointing to each other because it's all forward and back linked lists. So you can find out where the V node is actually mounted because the V node may be down somewhere but it's mounted up on a root or a newer. You know you have a mount point, yes. And then if it's the mount points there you may be covered so you have to look at this which we of course modify. And then in certain cases you then have to follow right through and I look at all these to get the references and locks correct. So what things can go wrong? Well I thought I was good at security and I had called which kind of was working for a long time and I didn't think about well if I let somebody do pivot route out of a true environment this is gonna let him see everything and everybody else will see nothing so that's actually very bad. So the first thing that this system called us is checks to see am I being called from a true environment and if so be prone. But this was not in there for a long time so it would have been a gaping hole. When you were validating input I first was using locks and then in certain situations where I would give it mount points and new route was on the same path so it would lock except the system would freeze. And then I would have to go into the debugger and find out why is it doing that? Oh it makes sense and so on. And there's also a problem with when you're taking two v-nose and you shouldn't lock two v-nose at the same time because this opens you to a potential race because you're trying to get one and another it's just a course of somebody else is getting the other one in your deadlock. So I haven't found a way to get this solved completely but since I'm breaking the rules in pivot route I make a lock around pivot route and I also have the lock on the mount list and I try to keep that area very short but I need to have both the put old and the old route locked at the same time to the flip the certain values. Can't do it one after the other. I may be able to fix that still but to me it seems I need to have both locks at the same time because I'm getting a value which I then put in the other one. And okay phenomenally what happens, when you look at your system after you've run a buggy pivot route a few things can happen. For example, it may seem to have worked but you panic when you reboot which means that some reference has not been counted down correctly and it will tell you when it reboots. Oh I couldn't even unmount this thing it's too many references and then it panics while it's shutting down. So that means there's a problem in your code and it's latent there but it doesn't really seem like there's a problem until you do the reboot. Or obviously if you access the old route under the put old it may lock up your system. I had that too because you've left you've left the reference too much and then you see it just it locks on you. So this actually then shows you how complicated VLS can get if you try to fiddle with it. I talked about the processes which run on they get moved, their current working director but their file descriptors of course the processes have to contend that with themselves. I mean there's no way for a secure shell demon to know what to open on the other side so you have to restart these services. So that's something that actually the administrator is left to do. He only has the possibility to work with put old and make sure that all the processes remove their references on the old file system. So that's how much I've done till now and this is what I tend to do. I'm still not so sure if all the code it's not really much code actually it's one or two subroutines that I could actually put into the mount BFS mount code and just support the minus T pivot option. But the drawback is that then it would always be in the kernel and making assist control turning it on and off. I mean it's just added aggravation for root to have to do because either you lock it out and then you have to reboot anyway which is kind of pointless because you were trying to avoid to do that. And if you make a kernel module then of course it's isolated. It doesn't, your kernel proper doesn't have this but anybody can load it, right? So I have this right now which means it's easy to do this. It's probably harder to do this than this. It's better to isolate from the rest of the tree. Then once that's done obviously it's nice to support the pivot root system call. So you would have it in the system calls that we have the compatibility library which is have a little shim to also support that. And I still need to update these persistent variables. And when I have running code for NetBSD but I do plan to do it also for a previous year with some help from previous people and then this last step has to be done. But this is actually the easy part because it's just moving one thing straight forward. And in NetBSD you could ask well should the kernel also look at is there a Dev PTS and if so move it over because there may be network connections and so on but I'm not so sure I'm gonna have to discuss this on the mailing list if we want to do this. I think it's the Unix philosophy is to do as little as necessary and let other programs put them together, these small imperatives put them together to make the complete solution. So I'm gonna show you another demo a bit longer. Yeah, turn off time. And what is it? It's actually a very large NetBSD kernel, 23 megabytes. I didn't compress it, I didn't strip it, nothing. And it's a generic install kernel, current head. And it has an eight megabyte RAM disk built in. It's a custom RAM disk with a secure Sheldemon running and a root file and a root home directory and SSH or thrice key so I can log in to the system from the outside, I'll show you this. So I'm in the RAM disk. And then I will, well PivotRoot came out as already loaded or I could load it and then we'll switch like I did before. And then I will restart all the processes to in fact get rid of all those file descriptors still hanging off my RAM disk this time because this is my root. And then I unmount the RAM disk and you will see that my system is still running and I can still log in from the outside. So this is a way, this demo is showing you step by step the init RD of Linux, I mean 10 years later. But it's instructive, it also shows you that it works. And I'll just now drive this through and then later on we can maybe do some questions. We probably do one question now. Reason being that when I boot this thing into the RAM disk, you will see it will take a very long time. Well a very long time. I think it takes about 30 seconds. Because NEPISD when it boots and it discovers that slash def is empty. Def console is missing. It says, oh I have to make the def file system. So it makes the def, it uses shell make def it makes all the devices. And that takes a long time. Because I mean this is a virtual machine so it's not fast. And eventually we'll see it making these devices that unfortunately takes some time. So you can't really see it but it's all green until it discovers the root file on NEPISD and then it's doing this thing which takes 30 seconds. And then once it's done I can then log in. Also in virtual box when you start with a different DHCP setup it changes the IP. So it's not 101 anymore. It's 14. And I'm gonna have to log in as root here because I only have root on my RAM disk. And I have a secure shell secret key so I don't need to type a password. So did something happen back here? Yes, it's finished. It started secure shell demon, blah blah blah. It actually loads the pivot root came out because if I forget I get caught on this kind of ugly in a demo. So it should be listening here and I can then log in. Yeah, first time. Oh, yeah, well actually I shouldn't do this. I have this thing called ish which it's a stupid little program. It actually just doesn't store the secure shell keys because if it's a one time thing you don't want them stored in your known host because it'll just pollute it. So ish, it doesn't do that. So see I got in without answering the stupid question. So I'm in the RAM disk here. Root is quite full, 99%. But I have a nice selection of programs here. It's crunched of course. So, whoops. Yeah, typing with one hand and then turn caps a bit bad then. All right, so there's 116 commands that are called crunch together here. It's very similar to rescue but a few have thrown out and a few have put in to have the networking part in. And it's all crunched into one program obviously. And pivot root's crunched in there too. So now we have the MFS, this eight megabyte thing and we have, well there's nothing here yet because it's all in memory and now we mount the device W0A, the one that we looked at before. Yeah, slash is missing. So we, excuse me. Yep, good hint. This is far up as it can go. All right, so yeah, thanks. Hold up plus. Sorry, so this is the root we had before. Two handed works better and we pivot into it. Okay, so that's what we expected. But actually I should have showed you the PS, the which processes were running because we have to now take care of these. So yeah, what do I do now? I actually, this is the last command I'm gonna type in. I'm gonna start ETCRC. This is what actually it would do. And the funny thing is actually it's reconfiguring this network again obviously because it was the system with the config in RCConf. And I get to keep, this is a network connection obviously. So I get to keep that network connection and now I could actually go in. This of course will fail because I can't check root. It's a rewrite but that doesn't bother me. But now I have all the stuff already running in the new root. And I could now unmount the old stuff. But I'm not gonna show this. So we're done and maybe it's actually time for one smart question, maybe. And then we can go to lunch. So, right. I'm intrigued about the possibilities of the Linux simulation of doing the Cisco. Can you think of a use case? Well, intrigued. The thing is we have a list of Cisco's that we can emulate with changing a few arguments or something to make them work the same way. Now, I don't think there's a large use case because Linux, why would you be running PivotRoot on a Linux application? This is operating system infrastructure. But since we have Pivot, we would have PivotRoot. We also put it into the compatible library just because we have it. Not because we expect an application to actually use it. And if it would use it, it would not fail. It would work. So I agree with you, there's really no real use case because if you're doing PivotRoot, you're probably running Linux anyway. I had a question myself. And I don't have an answer to it, but we have a very cool testing environment on NetBSD. And I don't know how we're gonna test this though because when you test PivotRoot, you're changing everything and the whole test automation will then, of course, start failing because it's in the wrong route. So I don't really know how we would test that. With the freebies, GGL, it would be easier because you put it in a GGL and it would do it in that level and you could look at it from the outside. So, what? Yeah, but still you're changing the route. But how would your Anita's test suite then continue in the new route? Okay, so the basic idea is you can use one to one user land kernel and you could mount some parts to it. You can also bind additional processes to a degree and then you can do the PivotRoot and test that. Okay, so the missing link is having the kernel running on the rump. Yes, okay, yes, that's actually a very good point. So I have a way to test it, I have no excuse. Okay, I think otherwise, feel free to come and ask me questions some other time. And have a good lunch.