 Rhaid i'n gweithio i ymddangos ar y cyfnod y tîm yma o'r llwyto ar y rŵl y cwntain. Fy fyddwn i'n Llywodraeth Cymru, ac rwy'n gweithio ar gyfer y tîm sy'n gweithio'r cwntain ar gyfer Clav Andrew. Felly, fel ydych chi'n gweithio, rwy'n gweithio i'n gweithio'r ystod. Felly, rwy'n gweithio ar gyfer llwyto ar gyfer Llywodraeth Cymru, ac rwy'n gweithio ar gyfer Llywodraeth Cymru. Ond dweud â bydd wedi eu cael ymgwrthoedd gyda Llywodraeth Cymru yn aeth, mae'n gweithio'r prgynod lleonill yn ei dweud. Felly, mae'n gweithio, roeddwn i'n meddwl, chwy adnwyddoedd y neud hynny. Rwy'n gweithio'r teimlo'r wneud. N-rwt, rwy'n gweithio'r roedd, mae'n gweithio, n-rwyth, doedd yn ein teimlo. Sgwyddwch! Felly, rwy'n gweithio'r cwntain. Ychwanegon o'r gweithio ziw yw'r wych Poes HelCA oherwydd rydw i gyd yn hynau. Yna ceisio, dwi'n gweithio'n gweithio a'r bobl? Yn ni'n gweithio'n gweithio fel rhoi ar gyfer gweithio'n gweithio, rydych yn gymryd y cyffredinol ar y cyffredinol yn dweud. Felly mae'n gweithio'n gweithio fel roeddiol o cookl there. Yn eu cyffredinol yn gweithio'n gweithio, yyn ni'n gweithio'n gweithio'n gweithio gyffredinol. i'r ffair o'n rhoi ar scrapwyrn bach y cyflod. Byddwn ni wnaethaf â'n gwirionedd hon. Rwy'n meddwl o'n gwirionedd. Mae'n bach so fawr, na wnaethaf panndwch astu'r iawn.isticallynydd yr ysgolter i'r cwestiynau. Yn Ffair yw'r ffair, mae'n gyfly saidd o gyflyneddau o'r blwyddyn sy'n gwirionedd gyda fi Klawfandri. Actually, that's exactly why, just before we start talking about rootless containers, we first need to talk a little bit about containers. And then before we talk about containers, we first need to talk a little bit about Cloud Foundry. this talk we're mainly focusing on the cloud foundry seeks application runtime as opposed to the container runtime. But actually like a lot of the topics they still relevant across both. It's just that if we're referring to cloud foundry we're kind of specifically talking about the application runtime here. So what is cloud foundry and we're assuming that we're at least partially familiar with what this is given. Where we are but just to sort of set the scene cloud foundry is an open source platform as a service, so it allows a chymlu'n gwybod i ddim yn y gwleidydd, ac mae'n gwleidydd yn y ffordd i'r gennydd yw'r ffordd. A oherwydd mae'n gwybod i'r gwleidydd a chymlu'n gwybod i'r gwleidydd yn fawr, mae'n gwleidydd yn bwysig yn lleiwyddoedd, a oes i fynd i wneud yn gwleidydd. Mae'n gwleidydd yn gwybod i'r gwleidydd. Ac mae'n gwleidydd yn fawr, mae'n ddim yn nhw'n meddwl ffawr o'r cyfrifio. Dwi'n gwybod i'r ffordd i'r gwleidydd, nad yna gweledd i'r gwleidydd. platform, oherwydd bobl sydd wedi cael ei fod yn oed â omlwch. Y dywed y ddweud o gyffredd gyfroedd y hollredd a'r rem ychydigion y cyffredd Platform, sy'n oed o fewn adeilad ac yn gallu Yn'r Ddenbyll Cymru, Pan-Famol I, i gyffredd. Y dywed y ddweud o gyffredd gyffredd cyffredd dywed o bobl ar gyfer cynllun a'r dŵel mwrdd. Mae'r ddweud eisiau bod chi'n ceisio ar gyfer gweithio ymddiol, oherwydd o gweld a'r grifelau ac yn grifelau, into the cloud and away you go. And so we look at this and we think, like, that's pretty concerning, actually. It's actually like a worst case scenario, if you think about it, completely public, completely multi-tenant, and we allow you to push whatever you want. That's pretty terrifying. And so it means that we have to make sure we are on the absolute forefront and the bleeding edge of container security. We have to be prepared for the worst. So how do containers fit into this? What even is a container? Let's start with that. And the first thing to know about containers is that there's no such thing as containers. Containers are a lie. What do I mean by this? Of course, containers are real. All of our applications are running in containers. So what I really mean is that containers are just Linux processes. There's nothing magic about them. They're just processes running on a VM somewhere. And the only thing that differentiates them from any normal process is the fact that these containers are isolated. They have layers of isolation around them so that they are isolated from each other as well as any other processes running on the VM. And it's that isolation that helps us here with the security aspect of this. So how does this work? Well, we use some kernel technologies, namely the Linux namespaces, which provide isolation of global system resources, so things like PIDs, mount points, et cetera, et cetera, as well as C groups, which provide limitation of physical resources, so things like CPU and memory. And there is a third one that we have to especially think about given Cloud Foundry's multi-tenant nature, which are disk quotas. So we need to make sure that applications aren't able to just write to disk and fill that up and provide an attack surface like that. But we're going to come back to that a bit later on. So that's kind of the intro. That's why we care about this. This is why we think it's important. What's coming up? So to begin with, we're going to take a bit of a deeper dive into exactly what makes a container a container. So we're going to look at namespaces, C groups, as well as dependencies. Then we're going to talk a little bit about some extra security layers that we apply on top of that to make sure our containers are as secure as they possibly can be. And then, finally, we'll get to rootless containers, what they are and how they fit into all of this. So, Claudia, what makes a container a container? Thank you, Ed. That's a very good question. Let's start with namespaces. So namespaces are a Linux kernel feature which allow you to isolate global system resources for a process. So these are things like mount points network devices and process IDs, et cetera, as Etta said. It's basically about creating illusion. When you put a process inside a namespace, you can make it think it has exclusive access and sole control of those resources. So there are seven namespaces in Linux right now. These are Mount IPC, PID, NET, UTS, C group and user. And this last one, the username space, is particularly relevant when it comes to rootless containers. So we're going to come back to that in much more detail later. But for now, let's just pick a couple which are used to create just containers in general. And those are the PID and mount namespaces. So let's have a look at how they work. Let me move my mouse. There we go. So it's up with the PID namespace. So with the PID namespace, two processes running on the same VM can have the same process ID in different namespaces. It's what it appears to be for the process inside the namespace when in reality that process is then mapped to a different PID outside these namespaces. So if you take this awesome diagram here which I drew myself, you've got two child namespaces there, child namespace one, child namespace two, and they both think they are running PID one. So as you know, when you boot up your machine PID one, it is the first process that runs and does everything and all the other processes are then children, grandchildren, et cetera, of that process. So it's the big dog, it's God. It's the God process. But the parent namespace knows that these are mapped to other processes inside the host. The child namespaces have no idea they have a parent, none at all, because PID namespaces are top down. The parent has clear view down into its children and its grandchildren and everything. The child namespaces can see its children, yes, but it doesn't know it has a parent. It doesn't know it has siblings. It just has only visible control over its own given world now. All right, so the next one we're going to have a look at is the mountain namespace. So processes which have been put in a new mount space have a different view of the mount table. When we use pivot root at the same time, which I'm going to come back to in a minute or two, that process will see something different mounted as slash. So mountain is root fs. You'll notice here that I've done df minus m print type to see the file system mounts. I just did that because it gives you a pretty output for slides. In reality, you probably do like at prog PID mount info or mount or anything like that to see the mount table. If you want to see more about what namespace your PID is running in, you would do your ls minus l, prog PID ns, and those links would give you the namespace IDs for the namespaces that your process is running in. All right. So if namespaces alter a process's view of the system and its resources, then C groups can actually be used to enforce certain power over physical resources. As I already mentioned, those are things like memory and CPU. Control groups are again the next kernel feature. And so every time we create a container, we will put that container process inside a C group and then apply limits to that C group so that it can't access more CPU than it should to ruin the experience of other people. As I've said, the other thing we care about limiting is discreta. And this caused us a lot of problems. So we're going to come back to that again in more detail in a minute. So the last thing that makes a container thing, we're all very familiar with and we love, and different from a standard Linux container, is the encapsulation of dependencies. So a few years ago, Docker figured out a way to encapsulate dependencies into shippable units within things called images. And they figured out how to do this efficiently. These images could then be moved around between machines and this meant that we could run many different containers based on these identical blueprints, which underneath we're using the same Linux primitives and mostly we're sharing dependencies. And this is the key difference between Linux containers, which have existed for a while, and the containers which we talk about today. Isolation by itself gives you a Linux container, but isolation along with the encapsulation of dependencies gives you a container. Container. So let's look at how dependencies and encapsulation works. To do this, we use a layered file system and pivot root, which is what I mentioned earlier when we were talking about the mountaining space. Let's start with pivot root. So pivot root is a system called, which changes what a process sees when it looks at its root directory. So instead of seeing the host file hierarchy, we can tell the container process to see somewhere else as its root file system. So say for example, you had a program called run.show. It asks a question, what's in my root of s? And what comes back is whatever is on the host. Something boring, because it's not a container, it contains a call. But then the next time it runs, what's in slash, sorry, everyone has to talk, slash we can do pivot root with a path to somewhere else. And this time what comes back is something much more exciting. Busy box. It's a container root file system, much more exciting. But we can't just point to any old path. Actually we could, but it wouldn't be very useful unless it actually does what a root file system is supposed to do. So we have to give it an actual root file system. And we do that with layered file systems. So in the past. Could you speak a little bit slower? Yes, of course, very sorry. This is your job. Team members, this is your job. Shout, you know this. I'm very sorry. I do, sorry. All right. All right. So it's fine. I'm trying now to count seconds in my head. So this is where layered file systems come in. So in the past, the way that we, by we I mean the community as a whole, used to handled dependencies was we would package everything up into an AMI or a VMDK or whatever and that was fine. If you didn't care about things like boot speed and filling up your disk with junk. So imagine you had a container which needed Ubuntu as part of its dependency package. So that container would come along, you would download Ubuntu, cool. Another container comes along and they're like, oh, hi, I also want Ubuntu. So you download that Ubuntu again and then a third container comes along and you kind of see how this spiles out of control because Ubuntu is not small. A lot of times dependencies are quite large. And this is where Docker really saved us a lot of pain. They realized this thing called layered file systems which is not a new concept. Could be used to get containers, their dependencies. So how does this work? Another awesome diagram. If you picture standard Docker files, the first line begins with from something we've taken Ubuntu. When a container's orchestrator reads that line, it will go ahead and download the contents of Ubuntu into a read only directory somewhere on the host file system. The next container comes along and it also wants Ubuntu. But this time, because the orchestrator already knows that we have one of those, it doesn't do it. It doesn't need to download it again. It can just point to it and say use this one too. So now they're sharing. We've already saved time and space. Doctor Who reference. And the next line, they run at update. Again, only one of them needs to do this. Whichever one gets there first will download the things that are required in that run at update. And whoever gets there second will just use the one that's already been downloaded. This third line, now they're doing something different. But this is still the first time that that second container created called has actually had to download anything. Until then, we've been sharing, we've been saving space on our disk. Finally, these are all then mounted into a read-write layer which becomes the container's root file system visible thanks to Pivot Root. So here we can see how layered file systems and Pivot Root allow us to share dependencies and stop container processes from interfering with each other's root file systems while they are running. Alrighty. All right. So at this point, we might be thinking, okay, we've got some pretty awesome isolation in place. We've got our C groups in place. We've got our dependencies sorted. We should be pretty good, right? Like, what else do we need to do? And of course, like, nothing's ever that easy. And so now we kind of need to think about, okay, well, we've got our isolation, but what happens like if or more likely when somebody breaks out of that isolation? What happens then? And what can we do to protect ourselves in that instance? And so to that, I say security onion. So of course, this is about applying layers. So there's some additional layers of security that we can apply to these containerized processes in order to protect ourselves here. So namely, I'm talking about capability dropping, set comp, and app armor. Let's just quickly run through those now. So capabilities. So historically on Linux systems, privileges tend to be divided into two. You had the all-powerful root user, UID0, who had the privileges to do absolutely everything. And then there was every other user on the system, so all of the unprivileged users, and they weren't allowed to do anything at all. And if an unprivileged user wanted to do some privileged operation, like do amount, do a pivot root, they had to sudo su up to the root user, at which point then they've gained every single privilege available. So it's not great. These days, those privileges have been split up into little chunks known as capabilities. Here are some examples. So we've got, for example, CapChone. So historically, if you wanted to chone a file to another user, you had to be the root user. But now that these capabilities have been split up, it means that actually any process is now has the privileges to do this as long as it has that CapChone capability. Similarly for CapSetUID. And there's about 40 of these in total. And you can see that that gives us a much nicer separation of concerns there. But there is one that we need to talk about, which is CapSysAdmin. And this has kind of become like a catch-all for any of the privileges that don't fit into any of the other capabilities that we have today. And so as a result, it's really overpowered and really to the point where if you have CapSysAdmin, you are still essentially the root user. And this is going to be really important later on when we start to talk about the rootless containers. So keep that in mind. But the reason we're mentioning this now is because it's actually possible to remove capabilities from our processes. So what this means is that when we've got our containerized processes that are running, we can actually remove CapSysAdmin as well as all the other capabilities that we want to. And it means that if that containerized process is able to break out of the isolation, we've actually limited the amount of damage it can do because it doesn't have the capability to do anything. So next up, we have SecComp. SecComp stands for Secure Computing Mode. And this is pretty awesome. SecComp basically allows us to limit the number of SysCools that our processes can make. So this is great. So if you think about it, like all of the security here, it's kind of relying on the surface area of the kernel. And the kernel's a pretty big thing. And so by giving, well, the fact that we are able to remove some of the SysCools from our containers or the containerized processes, that's awesome because it means that we are dramatically reducing the surface area for an attack there. And it means that, yeah, any vulnerabilities in any of those SysCools, that's no longer a concern for us because the containers don't have the permission to run that SysCool anyway. Finally then, we have AppArmer. So AppArmer allows us to restrict these containerized processes even further. So AppArmer is a form of mandatory access control, kind of like SE Linux. And the way this works is you create yourself an AppArmer profile, which contains a bunch of rules. So there's an example rule there. Deny at proc slash star w, which means that when applied, this process is not allowed to write to any file inside the proc file system. For example. And so we build up these profiles and we apply it to the containerized processes. And again, even then if they're able to break out of the isolation, we're limiting the damage that can actually be done in that situation. And so we have the security onion, capability dropping, set comp, and AppArmer. It's probably worth mentioning that in order to apply all this lot, you do typically need to have like Capsis admin and those root level privileges, which is going to be important later on. But the other thing, just want to quickly mention is we've actually covered loads of stuff, right? We've seen like namespaces, C groups, set comp, dependencies, AppArmer, like all this stuff. And it's actually really, really tricky stuff. It's quite low level sort of working around the kernel. And historically that fell to the garden team to program around that and to be responsible for that. But fortunately with the introduction of standards a couple of years ago, so particularly the OCI standard, this is now no longer such an issue for us. And in particular a tool called RunSea, which is just a very small binary tool that's published by the OCI. It basically abstracts away all of that hard work for us. And it means that we as garden can just cool down to the binary and it does all of that complexity for us. So it's been absolutely invaluable. I think it would be hard to overstate how good this has been for security in general. So thank you to RunSea, cheers. Okay, so at this point, as I said, we've covered the namespaces, the C groups, isolations, set comp, AppArmer, all this stuff like surely, surely, surely now we must be secure enough, right? So yeah, sure. So far we have suffered no major exploits. There is no word here, I can't touch word, I'm really sorry, sorry Molly. But it doesn't mean that we're invulnerable, not yet. If you picture the various exit points from a container as a door, then yes, we've made a really solid door. But how likely are the attackers to stop there? Pretty unlikely. Once you've sealed off the door, they're gonna step back, they're gonna look for the windows, which means that while we think we've done this, if we step back, what we've actually got is that. So we've secured the containers, but now we need to make sure that we secure everything else. We need to secure the things that are managing and running those containers. We need to secure garden itself. We've garden to not run as root. Right now, garden and all the other components, we're all running with mad privilege right now because we've had to be in the past. But things have changed, and now there's all sorts of primatism tricks that we can use to get to this. Yeah, so this is where rootless containers finally come into play. So the nice thing about rootless containers is it means we can actually create all of those containers without needing to be root ourselves. And the key trick here is username spaces. So we kind of touched on these earlier. So in the same way as we saw with the PID namespace, which allowed the PID of a process inside a container to be different from the perspective inside the container as compared to outside the container, such as the case also with the username space. But with the username space, we're talking about UIDs rather than PIDs. And this is really great because it means that inside the container, a process can appear to be running with a UID of zero or root. But actually from the perspective of the host and all of the other containers, it's just running as some completely random unprivileged user. And what's even more awesome about this is that actually any unprivileged user has the privileges to create a new username space. And so I'm going to put this up here in sort of giant letters because this is kind of the key point of the whole talk here. Any non-root user can create a new username space in which they are the root user. And once you are then the root user within that username space, you have capsis admin, you have the privileges that you need to go and set up the rest of the container. So that's pretty awesome. But username spaces can be a bit confusing. So I've got an analogy here. Hopefully this will help you. So let's imagine that in the host, we are an average frustrated user. We have no privileges. Our UID does not equal to zero. But in the container, hey, we're Brad Pitt. We are the root user. We have all privileges. Our UID is zero. And yeah, so this means that we've got that capsis admin. It means we can go and set up and create the rest of the container with all of those layers of security that we've just been through exactly the same as before, which is great. But the key thing here is that it's only, that those root level privileges are only applicable with inside that username space. So outside the username space, i.e. in the host and in the other containers, you're just an unprivileged user. Which is pretty awesome. So how does this work? Well, there's a map, a UID map. So as we can see there, we are basically mapping UID zero inside the container to UID 429 blah blah blah inside the host with a length of one. And that UID there, that's basically the maximum UID that's available on the Linux systems. We chose this because it's the one least likely to have any privileges or to clash with any other sort of UIDs. But there's a problem. And the problem is that by default, you're only allowed to map one single UID. So let's take a look at what this might look like. So let's assume that in the host, let's say we are unprivileged user UID 1000, Alice. And we say, okay, I'm going to create a new user namespace and I'm going to map myself to UID zero, the root user. That's great. But what about all of the other UIDs inside the container? We haven't applied a mapping for those. And so it means that if you were to look in it, look inside it, they would just be unknown and they wouldn't be allowed to do anything. And obviously, users expect UIDs to work. So how do we get around this? Well, fortunately, there is a solution, new UID map. And this is basically a small set UID binary. So set UID is basically, it allows you to run a binary with the privileges of the owner of the binary rather than the currently running user. And so if that binary is shown as the root user, it gives us a way to temporarily elevate our privileges, write the entire UID map, and then drop that down. So it's kind of cheating. It's not like 100% rule-less, but we kind of think it's okay. New UID map is a very standard binary that you'll find on most distributions. What about rid of s mounts? Yeah, so earlier I talked about how encapsulation is the thing which makes containers nice and portable and stops their dependencies from flooding your disk. And this is done through file system layering. Well, unfortunately, doing a root-first mount often requires root. So that was our next target when we wanted to become completely rule-less. Just quickly clarify by what we mean when I talk about our file system. There are two meanings associated with this word. One is the structure, just the files and directories, what you see when you go on and do LS and move around. The second meaning is the way that data is managed within that structure, and that is done by a file system type. So today I'm going to be talking about the second one, how that data is managed. So let's start in the beginning, back in the day when Garden was young and we weren't even called Garden. We used AUFS to create those root file system mounts for our containers. And we used AUFS for ages, and it was a headache. But we stuck with it because nothing else was mature enough for our needs. Well, either that or it was under proprietary licence and we couldn't use it. Our main problems with AUFS were that it was not in the mainline kernel and compiling that module went wrong more often than it went right. It was maintained by precisely one person, which meant asking for patches. It was not quick. And you have to be root to do an AUFS mount. But we were stuck with it until two years ago. Another option became very attractive. And because there was this huge push for us to get all of our components rootless, we created an entire team around our new root file system creator. And we called it GrootFS, which is a terrible name because it implies we created a new type of file system which we did not. We created something which downloaded the stuff and then gave you a directory at the end of it, which is still cool, a lot of cool stuff going on, but we did not make a file system anyway. We chose BetterFS to actually buy our new file system type. And it looked good to us because it had this quota tooling built in, which, as we know for Cloud Foundry, is really essential, multi-talented. And aside from needing a bit of privilege for setup, which can be done before we start our container server, it could snapshot our root file systems as an unprivileged user. And so we were very excited about this. But it turned out that that quota tooling we were so excited about ended up not really being very good in production. It could not stand up under load and the whole system ground to a very unpleasant halt. So we had to quickly turn it around and we chose our next favourite file system, which was OverlayFS, which is very similar to AUFS in that it performs a union mount, which basically means it takes the contents of several directories and make it appear as if it's under just one directory. The big difference is between AUFS and Overlay is that you can do mounts as an unprivileged user on Ubuntu. So Kononky Ubuntu had a chat with the OverlayFS maintainers and decided it was not a security risk to perform mounts as an unprivileged user, because not many, I don't think any, apart from is it ShiftFS does or soon does mounts as an unprivileged user. But Ubuntu now compiles a patch version of its kernel with the OverlayFS system white listed for that action. So it's very cool. It does tie us down a little bit, but it does get us by on our quest for rootless containers. We still use Grutifes to download all those layers and to chone all the directories to be owned by our Maximus user. But Grutifes will not do that mount. Instead, it will return all the information needed to do that mount. To RunC, in the form of the, is it the OCI spec, then RunC will perform those mounts in order. So we just need to make sure that the Grutifes mount is at the top of that list, that array, and RunC will get that to those details. And inside the username space, where it has all the correct powers to do everything it needs to do, it will mount our root file system for us as an unprivileged user. So it's very cool. So to sum up, we are running Garden rootless. We've got user NS and mappings. We have got Grutifes running rootless, which is creating a rootlessly owned of Grutifes. Are we now? Good. Not entirely. So there's still a couple of roadblocks left. I'm going to quickly run through these. Just aware of the time. So the first one is disk quotas. So we mentioned this earlier. We need to apply disk quotas to our applications to stop them from filling out the disk. And the way that we do this is with XFS. Fundamentally, XFS disk quotas require root-level privileges to apply. So the way that we worked around this was to just extract the tiny piece of code that was actually responsible for applying these quotas into a tiny binary. And then again, we applied that setUID bit to it. So very similar to how newUID map works. Next up was networking. This is kind of like elephant in the room. We haven't talked at all about this. And that's because, again, it requires root-level privileges. So what do we do? Well, we extracted the piece of code that does the networking and we applied the setUID binary to it. SetUID bit, sorry. You may be noticing the theme. Just quickly mentioned there is some work on going here. That's Slurp for NetoNest tool, which has got a great name. This is like user space networking. You can do that entirely unprivileged, but it's really slow so we couldn't use it for production. And then finally, there's C-group churning. So by default, C-groups are exposed as a virtual file system. And if you want to apply a limit, you have to echo some values into the C-group files. And the problem is that those files are owned by the host-root. So our container root doesn't have permission to do that. So to get around this, we actually have like a very small privileged setup phase that goes and tones the C-groups to our container root user. And this is sort of okay because that happens before any container is ever created. So there's no risk of a user interfering with that process at all. Are we production ready? Well, Garden has been sort of production ready for a while now, but of course, Cloud Foundry is a big thing. There are lots of other teams that need to align before we can get out there. So no, we're not production yet. We really hoped that by the time we got up here, we'd be able to say that we are so close. We can see it. It's painful to think how close we were being able to say that we were in production, but we're not yet. But hopefully this time next year, we'll be back or someone on the team will be back to give you the latest updates on all the fun new bugs that have come out from running rootless in production. It's going to happen. So where time is going to do this all at once. Here are a bunch of links. If you want to try rootless yourself, if you're deploying by SCF, just turn on the experimental rootless mode on your garden job, and then all your application containers will be running rootless. If you want to run the binary, just on Ubuntu stem cell, just to play around, you can create some rootless containers there. And the last one there is a link to the rootless containers roadmap, which is maintained by Alexa Sarai, who is a contributor to Run C that component we talked very highly of earlier. So that's just how rootless is going, generally in the community, which is very important for us because we work alongside the community. And I just actually do want to shout out to people in the community who have done so much work with rootless containers. Without them, we would probably have nothing to even talk about here today. So Jess Frizel, Lexus Rai, Acahyrus Suda, and many, many others, thank you so much for your work. And that's done, just under the wire. Any questions? Thank you. Do you have a 30-minute break after this talk, I believe? So we can take some questions if people want to stick around. Not 30 minutes of questions, hopefully. But I'll pass you the mic if you have any questions. Going once, okay? I think I'll have one. Thank you for the presentation. And did you consider ZFS instead? Yes, we did. My understanding is that the licensing around ZFS is a little complicated. And so we kind of got scared off from it. But you're right, I think you're right, actually. I think ZFS, you are able to do the disquadring unprivileged as well. But yeah, we can really consider it for production in the Cloud Foundry case because of that licensing. Anybody else? This will be a quick one. I'm curious about how to test these kinds of changes. So it sounds like a lot of these implementations come from like a theoretical kind of observation of what it would mean to be rootless. But are there any tests or automation suites to try to break out or try to really exercise and be really mischievous to your apps? Yeah, so I think there's a couple of parts to that. So first of all, the vast majority of the work here is actually contained inside RunC itself. So again, we get a lot of this for free. We've actually been helping out with some of the rootless stuff. So a year or so ago, we were the ones, well, we were helping out to get the features that we needed to make it work for Cloud Foundry. And as part of that, there was some testing stuff that we added into there. So I think RunC has a set of tests. Garden also has a pretty awesome set of security tests, integration tests as well. And yeah, we spent a lot of time talking about this and just any idea on the wall, like what we wanted to make sure was that we weren't making the security worse accidentally. Which was a possibility, right? But yeah, I think we got the RunC tests, the garden tests, and we're pretty sure it's pretty secure. If anyone could spot anything that we haven't noticed, please come and talk to us. But yeah, I think we've pretty much got it covered. The username spacing seems to provide a way to inside containers of different users and maybe add some restrictions. Can you clarify whether it's already possible for Cloud Foundry developers to set those restrictions to create new users and to set those? And if not, is it something that could be envisioned as a future? So I think in Cloud Foundry, I didn't quite hear the question, but I think I'm going to answer it. Let me know if it's not. So I think when you're pushing apps to Cloud Foundry, like we are running them as a very specific user inside that container. And so we have ensured that the UID mapping for that user exists. We also, with regards to the UID range, so we've got the first mapping, which is the root user back to Maximus, but then for every other UID, we actually start the range at like 65,000 or something in the host. Above nobody. Yeah, above nobody basically. So the idea there is that hopefully, it's not going to clash with any of the more common users, UIDs that you might be running on a system. So a common use case is, for example, in PHP to run the PHP interpreter with low privilege, so that if ever the app gets compromised, it has limited capability. For example, it would not be able to make network calls or restrict the system calls it would be making. And so currently, I understand the Diego is creating VCAP user. So you're saying I could have some user provided script, such as a provided script to create new users and to restrict the way my app is running. Would that seems possible? So I think it depends. If you're just running a buildparks-based application, there's no chance of customising that. You are running as the user that Diego runs you as. But with Docker images and stuff, you can run as any UID that you want to. I'm not sure I'm totally hearing the question. Maybe we could chat afterwards and now. Thank you so much. I'll try to clarify. Thank you. Anyone else? Cool, thank you very much.