 Our next speaker is Stefan Graber. He's the LexD project leader and one of the maintainers and one of the core developers and he's going to talk about system containers at scale. Hello. Alright, so let's talk about about first system containers and LexD in clustering. So, what are system containers? Well, system containers are effectively the oldest type of containers. They've been around for quite a while, originating with BSD jails. Then as Linux vServer on Linux, it was a patch set from over a decade ago. Then Sorority Zones, sorry, obviously, Open VZ on Linux, which was also an insanely large patch set on top of the Linux kernel. And then as we were streaming things in the Linux kernel, LexD and LexD came as the runtime to drive that. System containers behave very much like standard load systems. You run a full Linux distribution. It's not like a single process type thing that you get with Docker and other application containers. You don't need specialized software images or anything. You treat them just like virtual machines, effectively. They're very low overhead, easy to manage. You can run thousands of them on systems. There is no virtual overhead that just take up physical resources. There's no need for hardware, accelerated or anything. As far as the host is concerned, it's just a bunch of processes. You can go on the host and you see all your processes. It's nice and easy. Now, what's LexD? LexD is a modern system container manager. That's written in Go. It is libxc to drive containers. It's got a rest API, a bunch of rest API clients. As you can see here, you can have multiple hosts running Linux than the LexD layer to drive the kernel. Next on top, expose the rest API and then we've got a number of clients that can talk to that. More of what's LexD? LexD is going to be simple. It's a very clean command line interface. Pretty simple rest API. We've got bindings in a lot of languages to make it easy for people to drive system containers through LexD. It is very fast. It is based on images. It's normal like creating a root file system with the boot strap or whatever. It's got optimized storage and migration over the network. It's got direct hardware access because there are containers and we've got nice semantics to pass GPUs, USB devices, et cetera. It's secure. So we use all of the kernel namespaces by default. We also use LSMs like Aparna. We use SecComp. We use capabilities. We use pretty much everything at the disposal to make it safe. It's scalable and that's what we'll see mostly in this talk. It can go from just a single container on the laptop to tens of thousands of containers running in a cluster. As far as what we can run on top of LexD, we've got a lot of images that are generated daily for all of those distros plus a few more that literally couldn't fit in the slide anymore. So we built for about 18 different distros, about 77 different releases all combined, which ends up being over 300 images we build every day that people can use to run on LexD. You can also build your own, but we've got a lot of them. LexD is printed, is effectively on Chromebooks. So if you've seen that Linux feature on Chromebooks, it then gets you like a Debian shell, that's using LexD. So we've got a decent user base throughout. And that feature includes integration for like snapshots, backups, file transfer, GUI access, GPU access, sand card access, webcam access. They really went with it on the Chromebooks. The little piece of what LexD is used right now is on Travis CI. So you can, if you run any job on Travis that is not Intel 64 bit, so if you use ARM64, if you use PowerPC or if you use IBM Z, all of those platforms are using LexD containers with like an extremely quick startup time of usually less than two seconds running on shared systems with all of the security in place and some of the SQL interception stuff that Christian demoed has been done partly as part of that. Now for the LexD components, go through this quickly. That's kind of the main, and the main things we've got in IAPI. Clustering is what we'll demo today. I said we're image based, we've got images and then image aliases to have nice names on images. We've got instances, so those are containers, but these days it can also be virtual machines. That's a new thing we added a few weeks back. We've got snapshots and backups for instances. We've got network management to create new network bridges that you can use for your instances. We've got projects that get you like your own individual view on a shared LexD server effectively. So you can have like conflicting, like there's no more conflict with like container names or anything of that, so long as they are in different projects. And we've got storage with a variety of storage drivers we support and you can create custom volumes in the snapshots and all that. Some internal bits are mostly to get notified when something happens on the next day or for access control and we support doing file transfers and spawning applications directly in containers and virtual machines, accessing console and publishing containers to images. So now for the main topic of this talk. LexD has had clustering support for about two years now. It works. It's really built into LexD. There are no external dependencies. It works on LexD's chip or no hire. Insertions can just be turned into a cluster member and you can easily join an insertion into the cluster. There's really no external component you need for any other. It works using the same API as you have for a single node. It's got a few more bits you can use through the API to say I actually want something to be specifically on this machine. But if your client is not aware of clustering and it just throws things at LexD, exactly like if it was a standard node, things will just work. The cluster will just balance things for you and you'll never know if you're even talking to a cluster. And it can scale quite nicely. So we can run containers and dozens of nodes. We've actually run clusters of like 50 to 100 nodes and they still mostly work. And each of those can run hundreds to thousands of containers. So very high density, depending on what you're running. We've also added, and that's very recent as a few weeks ago, support for mixed architecture. So you can have cluster nodes that are different architectures and when you schedule, when you ask for a particular image to be used to create a container or virtual machine, you'll just pick whatever node is capable of running that given architecture. Alright, now for an interesting part of this. Let's see how that works. Okay, so for this, I've got three systems. Actually, I need to connect to a third one. Okay, now that it's connecting. So what we'll do is LexD is installed under 6D 320 that we released two days ago. So just configure the first node. So you want to set up a cluster. Let's go. Yes, need to enter inside P because the link log all is not gonna be fun enough. This one. We're not joining and I guess in cluster, we want to build a new one. Yeah, let's set the password. Let's configure some storage. So let's go barfs. Create barfs. That's fine. Is there anything in special to do on this one? Yeah, okay, that one is a bit different. So just need to tell it what's the shared subnet for all of those. So that's my subnet at home. Okay. Alright, so right now you've got a single LexD part of the cluster, but it's the only one in there. You can see. Now let's go on to the next one and repeat this thing. It's going to ask less question because it's just joining. So once clustering, it's a p address is the art, I believe. Joining is in clusters. Yes, and the other node was on 1646. Okay, so it's asking for the password entered earlier. Yes, everything's going to go away when we're joining. Size, we don't care. Source, we don't care. Okay, so now we're joined. We should see that we've got two nodes and things to work. Now to make things slightly more interesting. So those systems were in delix86, nothing super special to the Xeon CPUs. Now we've got one that is not in delix86. So this one is running on 64. Same thing next to init. Cluster, name is fine. This is wrong. I forgot to connect that. So it's actually a nested container because I didn't have a spare 64 system, so I'm just doing next-to-nesting for that one, but I connected it to the wrong network, so I'm just fixing that. Okay, so let's do this again. The IP should be right now. Okay, so cluster and yes, name is correct. IP is correct this time. Joining this cluster is yes, and we set it 1646. Cluster password. Yes, we're cool with that. Size, we don't care. Because it's a nested container, it can't create a loop device. I need to actually tell it where the storage is, and it should be the end of that. Okay, let's go back to one of the x86 nodes. So now if I list the cluster, we see we've got three, and one of them is AR64 instead of Intel. Now let's show some stuff at it. So just create a container called C1. This one is... I didn't specify what architecture I actually want, so it's kind of surprised me. LexD will pick whichever it considers to be the least busy server and just schedule a container there. So it's probably going to be on one of the x86 ones. Yeah, it's on EdFu, which is one of the x86 servers. Okay, now let's do another one to send those. I think my guess would be it's going to go on neutral, which is the other x86 system, and then the third one will most likely be scheduled on ARM. Let's see. Yep, it doesn't have an IP yet, but that's going to fix itself. There we go. And let's do alpine. Oh, that name is already taken. And it's just out of order. Never mind. Okay, so C2, and that's going to be on ARM. Okay, so now if I go on there, so from... I keep forgetting that alpine doesn't have a bash. There we go. So yeah, I'm just executing a comment in there, and we can see it's running on AR64. So LexD is doing all the API forwarding for us. So I'm on one machine talking to the cluster and just go on to the right node and kind of query it there. The other thing that's somewhat interesting is we've got a tool to convert a system into a container. So that's what we've got here. That VM01 is a CentOS 7 VM. That's just doing nothing, but it's there. We've got a tool called LexD P2C that can take the address of the cluster. We'll ask for the same password we set, and we'll then transfer the entire thing over the network into LexD, creating a new container for you that uses the entire content of the system. That's going to take a little while, so I'm just going to let it run. Well, that's going on. I want to show the new cool thing we've added. So all of the LexD networking, storage, and configuration bits, because our containers act so much like virtual machines, the same concept really applies to actual virtual machines. So we figured, well, why not just allow running virtual machines as well, using the exact same storage and configuration? So that's what we've got. I can do launch, and notice I've got just an extra thing at the end, should just dash, dash VM. That's pretty much the only difference. In this case, I don't want it to go on ARM, because since it's that ARM host is a container inside a VM on ARM, there's no way I can run a VM inside there. But the X86 machines are running on physical hardware, so those will be just fine. We do support running VMs on ARM, but you need to run on the actual hardware, which is not the case here. So the images are a bit larger because there's no machines, but still downloading, unpacking that, creating the storage volume on the RFS, I think it was this time. And now if you do console, so console works fine on containers too, just to show you. If I do C1, there you go. So on a container, you get attached to the console and on a VM. What did I call it? No, not fun. Well, VM01 is the one I'm transferring in as a container. I would have expected console V1 to actually function. Where did it go? Is it just because it's confused? I swear that worked on it. Okay, all right. All right, so this is a bit picky. So we see the same thing, a VM was booting. I touched a bit late. Let's just go back to this guy here. We're going to just launch a second one of them. Let's see if this one behaves properly. Creating V2. Come on, you can do it. Oh, yeah, that one takes a tiny bit because since it's loaded by anything within the cluster and the node it picked doesn't have the image yet, it's doing an internal cluster transfer of the image. So it's not pulling it from the internet again, but still needs to move it around. It's optimized. It uses burrfs and receive in this case. It doesn't use rsync or anything. It's pretty optimized, but so we can see we're in the boot loader and then booting the VM. And lastly, just to show you that I'm hoping that VM is done transferring the CentOS thing. It is. So if we go here, we can start VM01, which is a container that was treated from the CentOS system. And there we go. How am I doing on time? I'm getting my two minutes behind. Yeah, so LexD is available on obviously Linux, that's where we run, but we also have a Windows and macOS client so that you can talk to a remote LexD if you've got another Raspberry Pi or Intel NUC or something you want to turn LexD on. If you want to contribute to LexD, it's written in Go. It's pretty translatable. We've got client libraries for a bunch of languages. It's a Part 2 license. There's no corporate assignment or anything in there. We've got a good community you can work with. And we've got usually a bunch of smaller issues, good starting points to contribute. That's it. If we've got some questions, we can take them now. And we've got stickers towards the exit when you leave if you want any of those. Questions? Sorry for speaking so fast, but as it turns out, 20 minutes is pretty short. Thank you very much. Two quick questions. What do you think about running Kubernetes inside LexD containers? Yeah, so we've got, yes, we can do it either way around. Some people have been, you can run Kubernetes, especially things like API servers and stuff inside LexD containers, no problem. You can even run Kubelet inside LexD containers because we support nesting and we support running Docker inside LexD containers. So that's possible and people have done it before. And you can actually do it the other way around as well where LXE is a community project that implements a CRI for Kubernetes, then that then drives LexD containers. So you can kind of do it either way, but yeah, it's possible. And then question like why cut a container exist if LexD is secure? Why what? Why cut a container exist if LXD is secure like system containers? It's always, so that depends on people. That depends what they trust. We've seen that hardware is not particularly safe either. Some people think that relying on the Linux kernel for for the entirety of the security story is quite fine. Some people think that VMs are the only option. Some people think that you need both. In fact, like on Chromebooks, Google is on purpose using both. So they're using a virtual machine layer and then running only on previous containers inside there so that if the kernel is busted, you're stayed in a VM. If the VM is busted, you're stayed on a previous user and user namespace. Because we've seen exploits against both in the past. Recently we've actually seen more CVEs and security issues around both the hardware bits of authorization and some of the hypervisor stuff than we have against Linux kernel as far as escape of containers. But I mean there's always a risk and that's kind of up to you what's fine with you, what's not. Combining both is also the slowest option, but it's there and some people have done it. Okay. Two short questions. First, do you support foreign containers on the host? I mean, running ARM containers on X64 or something like this? Sorry, I'm not answering the question. Foreign containers. Oh yeah, okay. So you architecture emulation on the system effectively. So like ARM on X86? Yeah. So we did that in the past. I did implement support for that in LXC almost, I don't know, five, six, seven years ago using QMU user static. It is possible. It is not pleasant and it's not something we want to ever have to support again. The main issue being that the QMU user static layer cannot handle properly threads or net link and some other things like that. So we had to do a very, very weird container where most of the binaries were indeed ARM, but the init system and the network tools were X86, which works, but it's really, really weird. And as soon as you start doing updates and stuff against those containers, just quickly get into a really weird state. So not something we're particularly keen on revisiting at this point. It is possible. You can make it work. You could create a custom image that bundles QMU user static at the right location and with the right bin formats configuration, LXD will let you do it and it will just work. But not something we want to support. Actually, I do just a bind of host system works. I want to see a native implementation. But anyway, and second question, you talk about clustering. What about roaming clustering nodes? If you remove your notebook, for example, with a node somewhere else. That's kind of tricky. So LXD has my move support to move containers around. That works fine, but usually you want to stop them. If they're running, then you just create that Adrian talked about earlier, which can work in some cases, but can also tends to fail with a lot of modern software. As far as the storage bits, one thing that's interesting is that LXD does support SEF as a storage driver. And so if your container is backed by SEF, at least if a node goes down, you can always start the containers back up anywhere as you want, because the data is on the network. So that's kind of what we have there. Anything we're out of time? Yep, we're out of time. Thanks very much.