 Hello, I'm Christophe de Dinsin. I'm working at Red Hat on virtualization, and today I'm going to talk about Kata containers and QMU mini. So as most of you probably know, Kata containers run containers in their own virtual machine, and that's in order to provide additional isolation features, et cetera. So this talk is mostly about packaging that for Fedora. So let's start with a short story, why package Kata containers on Fedora. So you want to put containers in virtual machines to provide isolation additional features, and I'm taking the value proposition here straight from the Kata containers website, the speed of containers, the security of VMs, and it's a drop-in replacement for container runtime. So where you used to have Run-C, for instance, you just have Kata runtime instead, and the rest should work more or less the same, and should work with Docker, Podman, Kubernetes, et cetera, OpenShift, and so on. So of course, as engineers, we may ask questions, do we really get everything at once? Security, compatibility, performance, simplicity. You probably guessed what the answer to this is. You see on the top right, square peg, round hole, doesn't fit. So how do you actually run Kata containers? Well, if you have any experience with containers, you know that the answer is, it depends. So for instance, with Podman, it's fairly simple. You simply add a dash-dash runtime option, and then you specify Kata runtime after that, and that's basically it. Well, I'm sorry, I lied, and that's not the last time. Because for now, you need to also disable AC Linux, so dash-dash-security-opt label equals disable, so that's probably going to be fixed relatively soon, I hope. For Docker, it's just slightly more complicated because you need first to edit your Docker service file to, so that it knows about the new runtime. Then after that, it's pretty much like for Podman. For CIO, that's basically you need to add an entry in your CIO.conf, and that's how it knows about the runtime. Okay, so installing Kata containers used to be what you see on the little picture on the top right. With actual federal packages, it's a bit easier. It's basically GNF install, and you have five packages though Kata runtime brings, as a dependency, most of the others anyway, but we'll talk about why we have five packages, what they do. So Kata runtime is the OCI runtime itself, so that's the part that takes the OCI commands and forwards them. Then you have Kata OS builder, that's the second most important component, and what this one does is take the existing kernel in your system, so that's the federal kernel. In that case, and build an image that Kata containers can use after that. Then we have Kata Sheen, that deals with the IOs and signals for your running container. Kata Proxy, that routes the data to the agent inside the guest. And one that is sort of outside of this picture, which is a same page merging throttler. So you may have heard me mentioned the agent, so you might wonder why is there no Kata agent package? And that's because basically it's integrated as part of Kata OS builder, so the builder builds the agent, the Kata OS builder includes the agent and puts that in your image. And if you know Kata, you know there are other components that are not packaged yet, like Kata netmon, but at this moment I wish this was the worst of our problems with what we need to do with Kata containers. So in terms of configuration, there's a lot that you can configure with Kata and it's found in the, oh, no, that's weird. Is that true for, okay, so I tried to push some things on GitHub and I think I did a mistake in my rebase because you're supposed to see the configuration to mel file there and it's not there, so sorry. And a variance for Firecracker, for instance. So configuration.tomel is where you have your various configuration options. So why is all the config, okay, I don't know. So the runtime, for instance, the runtime section looks like this. These are pretty standard config files, so I'm listing them so that you know where to look for each of them and I'm sorry that they show up empty. That's the thing with having the presentation generate that in real time. I don't know, I think I must have moved the files away or something. And there are many other knobs. So Factory is one that is interesting, I'm sorry it doesn't show up, but that allows you to do VM templating, for instance, and you have configurations for all the subcomponents, the shim, et cetera. All these are in different sections in the same configuration.tomel file. So, okay, now that's going to be really annoying. Where is, sorry, but now you see the secret source. Where is the file? Okay, the files are gone. No, that's annoying. Sorry? Yeah, so that's where they come from, and let me show you what they look like in real life, not in the talk. So for instance, the main one is here, and so the section I had was basically, so what I'm looking at right now here, so this one is configuring.tomel. Sorry, real life, real life, now it's there. Hi, it's still in my Emacs buffer, so I'm saved. I'm not saved. Oh, I'm saved. So you have the path, you have the cam, so basically it was ready, I think it's my git push to git have that. I tried to push the presentation on GitHub, and I had to do a rebase, and I think I lost it in the rebase. So, the path to the values components is found in this hypervisor configuration file, so you have the initRD, the image. You have to choose one between initRD and image, and by default we select initRD because it's basically about 10 times smaller, because you don't have everything, you don't have the whole file system. And you have similar configuration for other hypervisors than QMU, but who would want to do that, right? So choices are Firecracker, NEMU, and Acorn, I guess you say that. And in terms of hypervisor, you can put QMU, but you can also put QMU Lite, which is the kind of version that they built in Catacontainers itself, or a QMU Mini, which is the other part of this talk. So, basically at this point, most of you can leave because what I'm going to talk about now is all the crap to get the, so it's mostly rambling and grumbling and being unhappy about things. So, let's start with a bit of history and how does Catacontainer fit with all the rest, and with all the rest, I mean the world. So, let's start with words of wisdom from folks before us that you will see will come back again and again. So, the first one is for every problem, there is a solution that is simple, clean, and wrong. You know, if you have done any kind of pearl, you know about there is more than one way to do it, which they use so often, they have a timetoddy acronym for it. The good things about standards is that there are so many to choose from, there could not be more true than in containers. You probably know about the second law of thermodynamics, which is that entropy, which means disorder, it's a fancy name for disorder, increases with time. So, do you know who the guy is on the top left? So, his name is Karno, and it's a very interesting life. You should probably Google him on Wikipedia. And our friend, Richard Gabriel, has written a nice essay way back, you know, last century called the rise of the worst is better. And in it, it says one thing that I find very interesting, which is in the future, we will have a good operating system and a good programming language. That's the good news. The bad news is there will be Unix and C++. That was really prescient. But he also said that, get half of the right thing available so that it spreads like a virus. And once people are hooked on it, improve it to 90% of the right thing, as opposed to trying to be the right thing right away. Our friend over here told me yesterday, you know, while we were discussing this, break it properly so that you can fix it properly. And I had to actually modify my presentation to include that great quote. And finally, you know, because so far I only coded folks from our world, your computer science world. So who knows Gretchen Rubin? Who had heard of Gretchen Rubin before? So the fun thing is I was looking for great quotes about containers. And that's actually the one that popped up. And this one is into, you know, home organization and cleaning up your home and stuff like that. But I find this really, this applies so well to our world. I love cunning containers as much as anyone, but I found that if I get rid of everything, I don't need. I often don't need a container at all. So now you're more knowledgeable about things. Okay, so let's go back very, very far in history and talk about software, containment, boxing, sandboxing, all this kind of things. And I'm going to show you that this is all us being controlled freaks, you know, trying to contain things, everything. Do you remember that thing? Way back when mainstream operating system were running directly on the hardware, and that's what you got, basically, you know. So this is back to this quote, right? Running directly on the hardware is the wrong solution. It's simple, but it's wrong. So then we invented hardware things like MMUs and so on, and you control memory access. And from there, IBM went to IBM machines and then in the x86 world, VMware and others. But the problem is that you need your sandbox, you need something in it that you own. And that's where things like Fruit, for instance, was one of the first instances where you'd actually get your own kind of private thing where you could do your stuff and not touch the rest of the system. Then VMware was basically how to control the non-virtualizable x86, which was a feat in itself. Initially it was to run Windows on Linux. And then we had FreeBSD Jails and what that gave is basically final control over other things like files, networking, et cetera. V-server allowed to control the ports, Solaris Zones controlled views on the file system and things like that. C-groups gave us control on processes and namespaces control on things like real access, stuff like that. And then you had to add, you know, user space access for all of these things. And basically you were ready to build Docker with it. Does that make sense? But there is a cost for control and you can see, you know, on the top right picture, you can imagine what lack of control looks like, right? And so there is a constant trade-off, a tick-tock between conflicted diamonds. For instance, bare hardware gives you full access but it's fragile, it's not portable. Virtual memory gives you memory production but then you had early on distrashing performance costs and so on. So it's always a trade-off between inexpensive or easy and something that you really control. Shrewd and jail give you an inexpensive and easy way to do things but you had limited benefits. It was basically only the file system and full virtualization would fix that. You could run DOS on Linux but it was so slow, at least initially. Containers went back to the, you know, lightweight set of things and you could then orchestrate a lot at large scale. But then you want to product that so you put Kata containers and again you're going to see something that is actually slower. And then back to try to make things lean and mean, you know, QMU, the VM engine that we use, the hypervisor is basically a huge beast and for years we searched for an excuse to get rid of IDE, we finally got it, right? If we want to run containers in it that don't need IDE, we can scrap that stuff. Okay, so what do we do with that, actually? What really matters is your software. It's the stuff at the center, that's what we care about. So, you know, if you were running on a Macintosh or in a unicannel, your software basically has direct access to the hardware and you're happy with it. Where this runs into trouble is when this other guy tries to run stuff on your machine. Now their software is going to steal the CPU or the memory or whatever and it's bad for you. So that's why you invent real operating systems that will schedule stuff and protect you from these other nasty folks. Your software is perfect, of course, but theirs is not. So you need to address that. So far, my picture on the top right is, okay, everything is fancy, I'm happy, and so on. But then you start saying, okay, I want my sandbox. So now you need to bring some thing that looks like a file system. For instance, inside your own sandbox. So you start trying to do that with the operating system and then that starts to be complicated. You have the same thing at two places, which means probably slightly different semantics at the two places. It's not going to do exactly the same thing. So if you wanted to really do the same thing, then you need to bring up the whole machine. So that's where virtual machines are invented. You know that now you can run inside and it looks really like a real machine. So it's a full machine sandbox, but starting to say to go home. So you know, the components in our case, like cumuliverts and the kernel component KVM. And that's too big, that's too fat, that's too slow. Okay, so let's try to do the same thing, but inside the operating system and that's containers. So it's OX, OX built sandboxes. I don't know who said that, but I think it's somewhat true, is that whenever you need to use virtualization, it's because there is a bug in your operating system. Which is, in a sense, it makes sense, right? And so the good thing about containers is that they scale pretty well, because you're sharing resources in an operating system that is designed to share resources. So that can scale well. And from there, you can build things. So now the next problem is, if you start scaling like this, you don't want to write all the stuff, you want to run their software, you want to compose things. And that was a pipe dream for so long that it's worth stressing software composition, software components, stuff that you could pick. It's another of these things that were predicted in the 90s and for kept dreaming of software components that you could compose, assemble, et cetera. That's exactly what Docker brought and that's why it's so successful. Now that we use Docker, I wish I could go back in time and for all these folks screaming, we need software components that are, no, don't do that. Well, too late. So to orchestrate all these things, then we have tools like Kubernetes on the side and so on that manage all that stuff for you. And then you need to talk to what is inside the containers. So that's why you need a common runtime interface and stuff like that. But if they want to talk to virtual machine, then it changes a little. And for instance, you probably lose, if you don't do something special, you lose all the benefits you had from Docker because the composition of software that Docker was providing, the overlay file system and stuff like that, you don't have that in VMs naturally. It doesn't fit there. So you can have containers inside the VM, but you can have your VMs themselves be containers. So that's why Kala container is trying to fix that. They are trying to bring the best of both worlds, basically to bring the software composition back to VMs. Does that make sense? I mean, this diagram is a bit busy, but I'm trying to summarize 20 years of folks thinking about this in one slide. And QMU mini is QMU optimized for Kala. So we are just trying to have something that does just what we need for that use case. So now my smiley on the top right is like, I'm lost. Are you confused yet? How did we get there? That's the predictable results of open source software is great, but basically we grow things step by step. There is no giant design. There is no one guy who said, we are going to do the right thing. And so that's exactly what we expect, but now we have to make it work. That's the bad news. So general architecture of Kala containers. So there was a great talk yesterday. So this is going to be a bit of a repeat for those who saw Matt's here on yesterday, but basically Docker talks to container through a number of layers. So it's a story of the guy who talked to the guy who talked to the guy who talked to the guy who finally talks to your container. And the part that is really replaced is at the bottom is this RunC red box that gets replaced with this Kala runtime red orange box. Let me digress because I think you're tired by all that stuff. So RunC CRUN, we really needed another language war. So let's look at what RunC is. RunC is a CLI tool for spawning and running containers according to the OCI specification. That's fine, right? Looks good. CRUN, a fast and low memory footprint OCI container fully written in C. How is that different? Well, why another implementation? Most of the tools used in the Linux container ecosystem are written in Go. I believe C is better fit for. Let's do it again. And let's name it CRUN rather than RunC so that newbies like me are completely confused. Okay, so that was not the end of my digression. And I can go all the way. Why stop digressing? Hypervisor-based runtimes for OCI. What is that stuff? RunV is compatible with OCI. Okay, the others too, so that's not different. However, due to the differences between hypervisors and container, the following sections of OCI don't apply to RunV. Namespaces, capability, device, namespace, does that? Is that useful for containers' namespaces? Capabilities, I mean, who needs to drop capabilities? Then what should be happy to have a runtime that doesn't care about capabilities, right? That's the moment where you want to bang your head like this. BAM! Okay, so now time to show a simplified diagram of how things actually work. What I like about this diagram is that, in a sense, it's like we had a time travel thing just before. In a sense, this is time going downward, right? So on the left, you have things like Docker. And they talk to a cubelet, and then they talk through the common runtime interface. And then you need to run on the other side. Oh yeah, you're right, Karen. On the side, it's on this way. The thing that this is backwards. So basically, this started with plugins everywhere. We want to do this to be very, very modular, so Docker, same Docker, container, D, and so on. And the problem, well, it's very flexible and so on, but it's a bit complicated to make all this stuff work. And so we'll see that many times, there was an attempt at simplifying by putting things back in the black box. No, sorry, Marie, not now. So I also use this slide as an opportunity for you to understand the differences between, for instance, CRI and Cryo, or CRIO. Now what, so there are CIs that are not OCI compliant? What do they do? I don't know. And there is a, okay. So the good thing is that all these things are perfectly standard. You just need to learn their names and what they do. So Cata containers is basically, I swap the runtime. Nothing could be easier, right? I just replace the thing at the bottom with the things at the top, fine? Where's the problem? Okay, there is actually more than the runtime. And that's why it gets complicated. I already showed this in a smaller version, but you can see in its full glory, it has picked up straight from, lifted up straight from the, I think, Cata Container website. And so one of the trouble things is that you need to talk to the VM and this is done through, you see this hypervisor serial interface. I think they can use vSoc as well now. So these things get complicated. That's basically yesterday's, the Matt Heans talk I talked about yesterday, this is it summarizing one slide. But the problem with all these are rows is that they can break. Anything in there can break. And so what do they do? They do a v2 that looks like this. You know, this is called the great black box simplification. It's really this exact same diagram, right? Except you locked the black box. Inside it does the same thing, trust me. Okay, so now I'm running out of time already. So there are plenty of things I would like to talk about, but I can't, because I have only 45 minutes to talk. So I can talk in details about ContainerD, DockerD and all the nasty little demons that do their stuff in the corner. Sheens and proxies and all the other, on agents, all these little helper processes I can go into much detail besides large diagrams on what they actually do. Cubelet, that's another story. How does it talk to pods and so on? All the fancy exotic runtimes, how do they fit in that picture? Frankly, for this one I just have no idea. And then Podman, well, so we have Podman, we have Docker. What is the difference? Why does that matter? It's a very interesting topic itself. I can't go in depth into this. And what is Moby? Okay, that's yet another one. Container image formats are also another interesting question. I won't touch that at all. Repositories build tool, don't ask me about this. I know nothing about that stuff. All these things are additional standards and ecosystems and communities that you need to know about if you're serious about containers. So the summary of all this is that the Rubik Goldberg School of Computer Programming, led us to this situation of there's not just more than one way to do it, there's more than a thousand ways to do anything. So back to packaging, which was the main topic. I digress a little for now. It was supposed to be simple, but now you can probably guess why it was not. So what did I start with? I started with some bits and pieces, a little bit like the picture on the top right. Some work had already been done. The most important one of use is that Kata containers itself was largely working, so you could build it manually and have most of the stuff run. There were some Suzy packages available and they were sort of working if you were ready to add fancy dangerous repos on the side. And sometimes you had interesting stuff, like QMU Lite, the name of the package in these repos is OpenBios Development Utilities. Copy, paste, error somewhere, maybe? Hello. But also, and that's much more of a problem, there are some interesting conflicts because that all relied on their own custom build version of QMU which of course is completely outdated. It was based on QMU 2 something. That leads to somewhat nasty and let's call them interesting conflicts. So bottom line is you don't want to do that. And of course these packages were not following any of the federal conventions. Some of them I'm going to talk about later related to Go which are really another thing entirely. And inside Red Hat, some test packages had been built but not submitted yet and so meaning they were not built on a regular basis. They were not really working, they were outdated, they were based on Kata containers 1.4 and the original packator had obviously stopped putting any muscle into it. So basically you had half of the packages and the half that was there did not work. The other bad news is that I'm a total newbie with respect to federal packaging. The first federal package I did was like six months ago, eight months ago, something like that. So that means that if these packages are broken, please tell me, sorry, strike that, I lied. It is broken, please tell me. Okay, so Kata package structure, why did we decide to build one package to build five packages? Well, the obvious answer is why build only one package when you can build five, right? I mean, I'm paid by the package. So you have seen that already but it's a refresher. The main package is Kata runtime then building the OS image with Kata OS builder. So can you think of reasons why you would want to have the runtime but not the OS builder or the other way? So for instance, it depends on where you store things, right? So you might want to have one place where you actually have your reference bills and then leave from there, things like that. So there are good reasons why each of these components you might want to upgrade them or install them independently because basically there are really different parts in this system and because they communicate to one another, they expose interfaces. So you might have cases where to fix a bug, you might want to upgrade one of them but not the rest. I'm not even talking about QMU yet but we're going to talk about it later. So why was it complicated? It's in large part a tale of two package managers. Go and RPM don't mix too well, much like Diet Coke and Mantos don't mix. More or less the same effect. Oh no, no, that's really nasty. Okay, so why are, there's no rhyme and reason on which files were gone, it's really real. Okay, so you can build a, let me show you what this was showing. This was basically showing that you can build inside a spec file. Oh, hey, I can use the same trick. I can have that in my, so it's called bidmanuage.spec and I have it there, ta-da, okay, that's going to be annoying. Okay, now you know that this is colorized in real time by the way, this is fancy. Okay, so in order to build a go package manually, you need to do stuff like that. And that's basically using the make file that was provided by them. And I highlighted this not check version equal blah blah blah which I added in the, so let me explain why. This if not check version that you see in there is disabling this have yq, yq I guess is pronounced yuck. And this yq is something that you can't find anywhere. It's a small little project in the corner that happens to be pushed in the right go repositories. And so implicitly what this does is it fetches stuff from the internet, expands all the sources for yq, builds yq on the side, and does all that stuff. That requires internet access which typically you don't want to do while you're building federal packages. And so in order to, so I scratched my head quite a while trying to bring the yq source file. There was no documentation. I did not know where to find them, et cetera. Until one of our colleagues, Dave Gilbert mentioned, you know, this thing is only checking the version number and you're doing that anyway while you're building. So you don't need that. And so, hey, yeah, I can set this not check version internal variable and that works. The rest of the package builds. That tells you the level of, high level engineering that went into that stuff. When I was done with that, then I learned, hey, you can't do that on Fedora, Rohide, or 31 at the time when this was introduced. Because now this go stuff is so crappy, is so dangerous, and they had so many issues with it that they decided to have packaging guidelines for GoLang and a brand new set of RPM macros called GoLang RPM macros that I can't say without. Because that was under development while I was doing that stuff. And the problem again is that Go has its own package manager and downloads its own stuff, Fedora won't have that. So of course, on the website documenting this stuff, it looks simple. That's the example they give. And that example works perfectly fine. But that's about the only thing that works. In real life, okay. Sorry. So in real life, that doesn't really work because that may file is doing stuff that GoBuild itself does not know how to do. Like setting the machine type, the kernel parameters, stuff like that. And you can't really do that with GoBuild. There is no simple way to, which is why they have a May file by the way. So they are setting files and turning in files into the actual files, the fields and so on. And all that stuff does not compute with GoRPM macros. So I have to use the half that is mandatory because otherwise they won't accept my package and then completely ignore the rest, which is a bit annoying. And then they have this nice tool called GoToRPM which gives a good starting point. The only problem with GoToRPM is that it did not exist when I started. And so in the end, I take my existing RPM and I run GoToRPM and I see this nice section of provides and that explains the problem with GoPackaging. That's the stuff that my package provides. Because it's building it, it's dragging that stuff. And so they are trying to find a way to tell all these bundled things are inside your package as well. So we all thought this would be easy, much like when you try to feed a kid, right? You think, hey, everybody can do that. And then you get a picture on the top right. It turned to be really messy. So having all these commits just for that, well, when you see what is in the commits it was. Okay, so by now you really want to exit this room and you hope I'm done. Well, I'm sorry, but there's still a bit of work to be done for this Kata container. Not everything is under control, much like when you have, it looks very much like my cat, coffee, cat, laptop. What could go wrong there? So one issue is that you have moving parts left and right. And the Linux kernel, for instance, for someone who is outside looks a bit like this. It's a fractal maze of things that move all over the place. And it's really hard to understand what's going on. So one of the issues is that they moved to C-group V2 and Fedora has adapted that quite early. This means that you need to do this work around every time you reboot your machine to make sure that container sees the C-group at the right spot. Except if you run for a while, then chances are that the CPU one is already mounted elsewhere. And apparently you can't mount it C-V2 and V1 at the same time. So now you're stuck. You have basically to reboot your machine just to be able to do that mount thing because there is no way to force and mount it. I tried, well, at least I did not find it. Then there is the support for Vitaio FS, which is much faster than 9P. I'm going to come back on this. But that's very recent. The commit for Vitaio FSD was last Thursday upstream. I don't know, something like that, right? So it's very recent. And the commit for the kernel was I think 5.3.15 or something like that. So it's in 5.4, but it's relatively recent. And the problem is that that forces you to go to recent kernels, which have C-group V2 back to the problem above. Then that stuff is running at root. And we mean real root without dropping capabilities or Linux or anything fancy like that. So this is really Dr. Ible at play. We can have that on a real system. So we need some kind of jailer. We look at it, we see that, right? Except the reality, it works like this today. And we are lucky sometimes, some things may be caught like the little, but not that often. So we would like ideally to leverage Invert. And I have the kind of Goldilocks, for those of you who saw Daniel's talk, he was talking about Goldilocks level of security. And Invert is sort of providing that. So we'd like to leverage that, but that brings in a whole kind of worms of who does the work. Are we asking to do that in Kala containers, but then that will be Kala containers only? Or do we find a way for Invert to offer a jailing feature that is easy to use from go? Or do we see that as a QMU option that where QMU would sort of double fork itself and do the stuff for you? Or whether, it's hard to tell what is the best solution at this point, we're still discussing it. So SELinux is both, which is that your containers at the moment look a little bit like this pipe here from a security standpoint. So again, not in production, please. And then there are a number of enhancements that we may want. And as this picture illustrates, one of the problems is that we have diverging versions of what enhancement means. So for instance, do we want to have memory mapped kernel? Do we want to boot PVA? So we had this nice experiment. We have various experiments on how to boot things fast, for instance, or using less memory, leveraging the kernel image between multiple booths, this kind of things. And how do we want that? And then we want to have a VM optimized kernel. I mentioned that QMU mini is a good excuse to get rid of IDE. So can you guess what this thing on the right is? It's the very first IDE drive from a compact of 19, I forgot, I think it's 1980s, I forgot. Way old. But the same is true for your standard federal kernel. It supports tons of stuff that you don't need. But out of the box, it doesn't build by default, it builds as module, all the virtio stuff, et cetera, which means that basically you don't get much help from mapping it or things like that. Okay, so virtio FS, there was a whole talk about this. I'm not going to expand in great details, but the thing is that networking in general is a bad thing, you all know that, right? Fake networking like the picture on the top right, it's even worse. And 9P is exactly that. So if you compare virtio FS to 9P, there is really no discussion, and it's even better if you compare for async IO. So you have a very, very good reason to switch to virtio FS, just need it to be upstream, available everywhere and so on. Okay, so I mentioned QMU mini a few times and I don't have that much time left, so I'm going to very quickly brush over what this is. This is really a QMU tuned for color. The point is to try to make it as straight QMU as possible, as little change as possible, no patches, no nothing. And that turned out to be actually relatively easy once I understood what I was doing, which took a while. For instance, in the existing configure script, there is this run configure disabled everything, which is very fancy because I spend a lot of time writing my own, till I realized that this was basically in the standard. And then after that, you can enable your own stuff. So there are some miscellaneous, if you really look carefully, there are some miscellaneous things in there, like for instance, VHOS Cuzzy, and what is this mini FDT, stuff like that. So there are still a few things that we don't really want to have in there, but if they are not there, you have unsatisfied symbols, you have missing bits, that means it doesn't really work. So then I was relatively happy with that, and I started discussing with other folks, who said, ah, no, don't add another binary, because we already have QMU system, blah, blah, blah, and 10 variants of it depending on the system you emulate, and then you have QMU KVM, and the spec file, there is a single spec file that builds all of this, which is already quite hairy. So I sort of did monkey see monkey do, you know, I took the QMU KVM part, and I did my stuff in this mini part, and I thought, hey, that's peachy. But that means yet another version of QMU that we have to explain, to have main pages for, to maintain, that might be a different version, and so on. So overall, we wish we could do without it. And the worst part is, I did that work, but I'm convinced by the argument, and I was convinced before I did the work, actually. So I was working stuff that I thought was. In terms of patches, there were only four trends in patches. By trends, I mean that many of these things are actually now in the existing code base, so if I rebase, I would probably not need these patches, but something else. And so, the thing is I had to learn over and over again, because it looks simple when you see the end result, but I had to learn how to get there, and that was actually my version three to get it. So, what do we gain from doing that? That's a much, much smaller binary. The size went from 13 megabytes down to 5.8, so that's pretty significant saving. That's not just ID that we dropped, right? And do we gain something in terms of performance? It's a bit unclear, frankly. That's the diagrams. So this is the best of five with 10 runs. So this means that what you see here is the range of time. And you can see that QMU KVM, for instance, on one vCPU with two gigs of memory, has a relatively large range. And yes, QMU Mini seems to be, it goes lower, meaning it's faster, but it's also sometimes as high or higher. So it's unclear that we really save much in terms of performance. Now, I did not do much more extensive testing than that. I took basically the worst case, which is to start up container, have it echo a number, and shut down. So that's the worst case, and my workload is practically non-existent. It's all boot time and shut down. And by the way, you remember the thing that was said at the beginning about it's being fast and secure, et cetera? Look at Podman on the left. Podman without color runtime. So yes, it's fast, except if you care about boot time. Inside the VM, there is not that much difference, frankly. But if you care about boot time, there is a difference. So will it ever ship? I saw this nice picture of a launch of a ship, which I thought was. So on the pros, should we ship something like that? It does reduce the footprint. It may reduce the start of time. It's built from a single source. It's the same spec file. It's basically the same version, et cetera. So we get all the patches and all the series solutions and so on. And it's definitely shorter to type QMU-mini than QMU-space-mini, right? That has to be a pro. On the cons side, we have multiple binaries when we already have too many. This requires some additional testing because it's a very different configuration, so it needs some testing. It's an overly specialized use case. Or maybe it is not. Maybe with some tweaks, it could also be usable by Qvert, it could be usable by other cases. That binary does not exist anywhere else. So it would be in Fedor, but that means that the same config files would have to have a different argument in another distros. And that's really the killer, is we think that we may do almost as well simply by having QMU-DL open the large stuff. Like for instance, Spice. Spice is big. It brings tons of other libraries with it. It brings in particular Orc, which is a code generation runtime, which you might not really want to have in your binary, stuff like that. So we want to do an experiment with the L open before committing to anything like QMU-Mini. My conclusion is we have only begun making this complicated. It's only the start, it's going to get worse, and that's logical because entropy always increases. Thank you, and time for questions. Thank you for being patient with me. So the question is whether there is any plan to support checkpointing and restoring? I'm pretty sure there are, but I'm not familiar with them. We had a talk just before that on, for instance, I'm not sure if a solution like what was presented before could apply in that case. What kind of interface would you use for your checkpointing? So that would be a throughput man. Frankly, I don't know. I don't know of any plans. Well, you know, if you shout loud enough, they will plan it, right? For it to be planned, you only need to open a GitHub issue, so. Any other questions? Yes? So the question is why do we provide CataOS Builder and not the kernel directly or Inetardee? The answer is that if you want to build your own kernel and Inetardee, you still can. But from this stuff in the long term, we might want it to migrate to RHEL, and there is little doubt that from RHEL's point of view, what will be supported is the RHEL kernel running in the container as well. So now what we want to have is that if you update your host kernel, the image gets updated as well. And so that, basically that spec file does that. It will, whenever there is a kernel update, it will reinstall an updated image. My Inetardee business. That's what happens. That's what happens. I see you frown. You don't like that? Only the image is rebuilt, right? And the old image is kept. So the old image gives its own name. You get a new image. Then you have to do it manually. And there are scripts inside this RPM. There are scripts that allow you to do that with any random kernel you want. By the way, you may have not seen that in the values config file, but there was a decision relatively recently, I think it was in version one, eight, one of the package or something like that, where we moved the stuff you build under varkash, where Scata containers typically put it under user share. And the reason is we see that as be your stuff. So we don't build stuff in user share anymore. So the question is, can you stuff the images in Quay or Docker or any other place? From that point of view, I don't think it's different. That the part of building the images and creating the overlays and all that stuff, it's pretty much the same way. From the user perspective, at least. The way they are mounted inside is different, but the way they are built is not. So the question is does Caddox support side cars? And I don't know. It does anybody have a reasonable clue about that? So in any case, the short answer would be no, because the notion of Caddox containers supporting is not clear, right? So the next question would be to a point where Red Hat could support that, which is a slightly different question. The feature does not exist as far as I know today, but I'll check. So could you repeat the question please? So the question is how does Caddox deal with volumes? Well, so you have two parts to this. One part is the initial image, the your start volume, et cetera, and the fact that you want to expose something that are sort of available initially to your image. And for those, basically that's why the choice of Vata UFS or 9P comes into play. Now, if you want to add any additional volumes, well, you have either the path of going the same way you would do for a container, but now because you're in a VM, you have additional possibilities as well. And basically stuff that you can do inside of VM by passing the additional options, you can mount an additional volume or do stuff like that. So you have more options, so whether they will be supported, whether they will be used, at some point it's not very clear to me. Did that answer the question? So if you mean whether you can pass the dash view option? Yeah, that works. The question is how that dash view option is shown inside. That's, so that's where the choice between, so one thing I did not really touch about Vata UFS is that because it's not a network file system, it's basically talking to the host file system. It's a shortcut. So now what you see is a real file system with full file system semantics and you have values caching options that you can tweak and stuff like that, but in the most conservative mode if you want, you can even, at least in theory, I'm not sure if it's supported, but you could even have things like locking work properly between host and guest. Does that answer the question better? I'm out of time. So if you have one last question, I'm just to annoy my moderator. Thank you for listening.