 Hello everyone, and thanks for joining us to our Open Infra-Live episode. This is our weekly one-hour live meeting where we discuss, share and talk about all the greatest updates from the Open Infrastructure community. My name is Samuel Ortiz. I worked at Apple. And I'm also a Cata containers architecture committee member. And as a matter of fact, today we're going to talk about Cata container use cases. I'm going to be your host today, and I'm going to be joined by some very prestigious guests. We have Peng Tao from the ANG Group. We also have Ubertus Franke from IBM, Brent Olligworth from AMD, and Xiaobao Feng from Huawei. Just so everyone knows, we will save some time at the end of this episode for Q&A. So feel free to drop questions in the comments section throughout the show, and we'll try to answer as many as possible at the end of the session. Alright, so Cata container use cases. Before we actually go into those use cases, I'd like to very quickly give you an update of the Cata container project. Alright, so what's new with Cata? Well, even before going with the next release, I'd like to give you a quick refresher on the previous one, 2.1.0. Then we'll look at 2.2.0, then the upcoming one, and we'll talk a little bit about two interesting points. The use cases, meetings and discussion that we're currently having, and the new stable branch measurement model that we're trying to follow with the project. Alright, so 2.1.0. If we can go to the next slide, yes, thank you. Yeah, 2.1.0 is the last official Cata container release. It was released on May 15th with a few interesting features. The Cata agent that's running inside the Cata continuous guest is now asynchronous, so it gives us improved performance, better IO utilization, and better resource utilization in general. We also decided that Cata deploy will be our main way of deploying and distributing Cata containers. As for tracing, and tracing is a very important part of Cata containers in order to follow the bug and look at what's happening among all the different pieces of the project. On tracing, we went from open tracing to open telemetry. We also improved the direct device assignment, also called a device pass-through, and we added support for IPv6, and also a virtual IOMM, which is a very interesting way of doing memory hotplug and hothuntplug. And last but not least, as we released 2.1.0, we officially deprecated Cata 1.x, the very first series of Cata continuous release. So thanks for all the fish, Cata 1.x, and Long Live 2.1. Cata 2.0.0 is the next Cata continuous minor release. It's scheduled for August 6th, but I think it may be delayed by one week. This is something that we're currently discussing in the Cata continuous architecture committee meetings. There are public every week on Tuesday at 3pm UTC. So if you want to join and discuss this, feel free to join on the meeting and discuss with the rest of the committee. So it's scheduled for August 6th, maybe delayed by week. And with the 2.2.0, we're going to see a few interesting new features. First of all, we work quite a bit on the stabilization of the CI. And it's kind of a hidden feature for end users, but for the community, it's a key feature because it's basically kind of telling us if the development... It's really helping development and accelerating development if the CI can be reduced and stabilized and so on. So PRs can get merged much, much quicker as we stabilize CI. We also continued working on tracing. We've seen a lot of improvement, stabilization. There's been a lot of discussion upstream on tracing. And so this is moving forward in a very nice way. We added a feature for watching volumes and monpoints and kind of being able to detect from the guest if the stone fires are changing on the mounted volumes and monpoints. And this is a very important feature for some production use cases. And also we moved the machine type for one of the app providers that we support with Kata containers, QMU. We moved it from the PC machine type to a more modern one, which is Q35. So that's for 2.2.0. Then we also started to look at a simplified release management. So we're now going to have a different release cadence. We're going to do a new release candidate for either the next major or minor release every three weeks. So this is a fixed cadence. So every three weeks, depending on if we decide to go with a major or minor release, we'll have a new release candidate. And then we do a one minor release every 12 weeks. So 2.2 would be released and 2.3, if everything goes fine, should come 12 weeks after that. And also we decided to simplify the way we handle stable branches. We used to maintain at least two stable branches. We now move to something simpler and actually more pragmatic and that maps more closely to what people do in production. Which is basically maintaining one stable branch. We only need one stable branch. And the stable branch is basically going to track the previous minor or major release depending on what the last one was. But this is going to be what our stable unique branch is going to track from now. And the last point I want to talk about before we go into the use case discussion with our guests are actually the data container use case meetings that we are currently holding on a weekly basis. So we looked at a few different interesting streams that are mapping to production use cases either deployed today or that people are trying to deploy or are interested to deploy in production. And we kind of narrow it down to two interesting use cases. The first one is performance isolation. And in this weekly meeting we're trying to discuss, present and look at the best practices for kind of keeping Cata containers in production, keeping the resources overhead that Cata containers bring and the noisiness keeping all of these constrained and making it as seamless as possible as far as capacity and resource consumption is concerned when people run Cata in production. And the next use case meeting that we're holding also on a weekly basis and feel free to ask at the end of the conversation, at the end of the session or reach one of us in the community to get the data on how to join those meetings. But the last one, the other use case meeting that we're holding is the confidential containers meeting where it's a cross industry meeting with the budget folks from IBM, from AMD, from Intel, Apple and Talibaba. It's a very interesting use case. I encourage everyone to join and join your discussions. Essentially, we're trying to build a complete end-to-end stack for running confidential computing workloads with Cata containers. And I've all of these integrated with Kubernetes so that people can essentially take a container image and run that through confidential computing. So yeah, that's it for the updates. I think we can now move to the actual use case presentation. I'd like to welcome Peng Tao from the end group. Peng, welcome. The stage is yours. Hi, thank you. Thank you, Sameer. Can you show my slides? We can see your slides. Go ahead. Okay, thank you. My name is Tao Peng from the end group and I'm a staff engineer in the end group. And I'm also a member of the Cata architecture community. Today I will talk about why we use Cata containers in the end group and how we use it in the end group. And also the key point we are actively working on on Cata containers, which is the Cata containers performance isolation. As mentioned, SMU will have a small working group to focus on the end topic. Can we move to the next slide? Okay, so why we use Cata containers? The most obvious reason is that we do not want to share a host kernel with different ports. So in one case, if a port has some bug and triggers a kernel panic, it will panic the entire host and affect all the ports on the same host. And another thing is even if it doesn't panic their host, it can panic itself. The application panic goes out of memory all the time. And all these scenarios will affect the containers, the ports on the same host. And we do not like different applications, different services to affect each other in our environment. So that's the reason. Can you move to the next slide? Okay, thank you. Another reason is performance-sharer. When we run different containers with a normal container, we will have to share many kernel resources such as kernel objects and different kernel streets. All these kernel resources are a point of debtors. So when we run different services on the same host, they will affect each other's performance. And as you know, we are an online internet company and we do online payment transactions all the time. And in these transactions, we do time generator. And RPCS run chip is very critical in our use case. And when we share the kernel objects with different ports, we will get higher performance generator and it ends up during our online service performance. So that's another reason that we do not want to run containers in the normal RPC container way. So can you move to the next slide? Okay, thank you. So our solution is to run cata containers. With cata containers, even if we still share the same host kernel, we have a hypervisor layer and we run different cata ports in different guest OASs. And then when one guest panics, it will not affect the other ports on the same host. Also with this guest OAS level isolation, we no longer share a lot of host kernel 3s on the host and our service performance is much more stable with cata. So can you move to the next one? Thank you. So in our group, with cata containers and many other technologies, we build a trust native infrastructure. Within this infrastructure, we run offline and online services and different jobs. We do our service containers, big data jobs, edge computing and many offline tasks on the same host. And we use cata to isolate them. We still run ccontainers such as since the isolation is not free, we want best performance for the online services. We have to run them still in the traditional ccontainer. But for many other jobs like offline computing and offline services, we do run them in cata containers. And below cata containers, we have container storage and container network all built on top of some open source and internal technologies. Under this stage, we run our infrastructure on top of some Alibaba technologies like Alibaba operating system and there are cloud bare metal services. Can you run? Okay. So just because we run cata is simple. We run different priority tasks. We run high priority tasks still in traditional ccontainer but we put all the low priority tasks in cata containers so that the indoor priority tasks will not affect the performance of their high priority tasks. Next slide. Thank you. So with that in mind, we still use cata containers the same way as the option cata containers. So we run a cata monitor demo. It will monitor all the cata containers with two processes. And then the same way to process it will expose all the cata metrics to cata monitor and this will be projected to permissions and we can do more analysis on permissions. So this is... Sorry. Okay. Go next slide. Next slide. So with our event, we still use another technology we call a net-ass image acceleration service. So the permit to serve with ZAT is when we start a container a lot of time is used to pour the container image and we create an open source project and the... And put it in a C&C of Stringerfly project. There's a link to it. It's called an image service and we call it... We name it a net-ass. And it's your specified system and it can... We can pour image very quickly and download all the image data on demand later. Then net-ass expose both fields but how it supports. So we run both traditional 1c container and also cata containers on top of net-ass. So can you go next slide? Okay. So with our use case, the most interesting topic we would like to see in the cata community is performance isolation. So we are having the weekly performance isolation meeting to look into how we can isolate different tasks with cata container. So right now we look into it in different angle. For the network part, we look into how we can direct assign a NIC with password capability. We kind of use a customized C&I to do it right now. And we'd like to see... to implement it in the cata community as well. Also we use block device password with customized CSI. And we are working with Eric from Apple to get an upstream implementation for this capability as well. For CPU isolation, we are looking at the mixture of CPU set and CPU share on the same host. And also for memory, we are looking at how we can use NUMA and RBT on the same node. So we're coming to join our weekly community performance isolation use case meeting if you are also interested in this topic. So that's all for my session. Let's welcome Brent and Herbert. Nick from IBM. Yes, next we have Brent Folliwod from AMD and Berges from IBM. Welcome to the stage and the show. And I think you're going to have some very interesting bits to share. Go ahead, please. Thank you, Samuel. Yeah, great to be here. Okay, this is kind of a fun picture. This actually sits behind me on my shelf somewhere. It's an interesting point in time. So this is all public record. May 6, 2015, AMD changed from the New York Stock Exchange to the NASDAQ. I actually had the opportunity to go and to ring the opening bell. What's not public record is what happened the day after that. And of course the day after that, I was on Wall Street. We were having meetings with various financial services companies and really coming to a deep understanding of a specific problem that they have, which is this. Many of these companies, and I'm talking about regulated industries, financial health, defense, they have to significantly over provision because at month end, when they need to run transaction processing, when they need to run burst transactions, it actually becomes very, very difficult for them to find any sort of opportunity to share their hosting, share the compute that they're doing. They tend to operate inside data centers that look like prisons and are highly siloed. So that's sort of the start of my team's interaction with confidential computing. John Hollingsworth, I'm a senior manager in our AMD Epic Software Ecosystem team. I lead a group that works with open source partners. We've got the Linux distros, the container orchestration, the libraries, tool security compilers as well. But we have been on a journey for the last seven, eight years to figure out how to solve that problem. This is a fantastic case study as well. Most of you may be familiar with this. I'll invite you to read it. Normally, I don't do that in a presentation, but this one's worth actually going through and parsing the text. Trend Micro runs a zero-day initiative, phone-to-own competition, and they have a virtual machine escape category. Think, go into a conference room. They set up a hypervisor, put a secret on the hypervisor, and then allow people to connect as guests and see if they can break in. This competition was famous because in 2019, the team Flora Acetate was able to win a Tesla Model 3 by successfully exploding the security on it. And hopefully you've read this. I love doing this presentation in person because typically somebody gasps audibly when they understand what this is actually saying. And so I'll play it backwards for you. Essentially, they set up this hypervisor, they gave the team's access to a guest, and the successful team was able to set up a website ahead of time, open their website inside Microsoft Edge, exploit a bug in Edge to break out of Windows 10, exploit a bug in Windows 10 to break out of Windows 10, exploit a bug in VMware to break out and completely control the machine. No software installation and essentially all configured ahead of time. It's horrifying. You think about what happens when bugs like this are discovered. Microsoft, of course, patches the bug. VMware is going to patch the bug. And Flora Acetate in 2019 did the exact same thing again. Still the same website configured ahead of time. It's terrifying that this is entirely possible. And so sometime around March 2017, I started building demos, talking to partners. We're looking at the challenge that we've got. And along the same time, AMD came out with a memory encryption technology called SEV. In 2018 was our first implementation that went upstream into the Linux kernel and very shortly afterward, we had a prototype of Cata in GitHub. It's been something that we've been engaged in. We've been looking at. We've been kicking the tires on for the entire time that we've been working upstream inside this SEV project. I'm excited to be here with Hubertus. IBM and AMD, it's public since 2020, have been working together to figure out how we improve our security with specifically regard to container, working together on the container project that's the Cata container. Hubertus, say hello. I guess I give your introduction as well and we'll dive in from there. Okay, cool. I'm Hubertus Franck. I'm in research. I lead a team here on confidential computing, in particular integration into the cloud and integration into the cloud run times. And just as Brandt and Peng already mentioned, I mean the main thing is the customer has to have absolute trust in what that data is not being exposed, right? In particular, as we're going into regulated environments, governments are putting stricter rules, for instance, on health data in place. We have to be able to ascertain that the hypervisor as an example, although it can get access to the data, sees no value in it. That means encryption is a key point as Brandt was presenting a key point for us to basically marry the confidential or the container-based workloads with effectively the technologies that are coming out like AMD, SAP. Back to you, Brandt. Awesome. Thanks. So a couple of thoughts. The questions that we got coming into the podcast here, why are we doing this? What are we looking for? What's interesting? How can the Cata community help? So, Aaron, thank you for grabbing this slide. This is an interesting place to drive the conversation. We often talk about three pillars of security, being, of course, confidentiality that only I can read my data. Integrity protection is the idea that only I can modify my data, right? If I'm borrowing a machine from Hubertus, I don't want him as administrator or as root being able to read what I'm doing or modify it in any way. And, of course, attestation is this idea that I can prove it. And then we talk about the three states of data. We've got data that's in flight. Most often, that's network, as we're talking about it. We've got data at rest. Most often, that's the disk. And then data in use. The three states of data, the data in flight and data in rest, these are fairly well understood, fairly well solved problems. It's the data in use one that's specifically a challenge. And one of the things that I find really exciting about the Cata Container Project and Samuel mentioned this earlier, right? A complete end-to-end stack for putting these pieces together. So, Aaron, let's take a look at the next slide. This is really my motivation here. So, this is an interesting example. This is, of course, the Chrome browser. It's possible this is Firefox, actually. But I've gone to amd.com and, of course, I have the wonderful little lock symbol that has become ubiquitous. People understand that now. You know, even my grandmother can go on to a website and look and see if there's a lock to figure out whether or not she's secure. Of course, if you click on that, you're going to get a reporting of the certificates that were used for attestation. And so, it starts with attestation and it works backwards, right? I've got a certificate. I've got it signed by a trusted authority. I know that I trust that authority. I can look at the set of algorithms that were used to encrypt my transaction. I know that a man in the middle can't read that data. I know that they can't modify it without me finding out about it. And so, I know that I have confidentiality, I have integrity, and I have attestation. The thing that's exciting for me, specifically about the Cata Container Project, is I see leadership in the ability to do this in a way that's going to actually flow through to the end user. And so, with this 2018 prototype that we've had in place, and with the SEV technology, it's been possible for us to offer confidentiality. Meaning, you could just turn it on and the containers would run in an encrypted mode. We're currently in the process of investing in Cata containers. We're working with the community. We're working with folks like IBM to go do all the other things as well. And it's critical that in the next couple of years, we've got the ability to build an infrastructure that can do all of this, as well as participate in things like the Confidential Computing Consortium that helps us understand how to flow this through to the end user. We need for everybody to be able to run in a secure mode. And just like the lock screen that's on your web browser, be able to see that you're actually implementing a completely secure stack. Hubert, just comments, or what else would you like to add there? Yeah, so there's some interesting new developments, right, that our teams are working on. Samuel is aware of those, right? Particularly, you want to also bridge between the storage and the network and the memory, so to speak, right? And one of the things that I think the community is very receptive from is if you, for instance, think about, as a use case, the clients effectively would like to have their own kernel image, as an example, right, and their own VM image. Today, that's basically still governed, so to speak, by the Kata community, right? But at the end of the day, if, for instance, the client would like to have their own VM, such as a micro-VM, to drive down the image size and things like that, we want to make sure that we're going to bring in the technologies like encrypted boot inside the VM itself, right? And that actually creates quite some challenges, right? Because we need to effectively make sure that the boot image itself is not visible to the host itself, right? And so by building, for instance, the VM coupled with the SAP technology, we are creating the enclave. And so many of the technologies that we know how to do from the VM perspective, we want to now also bring into the Kata community. And that's exactly where the confidential container initiatives is going to be. Thank you. All right, I think last slide. This is just interesting to offer as SEV roadmap from an AMD perspective. All of these technologies are public and available. Again, speaking specifically about the ability to do confidentiality and integrity protection, and when you look across the roadmap, we now offer technologies that will do confidentiality and integrity protection for memory, for registers, for cache. We're very excited about the roadmap. Of course, there's a Linux project that's terribly interesting to go work on SNP. And we're excited about the technology that we're offering here. I think that's close to the end of our time. Where does any other final comments, and then we can probably wrap it up. You know, you wrapped it up. Very good. Thanks. Awesome. Thanks, guys. Hey, thanks a lot, Brent. Thanks a lot to Bertos. It was very interesting. And as a matter of fact, we're really much involved with the confidential computing for those who are not aware, the upstream code in Kata containers supports memory encryption for a few hardware technologies. SCV is one of them. So a lot of progress on that front in the upstream project. So if you want to join this discussion, we actually have a meeting right after this live session about confidential computing. So yeah, thanks again. This was very interesting. Our next guest is Xiaobao Feng and he's going to talk about how Huawei is using Kata containers. Feng, welcome to the stage. Hello. Hey. Okay. So I'd like to share our deployment of Kata containers on Huawei public cloud service. I'm an engineer working in Huawei cloud and my job is focused on the container runtime. So these are the contents of my presentation. So first I will make a brief introduction on two cloud services. It's container services on the Huawei cloud, the CCI and the CC Turbo. And both of them use Kata containers as the container runtime on the nodes. And then I will go into more detail about the deployment of Kata containers on these two services, including our enhancement of the Kata containers. Okay. Next slide, please. Okay. First, let's take a look at CCI. CCI is short for cloud container instance. I have to say it's the first service Kubernetes service on public cloud. The whole clusters and nodes are managed by the CCI team itself and the customers care nothing about their... They care nothing about the nodes and the global objects. They just create the workloads directly through the Kubernetes API and control commands. And the workloads are built per second based on the running time. So this makes the CCI very suitable for running batch jobs. Actually, most of the CCI customers are running batch jobs like gene computing, AI training, and inferencing. So for better support of these jobs running, we enhanced Kubernetes and Kupelet to better support the networks, storages, and the physical devices like GPU or MPU. So the CCI customers and nodes are managed independently when we started the project. It's managed by the CCI team independently. But now it's evolving to share the same nodes with the ECS service. The ECS service is the Vodal Machine service provided by public cloud. So the secure containers of CCI and the VMs of ECS can now be scheduled on the same cluster or even on the same node that will greatly increase the utilization of the data center. Next slide. So now let's talk about the CCI Turbo. CCI is short for the cloud container engine which is a complete Kubernetes service provided by public cloud. The whole cluster including the nodes and the global objects are managed by the customers themselves. The Turbo here means that it's enhanced version of CCI. Actually the big enhancement of CCI Turbo is that it can manage the physical machines provided by a BMS service on Huawei and can launch cloud containers on it. We have the offload card which is also made by Huawei installed on all of these BMS machines. And the networks and the storages are offloaded to the card which makes a high resource utilization of the host and also it will improve the performance of storages and the networks. Oh sorry, the last slide. So with CataCandenders enables our customers can... Sorry, I think it's the last slides. Sorry, no, no, no. It's the next I think. Yeah, CCI Turbo, yeah. So with CataCandenders enabled our customers can use CCI Turbo as a platform to provide their own SaaS or past services and without the security concern of their run C container. Okay, next slide. Okay. So now let's go into more detail about the Cata deployment on these two services. First let's see how Cata connects the network on Huawei cloud on CCI. This answer is our implementation of CNI, of course Neutron is deployment on the same node of CCI and it costs Neutron to allocate the Neutron port so the IP address is also allocated so there is no overlay network. It also costs the OVS commands to create bridges and the VHOS user port and this OVS is a is a TPTK version so it can create a VHOS user port and then the yarns will add this VHOS user port to the CataCandabox with the network sub commands of Cata Runtime. So with the TPTK enabled the network packets are transmitted through the shared huge page memories between QMU and OVS which makes high bandwidth and low latency for containers a special point here is that the pre-allocated huge pages should be recognized as a locatable memory for Kubernetes as we all know that this is an important parameter for the Kubernetes schedule. We add a feature gate which is called a locator memory from a huge page to Kubernetes. When it is enabled the Kubernetes will convert the huge pages to a locatable memory when reporting to a scheduler and the memory limit on the node will be converted to the huge page. Okay, next slide. The containers running on CCI also have to access the local and remote storages on the cloud so we made support of it. CCI support of local storages differs from other services in that it should ensure IO isolation for each port because ports of multi-tenants may be running on the same node so we enhance COPLIT to support IOPS and BPS limit limitation for each port. For remote storages we implement our own CSI driver to support the CSI driver is called FUSI to support the block storage which is an EVS service on Huawei cloud and NAS which is a server service and object storages. All these storages can be imported directly as a PV to CCI and also the customers can create PVC and CCI can provision the storages and create PV and bind them to the PVC automatically. We also enhance the CATA to support these remote storages. For block storage CSI driver will attach it to the host but rather than mounting it on the host file system COPLIT keeps the password block device as the mount source directly to the CATA runtime. The CATA runtime will recognize the block device source as a block device so it will attach it to the virtual machine via VDIOS SCSI or VDIO BLOCK and the CATA agent will mount the device to the guest file system so it leaves nothing mounted on the host. For NAS and object storage COPLIT will give the UIR directly to CATA and the CATA agent mounts it in the VM and it also leaves nothing mounted on the host too. Many of the CCI ports are running AI jobs so they need the GPU like NVIDIA or MPU which is a Huawei ascent it is made by Huawei and it also supports the AI computing. So they need these physical devices to parallelize and accelerate the computing task. So we implement our own device plugins to support GPU and MPU device in VFI O mode. It can discover the devices through the device ID or an event ID and bind them to the VFI driver and then report them to the Kubernetes scheduler. In CATA we recompire the NVIDIA and the Huawei driver to allow them running on the customized kernel. We prepared all the dynamic libraries of driver installation package in the container file system so everything is prepared and containers just start there TensorFlow or other AI frameworks directly like what they do on the host. And to support the large-scale AI training jobs CCI also made it support NVIDIA link and even a band for customers to build a large training cluster. Okay. So now let's look at the CCTurbo. The CATA-Kanana running on CCTurbo tells more about its memory overhead in a good time because customers are directly aware of them because the clusters are managed by the customers. We need to optimize them. We rewrite the CATA-SHIM V2 in Rust to reduce the RSS of the SHIM process to under 4 megabytes. And we also made our own CUMI micro-VM it's tailored to CUMI to make it lightweight enough. The overhead of CUMI process is reduced from 30 megabytes to 10 megabytes. And the VNPUT time is also optimized to about 100 millisecond. We tailored the guest kernel of CATA based on the OLAOS, which is OLAOS which is a Linux distribution made by Huawei. This also reduced the memory footprint of guest kernel to 30 megabytes. And the kernel startup time is about 30 milliseconds. So after all these optimizations the memory overhead of each port is smaller than 50 megabytes. And the end-to-end startup time of each port is under one second which is basically met our requirement, but further optimization I think should be done. On CCTurbo the CATA containers are running on the physical machines of BMS service as I said. These physical machines provide many physical network devices in the form of VF of SIR-IoV in this to support the ENI. So we make these VF interfaces pass through to the sandbox of CATA via VFIO so loss of network performance is minimized. But Kubernetes surface traffic, Kubernetes surface traffic cannot be directly sent out through these interfaces. So we add an extra VF interface to each port as the channel of Kubernetes surface traffic. CNI will configure the IP tables in the NetNS to redirect the surface traffic. So CATA should sync it to the sandbox. This is also our enhancement of CATA. After doing all these things the network performance of CCTurbo is close to the bare metal machine. So that's all what I want to share. Thank you all. Thanks a lot Feng. It's very interesting to see how we can of all try to fix similar issues in different contexts. Thanks a lot. This is great. I have a few questions for you and for the rest of the audience. But I think we have around 10 minutes left. We can now take questions from the common box. I think one of the first one was about the amount of resource utilization of the lightweight NPM question asked by Gemal. Thanks for asking Gemal. Yes, exactly that one. I'm going to ask Tau, do you have an insight on this one? I know you've been driving the use case meeting for kind of resource isolation and I think this is kind of your turn. So what do you have to say about this? Do you have an insight? Yeah, for number of resource utilization it's actually dependent on the scale of your container size, especially when we look at the memory footprint. We have a very small fixed overhead for each port. But it also scales up with the size of your memory your port memory. I see we can see it also depends on which hypervisor you are using. Yes. You are in a hypervisor or this hypervisor have different resource utilization cost. Yeah, and the bigger the container the overhead of the VMM and the runtime becomes more marginal, obviously. Do you do some sort of sharing of memory between containers? If I'm running like 10 containers running the same image do you do some tricks for sharing memory between those containers? Yeah, in containers the guest is shared or shared with a text technology and all the containers will be used in the same host page cache and instead of consuming any guest page caches. Yeah, and that leads to interesting data when you start comparing the memory conception of Cata containers versus RunC for example where those kind of memory sharing is less possible. You see some interesting data with some containers after some size Cata containers memory overhead is less important than the RunC ones in some cases so that's pretty interesting. What about CPU overhead? CPU overhead of Cata is pretty low because I can see in your use case you already lower than 5% normally it also depends on what kind of workload you are running but normally it's below 5% in some cases the overhead is almost non-dictable. Yeah, especially with encrypted memory then once in the memory sharing it would be out the window because obviously you can't do that anymore, right? You can do these kind of tricks obviously with encrypted memory and computational computing you do not want any of your container memory to be shared with other containers but I guess it's kind of the price to pay for the additional security that you get with okay, thanks a lot thanks a lot for sharing this we had another question on emulation and VMM what kind of emulation do you run QMU so I'm going to answer partly and then ask Feng to chime in as well but we support many VMMs we support QMU obviously we also support Firecracker Cloud Advisor Acorn and essentially we have an architecture where we can potentially extend those to other app advisors but interestingly Feng mentioned about using lightweight QMU or a modified version of QMU I'm interesting to know more about this Feng, is that a fork of QMU or are you using some of the latest and greatest micro-VM machine type for example yeah it's a fork version of QMU and I think it has changed the machine type it has implemented its own machine type and also it also uses the VM Linux instead of the busy image to compress the kernel so it can optimize the boot time and also the machine type is not Q35 so it reduces many memory overhead of it in our environment the memory overhead of the common QMU is about 30 megabytes in the lightweight QMU it's under 10 megabytes and also in Huawei we also made another implement our own a new hypervisor called StrataWord it's also written in Rust and it's more lightweight but we will try it later but now we are using the QMU again so the Hubertus you want to try it? I also want to add in our own experience if you properly configure QMU rather than taking it out of the box binary you can really bring it down to sizes that are familiar with the firecracker and things like that, number one number two there is an effort within the QMU community to be much more modular the moment you start modularity you can effectively analyze and build in terms of virtualization and bring that image down even further and I think the approach to go to the micro VMs is a very important one and I think this effort that we are seeing in the QMU community and also the various initiatives of building new VMMs lightweight VMMs it's kind of been driven more and more is basically driven into KVM so what is left to emulate at the end of the day it's the memory and maybe the keyboard right? Yes, yes and then you build a very very tiny device model in user space and that's how you reduce things definitely thanks a lot for the insight we have a question from someone called Eric Ernst I think I know this guy on the directs blog device storage direct blog device it's a very interesting question I think many users are trying to do the same thing essentially not mounting the blog device on the host but doing the direct mount in the guest sorry and there's some efforts currently going on upstream for supporting that use case Feng, I think you mentioned you were working on this as well can you elaborate can you share if you're also looking at the upstream effort for this or are you aware of this kind of effort? As I know that Cudder containers only recognize the blog device as a device parameter of the container and it won't mount it to the guest directly I think I don't know if the version of the Cudder is too old for me to read I don't know okay another oh yeah go ahead you wanted to add something I just don't quite get the question so do you mean that the Cudder container in the upstream Cudder container already support the blog device mounted on no it doesn't but there's been a few discussions around it and how to do it there was a proposal that was not accepted there's also some efforts for pushing that at the Kubernetes level if I remember correctly to have a more transparent integration I think in our scenario this is more I think it's better for us to mount the blog device to the guest directly yeah I think it's the case for many use cases so people are looking at this we only have a few minutes left I'm going to take the last question from Niels it's cool to see there's several contributions to CADA all those extensions mentioned and made by AMD and why they are fed back in the upstream or do we have to pick this or that solution maybe Brent you want to talk about how AMD is upstreaming their efforts and contributions yeah happy to do that so for quite some time we had a project that was in our own GitHub but now that code is all checked and the mainline CADA project and our intent is to continue to develop that there so you don't have to go grab some sort of a branch that's right the confidential computing contribution for AMD have been merged and so I've been merged with a few other hardware vendors contribution to that effort so it's great to see and make sure that the project supports many different hardware implementation so that's pretty good thanks a lot Brent we only have a couple minutes left I just want to mention that next week we're going to have another episode of the opening for live session and this is pretty exciting we're going to have a few members of the Kubernetes both the Kubernetes Steering Committee and the OpenStack Technical Committee will be joining the episode and answering questions about integration cross project support and how you can keep with the updates on those different projects so this sounds pretty exciting I invite you to join next week episode if you want to hear about this and also if you have IDs for future episodes you can submit your IDs at www.openinfra.live anything is very welcome there I think we're running out of time I'd like to thank all our guests today for joining and participating to this session it was very interesting and thanks a lot for listening take care