 Some people are still having coffee, but obviously I'm not Dan Walsh. Dan unfortunately couldn't make it here. Most of you have already seen the talk before, so I'm repeating myself maybe. If not, my name is Valentin. I'm working in Dan's team, so he trusted me to give the talks on his behalf. I'm not sure how long this talk will take because I never gave this talk and Dan didn't either because this was the first time he planned to do it, so it's a world premiere if you if you want. So in in the talks before I was talking about the containers philosophy of Red Hat and that Dan and his team and teams across the company were somehow reflecting on what container tools should do, how they can be built, the architecture they're based on and the different use cases. And we were talking a lot about Potman today already. We were having a brief look at Builda, right, how we can use it, that it's meant to be a core utils library and that it still supports stalker file, but we extended it a little bit to support include files and things like this. And this talk targets Builda specifically, how we can run Builda in Kubernetes building pipelines, for instance, how we can do this in a more secure fashion, how it's meant to be used and also how we can speed up things. In the end, we always have done or a very common optimization problem is security and performance and the team was working a lot on optimizing this as much as possible with what the kernel can do at the moment and what the standards have to offer at the moment. So Builda, Builda I think is, I think it's around two years ago on Defconn, Dan asked the team to, besides Potman, also create a tool that builds container images. And I'm not sure if you have met or heard Dan before, he has a very, very, very, very strong Boston accent. So he said, instead of Builda, he said Builda. And this is how it came to be. Right, there's a cute logo also right of the Boston, Boston dog. And this one, I think this is the old logo, the new one, the head is whiter, because people came to him and asked why the dog is wearing underwear on the head. So as I said before, Builda is specialized into building containers, container images. And it's meant to go beyond what Docker files have to offer. Right, Docker files do a great job. And I use it on a daily basis, maybe you use it on a daily basis. But there are certain needs and use cases where Docker file is too constrained, too limited. And Builda is targeting exactly this. So for instance, here we can with Builda from Builda creates a new container and starts it. Here, we create a new container that is based on Fedora on Fedora latest, and Builda will return the container ID, a Builda mount mounts the root of s of this building container on the host and returns the mount point, which we're storing in the MET variable. And then we can use the mount point and operate on it. So in a talk before I was mentioning, for instance, instead of Fedora, we can use scratch. So it's basically an empty container. There's nothing on the root of s besides some, some mounts and devices that are required to actually execute the container. And then we can use this or we can use DNF install install root and use the host and the host repositories to install the binaries in the container. So this is really a good way to create a very trimmed down minimal container image. Before, at least I think this is what Dan wanted to say with this slide is pointing to Docker CP. So before if you wanted to exchange data between the container and the host, you either had to use volumes, which are essentially mounts, or copy, you could could use network storages as well, or you copy data between the container between containers, or host and container, then you ordered data data and copy it back. With a potman mount, you can just have or you have full access to the root of s of the container and do whatever you want. And, and the cool thing is, you don't need to write something specific. You don't need to wrap around a tool like Docker that you can use, you can basically use scripts that you already have, and you execute on the host, and maybe have to crude inside it, and then execute it there. But it will work. It's a very transparent, transparent way of achieving it. Right here, for instance, we copy or do a recursive copy from some sources into the mount point, then we do a DNF install route. So what install route does it changes some paths of DNF, which will surface into DNF installing the HTTPD HTTPD package under the mount point instead of the host paths. Right here, or we can do when we're in a, in a source project, you compile, you compile your whatever, let's say potman, and then you can change the destination directory onto the mount, and then it will be installed there. So it's a very, very easy and transparent and portable way of working with containers. I referred to, to the person in the red t-shirt in the back already a few times before, but it's just awesome that you wear it. Containers are Linux, and this is, this is it. There is, we try hard not to make working with containers something special. It should work just as working on a VM on your host. There should be nothing, nothing special about it. And it should be easy to integrate. It should be compatible and also interoperable. And this is a basically what containers are Linux can mean as well. Right. And builder also supports the builder config command. What config does it alters or allows to alter the configuration of the resulting image. So here, if we do a builder config dash, dash entry point, and if we then commit the image, create an image from that in the configuration, we will then change the entry point to whatever we have done. We can also change environment variables. So this is basically something that the docker file exposes already. And that we still need also when we're just working on the CLI. And this is a, a neat way to, to achieve altering the config of the, the resulting image. And last but not least, we can commit the container. So we use the current state of the container and create a container image out of his, out of it. And we call it my HTTPD. And after that we can, we can push it to, to Dan's repository on the Docker IO registry. But I'm not as entertaining as Dan. Usually he wants people to talk all the time. I'm not a cowboy like he is. So I just say, Dan, wait, what about Docker file? For sure, builder also supports Docker file where the build using Docker file command, you can use it in the same way as with Docker. Or for the lazy ones, you can use build a bot, which is short for build using Docker file. And does builder have a scripting language, perhaps builder file? It's bash. I already made the joke before, but well, the, the slides are not very old. So what do we mean by bash containers are Linux? We didn't want to introduce a new concept that people have to adopt to, they can just use the tools they're used to and the scripts that are already out there in the wild. So we want to offer a core utility like experience. So we want and encourage people to build their tools on top of builder. If they have certain needs or certain requirements or specific use cases to build container images, then builder is a good choice to use as a low level tool to basically achieve it and not worry about the rest. Right. At the, at the moment, the team is working, especially the, the builder folks, obviously are working on making or using builder in OpenShift for the build pipeline rather than Docker. And also there's a lot of work going on with Ansible containers. And right. Later in the talk, Dan and I will talk about how the team makes it possible to use builder in the OpenShift Kubernetes pipeline specifically. So usually what, when using Docker and when people want to build containers, inside containers. So when you, when you have a building pipeline on Kubernetes necessarily all your workloads are running in a pod, so in a container or a container in a pod. And if you start Docker there, you will have a new Docker daemon. If you want to pull something down, everything needs to be started again. And most likely you won't have the privileges. So a common approach to allow it is Docker in Docker. So you mount the Docker socket into the container, install the Docker client, and then you can conveniently use Docker inside there. But this, this works well, but it comes at a cost and especially at a security cost. And using builder, so because the security cost is because you have basically access to a socket on the host, and the, which effectively gives you root access on the host. And this is a bad thing. Not necessarily when you know your workloads, when you trust the workloads, but when you want to execute random Docker builds, or when you're afraid of corruption, then you cannot trust basically this architecture. Builder works differently. As I said, it's not a daemon. So there's no need to talk to the Docker socket and it's especially not leaked. The problem with our security issue is basically information and data from the host leak into the container. So there's always a door open. Some people say, well, it's always with containers, but this is specifically, right? There, you know, this is a way how I can access the host. I can exploit this. So builder can also run as a non-root on, on the desktop, I think also on the server at Justice Potman. The architecture is very similar. The containers are, look a little bit different because for building a container, or building a container looks a little bit different. You have different requirements also with respect to the capabilities and things like this. Then for running them in production, such as with Potman. And well, I've said this before, we can use or mount the root of s on the host and then just install it there. The package is there. I was talking about this already before. So traditionally, people leak the Docker daemon, the socket is specifically into the container. So then in the container, in a Kubernetes pipeline, you can execute it. And this was a very bad idea from a security perspective. And Dan wrote an article a couple of years ago about how granting unprivileged users access to the Docker socket is the most dangerous thing you do on a link system. So you can basically, or you have root without the password. So you can, you can do a lot of nasty things. The bottom line there is if you give me access to the Docker socket, I can become full root on your system and cause harm. Then as I'm rude, I can basically erase all the traces on the host. So I can corrupt data. I can do very nasty things and then clean up afterwards. And then the folks on the other side are left alone, figuring out what has happened. And most likely they will not find out. Right. So builder is targeting also this use case, building container images in the pipeline. And conveniently, if you want to use it into a Kubernetes pipeline, it would be nice to have pre-built images that you can use. And here we have some on Quay. This is basically where you can find it on Quay.io, repository builder. You can download it. We offer it in three different flavors. So builders pre-installed on this image. You don't need to create your own images. This is something that we maintain officially from upstream. So there's a stable one which uses Fedora, the stable Fedora release and also the latest stable version of Portman. Then there's a testing one with updates from testing if they exist and then an upstream one, which basically uses the latest master. So it's head, it's the bleeding edge. Obviously there can be bugs introduced. So if you use it in production, I highly recommend using the stable one. If you're curious about the latest one or want to play with the latest version, use the upstream one. I'm not sure how often it gets propagated but I'm certain it's at least once per day. So currently we use the, if we look at the images at the stable docker file that we use to generate the builder image on Quay.io, currently we use the latest Fedora release for updating the container images. But in the future we could also use the universal base image from Red Hat for instance if customers want to build on it or if the community wants to build on it. So there are thoughts to also offer it with UBI images. So here, from then we, now we install builder, we also install fuse overlays, FS. So why fuse overlay? We want to run the containers without root inside the container. Which means we cannot use natively overlay because the overlay file system requires us to have the capsis admin capability which effectively can be more or less compared to having root. So still we want to use the mechanisms of overlay and the concept of layering file systems. So Giuseppe Scrivano, colleague from the team wrote fuse overlay FS which is basically overlay implementation in user space which runs in user space and which uses fuse. And right, we remove the container as e-linux here to keep the image really, really small. And after that we do the typical cleanup, we remove the cache, the locks of DNF and YUM to get rid of all the traces and metadata that we don't need. So we really need to keep it small. So in the third step in the stalker file, we are configuring container storage which is used by builder potmans, copio and also cryo. And specifically here we use it to use fuse overlay FS. So if you see the first SED command, we're uncommenting the mount program which basically tells or instructs the container storage backend to use this mount program which is then pointing to or by default to fuse overlay FS. So then fuse overlay FS is used for mounting and we also set the additional image stores to var lip-shared. So the additional image stores usually when we run builder potman yada yada, there is one central storage for images. Sometimes it's nice or maybe what additional images does is it allows for also looking up in other paths and other stores searching for the image which is a nice use case for instance if you have a shared environment, maybe an HPC, you want things to be fast, you don't want users to consume a lot of disk storage, you want to avoid redundancy, you may have a shared network storage, you can use additional image stores and point it to the shared network storage and then users can use the images there directly. So when you do a potman or build a poll, it will first look into the image stores, see if the image is there, if it's there it will use it, otherwise it will contact and ask registry for polling. And in the last step we're just adding some files that otherwise build will add at run time to tweak a little bit more performance. We can save a few cycles by adding the log files for instance. So the log files that you see here are used to synchronize processes to avoid data corruption. So if you have multiple builder processes running in parallel, all talking to the store, they need to be serialized at some point to avoid data corruption, they could be reading, writing at the same paths simultaneously. We can't use mutics as in semaphores because we don't share the same memory space, process space so we have to defer those things to file locks. And last but not least, we have to set the builder started in user and as environment variable and set also the builder isolation crude. So if builder detects that it's being run without Capsis admin, it will attempt to set a username space as part of setting itself up to run its rootless mode, which is just its behavior when it runs without Capsis admin because we need to create a new or unshare from the current username space to get a little bit more privileges out of it. And setting this environment variable stops it from attempting to do it because we're already running in a container so we don't need to do it. And we also tell builder to set it to the crude based isolation rather than full namespace isolation. Well, again, we're already running into a container. We're already in a locked down environment and running a screwed makes it a little bit less redundant. And also we tweak a little bit more performance. So this is a really nice way of doing it. Some of you will understandably ask, how can I possibly know about this? Well, there are some technical details or many technical details. And this is first why we're talking about it and also why we are officially supporting this image. So users don't have to worry about that. So as I mentioned in the introduction of this talk, basically we have an optimization problem, security versus speed. It will haunt us forever in it. And yeah, the speed that a process can run versus amount of security we can wrap processes with. So for instance, in a recent report, I read that depending on where the container is being executed using second filters can decrease performance by up to 10%, depending on where you run on. Why? Because the second filters are really big, right? Traditionally, if you were in the last talk, I said that traditionally container engines use a wide list approach, so the filters get really, really big. So we enable, I don't know, 300 syscalls or 180 syscalls for each syscall. The kernel has to go through the list and check, Oh, am I allowed to do it? Am I allowed to do it? Am I allowed to do it? Am I allowed to do it? And so on this comes at a cost. So security comes at a cost. And in this talk, Dan explains how we can find the right balance when using Builder. So we designed the Builder image to allow users also to experiment with the correct mix of them depending on what they need. Later, I'm going to show some code examples and also a demo that Dan recorded yesterday to show how users can use it to and to fit it to their specific needs. So certainly the totally locked down case is the slowest one. We have as much isolation as the kernel can get us nearly as much and a lot of resource restrictions. So we don't have access to the mountain points of the host. We don't see the pit namespace. We have a different network namespace. All these things. So now we use Podman to execute this image. But we could have just as easily used Docker or Cryo in Kubernetes. Here we need to mount depth fuse because Overlay of as is a fuse implementation and needs fuse to be present. This is the Builder image that we're using. Cryo repository builder. It will automatically pick the latest. So which is the point to stable. Here we mount the Docker file. Sorry, the Docker file that I was showing before. And we're running on a SELinux enabled system. So we need to pass the option colon set to instruct relabeling. If not, SELinux will tell us, ah, no, my friend, you're not going to read that. And then we're going to build the test image. Again, build a but short version. We're lazy for build using Docker file. So the container starts with basically everything is empty. The container starch is empty because there's nothing. We clean everything up from the base image. All the DNF caches are cold. There's no metadata. All the files have to be created. This takes a little bit of time. It makes it slow. However, it's it's most secure. It's totally locked down. Although it's running as rude inside the container. This would work within a username space. And it's totally isolated from the from the host. necessarily. This is the the most secure option that we're having. There's no information leak. However, if we there, there's no sharing. If we have no leak to the host, the containers and the host cannot share anything. So when you have all images on this planet, on your host, and you know, well, I have it on the host, you will redundantly store the data there and it will make the build slower. Yet we chose or optimized for security. And this is the cost that we have to pay. I'm not going to start demo now. I'm not a cowboy as Dan is, but I'm going to show his demo later. So the totally, so not sorry for the hiccup, right? I mean, totally privileged is, is the fastest one. We can share everything with a host. It's basically just a glorified process that happens to have another root of s and we use builder from there. So again, we have the same command as before, but now we're going to mount varlip containers, which contains, sorry, the images that are already pulled down. So this means that if we pull down an image once, we don't need to pull it down another time. It will be shared with a host and all containers that are also mounting or volume mounting varlip containers. To make this less of a pain, we disable as e-linux and I'm pretty sure Dan will, will talk a lot about that this is not a good idea, but it really just depends on the use case. If we know that no bad things will happen because we trust the builds that are the images that we're going to build, then this is a fine thing. So right here, it's the fastest and it can, since everything can be shared on the host in the containers and other container engines also scopio build an apartment and so on. They can, can instantly use the new images from the store. However, it's the least secure. Since the container storage is shared, yeah, we can, we can write content that would affect other containers in the future. We could also just remove things and maybe break running workloads at the moment. If container A is downloading an image and malicious container B is detecting that a new image is there, it might just remove it. And then if container A wants to run another container in the container referencing the image that has just been deleted, well, this is, this is not going to work, at least not in all circumstances. And disabling as e-linux, well, I mentioned it before Dan, Dan really does not, does not like disabling this as e-linux. And it's one of the most important security mechanisms on as e-linux enabled systems. It has prevented a lot of CVEs and security issues in the past and it will prevent them in the future. In some cases, as e-linux even prevented kernel CVEs, because, I mean, we cannot exploit a kernel bug if we're not allowed to execute. So there's a lot of good things and wisdom in using as e-linux. Still, it's more secure than Docker and because it can only, we only write to the container host. So, right, we're volume mounting varlip containers and not socket that allows for root execution on the entire host. Demo I will show later. I'm not as a cowboy as Dan is. So now we're, we're trying to find a middle ground. We don't want, well, we want to speed up things because maybe we, we want to have things fast and maybe sharing things read only is acceptable. Well, if I cannot write to it and the kernel makes sure I'm not able to, to corrupt any data, but just consume it, this is something that may be acceptable, maybe not for every user in all, all circumstances, but for some. And now we're writing here varlip container storage, storage as read only and put it to varlip shared. If you remember, we configured our builder in the container image in the Docker file to set the additional stores to varlip shared. And what we're doing here effectively is mounting the storage, this image store or container store from the host, everything that's inside there to the additional store in the container. So when in the container builder is pulling an image that is being used during a build, it will look up in the additional store, see it's there, use the data there, and it will have quite some, quite some performance improvement, right? It's obvious. We don't need to, to download hundreds of megabytes, but we just access it. That's already on disk. Alright, basically I said everything that is here, and it's also very secure because we're, we're still totally locked down. Still it, the image inside is running as root. Well, we can use username spaces, but this would not work within a username space. Most isolate. Yeah. And it's most isolated from the host. Only information leak images from the host. So what we need to be careful of is, although we cannot write, but if you're storing confidential data on the host image store, then you shouldn't do it because the containers have access to it. They can read it. So if you have a secret, don't put it there demo. I'm going to show later. And now Dan wants to talk about some more advanced use of the additional stores. So one question is when it comes to improving the performance of builder in such a Kubernetes open shift environment is why are we pulling potentially huge images to every container node, right? We can, I mean, we're in a distributed system. We have that. We then have distributed problems, which in many cases are the same. We have on the host across processes, but now we just somehow lift it to yet another level, which then spans across machines. And one example, right, is HPC high performance computing. Those folks, they have really gigantic images and they are running on thousands of notes. Yet they want to avoid redundancy in a very, very locked down environment. So usually you don't get or nearly all circumstances, you won't have root access on any node. You have an HPC. This is why many HPC targeted container engines are from the security perspective. Well, not while there's barely any security because most are recruiting or just use a crude or CH route, depending on how you want to want to pronounce it. So for HPC or for this specific use case, we can put additional stores on, on a shared network storage, right? It could be NFS, then there's Cephs, S3, Iskasi, all the things that are out there, we can use it. Additional image stores, the admins decide what's there and you can use it. And the nice thing is, all container tools that we are building, they're, they all share the same building blocks. So LipPod, Podman, for instance, uses code directly from Builder. It has a dependency on Builder, go packages from Builder that it's using for the Dockerfile builds. Same goes down to the image library and the storage library. So all of the tools instantly benefit from it. So there's a really nice things. Shortcomings with this approach is, well, there could be potential problems with network latency and hiccups. So this is somehow also a little bit a shift of the responsibility, but we're not to blame anymore, so that's nice, right? And yeah, we, so an argument to do is, to do that is, well, we already share network storage in those environments, so we should also do this for images. Again, containers are Linux and this is just normal data for processes that look a little bit different than commonly on your system. Another thing we've been looking at at builds is why does DNF or YUM take so long to run? Sometimes this is really a bottleneck during build because, well, downloading things can take time, right? So both YUM and DNF, they need to update their cache. So they have a cache and they look, is the cache still hot or do I need to update it? And this includes basically information on all RPMs on all paths used in the entire distribution and it's apparently it's a very, very big XML file and parsing XML, it takes also some time and this can take more than 30 seconds. So if you do a YUM yada yada, you can add 30 seconds to the basically the execution time of the command. So if we look a little bit deeper at the problem is also the way how most people write Docker files. So here, for instance, we create a new or we base a new image on UBI aid, then we do basically what everybody's doing with any package manager on this planet. We install something and then we remove the metadata again. Well, every time, okay, first, we remove the metadata because we don't want to have this on our layer, right? We want our images to be small or the layers that we use in our images to be small. So we should only keep data on the container image layer that we really need. For execution, we don't need the metadata. So we remove it. But here we have at least two calls of YUM or DNF. So you can add 60 seconds to the build time of the image and it's not really necessary. So we were thinking a little bit about how can can we do this? So we cannot really do it read only. So we can't use the trick we've been using before with the read only image stores. Because in some cases, maybe DNF and YUM want to write there. So then it needs to be writable. And here comes the concept of overlay volume mounts. So overlay volume mounts allow you to volume mount into a directory from the host into the container. It's writable inside the container, but read only on the host. So the mount point for the container, we can write there. We can read the data that is already there on the host. But as soon as we write something, it won't be propagated down to the host. So the data on the host is secure, but we can read it. And right, a newly written content will be destroyed when the container exits. Basically, we unmount, we unmount all the junk that we don't need. And we're done. So in this case, if we mount the metadata or the caches from the host into the container, awesome. If assuming the cache is hot, then we can just use it 30 seconds. Done. Another YUM. Done. And if we remove it, we don't care. It won't be written to the host. So this Docker file will still work. If we run the second run DNF, the cache is still hot. Don't need it. And if we only use it, we don't even need to clean afterwards. Or maybe we still want to clean because we don't want to commit it. But it depends really on how many, or how many paths we mounted inside. How do we do this? We add the colon capital O to the path that we are mounting or the volume mount. And this is basically achieving exactly this. So we have the overlay volume mount then at VAR cache DNF that we use from the host. And talking in overlay terms, right, we have a layer overlay file system. The host one is on the lower level on the lower read only layer, not level layer, sorry. And the one in the container is an upper layer, which is rideable. So this basically does the trick for us. So we're just using overlay concepts to achieve it. And well demo, I'm going to show, I think we still have 15 minutes, which is awesome. And after that, I guess I need to clear my throat. Oops. Oh, right. I'm not. There we go. So I'm not now mirroring. Can you, can you still read it in the back? If not, I would ask you to to move forward. So, um, Dan said, well, I'm going to prepare a demo for you. And I said, please don't prepare a demo for me. Things will go south if I run it locally, or maybe the network is down. So he said, okay, I'm going to record, record the demo. So thank you, Dan, in case you're watching. And now I'm going to show you what's going on. I try to show you what's going on right there. I can go full screen. Wow, this is much better. Oh, nice. Okay. Running Dan's demo. This is a nice demo, isn't it? Okay, it's showing technical issues. I'm going to scale it back. Okay, maybe it doesn't scale as much as I want to. There you see, I'm really a terminal guy. I'm not okay. This is as big as we can go apparently. Does it? Well, let's let's do it like this then. So here we have the three examples, right? We have the totally locked down container, we check how long it will take. Then we have the totally privileged container, which also has a right access, which we expect to be the fastest. And then we have the middle ground where we can mount the cache read only. And I think Dan also wants to demonstrate the overlay mount. So now Dan is using the UBI 8 universal base image from rel 8 and use it in a clean container storage. So here we are now pulling or we have to pull the image or the two layers and the image config. So this takes already 17.6 seconds. I think Dan wanted to make sure that I'm not skipping the 17 seconds. So now we use the totally privileged one. So we mount varlip containers directly into our build container. So well, okay, well, this is, this is really fast, right? So we spare 14 seconds just by allowing for sharing data. But I'm amazed that Dan actually did this in one run and didn't script it. I'm sure I would have made plenty of typos when one typo. It's amazing for me. It would be every second word. So here this time it was obviously faster. We were around 40 seconds faster because the image was already there. So we can just reuse it. But we had to disable as Elinix entirely. And in our world, we don't like disabling as Elinix at all for the aforementioned reasons, right? Especially the problem is now we can override data in the host. So in the container storage of the host, we don't have access to the host, but to the container storage. So with respect to this data, well, we can corrupt it. So in the, in the last, in the last example, we use the additional stores. So this is what I was talking on before where we mount the container storage as read only to the path of the additional stores within the container image, which is build a stable, right? Okay, now I'm going to put this down. I guess we lost a little bit of things here. Yeah. So we, we volume mount varlip container storage to varlip shared as read only and varlip shared in the build a stable container image that we use. This is configured to be an additional storage. So when pulling images, builder will look up into the additional store, detect that the image is already there. And now this is actually faster. Well, it's, it's not a representative benchmark because it has only been executed once. And maybe we, there was some scheduling going on or Dan was listening to heavy metal while doing this and Spotify was downloading at the moment, um, or writing a lot to the disk. Um, but you can see that with the concept of additional stores and mounting them as read only, we have a very similar speed, a very similar performance as running in a completely privileged container. So in, in the end, this optimization problem in this specific context can, can really, can really be tamed. So we can be secure in the sense that we are not able to corrupt data on the host. At least we're protected by the kernel because we can trust the kernel or have to trust the kernel that it will be mounted as read only. But we have the same performance as with running completely insecure privileged where we could corrupt the data. So then thank you for providing this demo. You saved my life and I guess I saved yours. So maybe we're even, and that's pretty much for this talk. I'm happy to answer any questions. Maybe you want to see more slides. You, uh, I want to rerun the demo to share it. You mean, uh, yeah, I will. So I asked Dan, uh, to upload the slides on scat.com. So we will make sure that we will upload all, all the slides to scat.com. So you can, you can have a look and I will, I will ask Dan also to, to put a link to, to the demo or it will expire at some point, but what I will do is I will sneak it. There we go. So here you will, now the link is there for sure. Um, the, as Kinema here, it might expire at some point. Um, although I'm not sure, I will, I will talk to Dan and he will make sure. Any other questions? Yeah. So I'm, so the, the question is, um, basically the, the, the status of sharing data on network storage, right? Um, correct? Well, I'm, I, I don't know of any problem so far. Let's put it this way. I was, uh, personally, I was not involved into this development. Dan will know for sure. Um, so then if you're listening, um, but I haven't heard of, of any complications so far. So another way to speed things up would be to let the host and the containers, uh, use, uh, mirror, which is close to them. So it can be a node local registry mirror where, where, if you pull from Docker IO or registry red hat.com, uh, or registry access red hat.com, uh, everything will be cached there. This would also speed up things, but we, we chose this one, this one for now. So long story short, I cannot answer your, your, your question, uh, for sure, but I, uh, or with certainty, but I haven't heard of any problems so far. All right. Then I think after three hours, I'm going to take a shower now and going to talk to you later. Thanks for joining.