 Good morning, good afternoon, good evening, and welcome to Red Hat portfolio. Welcome to this week's Ask an OpenShift admin office hours. I am, as always, extremely happy and very lucky to be joined by my co-host, Johnny Ricard. Happy Wednesday, and I hope you had a good holiday last week. Oh, absolutely. You know, my favorite time of the year is eating time, and so I got to do plenty of that. How was yours? It was great. We roasted a turkey for the first time in four or five years. Usually we do the fried turkey thing. As you can tell, I enjoy eating, and I'm from the south, so nothing like a little fire hazard for the holidays. But this year, just through, I don't know, I don't know if it was luck or bad luck, our turkey was too big for the fryer. It was a great holiday. Stephanie, in the background here, I hope that you had a great holiday as well, and it's good to see everybody. That's awesome. If it's not fried, it's not southern, so we did the fried turkey thing. Yeah. So we are very happy to be joined by Adele Zalouk today. I hope I'm pronouncing that correctly. As my children will tell you, I'm terrible with names. So today, we are going to be talking about Sandbox Containers, which is the red hat, the open shift name for Kata containers. So Adele, if you don't mind introducing yourself. Hey, everyone. Very happy to be here. My name is Adele Zalouk, and I'm the product manager for the open shift Sandbox Containers, as Andrew just mentioned. So I know this is something that is interesting to me. It's kind of close to the things that I am involved in on a regular basis because there is some fanciness involved on the back end, right? We're going to be talking about hypervisors and KVM and all that other stuff a little bit. So looking forward to really defining and digging into Sandbox Containers, Kata Containers, and how they fit in with open shift. But before we get to that, let's talk about, you know, let's do our normal opening stuff, Johnny. So first, the ask an open shift admin office hour live stream. I'll get all of those words out eventually. It's like riding a bike. You know, it's been two weeks and it was two weeks before that. So I'm feeling a little rusty. Just a little rusty. Yeah, I gotta get it back in gear. Yeah. It's, you know, as long as it's, you know, like riding a bike and not falling off a bike. Exactly. So the the admin ask an open shift admin office hour live stream is one of the office hours series of live streams here on Red Hat live streaming. And ultimately our goal here is just like if you had a professor or a manager or anybody like that who had office hours, it's an opportunity for you, our audience to come and ask us anything and everything that you would like. You know, we are open shift experts. Johnny and I are both, you know, we have a strong administrator background. We've been doing this for a number of years. So we want to answer those questions for you. You know, we always have a topic and today we are privileged to have a guest join us as well to elaborate on that topic. But don't withhold or don't restrain your questions to just that topic. You can ask us about anything you'd like. Worst thing that happens is we don't know and I will say that that happens a lot with me and we go and find the answer. We try and get out a blog post after each one of these episodes. I know I'm still running a little bit behind we're catching up on the blog posts. But we try and get a blog post out after each one of these. So that way we can, you know, we can answer those questions that we might not have got to while we're here on the stream as well as provide links and all kinds of other stuff that you see here on the stream. So the other thing that we tend to do or that I like to do here at the start of each one of these streams is have a few top of mind topics. These are things that kind of Johnny and I saw come across our desk, right? Things that we've seen come up recently that we feel are important to you all, our audience. And we want to make sure that you're aware of them and you know, can either be prepared for something that's coming down the line or just, you know, general awareness type of stuff. So the first one of these top of mind Christian is awesome. Oh man, I forgot I got a coffee cup that I keep meaning to show you Christian. I'll have to send it to you offline because you're going to love it. So the first one that I want to talk about is or highlight is if you missed KubeCon live, if the sessions are now available online. So let me grab the link here. And I will paste that into the chat here on Twitch. So inside of there is kind of the whole list of sessions that are available inside of there. I don't have the link handy, but they also publish their transparency reports, which included the top 10 list of sessions. I think there were two or three sessions that included Red Hat folks. There was one on Helm charts that had Karina. There was one on storage and disaster recovery, which had several of our storage folks on there. That's a great one by the way. I'm already talking with those folks about coming on to discuss ramen. Not the dish, but the service for doing effectively volume replication with OpenShiftData Foundation. So an entirely Kubernetes native way using CRDs to set up volume replication between two clusters. So lots of really cool stuff going on there. But yeah, I know there was a bunch of sessions. I don't remember how many sessions were in KubeCon, but they're all available on YouTube now. So definitely check those out if you happen to miss any of them. I'm going to skip over the next one and save it for last, Johnny, if you're looking at the notes. So another announcement that we have in this one, I thought was kind of quiet was that ARO, Azure Red Hat OpenShift, has been, it is now HIPAA compliant. So I'll post the link to the blog post inside at here. But for those healthcare customers, for anybody who needs that type of regulatory compliance, ARO is now yet another solution that you have available to you. I think IBM cloud is the other one. So I don't know what the name for it is now. We used to call it ROX, Red Hat OpenShift Kubernetes service with IBM, but IBM cloud is also HIPAA compliant. Now Azure Red Hat OpenShift is HIPAA compliant. So by all means, if you want to manage service, a managed OpenShift service, regulatory compliance is no longer a barrier. Our hope nine, so yesterday's slides, I will, I know they updated the link and we'll talk about that in just a moment. They updated the what's new, what's next link with the video, they have not posted the slides. If they, I'm going to send an email as soon as we're done with the stream today, I can post those. I now have the power, I have the login. So I'll grab those, if nothing else, I'll grab those slides and we'll get them at least posted to SpeakerDeck and then get them posted on the webpage as soon as we can. So the next one, this is a question that has, it's come up kind of periodically over the last couple of years, but there's been a rash of these questions that I've seen internally over the last couple of weeks. And it's mostly focused on Rev and vSphere customers who they want to move, open shift compute nodes and sometimes control plane nodes between clusters. So moving between two DRS clusters or two Rev clusters or compute clusters, hey, I am maybe life cycling the hardware inside of there. I need to expand capacity for whatever reason it happens to be. And effectively the question is, is this possible? Is it supported for me to be able to do this? And the answer can be it's not necessarily straightforward. So Johnny, keep me honest here, make sure that I'm not talking out of line. So if you're using a non-integrated or at a platform, none deployments, then absolutely. You can of course do that. We have no visibility into the underlying infrastructure. You know, you could, you could put them on different continents if you wanted to. So long as the latency and all the other requirements for open shift are there, open shift itself wouldn't know the difference. You would just have, you know, probably other issues if they're on different continents. But so if you are using UPI or IPI, when there is a cloud provider involved, essentially you need to be cognizant of what integrations are being utilized. So what do I mean by that? If you are using, say the CSI provisioner or the, you know, the vSphere CSI provisioner or the vSphere entry provisioner, it needs to be able to access those PVCs regardless of where the nodes are at. So if you have two clusters or three or five clusters and you have one data store that all of those PVCs are provisioned into, that data store needs to be accessible across all of those different clusters that the nodes may potentially be at. And I will note that this is not a tested scenario. So it is in theory supported, but it is probably going to be a gray area because it's not explicitly tested. So the safest way, if you need to say move my cluster or move my open shift cluster from say vSphere DRS cluster one to DRS cluster two, if you're using IPI, is to add a new machine set with the new cluster and then stand up nodes in the new cluster and destroy nodes from the original cluster, including doing a, basically a disaster recovery, right? To destroy a control plane node and then create a new one and do a recovery on it and everything else. That is going to be the safest and probably the recommended way if you were to pick up the phone and call us. In theory, it is possible to shut down the whole cluster and migrate everything over. That gets complex. Again, if you're using PVCs, because the PVCs won't move. If they do move, they will effectively be lost, if you will, by the storage provisioner and it won't be able to connect them to the nodes. So just make sure you disconnect those PVCs, make sure wherever they're at, they're accessible in the new location as well. And then you would need to, of course, configure or reconfigure the vSphere stuff. There's a KCS for that. Let me see if I can find it real quick. But there's a KCS for both Rev and OpenShift on how to reconfigure the machine sets, et cetera. Here's the KCS. I'm going to post that into Twitch here for those things. So I think that covers the basis associated with that. By far, the safest thing to do, stand up a new cluster. Next to that is deploy new nodes in the new location and basically recreate the cluster in flight, if you will. I won't say it's the safest, but the next option would be to power down and move all of those things, taking into account any of the dependencies that exist, particularly PVCs and networking, of course. Those are kind of the options. You cannot do, I mean, technically, you probably could do a live migration. I would say that that is definitely not tested and probably not supported if something goes wrong. Definitely same for a storage live migration. The risk with storage migration in particular is mounted PVCs. If they're coming from the VMware CSI driver or entry driver, it'll storage for live migrate those PVCs with it, and again, those will get lost. So just be careful. Okay, Johnny, any other thoughts? Yeah, I mean, just with that in mind, too, if you're moving from one cluster to another, you're going to have to update any secrets that are out there to reflect that cluster name and all that stuff, too. So just something to keep in mind that it's a little bit more involved once you start doing that. Yeah. There was one question in the chat. About, you know, forwarding logs from, you know, from, you know, like your Havana logs up to a third party logging. Our hope nine, thank you for posting the link to the documentation. But yeah, that's the right answer. So for the, there's an external forwarder that goes with OpenShift that you can configure, and it'll push your logs out to, you know, whichever supported seam you're looking for. Yeah, there's also, if I remember correctly, with the log forwarding API, there's a filter option. So you can, you know, essentially set it to forward all logs that either match or don't match the filter, which might help cut that down. I know we used to have customers talking, you know, the, I pay for my third party, you know, log analysis tool by ingest bandwidth. And, you know, there's, there's a lot of stuff here that I just don't need. You know, how can I cut it down? So the filtering option is an option there as well. There's a 404 error inside the blog post. Okay. Thank you, DMI3. We'll, I'll take a note on that and we'll see if we can get them to update that link. Be aware that you enable external logging. You also need to include a rule to send it to your Elastic locally. Yes. Very good point. Thank you. Yes. That's Andrew did that himself. So yeah, why aren't I seeing all of these logs? What happened? Okay. So the last thing that I held off until the end there is, in case you missed it and somebody maybe was our hope nine, you know, mentioned it, we did do an update to the OpenShift roadmap yesterday. So it is posted into the, into the YouTube channel. You can go back, you can look at it. Do be aware that the first roughly 11 and a half minutes were bad. We had some issues with the, so the way that it works is we do the roadmap broadcast internally at Red Hat using Blue Jeans. And then we take that Blue Jeans and we stream it publicly. So you all, our public, you know, streaming audience and the internal Red Hats, you know, employees all see that presentation literally simultaneously. It just so happens that the machine that was being used for that Blue Jeans Restream was having issues for whatever reason. So it took them a little over 10 minutes to get that figured out. Fortunately, during that first 10 minutes, it was just kind of an overview of, you know, high level OpenShift strategy things. It wasn't detailed roadmap. So you didn't miss anything there, but we'll get all of those slides posted. It'll have all of the slides when we get those up there. Slack DNS Post Modem was great. Yes, Christian, you posted that yesterday into our internal Slack. Let me see if I can find that, or if you have the link handy. You know, I know you're a DNS nerd. I'm a little bit of a DNS nerd. I can spell it usually. But yeah, if you remember, Slack had some issues back, I think it was August, August, September timeframe, when they were trying to implement DNS Sec. And that blog post does a great job of walking through all the issues they saw. You know, it's a really good example of what a post modem and root cause analysis should look like in my opinion. So if you happen to have that link handy, Christian, please, please post it. Okay. Oh, wait, sorry. Johnny, is there like two of these things in the list that you want to cover real quick, like highlight from the what's new or sorry, from the what's next? Yeah, the big one is just kind of continue on from a topic that we covered a couple of weeks ago with the Metal LB. Coming up is going to be Metal LB and then the BGP integration that's coming through. So that's going to be really awesome. And then the other one is the Aero and Rosa user experience where you're deploying like it's just that whole workflow is going to get a lot better from a from a visual standpoint and just like, you know, just making it easier for everyone to deploy. And then the last one is really for a friend of mine, an old customer of mine who's really been looking forward to this for a long time. And that's OCP on ARM. That's going to be huge. I know that there's a lot of people out in the world looking for that. And, you know, I'm really excited about that, especially from our friend Derek who's who's been all over the stuff from, you know, getting getting as far out on the edge as you can with the, you know, the minimal footprint as he possibly can. So all I'm hearing is send all of your open shift on Pi zero questions to Johnny. Yes, and I will forward them to somebody else. But yeah, send them away because I'd love to try and figure out what's going on. The only the only one that I'll highlight again, you know, it was they did a great job with the time yesterday. It didn't we used to those would run like two and a half hours. They did a phenomenal job yesterday. I think it was like an hour and 10 minutes. So it's not too much to to watch in the background. The only one I'll highlight is the targeted edge upgrade or targeted update edge blocking. So we've talked many times about, you know, hey, they pulled this update edge and, you know, it's a bug. Most recently, it's been bugs and things like the VMX net driver that's, you know, specific to VMware. And, you know, a lot of customers like, well, I don't use VMware. Why is that blocking me from upgrading? So engineering is they hear you engineering and product management, they hear you, they're working on it. So that's one of the things that they that will be adding in the hopefully not too distant future of, you know, hey, here's, you know, there are there are potential issues if these don't affect you, you know, and we see that maybe these don't affect you. I don't know exactly what the workflow will look like there. But, you know, you can go ahead and upgrade anyways. So that that I think will be a big one. Let's see just looking at the chat here real quick. I would like to put open shift on my Raspberry Pi cluster running K zeros at the moment. Safis, that's a that's an interesting one. So open shift on ARM is in tech preview today with AWS and whatever their ARM variant is. I know we're working on expanding that the biggest issue with open shift on things like the Raspberry Pi is, you know, quite simply, you know, memory compute right now. The smallest footprint that we've crammed it down into, if you will, is you can at the command line force a single node open shift into I think it's two or three VCPUs or CPUs and 16 gigabytes of RAM. But you don't have really any room for anything else at that point. So it's possible to get down to that small footprint today. You know, I know that I'm sure engineering would love to get it smaller than that. But unfortunately, today that's where we're at. So I'm interested to see over the next, you know, six to nine months where we go with that, and especially getting it away from just data center ARM. Because like everybody else, I was actually this morning, while I was waiting for for the stream to happen, I was perusing Amazon looking to see, you know, how much would it be for me to get a nut so that I can just always have, you know, an Intel next unit computing, I just always have a single node cluster running in the background. So anyways, we all are if you if you're really crazy about the nuts, also look at the ACES because they have a small form PC as well. That's like it's a competitor to the nut. And hopefully with like the the pies as the pies are, you know, advancing, you know, right now it's I think the quad core and eight gigs of RAM. So soon we'll hopefully get to 16 gigs, and then we should be able to support, you know, some of these bigger, bigger platforms on top of them. Yeah, yeah, I see Christian using an Erichism there of putting 10 pounds of open shift into a five pound bag. And you just got to make sure that the, you know, it's not not they're small holes that way doesn't squish out the holes and you don't end up losing things all the four. Can you talk about introducing a service mesh like Istio to an already running production cluster? We understand the benefits, but is it worth the risk any advice? So in theory, you can deploy and to be clear, Andrew is not a service mesh expert. And I have not deployed it to a production cluster. I've deployed it to a couple of test clusters, stuff like that. But from my understanding, when you deploy it, it doesn't actually do anything until you configure, you know, the side cars to be attached or, you know, the other basically you have to say now go use it type of thing. So if you wanted to, you could effectively have so yes, if it's one one cluster that runs production and everything else, you can limit it to certain namespaces, certain projects, so that you can kind of do a phased or a test roll out first before adding it to the main application, the production application. So that would be my recommendation. Johnny Adele, I don't know if you have any other thoughts on that. Yeah, I think that's about right. We've used it before. And I've used it in different lives. And I think the networking is a bit complicated. So you have to make sure that, you know, you're getting what you want. Or you have a goal in mind before you actually opt in and solve them in your cluster, not only is it extra resources, but also makes debugging a bit more complicated. But we're working in OpenTip to make that much better, whether through the layer three layer two stack was the CNI plug in or upper layers. So it's it's worth it if you know what you're doing. And if you understand why you want like always. Yep, absolutely. And you made a good point about cost, right? Like it's a resource cost, right? It's free, but it's not, you know, so just be aware that it does take up some you know, scarce resources for most of us. And just for information, like in OpenTip, we did a lot of effort to actually move a POS container, which was not a sidecar, but was a like, I don't know how to call it, another container that persists networking. So we removed that now in OpenTip and cryo with spin NS. So that is not huge improvement, but we're optimizing at that level. So adding service mesh and sidecars to all your containers has to be thoughtful. But it's it's a it's definitely brings a lot of benefits and there's like observability and debugging and all these things. Yeah. Oh, Christian mentioned the POS container. The POS container, which I'll have to see if I can dig up the link for that, which describes the use for it. The POS containers are basically magic from my perspective. So Adele, I don't know if you had anything. Yeah, it's actually called MFRA container. POS container is the name that we like the slang name of it, but MFRA container is the more official one. It's still, I kind of think of it as a sidecar that helps, but it is a main building block for networking. It was, and now we have a workaround to prevent having it in a POS. So it's very clean. If you look at the processes running in your host, you just see the actual workload and nothing else. That's really cool. That would certainly make troubleshooting. Well, I guess in some cases easier. Some cases may be a little harder, but that's interesting. Yep. So yeah, I don't see so Zafi's I see talking about running memory compression. I have no idea how that would work with OpenShift and Kubernetes. Usually we say you don't want to have any sort of, you don't want to use features that can throw off the scheduler because the scheduler sees resources and it assumes that they're there and then it will schedule workload there. So if you're using compression or even swap space, that was the big thing a few years ago, swap space with Kubernetes. OpenShift virtualization, they went through a big debate on using the transparent page sharing feature for virtual machines and all that. And it was one of those like, we don't want to mess with the scheduler. But I can see certainly on a non-production cluster that that would be an option if you're just standing up something. Yeah. Okay. Q tries to do some system reserve and it used to consider that POS container and now it has less things to worry about. Okay. Andrew, there's one last question. It was from EV Lacan. It's about the Assisted Installer would like to configure RAID 1 for OS disks. Ignition supports RAID configuration from what I see for non-OS disks. How would you approach this configuration? Good eyes on that one, Johnny. I missed that one. Thank you. No problem. If I'm being honest, I don't know the answer to that. So what I think you can do, so Assisted Installer, it will generate a ISO and that ISO has, you know, you boot the node to the ISO and then it registers up with the Assisted Installer. I think you can interrupt that boot process and effectively pass it the same set of configuration and everything else that you would with a normal boot process and then eventually, you know, so use butane or something like that to generate an Ignition config and then interrupt at the boot prompt, append or tell it to look to that Ignition config remotely to do its initial setup and then add it there. But I don't know that for sure. So I'll reach out to the Assisted Installer team. We'll see if we can get an answer on that and kind of go from there. Assisted Installer starts on management. Now we're going to configure two production networks with bond plus static IP and install OpenShift using static IPs. So static IPs, if they're not already there, I believe that they're adding static IPs. As far as configuring bonds and stuff like that on those initial networks, I'll have to check with them on the same thing. So we'll take notes and I'll kind of collect these things. You can always reach out to me, andrew.solomonetredhat.com, if you want to follow up and send me those questions and then I'll just you know, forward them right over to that team and see if we can get answers. If not, then next week or as soon as I can get answers, we'll talk about those in the top of mind topics. So Adele, we're a little bit past where I normally start with the main topic of the day. I hope you don't mind. But yeah, Johnny, thank you for posting that in the chat there. These are really great questions. Please don't stop with the questions. So Adele, sandbox containers, cottage containers. This was something that I said at the beginning, right? It's really cool to me. It's really exciting to me. But when we first announced it internally, there was a bunch of questions and even a little bit of confusion around kind of what it is, how it works and where we should use it. So if you don't mind, I know, please. Yeah, so usually like I have an entire slide deck, but I don't think we have the time to go through it. So let's do it like, you know, ad-hoc-ly. OpenShift Sandbox containers is the product. Cata containers is the project. And what we do is, is basically wrap Cata containers and make it supported in Red Hat. We provide an operator with OpenShift Sandbox containers to do the heavy lifting and installation tasks, which I will show you later just like I have an entire graph that shows all the flow. But really it's not just like Cata containers, the only sandbox that we support with sandbox containers now that does not prevent us in the future from, for example, picking up another sandbox if we have enough use cases. But for now, it's like, I'm saying this because I usually see the mapping, the one-to-one mapping between Cata containers and sandbox containers, but it's really just an umbrella term like the sandbox containers for a broader set of use cases, which might or might not evolve in the future based on the use cases that we see and the questions that we get asked. So that's, in a nutshell, what Sandbox containers is. Cata containers, on the other hand, is the project. It's an OCI compliant runtime. It feels and looks and does everything like a normal container. In Capsule, it's like implements OCI, interacts with the CRI interface and cryo in OpenShift. Again, we're going to be seeing that more later. But yeah, it's a compliant runtime that you can run no additional effort, just run your workloads, mention the runtime class, and there you go, running in a sandbox environment. So I want to unpack a couple of things in which you just said. So one, if I'm understanding you correctly, sandbox containers is kind of an umbrella term we're using for sandboxing technologies. And there's only one right now, Cata, but there could be others potentially in the future. And I don't want to, I have no idea whether or not there will. I'm just saying that it sounds like that we're leaving that door open, if you will. Yeah. Now it's a one-to-one mapping, but that umbrella term that gives us the flexibility. There's a lot of sandboxing techniques there. And now we have a generic one that works for us and for our use cases in the future if we decide to go another way based on users' requests, then we can. But it's just not one exactly one-to-one mapping. Okay. And the other thing I wanted to kind of poke at is something that I didn't realize in that it is a full like runtime replacements. It's a full CRI. I always thought it was just kind of a layer on top. So that's really interesting. So it's actually what we have is we have the container runtime interface. That's the interface that higher layer runtimes like cryo or container D in other environments implement. That's the high level runtime. It gets the command from clients like crycuttle or cubelet in cube and open chip, and then basically does multiple things. It delegates the task of actually containerizing a process or sandboxing it to other lower layers runtimes and does other things like image management, life cycling, and all these things that are additional to just the containerization part which is delegated to something like RunC as a low layer runtime or Cata or any other lower runtime that is OCI compliant open container initiative compliant. Okay. Okay. Yeah. So that's a good point of I think I confused maybe a little the OCI and CRI too many acronyms there. TLA's. So yeah, Khalid highlights G visor is another form of isolation and that kind of brings me to the next question that I had in mind which was, you know, yes, there are other isolation techniques out there. You know, Amazon made a big splash with firecracker not so long ago, which I think a lot of people lumped these together, even though they're not quite the same. Internally, there was a lot of questions around, well, what's the difference between Cata and open chip virtualization, for example. So if you could elaborate on that. Yeah, sure. I think I think one step that I want to take before elaborating on that is, you know, I would like to always answer why would you use sandboxing, right? We there are different forms of sandboxing. The container world that we live in today is a form of sandboxing. Some people even call it software virtualization. It's basically you're taking a process and making it a super process by adding, you know, process namespace, PID namespaces, all network namespaces, all these things that basically take a container in a process and make it a container. Now, the problem or not the problem that the thing that is left out in that equation is that containers by default share the same kernel. I'm using the kernel features to do the namespacing but the isolation bit. However, in some scenarios, right, that is not enough. There's there's even like some people might argue that, you know, there's new user namespaces, which is a new nice feature coming in that allow me rootless access. I'm not root in the process. But still, there's a lot of limitations to that. It's an ongoing effort. It definitely reduces the risk of that container being escaped for that container. Like if you're root in that container, you escape the to the host, you root on the host. But if you're using user namespaces, the UID gets mapped. And then you're when you escape that container, you're not root there. But then you can't really do privileged things inside the container because the UID gets mapped. And there is a lot of dependencies to help you map that that UID to another non root user. And these dependencies actually require root access to do that work. So it's not 100% rootless. There's a lot of layers, like there's the demon or the socket or the runtime that needs to be rootless. And then the process that needs to be rootless. And then the shared kernel problem, like, I'm mapping to actual user IDs in the host. So I cannot really do two containers gives them root ID inside because it's still a map to the same user ID on the host. So again, there's a lot of things to make that mapping to mount file systems. There's a set of file systems that are now supported, like VFS or FuseFS. OverlayFS is not usually supported. C Group V2 is the only supported version of user namespaces. That's why you don't see it mainstream and Kubernetes or other environments because so far some environments are using C Group V1. And you see a push for people moving to C Group V2. So all of that kind of brings us to adding another tool in our toolbox, which is I want to add a virtualization layer. I want to add another sandboxing layer to my equation that I could use for multiple reasons. One reason could be I'm running untrusted code. Why am I doing that? Well, I kind of like, by default, if you're using a dependency from other code, you're kind of relying on something that you haven't fully qualified yourself. What is that thing with compromised NPM modules? Yeah, exactly. And then other people might run CI CD jobs. These jobs are coming from different places. They are hosting those in their environment. And you cannot 100% guarantee. But we have, of course, like we have that entire stack with image registries, private registries, vulnerability scanning, ACS and all these things. That's just an additional layer that we're adding to run this set of untrusted code. Or if you have trusted code that, for example, acts like a SAS. Imagine I have a service that takes PDFs and transforms them into markdown. That PDF is code. It can have code injected that you're running your application, which is a trusted image. But it takes input from a user, and that user could be malicious, or it could be an attacker. And then basically injects code because they know a lot about your environment. And there's a lot of ways to know about one's environment through exploration and so on. So, Adele, I want you to finish your thought. And then you've sparked a number of questions here. So we'll answer some of these questions. I just don't want you all in the chat to think that we're forgetting about you. Yeah. So basically, yeah, that's that. If you're even running a trusted code, and you want to really make sure that if you have sensitive information that this code is running, you might think about another layer of sandboxing. In that case, we're using like Catholic containers. And Catholic containers has a stack that makes use of hardware virtualization, makes use of kernel virtual machine, KVM. It's a Linux module, unlike other virtualization platforms that has, you know, L zero, let's say virtualization, this one is running on the OS. So any VM that you spawn, we use also the VMM, the virtual machine manager for us as QMU. And that basically just initiates vCPUs as threads in the kernels. So you're reusing kernel scheduling. And that's one benefit of OS virtualization, compared to L zero virtualization. But yeah, adding that to a cube native world, where you can just click a button and get a sandbox container for your untrusted workload for your CICD jobs, for your cloud native network functions is considered as a as one extra tool in your toolbox to protect yourself and your environment. Yeah. So, so JW, I'm going to ask your questions in in reverse order here. So first, would you choose Cota containers, sandbox containers over OpenShift virtualization? And does it also require bare metal? Yeah. So I'll answer the second question first, because it's easier. Yes, we currently require bare metal. And that's that's a requirement for our GA as well. For multiple reasons, because, you know, in the cloud, there is not really a clear, consistent way of having nested virtualization supported. But in the future, if there's a use case for it, we might think about like, clouds that enable and help us support nested virtualization there. Coming to the OpenShift virtual, yeah. Sorry. And just to because Boyle's Infosys asked if nesting is supported, and basically no, not yet. No, it is not. But you know, it's if for development reasons, or your testing stuff that we don't prevent you from doing that, but as formal support that will not be supported in the near term. Yeah. And I know for OpenShift virtualization, there's been a lot of questions around nested verts. And there's some very good and valid reasons why they don't support it. And they tell me that they will never support it. There's sometimes foot stomping when they when I ask them again and again and again. But so yeah, nested is hard. So I just want to be cognizant of that. But sorry to interrupt you, though. No, no, no worries. I wanted to answer your second question. And the second question you were asking about OpenShift virtualization and versus sandbox containers. So OpenShift virtualization for me kind of brings, maybe I can share my screen and then it's easier to show photos and pictures to explain stuff. Right. Can you see my screen? Okay, let me put it in present mode. It's a bit slow. I'm not trying an M1 like you, Andrew. Not yet, not yet. Yeah, not yet. So yeah, it's basically like this is this is kind of like confused person's flow to the comparison between OpenShift virtualization and sandbox or normal containers and sandbox containers. Usually, if you're running a normal container, workload, and we can call it trusted, so you know its source you have done your vulnerability scanning, you know that there are no zero days attacks. It's not a business critical application that has CVEs discovered, but you want to run it. Then you can go the normal default route, if you want, with running normal containers. That's high layer runtime as cryo low layer runtime as run seed. If you, on the other hand, want to run for whatever reason, like for the reasons that I mentioned before, you're running untrusted code for other people, you have a business critical application that the CVE was discovered for, but you have no fix yet for an easy way is to also put it in sandbox containers because then you figure out how to refactor your code and enters that CVE and figure out a fix. You can run that in a sandbox containers or if you're running a cloud native network function that is shared in a host where other cloud native network functions or other code from different vendors run on the same host, you could use OpenShift sandbox containers. On the other hand, OpenShift virtualization is this is what I call a general purpose use case where I have a VM, it's either running modern code or legacy code, but it's running in a VM environment and I want to move that as is. I don't have a container image for it. I haven't gotten into yet to my cloud native journey and I want to quickly catch up and run my stuff on containers and then basically refactor my application or just wanting to rehost. I would think of OpenShift virtualization as a good venue for that. It's a lift and shift basically. A VM here on the legacy environment can run as is on an OpenShift environment or a containerized environment versus sandbox containers, which is I have already gotten through my cloud native journey. I have containerized my app. I have an image and I just want that extra layer, extra tool in my tool set for running my untrusted applications for running code that has zero-day CVE that I didn't figure out how to fix yet and so on. Does this answer or at least remove some confusion? It does for me. Do you need just that security isolation aspect or do you need an entirely separate operating system to do all kinds of things inside of? Is it a pod that I want to treat as a pod that I want to use as a pod that just happens to need additional isolation or is it something more than that? I was going to move on and ask the next question. If you had something more to say, please don't let me interrupt you. No, it's fine. I said what I wanted. There's questions around that we can answer. Actually, I think the questions that are kind of popping up can be answered kind of through seeing how it works. So, you know, sees Santanapur. I'm really sorry about names. So, had asked and Christian kind of plus wondered of, you know, can we or how do we use a tecton task use a Cata container? And I think the answer to that is basically the runtime that it uses, whether it's the standard one or Cata is a part of the pod's definition itself. So, it doesn't matter if it's a human entering YAML or if it's tecton or if it's something else, Ansible is the word I was just searching for. That's more or less how you choose how it's going to run. Yeah. And I think tecton now supports it upstream an Argo for like that runtime extra runtime addition. We might think about also supporting if we see enough use cases and get ops and pipelines and an open chip, but it's really just, you know, a workload, your jobs are workloads, and the granular bit is the pod. If you switch the runtime class to Cata or sandbox containers, you're going to get an isolated container for your entire application. So, yeah, it's as easy as switching once you enable it. Okay. So, I'm actually sharing my screen here because I didn't want to forget that in case we didn't cover what we wanted to cover, but this is a one-time, one-stop shop. Let's call this where an index page that reference a lot of other places, including the open chip block. We're updating that, and it basically contains, will contain all the resources that we have, whether it's blog posts, videos that we publish on YouTube, demos, and so on. So, I'm going to share the link for, yeah, and under, yeah, cool. Yeah, you're one step ahead of me, Andrew. That's not me. That's Stephanie. Stephanie, okay, cool. Thank you, Stephanie. Okay. Right. So, I think there are two ways to see how it works. There's the, right, I had, I don't think I'm going to be able to cover that, but I usually have these kind of drawings that explains stuff end-to-end that we can explore, or we can explore from the CLI. So, you know, that will probably take about 10 minutes, which we all know what we have, but we can go through the CLI and do some magic. What do you think? I'm always a fan of seeing things in action, although I have to say that I always love your drawings. Anytime I see you post drawings and stuff like that, you always do a phenomenal job of explaining difficult concepts. But, yeah, I think, you know, proof is in the pudding, so to speak. So, if we can, let's use the CLI. Let's use the CLI. Okay. Cool. So, I have installed Cat9 in the environment. It's in a, or sandbox container. Sandbox containers is an operator. So, if I would basically look around, I should be finding, if my network allows it, sandbox containers. Doing it live is always, it's always interesting. Ask Johnny. I'm just going to show what got installed by OLM, just to give you an idea of what the operator does. And so, basically, that's the OLM stack. We have a subscription, and once you apply that subscription in an operator group, it kick-started the whole installation. It solves an operator. And as you see it here running, an install plan that is automatic by default, but you can also do it manual, and so on. So, this is the OLM stack. But then, really, what it created is, it created a machine config. And that machine config you're going to see here is to enable sandbox containers. It's running as an Arcus extension. And what that Arcus extension does, it just tells it to install, we have a mapping inside the machine compiler code to map the Arcus extension. Oops, yeah, here it is. To map the Arcus extension called sandbox containers to an RPM that contains all the things we want. So, if I actually would go into a node, and up here it did also create a machine config pool. Yeah, that actually got rolled to all the three nodes, so all the two workers that we have in that cluster. So, our two workers are now Kata, supporting Kata. And if I look at the node, and, oh, forget. So, I'm looking, changing that. Okay. So, if I RPM, I see here that RPM is basically what the extension maps to. And that RPM contains all the, let's see, if we can, contains all the things that we install, including the shim, including, yeah, I would have loved if it can show you the stuff here into, like, but this is the entire stack. So, we have a cryo, we have a Kata monitor, all these, our packages that got installed by just enabled the extension here. And we can see here cryo, this is a custom configuration by, we installed to allow us, you see here, that's one very important information that we're switching the runtime, we're adding a handler called kata. That handler is what tells cryo or the runtime what to do when it gets a pod that has a runtime class called kata. So, if I would map the real world as a client, then I would basically just a runtime class. Right. What I expect to see is I expect to see a runtime class called kata that should match that last word in the dot dot dot kata here as an exact match. And, oh, I gotta say that this is pushing all my like low level interesting things buttons. So, I'm enjoying seeing how the, how this was made. I would have loved like if we have more time. So, somebody did ask if they could get your Z shell theme for your CLI here. Yeah, I will share it. It's called QPS. I will share it maybe after like, but I will type it here in the, yeah. Cool. Yeah. And yeah, so basically, let me do something else. Thing is going to be removed by default. I like to remove the managed field and it's cleaner. So, basically, we do two things here. We define a runtime class. It's called kata. That's mapping to the runtime handler. And then we add things here. Oops. Okay. What? I did something strange. I see, I see Peters on. He's usually my, you know, the person who encounters all of the errors with Mac OS. Yeah, just some helper randomly. Apparently, I thought I needed help. I always do need help. Is that the Mac OS clippy? Yeah. Yeah, that's the Mac OS clippy. So, we have here the overhead bit and that basically tells us. So, we are not running. We were talking about the pause container and the overhead and introduces and that we're removing that. But, you know, with virtualization, that's the cost that we're getting. It's a trade-off, basically. It's a fixed overhead. We're basically telling the scheduler, you know what, a custom for the fixed amount of memory and CPU when you're scheduling workloads using that runtime, because otherwise, you're going to do wrong calculations. So, this is just to safeguard the scheduler to make correct calculations. These are values that we have studied and it's documented in our documentation page. If you go there, you're going to find what, what, how we exactly calculated in open to sandbox containers documentation. But that basically maps to that runtime here. So, once I create a pod using that runtime, it's going to use cut up. So, now let's, let's really create some, some, some stuff. Oh, sometimes I confuse Qtl with actual commands on Linux, LS and so on. Oops, yes, it's slow. All right, what do we have here? So, and let's go to pods as well. So, what I wanted to do is basically show you how, you know, it looks like if we deploy a host shared IPC shared pod, but let's, let's do something. Let's go one step back and apply that entire file and see f and then let's see pods. Okay. Oh, and I applied it in the wrong place, but we can reapply them. I applied them. Yeah. Yeah. Okay. So, one thing that you're going to notice off the bat is host networking does not really work with Scabat, which is a, you know, by design, in a sense, we don't want that to happen. You don't share like you're in a, you're in a VM, right? So, once you share the host network, you're not really sharing the host network, you're sharing the network of the VM. So, by design, that doesn't mean anything. And so, you're not going to find that working by default with, with Cata by design. Other things that we can explore now is, let's log into privileged and here, I forgot which host is this, but so, if you look here, this is all the devices in my host. If I'm running non-Cata privileged container and what this looks like is here. A non-privileged container. So, that's a privileged and what I basically say in a setting, I'm setting the security context to true, privileged equal. Now, what this does, it basically disable all my security defaults and accept process namespace sharing. So, I have access to my host devices, my mounts, everything, as you can see here. So, I can basically do anything, I can be rude on the host. If I actually do the other way around and... So, I'm going to take an opportunity to interrupt you real quick. One, to say that my dog is finally quieted down. She was very unhappy with what appears to be the pest control guy who is on our porch. So, apologies for the barking. But two, since you're talking about, you know, what's going on inside of there. Earlier, and my apologies, I don't remember who asked, but somebody was asking about, you know, can you do rootless pod man inside of a Cata container? And I think that we're flirting with the answer to that right here. It's not really like... It touches it in a sense. So, rootless here in the Cata world means that I am rude in a completely different kernel. So, you're actually rootless on the host, but you're rude on a guest kernel that has nothing to do with the underlying kernel. So, if that answers that, rootless usually means I'm using namespace, user namespaces, but sharing the same kernel, and I'm doing UID mappings between the container and the host. Podman is not... So far, it's not running through Kube, right? It is considered a mapping to CryoCuttle or the Kubelet. Podman is on the same layer. It calls out to low-layer runtimes, and it does a lot of the work that Cryo does. So, it's in a parallel line as Cryo. So, it does run with Rails, and Cata runs with Rails, but it's not supported. So, Scott McCarty probably have touched on that before. Yeah, I don't think it is supported in Rails now, but it works with Rails, and it works with Podman. So, you can spawn Cata containers with Podman. So, that does... In my mind, it opens up an interesting question. Like, there's the whole concept of Docker and Docker. Can we do Podman and Kubernetes or Podman and OpenShift with something like this? I think we can, but we have to do another wrapper to allow the Kubelet to talk to Podman instead of Cryo, and that would require a lot of effort, I would say. So, I'm going to say no. No, definitely no at the moment, but maybe in the future, who knows. All right, I'll stop interrupting you now. And we can't go a few minutes over if you have a few minutes, Adele? Yeah, I can. So, what we can do, maybe let me try to see if I can show you how. So, what it looks like on a container, I'm just trying to find out which one it is here. Let's see cry couple pods, right? And then, basically, our test privileged Cata. So, that should be... I want to show you how it looks like from the host, and then, basically, what we can continue in a part two sometime, if you have time. So, basically, that's a container that is running. And second, we need that. So, let's try to find the pod ID here that maps to here, Cata, that one, cry color, inspect, circle. Okay. Okay, that's a lot of things. Grip send, we hide this, send box. You're going to see here the sandbox ID. Now, let's skim through the processes, or actually, let's grip through here, that guy. And you can see here two things. You're going to see that sandbox ID corresponding to the container that I'm running in Cata, running as a chemo process, or a chemo process. That's our virtual machine manager. And that chemo process corresponds to my container. So, if I would, it was called test Cata, what was it called? So, that corresponds to that guy here, test privilege Cata. Okay. So, this is here, test privilege Cata. And one thing that we can see the difference is if we look at the process and CPU info, we can find here, for example, that I have two cores, if I exact. So, now what we're trying to do is we're trying to show the difference between Cata container and CPU time. So, you see it, this is a different kernel. One thing that we can also do is, it uses, by the way, the same kernel version. But if we look at, what else can we look at, yeah, for example, the kernel parameters for that, that could be something we can explore, right? It's a different command line for the kernel. So, different kernel parameters. So, it's definitely a separate kernel. However, we're still using our costs. So, if I would, right, if I look here, I'm going to look and find the same kernel version. So, that's 4.18. And that's here exactly the same. So, is there any possibility? Like, would it be considered a, I don't know if it would be considered a feature or not to have a different kernel, maybe? A different kernel. Well, some people consider it a feature, some people consider it a curse. Yeah, a feature in the sense that you can actually play around with the CTL modules and so on, which we can allow you to do. But the cursed part is, we have done a lot of effort to actually iron out and harden the Red Hat Core West for you, right? So, we're trying to take that, you know, give you another option for isolating a workload, but still keeping that level of security and immutability and all the kind of see nice things that we give you by default with ARCOS. And immutability might not be, you know, that most secure, but it's definitely one step. Like, we're always talking about layers here and immutability and what ARCOS does in terms of SE Linux and all these things gives you a default saying, even if you're in a VM, so we're giving you like isolation to the absolute top. Well, yeah, that said, like, if there is a very convincing use case that someone has, I'm open and happy to hear about. Right? So, basically to summarize what I just showed you, I probably wanted to show you also other things like we have a list of like five, six containers. What I was trying to do is basically show you what, enabling each of the non-secure modes in a bad pod, like whether it's host ID, a PID or host network and so on means, would also mean. So, for example, I can give you one example, but before that, we showed that, like in a sandbox containers, you have a separate instance of the kernel, it's a guest kernel. It's a different CPU characteristics. Whatever is running there is completely isolated, you're not sharing the kernel. And all the process, I think this was a privileged container. Let's continue and see what happens here. If you remember what I showed you in the other privileged container, so let me exit that one for a sec on the node, right? And go to that other privileged container quickly. And JW asks kind of an extension, I think of what I mentioned before, which is, can you build and load extra kernel modules inside of the Cata container? Yeah, I think you can. You can, but we don't support that at the moment. So, for the same reason I've mentioned, because that exporting, if we allow for it, like we can think about certain kernel modules or kernel extensions to enable, but we can't allow a broad view, depending on what we want to do. So, I think it's a good idea to do that. So, I think it's a good idea to do that. So, I think it's a good idea to do that. So, I think it's a good idea to do that. To enable, but we can't allow a broad view, depending on the use case. So, that's also for the same reason I just mentioned before. So, I wanted just to show you the test host, or yeah, test privileged. That's privileged, but not Cata. I think I have like a visceral cringe to privileged containers after having worked with OpenShift for so long. Yeah, I see. So, this is privileged normal container. You can see I get access to all the host devices. I'm basically absolute root. And here, I don't, I get access to the VM, which is the separate guest kernel. So, that's one example. Yeah, do we have time for another example I can also show? Yeah, I think another six minutes or so sounds good. If there's any questions from the audience, please don't hesitate to send those in. We'll address those as soon as we can. Okay. So, we looked at the privilege. We can look at host PID, right? So, let's exact here into host PID Cata. That means that I'm actually sharing the host PID of the host, or else would then do. So, I see here three processes. That's probably, I don't think I'm even going to, yeah, it's just my current process and the master process, the shell that I'm working with. So, it's definitely, yeah, definitely an isolated instance, just like you would expect with a container. Yeah. And if I look at a normal, of course, this whole assumption is I'm running, if you see as an admin, I can do that. Usually an open shift, I'm not an admin. I don't need people admin permissions. And we have, just for your information, what is the namespace that we're in, we don't usually allow people to run as root. We usually have, you know, UID range that we tell you to start from and there's an SAC that actually enforces that through mutating functionality. And you see your SA Linux codes. And so, we basically apply all these defaults for you, but I'm admin and I can do whatever I want and I can show you, like this is to show you that although you have that layers for these set of use cases, Fata might be helpful in that scenario. All right. So, back to the, we were in the PID thing. So, PC Lauterbach asks, so the kernel in the sandbox container can be dramatically different than the ARCOS, the CoreOS kernel, like a REL7 sandbox on a REL8 CoreOS host. Well, the sandbox can be dramatically different than an ARCOS. No, for open shift sandbox containers, it's the same kernel. It's a different instance of the kernel. So, whatever I see in that guest kernel, I don't see in the host. That's the isolation that we're talking about. That's the virtualization layer. Over the version of the kernel is the same. I'm using the ARCOS here. I'm using ARCOS here. I'm using Linux 4.8 and Linux 4.8 here. So, it's the same image that I use for the VM as I'm using on the host. Okay. Does that make sense? I wonder if that, is that a, like technically, yes, you could use a different kernel, but from a supportability and security standpoint and other reasons, we Red Hat say, no, you must use the same version. Yep. That's a technically speaking catech containers. The project allows that. We in the product do not for now. Got it. Yeah. So, looking at the proc, because everything in Linux is a file. So, looking at the processes here, I have all the host processes that I can see with host PID sharing here. That's the disable body folder. Even if it wasn't able, I would see that the processes is running on the VM. So, I guess we could go ahead and show all the host namespaces, but the outcome will be the same. Whatever you're going to see, you're going to see it in isolated kernel versus sharing the kernel was the namespace. So, I don't think it would make, like we can go ahead and show the rest, but it's basically adding, let me open the slide here just to show you what I'm doing in a slide. Yeah. It's basically enabling all these things and running them in a catech container. And we kind of nullify their effect or their negative effects. And as I said, in OpenShift, usually we have, we have admission controllers, we have SECs and all these things that ideally per project prevents you from actually configuring or misconfiguring your pods, but it never hurts to be safe rather than sorry and add an addition layer of isolation. And it actually makes sense for multiple use cases, as mentioned before. So, yeah, what we're trying to do is trying to mix, you know, merge Goku and Kakarotto into, like with OpenShift sandbox containers and OpenShift into one, uh, mixing all the pieces here, whether it's SEL, Linux, user namespaces, SIGComp and adding that addition layer of isolation with sandboxing and sandbox containers gives you, you know, I think puts you in a very good spot from a security standpoint. Yeah. Yeah, there's several requests for your slides, Adele. So, I don't know whether or not you're able to share them. If we can, we'll get those posted and I'll link from the blog post. Sure. So, we are out of time. So, we went about 15 minutes over. I hope not. I think you were suggesting a two-hour session or a regular two-hour block for the Ask an OpenShift admin. So, we may consider extending it, you know, now that we don't have, there used to be an OpenShift Commons briefing that was right at the end of or right up against the back of our show. So, we might consider extending it. I know sometimes it's hard, you know, folks like Adele, I know he's very busy. So, sometimes it's harder to get guests and stuff like that. But we'll see what we can do there because the last several, we've run at least five or ten minutes over every single time. Yeah, and we skipped through many, many topics, but yeah, I think we'll share it. Yeah, it's always fun to have you, to talk to Adele, to have you on, you know, I know you're very passionate and very knowledgeable on the subject. So, it's always fun and exciting. It was a pleasure talking to you, Andrew. Yeah, thank you. So, for our audience, please don't hesitate to reach out. You can contact me at andrew.sullivan at redhat.com or practical Andrew on Twitter. So, just like you've seen me poking into the chat on Twitch there, which should get rebroadcast across all the other platforms. You can contact me at any time with any questions that you have. We're happy to track those down. I'll throw Johnny under the bus as well. And I want to close out by saying that we, even though we're coming up on the end of the year, I think we've got two episodes after this for the rest of 2021. In the new year, we have a lot of stuff planned, a lot of stuff going on. I'm trying to get some more, kind of to our hope nine's points, kind of to today's topic, trying to get some more complex things in here. So, we might end up having to have some more extended episodes. But please, whatever platform you're watching us on, hit the subscribe button or whatever the equivalent is so that way you can know when we do have those topics on. All of that being said, thank you so much Adele. We really appreciate you coming on today talking about this. This is something that's exciting and interesting to me. So, I learned a few things and I've heard you talk about this over the last few months and bits and pieces. So, I appreciate that. And to our audience, thank you all, really appreciate your activity, all the questions today. I'll review all of those. We'll make sure that we have all of those answered in the blog post as well. So, in the meantime, have a great week, everybody, great weekend, everybody. And Johnny, I'm going to put you on the spot and give you the last word. Adele, that was so awesome, man. Thank you so much. Like you did an outstanding job, slides and demos were like just crazy great. So, thank you very much for showing up and really putting a show on for everybody. No, thank you. It was really awesome and fun having like talking to you guys. I think the questions were great. I would be happy to answer questions of line as well and happy to be here another time. So, yeah, it's great to be here and then talk to you guys.