 Well, hello everybody. Welcome to KBE Insider and with our fancy, fancy new graphic. It's always nice to have a new intro clip just to make our lives more entertaining. We also got some cool new swag. I've got the sweatshirt going so we're always happy about swag. So let's see, I have a couple of new things today. So first and foremost, I'd like to introduce my co-host for today, who is Josh Wood. And so we have a rule on the co-hosts along with me is that they must be named Josh. So even if it's somebody else, we're just going to call them Josh. Anyway, that way it's easier for me. Yeah, as I said, when we were leading up to this land, any Josh will do really. Exactly, exactly. So a little bit about the show. Oh, actually, so Josh, do you want to introduce yourself real quick and then I'll talk about the show a little bit. Sure. Hi, everybody. I'm Josh Wood. I'm a developer advocate at Red Hat, principally focusing on OpenShift and especially operators as an extension to the Kubernetes API and OpenShift in a way of delivering features that we add atop the Kubernetes core in OpenShift. Awesome, cool. So the reason we do this show is to try to give you an inside look into what's going on in Kubernetes land. And in particular, like the idea being that if you kind of talk to, let's say the lead engineers or the engineers who are actually doing the work that you'll get a much better insight of where Kubernetes is going in the future rather than the hopes and prayers of a product manager or a press release. So that way, because it's open source, there is a lot of contribution that's done based on the engineer's recommendations. And sometimes that's well reflected in the press releases and sometimes it's not. So we think it's a good idea to talk to the people who are actually doing the work and we hope you do too. And so feel free to put questions in the chat and if you have any comments or whatever, we are always happy to have more engagement as it were. So definitely let us know. But today's show, we are going to focus a little bit on virtualization. And so we have two guests. And the reason is, is because virtualization seems like something that isn't normally a part of Kubernetes being a container orchestration platform. And I think what we're starting to see is that it's more than that. It's an orchestration platform for lots and lots of different things. If you go back to our first episode, you can see us talking to Clayton Coleman about using it in general as a control plane. And it's kind of actually come up a number of times throughout our interviews. So we think there's a lot more going on there. So we invited two guests. One is David Vossel, I hope I said it right, who's from the Qvert team. And the other one is Bandan Das, who works on KVM or the kernel virtualization manager. Is that right? I was like, I don't remember what that virtual machine. Yeah. I was like, I can't remember what the expansion is. So and hopefully we can talk a little bit about virtualization and how it relates to Kubernetes. So let's start with David. Do you want to introduce yourself? I always say it's very difficult to remember or find out or discover or have any consistency to what titles and roles are within Red Hat. So I find it's much safer to let people introduce themselves. Yeah, sure. I'm David Vossel. I'm an engineer at Red Hat. I'm contributing to the Qvert open source project. And it really is kind of involved into an ecosystem. So I'm kind of contributing to the Qvert ecosystem at this point. I was involved with the Qvert project early on, got the opportunity to design a lot of the way that operates today. So I'm coming at this from a Qvert perspective. Gotcha. Cool. All right, Bandan, how about you? Hi, my name is Bandan. I work in virtualization. So as Langdon said, I work mostly on KVM, which is the kernel based virtualization module in the Linux kernel. And that's an ecosystem on the virtualization side as well. So it's QMU, KVM, and LiveVert, which thinks that usually my team takes care of both upstream and downstream. And then, yeah, I've been working. So I was always interested in systems. And I think the best way to deal with different kinds of systems issues is to work on virtualization because it has all things that you can think of, be it devices, be it CPU, be it interrupts. And so that got me stuck with virtualization for a long time. It has always kept me busy. And then besides my work at Red Hat, I also teach at BU, which I skipped this semester, but hopefully next semester I'm going to teach again. And I also work with Red Hat Research. We have a project going on on fuzzing QMU with Boston University that's going well. So, yeah. Cool. So one of the things we always like to run in this show is, sorry, Josh, did you have a question? No, I didn't actually. Go ahead. Okay, sorry. I was just going to say, we always like to ask kind of what brought you into the open source world. And so, David, I was wondering if maybe you could tell us what, like, how did you end up here? Yeah. So I've been on the periphery of open source for a really long time, like back in the 90s. So my dad's office would have old computers. And I would inherit these old computers. And I would need something to run on them. So I would use a Linux. That's all I could find. And at the time, I think you could find Linux in office stores, like the boxes or whatever. Yeah. So I might have gotten a few of those, like Mandrake Linux or something like that. But I think I downloaded some. Yeah, it did. Yeah. So I would download it. It would probably take me like a week to download an ISO. But I would download it and run it on these old computers. And I had no idea what I was doing. But that was fun. And then I never really contributed to open source as a result of that. But I was on the periphery. And after university, so I was using Linux throughout my studies in computer science, things like that. I needed a job. And I had lots of opportunities. But one opportunity kind of stood out. And it was a company called Digium that maintained the asterisk project, which is a telephony project. Oh, yeah. Yeah, telephony. I can't think of the what do they actually call those boxes? The thing in your office. My phone runs an asterisk. Oh, really? Yeah. Yeah. So I did that. And I got to contribute to asterisk open source projects and all, you know, or Linux stuff. And it kind of just took off from there. And I've been lucky to contribute to open source majority of my career at this point. I still remember I had a project a million years ago to basically, we're doing an asterisk implementation to lead us part of a backing system. And discovering that you couldn't actually run it in EC2, you know, for a long, long time. So that was a, that was a challenge. But it was actually related to virtualization, which is kind of was timing. Yeah. Yeah, I couldn't quite remember why exactly. Yeah. But that was an interesting experience. Yeah, the I still remember also, you know, speaking of old Linux distros, a stack of three and a half inch floppies that was literally this high, you know, slackware and installing it on a computer and being actually concerned that the monitor was going to late on fire. Right. We get the sync rate wrong and actually destroy the hardware. Yeah, like a similar memory to that I have is watching folks download those stacks of floppy disks to acquire Linux on what I think were like Macintosh, like LCs in the computer lab at the time, like loaded disk after disk after disk to copy them off, take them back to the rooms and do that when it can stop. Nice. All right, so Bondon, on to you. What got you into kind of the open source world? Yeah, I mean, isn't it amazing all open source stories are my story, I can completely correlate with David because I can kind of feel like, okay, that's my story as well, except that it started like a little bit later. I don't think I had access to, you know, I just I didn't even know what open source is in the 90s. And I think the first time during my undergrad days is when, you know, when I had to make a choice between either, you know, buying software that was available from a shady place for a much reduced price compared to, you know, going for a version that was available to download online, even though the speeds were kind of that really sucked, is when I started kind of realizing the importance of open source without realizing that realizing that that's free or open source software. I remember the first time I was kind of browsing in this in this marketplace back in India, where, you know, you could get software for a much reduced price. And I saw CDs for Red Hat Linux 9, not Red Hat Enterprise Linux 9, but Red Hat Linux 9 and I bought them. I didn't know that I could download them for free, but I just said I have to try these out. So that was my first time I actually kind of tried something which was, you know, open source and I installed them on my desktop only to find that I could either run my network card or my sound card, I cannot run my network at the same time. And that was how I got interested. I was able to look at the sources for Red Hat Linux 9. They were in the CDs and I did not understand a bit, but it was really, I felt empowered to know that, oh, so this is the code that kind of runs on the system, that's really cool. But I think the real contribution I made was, believe it or not, it was on Minix. When I was doing my masters, we had a project on Minix 3 and I found a bug in the network stack in Minix 3. And so I sent an email to, you know, Andy Tannenbaum with a patch saying that, okay, this is the change that is required. And he acknowledged it. I mean, I don't know if it got into a release. I didn't know that. There was no easy way to find that out. But the very fact that you could find an issue and can fix it and get it accepted, I think that was what got me kind of interested in the whole process. And the rest is history for me. I had an internship at Red Hat when I was able to work on kernel build tools followed by working in a hardware company in Massachusetts where we were working on device driver hardening. So that was my real kind of first experience working on contributing to open source software Linux based. So I was able to submit patches to drivers and later on in virtualization. So yeah, here I am. Nice. That's pretty cool. I like both those stories. I agree, though, Bond and a lot of them are, you know, I think a lot of the origin stories are often the same, you know, or the, at the very least, it's the, oh, you know, I have a scratch and I hitched it, right, is for sure. Like, and then like, like in listening to Ben and tell that story, I was, I was thinking a lot of the reason why I know the command line pretty well is because I have this janky laptop for a long time. I couldn't figure video drivers, right? So that'll do it. Yeah. All right, so talking a little bit more about what is going on in the kind of, you know, Kubernetes world. Moving on. So why are you or why do you think that, you know, kind of virtualization is so important in a containerized world? And feel free whoever wants to enter that first? Yeah, I'll jump in. Containers have to run on something. So what are they going to run on? I mean, of course, you can run on bare metal, but I think that virtualization has it's going to maintain itself as kind of the underlying substrate that containers and applications run on because they're easy to manage. You update them easy. They're kind of more flexible than bare metal in that, in that aspect. And they can do kind of interesting things. So you can like overcommit virtual machines on hardware where you can't really overcommit bare metal and metal. So as being this kind of substrate applications live on, I think that it's going to have a really long lived life within kind of this new ecosystem that's emerging with containers and even what do we call it serverless and things like that. Those have to run somewhere as well. So where are they going to run? Oh, I thought it was just being the cloud. Yeah, exactly. Well, it's going to move to the background. I think that's what we're seeing with virtual machines. They're moving to the background where maybe right as virtual machines originally started, even when infrastructure as a service originally started like DC2. We saw people packaging up their applications inside of virtual machines. So we saw Netflix do that. They used like these image creators to get all their applications into a virtual machine and they just kind of scale out that way. And then we saw that transition to painters. But that, where was I going with that? I don't think that that use case is going to make a whole lot more sense for very long. We are seeing a move away from the applications being packaged inside of the virtual machines. But still they're being run on top of the virtual machines just in a different way. So it's just kind of a reshuffling. So David, it's interesting. I hope you use the word flexible in kind of the center of your answer there. VM is more flexible than real hardware. In an odd way, and I wonder what your thoughts on this would be. In an odd way, I almost think about one of the, I agree with that flexibility thing, but then on the other side, if I'm a provider, I often think and answer that question by thinking about how VMs are actually a little bit less flexible for the user, meaning their security parameters and resource allocations and what rights I'm granting on that VM are more configurable. They're more flexible for me if I'm the provider, but I can lock them down in the form I deliver them to users in. I mean, I wonder in a world where we're mostly renting computers from a cloud provider to run our stuff. I mean, do you think that's an important dimension of what the VM surface needs to provide? Is that ability to lock VMs down, lock down their resource allocations, and talk about overcommitment kind of touches on that? Certainly, yeah. So isolation is one of those features that virtual machines provide that containers also provide, but it's a stronger form of isolation. And we're probably going to talk about Qvert a little bit, but I'll just say in Qvert, we actually have two layers. There's really multiple layers. There's lots of dimensions here. One of those is the hypervisor itself, which is a really strong form of isolation that we can give people. And then within that, we're also running the hypervisor in namespaces, so kernel namespaces. And then you also have SeLinux, or I forget the Ubuntu equivalents of that. So it's like isolation and security through depth that you're providing when you're using shared resources. And I think the hypervisor is an important part of that, especially when you're renting out shared resources like that, where multiple companies are running on the same hardware. You have no idea what you're running next to. Yeah. Is it App Armor? Is that the... Yes, App Armor. Yeah. So I was kind of curious about Bondin's... Sorry, our video seemed to have switched on me. I was kind of curious about Bondin, where are you seeing the growth in KVM as a reaction to Kubernetes and containerization? And that kind of is like, what seems to be more of the focus? So yeah, in reaction to containers, that's a good question. So KVM is... Is it a pretty low level in the stack? So that kind of makes it immune to... I mean, the good thing is we don't have to think about these things because we know that the API on top of us is going to take care of it. But that said, when you talk to me about virtualization, I don't think about only KVM. So I also think about QMU and LibVirt. And so with respect to how KVM is going to work with containers, I think there's a lot of kind of emphasis that's been put now that I don't think was in the past is, how can we expose LibVirt APIs that can help with these kind of things? And that was something I don't think used to happen previously. So that's one aspect. When it comes to the real kind of feature that KVM is exposing, which is the hardware isolation that David talked about, the mechanism itself is pretty transparent to containers because they don't really have to talk to KVM directly. And that kind of makes it very, very simple for us, not to worry about those kind of things. Yeah, I think that pretty much sums it up. So just kind of for the audience, what is the difference between QMU and KVM and LibVirt? Yeah, that's my question was going to be along very similar lines. It's like, could you kind of sketch for us just to give us a foundation for the folks who are listening? Like, where's the line where KVM stops and LibVirt starts being the sort of user interface to managing those features? How do they fit together? Likewise, I mean, I like with PMO as well. So, yeah, I kind of think about, so if you, does this really popular device drivers Linux device drivers book, I don't know if they still publish or release new versions, but they talk about mechanisms and policy. So they say that the kernel implements mechanisms and the user space kind of, you know, uses those mechanisms to implement policy. And I kind of think of KVM and KVM in those respects. So KVM is the code in the kernel that enables hardware virtualization. So whatever it is x86 on and a few other architectures that the Linux kernel supports, KVM is going to enable hardware virtualization and give us an interface to be able to use them from user space. And that's where KVM comes into the picture. QMU is the user space. QMU talks to the kernel via a device file, throw a set of iOctos to request services from KVM, which is in this case, you know, all things virtualization, all things hardware virtualization that is. A QMU also does a major part of QMU actually is also emulated devices. So even though you can pass through devices to the guest, that's not really the norm. I mean, a lot of devices that the guests use are actually just emulated devices that QMU implements. Or for that matter, in some clouds, it could be something some other user space, which might not be QMU. So that's another good thing about this mechanism policy separation that you don't have to use QMU with KVM. You could use your own user space that talks to KVM and gets those services. And then coming to LiveVirt is where we are saying that applications will find it really cumbersome to talk to QMU directly. So let's build an API on top of it, which will be easier for applications to use. And that's what LiveVirt does basically. It's a layer on top of QMU that helps applications to talk to QMU, which would have an easier, if you were a user, then you probably would have an easier for you to talk to QMU directly, like a human user. But for an application, it's LiveVirt. That's how I make the distinction between the three. And actually, Mike, go ahead, sorry. Just a quick clarifying point on that, because you mentioned a couple of times, wherever there is hardware virtualization support, like with BTX on Intel, is there or was there historically any support in KVM for virtualization on architectures without that hardware support? I remember some of the techniques in Zen for plain old X86 and didn't have BTX. And they're elaborate and like you wouldn't want to have to use them, but they're fascinating sort of from an implementation point of view. Was there ever any non-hardware virtualization that KVM could do? That's a good question. So when it comes to the real hardware extensions that implement virtualization, no, the answer is no. That said, KVM is really a hybrid model where it takes ideas from Zen as well. So KVM actually came into being after hardware virtualizations were introduced by Intel and AMD. So that's the first part. That said, we do have a lot of para-virtualized interfaces, for example, which is obviously brought from Zen. So for example, timekeeping is one of those things. So one of the things KVM does is it has this shared space between the guest and KVM, so which we are able to use timekeeping features. The guest is able to kind of get accurate timing without having to perform a severe penalty that you would have if you had an emulator device. That said, the other thing is in the initial days of virtualization, hardware extensions had a lot of limitations and what kind of instructions they can execute in what processor mode. So for example, in real mode, processor was not able to execute certain instructions when you were running in a guest. So KVM could emulate them. So it's a mix of all these things together. And my answer to you would be, it's yes and no both. It's no because, yeah, we had never had KVM when hardware virtualization did not exist. But now, even though there is hardware virtualization, but KVM also does a little bit of emulation that you would expect from a traditional kind of old hypervisor like Zen. Right on. That's crazy. All right, so kind of moving on into kind of talking about Qvert. So like where does that relationship live? Does Qvert use QM? Does it use KVM? Does it use liver? Does it use all the above? Where does Qvert and kind of, one of those tools, kind of where does that line drawn? And does that line stay still? Does it move around? Does it matter? Yeah. So Qvert is really just a fancy wrapper around all of this. So we are using liver, we're using QMU, we're using KVM. We didn't necessarily need to use liver. We chose to because it gave us some, it allows us to rapidly iterate on this stuff quickly because liver gives us a really nice interface like a user interface, CLI interface, but an interface to manage the life cycle of virtual machines that we've had to create ourselves. But at the end of the day, we're using QMU KVM really. And we are a wrapper around this. We're launching what's essentially just a QMU KVM process in a container, well in a pod, in a Kubernetes pod. And Qvert itself is just a set of controllers to manage the life cycle of that pod and also to provide that pod the kinds of resources, cluster resources that it needs. And so if we're talking about like CPU and memory, we're reusing the Kubernetes scheduler, provide those resources to the pod, and then we have the kind of small glue layer that's passing that on to the KVM process. Same thing with storage and network. We're using the regular pod network, getting an IP address, and we're have the glue in that pod to give that IP to the virtual machine. Same thing with persistent storage. You're giving it a persistent store PVC to that pod. You're attaching a PVC to the pod, then all of a sudden you have a boot image for your virtual machine. So Qvert is just layers and controllers around this kind of underlying technology. And actually, I mean, so for me, that brings up a big question, which is like, when I think about a container or by extension of pod, because a pod in a lot of ways acts a lot like a container, a virtualized machine expects a lot of things. It expects all the ports to work. It expects certain kinds of stores. It expects mostly I guess I'm thinking about networking. But so how does Qvert or wherever, like a certain tool chain, how does it kind of tell the virtualized OS that, oh, no, you're not really, you're not running in a pod. You're running just like you normally would, because I presume you kind of masquerade to it so that it doesn't, you know, you don't have to kind of change the inside of the virtualized machine? I kind of get what you're getting at. So when we launched the KVM virtual machine, the environment that it's in looks very natural to it. And I have my QMU is going to see the KVM device. It's going to see things like the IP address and the and persistent storage all kind of just, they just look like mounts or interfaces within that environment. It looks like you're just running on normal, like a normal non containerized environment. So we've done a lot to kind of make it appear that way. And that's where I talk about the glue. So we actually reach into that pod with a privilege daemon set to kind of set things up for us in a way that mimics what you would expect. So when we actually launched the virtual machine using, well, ultimately livevert to call QMU KVM, it just looks normal to it. It doesn't think anything different. So we've recreated that environment for the virtual machine. That was kind of what I was expecting, I guess. So are there trade off or are there negatives to that? Part of why, like you said, I go back to networking a lot, but part of why that networking works where it does is to kind of minimize the resource consumption and provide some level of different kinds of security. And so are there negative sides to kind of pretending that the virtualization or the virtualized OS is running in a normal, you know, kind of, you know, virtualization environment? Or are you dealing with those negatives on the outside? So kind of like, you know, yeah, you set it all up. So it feels like it's a normal environment. But when it tries to get out again or whatever, that's where you bring back, you know, that density or security or whatever those trade-offs might be. Let's say there's necessarily negatives. It's just, it's different. So like if you had complete control over a bare mill machine, you were running livevert on it, you could create your own bridge interface and your own network and do whatever you want. Here we are, you could say we're limited by the types of networks that we provide to the virtual machine, but really we're not. So we have, it's just a different layer. So at the cluster layer, we can create multiple interfaces and pass multiple interfaces to the pods today using multis. And we can provide like SRR, ROV devices and things like that. So we can do a lot of the same things that we could do if we were in complete control over a bare mill machine that we're doing at the cluster level using Kubernetes-like APIs to kind of manipulate those sorts of things and assign them to the virtual machine pods. So it's really just a shift of how we look at these things. It's it's still flexible, but just in a different way. So we still have that kind of control, but it's different looking. I got you. I got you. So David, I'm curious kind of at an implementation detail level to maybe help me understand that interface and where KVM infrastructure ends and Kuvert wrapper begins, right? Like where is that line and what's that interface look like? I'm sorry. I kind of lost my train of thought there. You talked about a set of custom controllers managing resources for these pods that we're going to launch VMs into. Do those custom controllers manage a set of CRDs? And is there a custom set of API endpoints represented in those CRDs or managing, communicating with and monitoring the set of VMs that we're running in that cluster? Like what's the implementation of that actually look like? Yeah, exactly. So we have our own API. And if you look at our API, you wouldn't know that it's necessarily KVM or KMU or libvert behind the scenes. We have maybe if you're really knowledgeable of what's going on behind the scenes, you might see certain values that make sense only for KVM and things like that. But our API is unique for Kuvert. You have a virtual machine API that describes how to create a virtual machine and things like that. And we have two layers of controllers. So we have a control plane that's living at kind of the cluster level. And it's managing the lifecycle of virtual machines at the cluster level. So we post your virtual machine to Kubernetes cluster. These cluster level controllers going to say, hey, I see a new virtual machine. I'm going to create a pod for it to live in. That pod's going to get scheduled onto a node somewhere with the correct resources assigned to it. So it's right now CPU and memory that you've requested there. And then we have a daemon set that's going to live on every single one of our nodes. It's going to see when that virtual machine's pod gets scheduled there. It's going to see that pod come up. It's going to reach into that pod. And it's going to manipulate some things. So it's going to be the thing that's helping facilitate network and some of our other things around persistent storage and stuff like that. And then within that pod itself, we have a really small daemon set that's just kind of gluing all those things together. So the things that the privileged daemon set went and set up for us, our shim process inside the pod itself is going to see those things and construct the libvert domain XML. And ultimately pass that libvert domain XML to libvert and actually start CUMI process. And then the result is because of this whole chain of operations and controllers, you have a virtual machine that starts in a pod. It really, it kind of haves like a pod in some ways too. Because the virtual machine can talk to all their pods and all their virtual machines. And you kind of just have a virtual machine all of a sudden appearing within this cluster and it looks like just a native application within there. So hopefully that answers your question. Yeah, there's a lot involved in multiple layers to get to the point of actually starting the virtual machine. And Kubernetes is kind of coordinating the lifecycle and kind of the management and resource placement of all that. Right on. So one quick follow-up to that. And this is also sort of for you as well, Bundin. And it is, if I'm working with CUMBERT, am I working with nested virtualization a lot because somebody's running the cluster on a VM in the first place and then trying to do CUMBERT things, is that a use case that's ruled out pre-Mafashi and we don't ever do that? Like, how often does that come up? I'm just like, it just immediately popped in my head. I'm like, every cluster I ever test is running in a VM in the first place. So what does it look like if I then want to run CUMBERT on most clusters? Yeah. So we definitely support, from a community standpoint, nested virtualization and use it every single day. That's what my dev environment is. We see people using it in production on infrastructure as a service because they can manage their virtual machines in really unique ways and provide levels of uptime guarantees that are difficult even with like EC2. So with CUMBERT, you can live migrate your virtual machines so you don't lose them and things like that. And that's nested virtualization. There's limitations to what you can do with that and it's complicated and you can hit some really crazy issues with that. But it's definitely something that the community uses. It's not something Red Hat supports right now and their OpenShift virtualization product. It's certainly a really strong case within the community. Yeah. So I had a question for David. So you mentioned this certain level of generalization if I understood it correct and the API. So is it true that people use something else other than LiveVirt? Or do you know of cases where LiveVirt is not in the picture at all? Not for LiveVirt today. So LiveVirt does have LiveVirt in the picture. It was designed in a way to isolate the usage of LiveVirt and really even KVM to that container, the pod that's actually running the virtual machine itself. And the API was trying to design it agnostic of what that underlying technology was. At this point, it's so ingrained into our design that I'd be really surprised if LiveVirt, KVM, were ever replaced or swapped out. But we have the potential to do something like that in the future. Yeah. And on the nested virtualization aspect, I wanted to add that, yeah, for a long time after nested virtualization came into being, I mean, the best use case that we could think of was testing and which is running a guest inside a guest and making sure we are able to expose features. Or for example, as you said, we don't have a bare metal system in hand. So we want to set up, even though I would say that nested word and first level virtualization are not really the same in terms of, so a bug that reproduces on nested word necessarily does not mean that that bug is going to reproduce on a bare metal system with virtualization. But this whole new thing with newer use cases that are coming up, they are very interesting because they are stressing nested virtualization in unique ways. And I mean, one of the things that I have personal experience with is we are able to find new and interesting bugs that we had never found out before. And they are not reproducible on bare metal or just through regular virtualization. In fact, we found, recently we found a hardware bug, which was only reproducible because of the nested virtualization setup. And so I think that's kind of the good thing of the whole nested virtualization getting introduced to the picture. And yeah, it's interesting. Right, because there is, I mean, in a sense, or at least in a subdivided sense, you are exercising different hardware when running nested vert, right? You got the second level address translator and like some of that's implemented in MMU, correct? So if I never run nested vert, I never touched that bit of hardware inside my CPU, right? So yeah. Yeah. New, I always love new and interesting bugs. So it's kind of moving on a little bit. So what's the future, do you think of the kind of virtualization within Kubernetes? You know, the, you know, when we were thinking about the setup for the shell, right, like the obvious answer for virtualization in Kubernetes is lift and shift, right? So I have a running application. I want to be able to manage it on the same control plane as I'm running everything else. Okay, let me, you know, let me copy that virtual machine over, you know, and I'm done. Eventually that'll end and or it may not be the best choice for all applications. So what's the plan? Where are we going to be in 10 years? I don't know, five years, two years? So there's a story arc here. Lift and shift. And it tends towards unicornals, right? Traditional vert is what we're talking about with lift and shift. And so you would take an application that's maybe it's a legacy application and you want to transition to a containerized or cloud native, I don't know, whatever this buzzword is, like infrastructure. So you want to move to Kubernetes, but you want to bring your virtual machine with you. Great. So you vert allows that. And I think that was the thing that kind of justified us building this to begin with. And we needed that. But that's not the full vision. So that's the thing that lets us do the thing we want to do. And that's where we're seeing a lot of traction, at least initially. And I think we're still kind of at maybe the first part of the adoption curve for Qvert and that's what we're using it for. The next step, I think, is using Qvert as infrastructure as a service. So having the same sorts of patterns that you would see in something like EC2 or Azure or GCP. And we're getting there on the development side where we kind of have that at this point. It's not necessarily adopted by users quite yet, but that's where we are. Yeah. Just to help me understand, to interrupt briefly, to help me understand kind of what that means. That's sort of I want to do Terraform kind of things, but I want to write Kubernetes style YAML because I've got an investment on the tooling and I know the terms of and the way the API works and is that kind of the idea? I could think of it like that. There's a set of operational patterns that we see in infrastructure as a service. So if you look at like EC2, we see auto scaling groups. And what people are doing with these auto scale groups is they have lots and lots of virtual machines that are scaling horizontally. And instead of, it goes back to whole pets versus cattle analogy. When something goes wrong with a virtual machine in their auto scale group, they don't care what actually happened. Nobody's going to look at somebody or something automation. It's just going to kill that virtual machine and no one's going to spin up to replace it. So that's what I mean infrastructure as a service like patterns. We're introducing those into Qvert. We have that today. And then once we have that, which we do, the next thing we are working on is for ultimately Qvert to be the substrate to run more Kubernetes clusters on. So now we have an ecosystem where traditionally you would have used like VMware and bare metal and run Kubernetes cluster on top of that. Now we have pure Kubernetes. So you have Kubernetes bare metal level running Qvert to launch your tenant clusters on Qvert virtual machines. And we're working on that right now. That's kind of the next step. And then the future step, the one that I don't really quite understand that's on the horizon is ultimately I think that Qvert is going to be the heart of multi cluster in a certain way, because it's the again, the substrate that you actually run multi cluster on. So if you're looking at a new project like ACP, which go look it up if you're not familiar with it, but it has the potential to launch clusters on demand for your workloads. So if you have a workload and the cluster doesn't exist for it and infrastructure doesn't exist for it yet, then perhaps we can have smart controllers that can spin up infrastructure in the fly for this application to live in and things like that. And I think Kubernetes Qvert has potential to be the thing that's powering that for bare metal. It doesn't necessarily make sense for public cloud infrastructure. Maybe it does. I don't know. But when we're trying to replicate those sorts of things on bare metal, that's where Qvert would be. So there's a long tail here, I think for Qvert. And the thing that kind of justified us making Qvert traditional virtual machine use case, that's the thing that got our foot in the door. And I think the potential is much greater than that. But it'll be behind the scenes. People won't know that they're running a Qvert bit. It's there. So can you repeat what project was that? Do you say ACP? KCP. Oh, okay. That's the United States stuff. That's literally the project we talked about with Clayton. And I think, yes. So Turtles all the way down is what you're telling me. Exactly what I'm saying. Yep. So that's really interesting, particularly the part where you get, if you have really sophisticated live migration, then you can have, you can actually kind of realize that dream where you can kind of say, oh, you know what, I need this Kubernetes cluster to be over there for performance reasons or whatever. And you can actually live migrate the entire cluster to wherever it's 9am so that you can handle all those logins or whatever. You could do really crazy stuff. So certainly live migration is interesting for lots of different reasons because it allows us to, live migration allows us to do like an update of the underlying cluster without impacting all the clusters that live on top of it, which is great. So you're shuffling around virtual machines or running Kubernetes clusters as you're updating the nodes that are underneath it. That's great. But another thing that's interesting is we can start thinking about suspend and resume of clusters. So you have like clusters that you spin up and maybe you've already created a quorum and everything within them. And then when you don't need it, rather than tearing down the like persistent part of that, you just kind of suspend it, give all those resources back, down the line clusters. Then when you need it again, it just kind of starts up again. And it's right where it was. So there's lots of just really advanced future looking things we can do here. So that was actually, I was kind of like had a related question in the back of my mind for a little bit there is that that brings it up well is that virtualization in general is slow to start, at least compared to a container. Is that an issue? Or is it more of a, I mean, the way I've dealt with it in systems that I've built is that I just account for, hey, I need a minute to start this thing over here. So that means if I'm going to do, if I don't want to ever be cold, I ever want to be out or whatever, I just start the thing and make sure there's a minute in there and then bring the other one down. Whereas with containers, I can kind of do it more instantly, which on paper, it's kind of like, you know, there's a lot of features that get written up by the tech rags that a lot of the time I'm not really sure I care about as much as the tech rags give it some reason to write about something. And that's one of the things that I always wonder about is think, but is there a performance issue there where you need to be able to start the virtualized environment more quickly? Is that something that needs to be addressed so that you can do some of this magic that you're describing over time? We like to limit, so there's, I can say that like latency. So the amount of time you ask for something before it actually comes available. And when we look at autoscaling, so if you're trying to autoscale your application and under the scenes, we're trying to provision infrastructure to meet that demand, then the quicker we can do that, the better. So the idea of like rewarming, good play a role, just things within the guest itself. So we saw a huge improvement back in like the, I forget what it was, whenever like SystemD came around, when you started having just the boot order in parallel rather than sequential, that improved boot times quite a bit. I think ideally anything under a minute is probably good enough for most people when we're talking about staying up infrastructure. When it gets to like five minutes and things like that, then you're probably going to optimize. I mean infrastructure as a service, when we look like EC2, they take a while to start up EC2 instances. Track cloud formation. Right, right. 40 minutes later. I think we're faster than that, hopefully. All depends on the environment. Yeah, it matters. That's the short answer. Right, but I guess it matters, but it's kind of like there's a threshold where it matters, right? There's kind of like a, almost like a step function, right? You know, if it's under a minute or something, maybe it's fine. But then there's planning around it or, you know, planning, right? So like you have to build into the tools some level of planning around it so that, you know, we can kind of tolerate that outage. I guess for me in my development experience, I need to tolerate when something goes away anyway. And I'm not sure how spin up time is different than going away. So it's kind of like, I feel like that's one of those things that I'm building it into my system anyway, you know, what differences. But I was just kind of curious on your take. Josh, did you have a follow up to that? Well, I was just kind of wondering, like, it is a follow up to that question. Like, so, so Lincoln was just asking David Bonden about a number of concerns specific to this use case and this kind of driving infrastructure as a service with Tuberk. When you're working on KVM and QEMU, QEMU, having a really hard time saying that this morning for some reason. Nevertheless, like, are these things, are these two teams and you two folks communicative about these issues or is that orientation you described before of mechanism rather than policy at the KVM layer mean that you necessarily stay loosely coupled from sort of what concerns are this project that's using KVM, what does it have versus other ones? Like, are KVM folks specifically thinking about these Kuvert and Kubernetes use cases? And in communication with that team, like, does that interface exist at the team level? Or do we restrict it to software precisely to make sure the coupling isn't too tight and that one is concerned with policy and the other concerned with mechanism? Yeah, it's, it's, it's, yeah, I don't think it's very, very tight at this point. And whether it's going to be any advantage, I don't know. We probably will. But that said, yeah, I mean, there are people who, you know, focus on these aspects, even in the, in the, in the, in the virtualization group. So there are people who, you know, spend a lot of time understanding containers working on them. It's just the nature of, you know, KVM work. Probably it's not a conventional to think about containers. But that said, when, when you guys were talking about this question, this interesting question that Langdon brought up, and Langdon asked me something similar a while back, and I said, well, yeah, we don't think about that. But yeah, it really depends. Sometimes we do because, um, for example, um, uh, you know, boot up times is, is, uh, is that's a, that's an interesting topic that comes up once in a while. And there has been changes in Q mu to actually improve boot up times. Um, like, for example, let's not get a traditional chipset emulated. Let's not emulate a traditional chipset rather boot up something that is like just takes the shortest amount of time possible, not initialize a lot of emulated devices. So we do get into those only, uh, but they, they, it's not part of the, I don't know, I don't think it's part of the, like a process as such right now, but it's kind of increasing, you know, it's, it's becoming more and more frequent and common. Right on. And then my next question is, uh, it is not so much a follow up on that, but it's more a, uh, a free skate. Let's call it. I'm just going to say it's a both of you. Um, uh, AR 64. I mean, what that really means is, can I use any of this stuff outside of the Intel architecture? What does that look like? Are there teams working on ARM servers? I know KVM runs there. So what's the Coot-vert picture look like for the future on? I can speak for Coot-vert. Uh, yeah, we, um, we have community members that already run on ARM. So we support it from a community. Uh, it's, there's two, there's multiple layers here. So we have ARM at the, uh, like a controller layer. So if we're going to run on ARM infrastructure, we probably need to build our controllers, even the cluster level controllers to, to run ARM. And then, uh, we have like the underlying and like the nodes that the virtual machines run on. Um, we need to have all of our, um, controllers that are at the node level to the ARM as well. But then there's this complex use case where we have containers, uh, that might need XA6 for the, um, control plane. And then we might have different types of nodes within our cluster, some XA6, some ARM, and that sort of things get really complicated. But we don't do really quite well yet, uh, being able to schedule things accurately to the right architecture and then have the right, um, components on there that are built with the right architecture. But, uh, we're looking at things like that as well. Um, yeah. So. So related to that, um, does, do either of you, you know, Bond and David, do you, either of you see yourselves, you know, maybe not you, but like your teams, right? Making direct requests of CPU manufacturers. Um, where, you know, like to, you know, Bondon's comment made me think of this as like, there's a, there's a lot of junk on that chip that you don't care about. Um, you know, and so if there's a way where, you know, AR64 knocked out, you know, half of its, um, uh, uh, calls, you know, blanking on the right word, but, uh, you know, like, would that be better? Um, and is there a way, you know, does that say something that you see going forward that there's actually specialized chips for this kind of scenario? So is it, uh, I, yeah, I don't know the answer to that. Um, all I can say is I have experience with, uh, x86 and the turnaround time to listen back from hardware manufacturers is really, really long. So by the time, you know, you get back to hear from them, the ship has sailed. So I don't know how advantageous would that be. That said, I, I think, uh, arm traditionally has been kind of more in, has been more, more working more in a sync with software, you know, providers, you know, listening to their requests, uh, just because of the nature of how arm works. And maybe it's, it's a possibility. Yeah. But honestly, I don't have an inside or good experience, you know, that I can share with you right now. That's, oh yeah, that's a good idea to talk to. Well, so then that kind of, David, unless you have something to add to that, because it, like, it makes me come to the, what I see is the analogous question on the software side, which is, are we going to run other systems than Linux? Are we going to run unikernels above Linux virtualization? How much call for that? Do you hear people doing that in the community? Is this a really popular idea? Is this stupid and crazy? We've got a good networking stack in Linux. Why do I want to run a, uh, a kernel that's built just around my application? Like, is that a use case that's important for, for the Kubernetes stuff? It comes up. I don't have a lot of experience with it. So I know, I think some people are even doing this already with Qvert. I don't know their use case and why. My instinct is that it's kind of a niche use case, but I could be totally wrong. And it's something that I don't understand well enough to really speak to. I know that it's something that exists in our ecosystem today. Yeah, it's, it's one of those like unikernels, like I'm kind of on both sides of the fence there. The, there's a lot, it makes a lot of sense, right, to use like an x86 chip because it's standardized and it's got a general use case and all this other stuff, right? But then there's also, you know, hey, I built this chip specifically for this particular scenario. And so you kind of have the same idea, right, with Linux and like a unikernel, right, is that I have kind of this general purpose thing, and it does quite a good job at a large number of things, but then there's also a small piece, you know, that maybe, you know, it could be focused on. So if you're running a golden image kind of world, right, anyway, you know, so we're, you know, we're building a virtual machine and then we're putting, you know, golden copies of it out or taking a golden copy and putting out instances, which is kind of a containerization idea as well. Maybe something like unikernels makes sense. The problem I see is that the unikernel doesn't get exercised anywhere near as well as the general purpose solution. So at least for me, I have yet to see, you know, there's a lot of good research in the unikernel space, but I've yet to see in the unikernel space kind of a good reason that offsets the fact that it's not getting it exercised as well, and that you have to kind of go through these extra steps. But I don't know, that's just my two cents. It's kind of like, so that was a, that's a nice easy closing question. I did want to ask quickly though to David, I noticed her website redirects to a music album, and I just wanted to ask what that was because I didn't actually listen to it. How interesting. Yeah, I guess that's still my GitHub, so that used that's kind of funny. That used to be my resume, and now it is me playing goofy banjo music, and that is about the, all there is to that story. Yeah. That's, that's pretty hilarious. So yeah, so I was looking for your Twitter handle for a tweet. It does not exist. There is no Twitter handle. There's no social media. GitHub is about as far as I get. Well, now it means, you know, if you're doing the music thing, now you can just start a TikTok and start there. Scared TikTok. Somehow, I don't have TikTok, but I feel like they already know everything about me. I feel this exact same way. I'm not even on there, but I'm afraid of their surveillance somehow. Right, right. My nephews are on there, and somehow they know that they're my nephews, and that somehow correlates to me. Well, David, you found the leak. The question now is how to plug the leak. So, I will say, we actually had a pretty good question in the chat, but we are out of time. And so, in the interest of saving time, we will not answer it, but basically is that, you know, should Cube slash OpenShift be following the Linux model of, you know, do one thing well rather than do all the things. And what I will propose is that we will try to bring that up on the next episode with our next guests. And we will let you off the hook because we know our guests have already planned to do other things today. Excuse me. And so, we'll wrap up there. But thank you so much, David and Bannon, for joining us on the show. Thank you to the guests or to the audience. I'm sorry. And we really hope to see you next time. We are the last Tuesday of the month. And we have a whole bunch of great guests lined up. You know, check out our website at cubebyexample.com. If you want to see more about the show, watch past episodes or see any and see our kind of upcoming guest list. And we'll see you next time. Thanks, y'all. Thanks to both of you. Thanks for having me. Bye. Bye.