 Well, hello, and welcome again, everybody, to another OpenShift Commons briefing. We're really pleased today to have folks from the Kubert project here to give us an overview and to talk about what you can do with it on Kubernetes. And from Red Hat, we have Itmar Heim and Fabian Deutsch, and I'm going to let them talk for about probably half an hour, do a little bit of a demo, and we'll have live Q&A at the end. If you have questions, ask them in the chat. We'll try and answer them. We'll read out the questions at the end so that everybody who is listening to the reporting afterwards can hear the questions. And we'll have some live Q&A as well. So without any further ado, Itmar, let's let's take it away and get started. Thank you, Diane. So I have one. I'm Itmar Heim. I run the engineering groups for system virtualization and container management in Red Hat and Fabian Deutsch, who is presenting with me, is leading the engineering effort around Kubert. So what is Kubert? It's an upstream research project where we're looking at can we use Kubernetes for converged infrastructure. And when we're saying converged infrastructure, we're talking about containers and virtual machines. Now, this is really early days, but there are some interesting concepts that we fought for sharing because we're looking at interesting aspects of how Kubernetes is doing things and leveraging different mechanism in Kubernetes that are pretty advanced around extensibility and trying to map different concepts from the experience that we have from scheduling other workloads like virtual machines on cluster to the Kubernetes world. So we thought it's, while it's still research, it's interesting to share and hopefully you'll be the same. So just as a background, when we're talking about virtualization in the Linux world, we're obviously talking about KVM and the nice thing about KVM was that it made a virtual machine just a user process. So we just get KVM KVM running as a user process. Now, just having one virtual machine, the fact that it's user process not good enough as a solution, as a technical solution, but not a full blown solution. And this is where a project like Overit came along to focus on, I want to run a group of virtual machines on a cluster of hosts with shared storage and full blown management user interface to orchestrate host life cycle, virtual machine life cycle, live migration, low balancing concepts like that. So an enterprise could do data center virtualization with KVM. Then OpenStack came along and tackle infrastructure as a service. So more, how do I provide APIs to virtualize not only the virtual machine but also my storage, my network and other services. But both Overit and OpenStack focused on virtual machines as the core end of the OpenStack is looking at other things as well. But virtual machines were really the start and the core of this and then came containers and Kubernetes. Now, Kubernetes is dealing in a similar area of I want to schedule war clothes containers on resources, fostered resources, hosts, storage network. And the nice thing about Kubernetes with containers, it's a bit more of a generic use case than virtual machine was. So the question was, can we take this generic use case that Kubernetes handles and look at virtual machines as well? Because, hey, virtual machines are just a user process. We can maybe put them in containers, for examples. And virtual machines already share some concept that containers use. Containers use SLNX and C Groups for isolation. They also use namespaces, which is not relevant for virtual machines, but really same concept. So can we look at this and get to a convergent infrastructure we put on Kubernetes, virtual machine and containers just together? Now, why would you want to do this? So, yes, containers are cool. Containers are all the rage, Kubernetes is great. And with things like OpenShift, it actually changes more than just scheduling war clothes, actually changing your DevOps, your development, the lifecycle, how you approach even software, CI. It's a broader solution than what virtualization on its own went to solve. But with that, there's a huge investment in virtual machines. They're just not going away anytime soon. Just like on-premise versus cloud isn't going away anytime soon. So if you think about it, we still have mainframes with us because the investment is so large. The virtualization market is still growing. So if you're investing containers in five or 10 years from now, would you want to have multiple silo deployments, or would you like to have a single infrastructure that you can put all of your war clothes on it? And think about how advanced it will get to be at that point and how easier it will be to maintain and operate. And obviously, would use cost if you can do it on a single infrastructure. Another aspect is as you evolve your applications that used to run in virtual machines, you want to start moving them to containers. So you have the database and the front end portion of your application. So maybe you start with the front end and you move it to containers. If you can move it to containers and it still runs on the same host and you get the local affinity that the container is calling the database, but it's on the same host, so no network latency. Everything is simpler to manage and control this way. So that's another benefit. As you migrate, you don't need to plan for different hardware footprints because you are now migrating war clothes from the virtual one to the container-wise one. The war clothes can stay on the same host. Now, for virtual machines, we can benefit from some advanced concept that Kubernetes has. So virtualization is very mature and has a lot of things that Kubernetes is still catching up with. But Kubernetes is, first of all, moving very fast and also brings some newer concepts. So if we look at demon sets, oh, that's a nice feature. We could get it out of the box in Kubernetes and get it for virtual machines suddenly. Rolling upgrades, things like that, are things that we could leverage there as well. The benefit to Kubernetes is that there's tons of experience on the virtualization side and tackling similar aspects that the one Kubernetes is tackling, especially when you're looking at on-premise and bare metal use cases. So things like host lifecycle, things like device pass-through, for all of those, there's tons of experience. And this is why we think this is going to be a win-win if we can look at converging those solutions into one. Now, when you're saying virtualization and containers on Kubernetes, sometimes there's confusion on what is the use case you're talking about. So I want to separate between two use cases. One use case is I care about containers. I want to run containers. I want to run them in Kubernetes. But I'm still not as certain that containers give me the level of isolation between tenants and workloads that virtual machines give me. So I want to run my container inside a VM just to use the VM as to isolate. And that's something a project like Clear Containers is trying to tackle and providing better isolation for virtual machines. Now, Kubert, by looking at virtual machines can also lend a solution to that problem, but that's not the focus. The focus is we want to run full-fledged VMs and leverage Kubernetes to do this. So it's not about the isolation story you might have heard of. It's we have VMs, same VMs we have in Overt and OpenStack, et cetera. And we want to take them and run them on Kubernetes with all of their features. So I care about running a VM as the use case. I'm a VM admin. This is the Overt dashboard and it's not about running Kubernetes with Overt, things like that, although we just published reference architecture on running OpenShift on top of virtualization, but that is today when you use virtualized infrastructure to solve gaps that Kubernetes doesn't deal with on host lifecycle. Hopefully in the future, we can look at the single infrastructure. We're using Overt here just to showcase features from virtualization and some of our thinking process. So virtualization, same concept, shared set of node storage and network and we want to run workloads on them and discuss virtual machines. Now, some aspects are similar between virtual machines and containers or pods and some are different. So we're trying to map them and try to look at those that make sense for containers. Let's focus on enhancing Kubernetes to close those gaps. Those that do not make sense, there's no point bloating Kubernetes to do everything in the world, rather relying on the extensibility of Kubernetes and separate the aspects that are not core to Kubernetes and just solving them for VMs. And Bob and Will later showcase in more detail the thinking around there. So things like memory sharing might be relevant to containers because if you're running containers, those are technologies we invented for virtual machines, but are relevant for containers as well. KSM will scan processes if they have similar memory pages, it will merge them to conserve space. Memory ballooning is a technology more specific to virtual machines, probably not relevant to containers. And Kubernetes, oh, this one, probably not relevant. Memory overloading, scheduling, all those aspects are relevant. And when you're looking at scheduling, so Kubernetes has scheduling, a pluggable scheduler and you can extend it. It's not at the maturity that OpenStack is where 80 different scheduler were implemented for different use cases, but it has the potential and the capabilities, it just need to be in the field more in those different use cases. Another aspect we're looking to solve, for example, when you're talking about running your own host as you want to conserve power. So in virtualization world, you'll try to consolidate the workload, say overnight, by live migrating workloads into fewer hosts and shutting down the other hosts. That is a concept that could be relevant to the container world as well. It's not all about just scheduling resources, workloads on hosts, a lot of the issues you deal with is actually error handling. And this is where things like fencing comes along. I no longer see a node that had pods running on it. Does that mean the node is down or no? I can somehow detect it's still running, despite not having a network ping to it, maybe through a storage mechanism or something else. Before I fence it, do I don't see the node or I don't see a lot of node that may do problems on my side? Now, why do I want to fence a node is because I want to make sure before I start that pod, that workload again on another host that is not going to crop my persistent volume if it's to Postgres databases. So, some storage support exclusive access, some don't. If we're talking about leveraging existing storage technologies, for example, an investment in them, it's something we'd like to solve. There are different fencing aspects. And you start by soft fencing, trying to SSH and restart the Kubelet maybe before you shut down the entire host, then go to IPMI, then go to the actual power interfaces to turn off the power of that host. Maybe you want to check if the host is K-dumping. If it has a terabyte or two terabyte of RAM, that's going to take time and you want that done for troubleshooting. So you want to detect it's dumping. You now know the workload is no longer running because it's dumping, but you don't want to turn the power off yet. Now, those concept might sound strange to you if you're focused on Kubernetes running on existing infrastructure, virtualize or cloud. But if we're looking at converge infrastructure, we will want Kubernetes to have concepts like this or augment it with solution, managing the environment to give us those features that we get today for free from Kubernetes. So we could get to a single environment. Another aspect is network management. Today, Kubernetes networking is, it's sophisticated on the multi-tenancy, but not on the physical layers. Maybe you have different physical networks and you want to map different networks. So that's an aspect of host lifecycle and network management that we would like to look at. And then we get to the actual virtual machine. So if you look at CPUs and memory, okay, that maps one to one, but we don't need to look in Kubernetes of things like Spice remote audio video that we do for virtual machines because the concept is not as relevant for containers. So there's no need to look at that. And our goal again, is not to push everything into Kubernetes, rather we're looking at what would make sense and where would each piece make sense. And this is where some things are relevant. We're starting, they weren't in the beginning, but hey, NUMA and CPU pinning are now in Kubernetes. Live migration exists in virtual machines for years. Being talked about from containers may or may not be relevant at some point, but hey, maybe we could look at those policies and do something about with the experience we're bringing from virtualization. So that is just to give concepts of why we're looking at this and what we're trying to accomplish. And now Fabian will deep dive a bit more into what we're looking at. Yes, so hello first and let me try to share my screen. I hope that everybody can see my screen now. Yep, that's good. Very good, so let me pick up where Ikema stopped. So Ikema showed us what we want to deliver on top of Kubernetes. And with those requirements, we looked at, so what are our goals? What do we want to deliver? And one of our, I mean, the foremost goal is to deliver a feature complete virtualization API which allows us to deliver all the functionality we have today in its granularity so with all the details, with Spice displays and with Numa pinning. And to do this in a way which doesn't contradict with containers or Kubernetes itself. By the way, Kubert is aiming to deliver an API so it's not about writing a fancy UI, it's rather to provide the functionality on the cluster level. Virtualization functionality on a cluster level. Besides that, we also want to play nice with Kubernetes. So we want to be well behaving in the community. We want to do that technically and community-wise. That technically means that we want to avoid hacks and rather see that we extend Kubernetes in a way in which it is intended. And then we obviously strive for production stable or stability on all levels. So that means we really need the production readiness on the low levels, so where the VM is really run basically on the bare metal on the node level but also up to the public API. So that UIs or automation can be built against our API. All of that should be looking like a native Kubernetes API on the other side to give the user a unified experience regardless if he's working with containers or pods. To achieve these goals, we had to understand a few things. So we need to understand what Virtualization API do we really need. We'll get to that in a moment, but there are a couple of options of where that could lie or how it could look. Then we want to understand what runtime do we want. There are also several options. Should we use Lippert and KVM or should we rather drop to QEMU because it also has its benefits and drawbacks. Then another point is how do we finally integrate. I mean, if we know how the API looks and how we want to do the runtime how do we integrate with Kubernetes. There are also several ways and we will also take a brief look at them but that also drove our research. And then in the end it was really to identify the technical gaps between the Virtualization functionality we know and have today and what Kubernetes can deliver, especially in the infrastructure areas or storage networking scheduling. So prior art, so before we started with Qivert there were already quite a few projects. Without looking at when they emerged, here are three I want to speak about. So one is Vertlet. That's a project which is effectively a container runtime for Kubernetes which can run VMs. The intention of the Vertlet is really to run VMs. I mean, it's not doing it transparently. It's rather providing some mechanisms like pod annotations to customize the VM. But on the other hand, it's using the pod API to define the VM. So you take a pod specification and the Vertlet is arriving in VM from you and you've got some freedom to specify certain parts of it. It's like what kind of bus you want to use for a disk. Then there's RunV, which is a different approach. Here it's not about running VMs, but rather to run VMs for isolation purpose. So what you have here is you also post a VM spec and if it's run by the RunV runtime then the VM will be spawned and that pod, so the containers are launched within that VM. But the fact that the VM is running is hidden from the user in any way and you don't have a way to influence it. The same is true for clear containers. So with OCI CC runtime, there is an implementation which you can indirectly use through the OCI CRI runtime. And that is also aiming at providing an isolation to pods and not to offer the user the ability to run VMs. So it's also implicit. So far so good. So we've seen that the existing projects offer different APIs to interact or not interact with the VM. At least, and the workload was so far the closest to our approach, but it does not provide the API we need. So what we need, so they were mainly used so and RunV and take containers were creating the VMs for isolation. And the workload approach has a problem with scale so far. So if we try to put all the details about VMs into annotations in the pod that is getting messy and unstructured and eventually doesn't even scale or we are reinventing a structured struct in the headers of a pod file and that did not look very well. So we said that we wanted to go with a dedicated API for virtualization. We actually have that today. We were based on TPRs which are now obsolete by CODs and we're actually working on our own user API server for custom resources and sub resources because this allows us to define our custom APIs or custom objects or custom kinds and resources related to virtualization. So far, for example, we've got an API to specify a VM but we also got an API to trigger migrations. And here we go with an example of the current VM API. So if you look at it from a syntactical point of view it looks like a pod specification. It also has a header, it has a kind field, it has metadata and a spec specification. And it can also be written in Yamal or JSON like a pod specification. It's just a different object you are specifying. To give you a little bit of insight of how this translates to our stack. So parts of these specification or parts of these definitions are used in different parts in Kubernetes. So if you look at the domain section further below where you've got the name and the type this is obviously taken to define the right VM on the low levels. But if you take a few lines further up for example the node selector that is a field which is taken to be passed to Kubernetes is to leverage the Kubernetes scheduler to do the appropriate scheduling. And right at the bottom you also see or you get a hint that we try to reuse existing API objects of the Kubernetes API in that example below it's that we are using a persistent volume claim as a source or the backing file for a disk. So that was about the API. And you saw that we are using LibVirt to run the stuff we handed that one or two times but there's more to it. So LibVirt is not run on the node level which is done by the Verlet Run VNK containers so they're not necessarily running LibVirt but at least they are run not within pods or containers but rather alongside the Kubelet as a service usually on the host operating system. So what we do is we said we want to containerize our runtime. So first before we get to that, LibVirt, the decision went with LibVirt because it is a proven stable and feature rich component which we're also using in other virtualization products or components in the open source landscape. And we put LibVirt and QEMU so all of its dependencies and things necessary to run the VMs into containers. That has benefits. The main benefits are that you can that the lifecycle of LibVirt and all the dependencies are not tied to the cluster or to the node life cycles. And that gives benefits when we think about releasing and updating strategies. Furthermore, you don't need to have two approaches to on the one hand manage the cluster or the application within the cluster. So LibVirt and on the other side, make sure that the right dependencies are installed on the hosts because then you have two mechanisms, two processes which you need to take care of to perform one update which can get complicated. Technically, however, what we do is that we, if you spawn a VM, then we'll be moved into the resource group of a pod. So not all VMs are running in one resource group but rather are moved to pod resource groups that into detail in a few slides. All right. So we spoke about the runtime and that it's living in pods. Now the question is, and I said that the other approaches put their runtimes alongside the Kubelet. So we also considered that approach, but for now we said that's not so nice because we would need to put stuff on the host. But that's not the only integration point. We've got additional integration points. So for example, we said that we need a separate API for a virtualization functionality. And I mentioned already that we were using CRDs and TPRs in the past. And that's actually the intended and blessed way of integrating with Kubernetes on the API level of using either CRDs today or custom API servers. The good thing here is that if you provide, if you do the integration on the API level, on the one hand we can use all the tooling provided by Kubernetes to do that integration. On the other hand, we can actually use other Kubernetes objects. For example, I mean in the API example I gave, they will reuse that persistent volume claim inside a VM to make that semantic connection between a VM and using a persistent volume claim as a backing to work for virtual disks. So with all this knowledge, here's now a technical deep dive. And if there's a question which I could imagine, please let me know. But with these informations we had so far, the architecture looks as follows. So on the left hand side and the cluster components, if you look at the white box which is called API server, that is the API server Kubernetes. Conceptually, we now run alongside of that our VRT API server. Today it's because the API server aggregation is really fresh and we are still updating that part. It is a bit different than shown in the diagram, but it's really close to that. Then to add our virtualization capabilities to the whole cluster, we use a daemon set to deploy the VRT handlers and leverage to every host in the cluster or to selected nodes. So here we can also use the namespace features or node selection features or in general, tanks and tolerations or labels to assign our cubert nodes to the real physical nodes as needed. The VRT handler in the picture, which is shown quite in the middle is responsible for watching DVM API objects and then performing the necessary operations on the host. So speaking to Libre to really launch DVM. And as you see, it's hinted here the VM is then actually launched in the pod of the handler in Libre but rather than the pod, which is assigned to that VM that is important to do because it allows Libre, sorry, it allows Kubernetes to track the resource consumption of that specific VM and not all the resource usage of all VMs is aggregated in one pod. So we're basically doing everything inside pods and that also also permits though we need to see if it's feasible or not to run other pods which just run container workloads alongside Cubert. With that API server aggregation which we saw on the left side on this slide so here you can really now or that allows us to really integrate also on the on the user level with the existing tools and Kubernetes. So Cube CTL will work for a pod like it will work for a VM because the API server will make sure that the requests related to virtualization will be redirected to to our virtualization logic to our word controller which was shown in the slide before. So I described it a little bit and now would go into a small example of how it really looks to illustrate what component is involved where and how does the spawning process of a VM look. So this slide shows how the waiting status so nothing happened no VMs are defined and we want to report. Once a VM is getting created currently that is a CRD the word controller will see that actually because the word controller is watching for VM instances or actually all kinds of virtualization objects which we introduced. Once that happens the word controller will schedule a pod the so called VM pod. This pod will access the harbor or the resource group for that VM. Once that pod is scheduled on a VM on a node the word handler will see that the VM were scheduled and will tell Libvert to launch the VM inside that pod. From that moment on Kubernetes is aware of the resource consumption of the VM so we can even set limits on the pod limits in the pod sense with regards to memory consumption and CPU limits to make sure that the VM really does not exceed those limits. This cannot be done for one VM but rather for all VMs so the process here is pretty simple and straightforward. I'm just looking at the questions to see if we can answer something. So the question is can the node in this case be a VM or virtual or only one of them? So it can be both but it depends. Primarily Kubernetes intended for VM that's where we traditionally run virtualization workloads but nesting is becoming more and more stable. I'm not sure if we will support that in production mode but if you enable nesting on your physical node then you can also run Qvert on virtual nodes. So that if you don't enable nesting you can still run your VM in emulation mode that has a serious performance impact but it's still possible. The other question oh yeah right that was an answer very good. So that is effectively the launch process that's all happening in the Kubernetes cluster. There are no underlying components involved so far. There are by grant there are some things to consider on the low levels which you cannot see here so which are a bit nasty and to be solved but the proof of concept works and we'll take a look at that not only the proof of concept but it's working reliably so far we still have to do some research that is rather the approach here. So technical gaps in general we saw while starting or since we started working Qvert we saw that a lot of functionality is there. Sadly as often it's the case that the functionality is present in areas which we don't need that feature rich yet and that areas which we require is lacking functionality. However wherever we see gaps a main intention of Qvert is not to work around them on our side but rather to the Kubernetes community to see that we can fix those issues in Kubernetes if it makes sense and if it's possible. Why? Because we want to strengthen and we want to improve the Kubernetes infrastructure. Like if we take core concepts like fencing we could possibly work around them on our side on Qvert but it makes so much more sense to put that into Kubernetes so that containers gain these benefits as well. So that is one of the conceptual also reasons for Qvert that we want to strengthen the infrastructure of Kubernetes to also support our workload. If you're interested in the details then we can take a quick look at all the gaps we encountered or the selection of gaps we encountered so far. So resource protection exclusiveness of fencing Itema mentioned fencing exclusiveness is really that you can guarantee that only a single node or single VM is accessing a volume at a given point in time. Currently this is done on the scheduling level in Kubernetes but that's not sufficient if you want to prevent that the database is getting corrupted. Host life cycle management which is strong at least in all word and is also something which is not so good yet in Kubernetes. QBitman and tools alike really improve the situation but we're still seeing some gaps there. Device management it's seeing some traction especially cause from the VGPU point of view so there are some concepts pushed into Kubernetes to support device management and device path through into pods and thus hopefully also into VMs. Compute for performance we need to have CPU pinning and Numa we're also seeing some advancement there. It took quite a while for Kubernetes to find the right approach here but it's really nice to see that they warmed up to it. Dynamic SLA is effectively that we can modify the CPU and RAM limits in the pod sense or that we can remove and add RAM and CPU for VMs. That seems also to be happening but it also takes its while until the concepts are ready in Kubernetes and once they are then we can piggyback that. I'm just checking the questions so in the sense of multi-QBitman clusters with Federation. Yes we need to see how that works. I'm currently not aware how that works with the custom API servers in general but we are aligning to what Kubernetes expects or how integration should look and so we hope that Kubernetes also provides a way of how this can look in the broader Kubernetes ecosystem including Federation Health Networking Handle between Condients and VMs within the same project. So that's actually what a coincidence so it's actually the next point of the detailed gap. Currently or work in progress and it will land next two weeks probably is that we our first it took us a while to find the right approach. So our first approach will be that we obey the networking of Kubernetes. So what we do is we request new interfaces from Kubernetes to be assigned to a pod. This is actually in line with what is happening in Kubernetes so pods will get multi-network or multi-interface support and we will be reusing that so additional interfaces will be used first to be assigned to VMs that is pretty straight forward but then we need to make sure to align with Kubernetes that we give that the VMs are respecting the IP addresses. We hope that this will work out with CNI so we're doing that on the CNI level and we effectively request new interfaces from CNI to be attached to VMs. This is under heavy research and we are prototyping this kind of stuff. We are in contact with the CNI or with several CNI people there to see what the right approach is. On the long run however we want to see that we get layer 2 networking. But that is conceptually difficult because layer 2 networking Kubernetes is not there yet and it has not been discussed so far and the question really is if it makes sense for Kubernetes. So it's a research research area which we need to tackle. For storage we identified that the Kubernetes is well at setting up storage but for advanced production ready configurations of multipathing for example that's still a way to go there to support that for every connection for every volume type it supports. Advanced operations like cloning and snapshots especially on the server side is also something. Snapshotting has been designed cloning I've not seen any proposal for that yet but that is stuff we need for the VMs scheduling. Resource driven so this is actually also emerging since we created that slide deck. Resource aware scheduling will make it into Kubernetes the first proposals are in related to device plugins but we also need to do the scheduling based on custom metrics. So far Kubernetes is using the pod metrics to do the appropriate scheduling we need to enhance Kubernetes or provide our feedback to make sure that we can also use virtualization metrics to do the scheduling. Rescheduling or balancing is to make sure that the pods today pods are only scheduled once they are created but to apply policies to a cluster for example for power saving as I mentioned before you want to do rescheduling or balancing so after pod creation time you might want to move a pod or a VM in our case to a different node to be able to shut down some nodes to save power I'm skipping the modularity on the infrastructure side we also obviously need some we need some enhancements to make it really to allow us to extend Kubernetes to provide the virtualization workload so we want to extend kubectl to be able to add verbs like live migration or starting and stopping of VMs which is not present in the pod concepts and we need the enat on formalization to make sure that we really are on the same page when it comes to integration between Kubernetes and and kubectl and then there's UAS native object storage in Kubernetes which effectively allows us to to really now that we have our use API server to store our objects still in Kubernetes and not need to provide our own key value store to store our data this got very detailed I noticed but I hope it gives an insight into what gaps there are and what we are looking into to solve so a quick wrap up before we get to a small demo so on the API level we really we're investigating so we start with a liberate like API including scheduling aspects but we still want to see and what to get a feeling for what granularity do we want do we want to be more pot like which was very simple in the beginning actually and this has now grown more sophisticated with all the enhancements like for example advanced selectors, states and tolerations it's adding more complexity to pot specs but we want to see if we want to stay complex or become more simple we also need to rethink or not rethink but continuously think and revisit if how the low level mechanics what I mentioned initially doing the implementation on the CRI level has benefits and drawbacks and we want to keep revisiting that process because we need to evaluate that for every single feature we have and how it matches together it's ongoing research and often you just get to the questions once you get to the implementation we noticed that Kubernetes still has the gaps we just mentioned them and we need to see that we we get the right ones delivered in time so that we can make progress and are not blocked because otherwise we are pushed to do workarounds and that is taking time and doesn't drive Kubernetes in the right direction. Nice note is that the operator pattern works pretty nice that is the pattern we use to track the VMs and was actually defined by CoreOS in the end we want to say that we need to continue with our research we see that there is potential for convergence and it looks promising like one nice side effect which is implicit is that it's so convenient that you just take a cluster or like we will see in a minute you take Minikube and just have to deploy POTS on it and you get a VM runtime for free and you can use the same infrastructure and that is really, really convenient the separation so far also looks like a win-win because Kubernetes can drive in its own pace focused on the container workloads and we on the cuber side can look and then we drive in our pace and just see that we align on the integration points in general it also looks like a win-win because we can which goes into the direction of the previous point because we can align on the same infrastructure there are attention points what we need to see what we need to see what role do we take do we drive for the absolute feature parity and even violate some Kubernetes assumptions or do we rather say we limit that feature but gain that we are more behaving like Kubernetes would expect or how Kubernetes is setting expectations for Numa for Numa we know we need to do a lot of detail work to get the real performance out of it and we are not so sure if Kubernetes will go the whole way so that is an area where we need to see where do we end up with or what do we end up with where do we want to put the focus on full functionality or better integration and finally we are seeing a lot of stuff is easy but really coming with stable and production enterprise ready solutions is hard and we see that every day after this small summary and now it's close to the end I would still try to give you a small demo to show how this is actually working and after the slide it has been shared you can also try it yourself at home the demo which we have prepared is based on Minicube or is leveraging Minicube and it will effectively deploy Minicube and we will be able to run IBM and connect to it hopefully before that I am taking a look at the questions but they seem to know for the questions they are doing a pretty good job so far answering the questions in line so let's see the demo and then we will take more questions after this very well so here I have checked out the demo repository which is shown in the slides to start you run the demo.sh script which is effectively just checking if you have got Cube CTL installed if you have got Minicube installed and then it is checking out Qvert after it checked out Qvert then it is deploying manifest including a test VM once that happened and you know deploying manifest is one thing but then it takes a while until all the container images are downloaded and that is what is happening in this line now we are waiting that the containers are downloaded and if you do it for the first time really it can take a few minutes until all images are downloaded there is a lot of debugging stuff added it is not optimized for size yet so it can really take some time at the moment but once that is done you can really see that the VM is eventually running and yes we do see it here so the VM is running and here by the way you see the output is not very sophisticated so that is where we want to improve Cube CTL to allow custom rendering for custom objects in our case to get some details we can use the JSON output and here we see already the VM definition in the domain part we see that a console is defined in a disk that an iSCSI target is used as a backing source for the disk and this should actually be updated to use the persistent volume claim to highlight that more and we see that a graphics device is attached including a network device further above just to show that it all really looks like a regular Kubernetes object you've got labels you've got some metadata annotations so it's really something you can use with your existing tools to see so we know that the VM is running what we can do now is we can take a look at the pods and here it should roughly match with what you've seen in the architecture diagram so we've got the HA proxy which is just an artifact because we don't have API server aggregation yet we've got an iSCSI demo target which is providing the image for the VM we've got Libred running a dedicated pod we've got a SPICE proxy which is sitting on the border of the cluster to allow accessing the VM we've got the VRT API which is providing the custom resources or currently providing the validation of the custom resources we've got a VRT controller which is reacting to the created API objects so VM in this case we've got a VRT handler which is running on the node and speaking to Libred and we've got the VRT launcher that is the pod VM which is opening the resource group for the VM and we've got a VRT manifest which is an artifact of the work in progress we currently have if we now look into the Libred pod we should be able to see that the test VM is really running but that's not what we want we don't want to directly go to the pod but we don't want to go to the pod to then see that the VM is running what we rather want to do is we want to connect to the VM itself if possible so what we can do is there's a small binary utility which we provide it will become a cube CTL plugin at some point once that is standardized and looks as it is now VRT CTL allows us to connect to the console and actually also to a spice so what we will do is we will connect to the serial console first we need to provide the write IP to connect to of the API server and it's listening on a specific port then we need to name the dvm and the device to use alright just a second I probably used the wrong port oh yeah there you see I had a mix up in the port number so and here we go it seems we're locked in so here we are running an alpine Linux that is actually the disk which is provided by the demo target we can do stuff like rebooting so it's really a connection through both CTL which is actually in the background the implementation is similar to the exact call of the kubectl command so here we really try to play along with how Kubernetes is doing it the serial console is not the only thing what we can also do is we can do spice we take the same line oh yes oh yeah I should shut it down so what we do now is we kubectl so I deleted the test vm and now I'm re-creating it for demo a vm is created and now we should be able to display the spice button here we go and we see that the vm is running it's a bit slow because the question was this is running in emulation mode we can actually change that but it's not I think it landed in the last version of minikube so it should be faster in future so and that was the demo are there any questions what to show in that demo I cannot show like migration because it's just a single note but it's sufficient to play around with other stuff there aren't any questions I have maybe it's a newbie question the issues you were talking about was NUMA the memory access bottlenecks and stuff is that anything that I know that's important to all the vm and virtual machines folks out there but is that anything that gets solved by kubernetes that bottlenecks I know you keep talking about it being very very important to solve that so open so sort of interrupting the delay is horrible so openshift is taking a lead here so what kubernetes says and that's also by the way an argument for putting VMs into pods is kubernetes says we want to support any workload and there's that area of performance sensitive workloads which openshift is seeing a demand for so some people from the openshift team like jeremy either that area and they actually tried for three quarters of a year to make kubernetes aware of these important tuning on the low levels the problem is that the kubernetes stakeholders set for a long time that is to find granular and we cannot solve that on the api level because we don't want to mess up the api with all those low level details so after an iterative process of like six or seven hours or proposals of how to solve the problem they finally settled down and found something so today it looks like they will be implementing an approach where you can where you can specify a policy of how your processes should all be distributed among human nodes which is a compromise so back to a question yes we do see that they're opening up to it it just takes time and especially in areas which have not been important to kubernetes before it really takes its time until the use cases understood and until the right solution is found and really can take a little bit of back and forth so if people want to get involved in this project where on github can they find you where can they find you and do there we go that's what I wanted to see thank you very much so first thank all of you for listening and yes we are really looking for people trying out kubernetes and trying it and filing issues also from other distributions so far we are trying it on centOS and fedora but we would really like to see that it runs on other distributions like debby and coroS as well there shouldn't not be it shouldn't be so difficult because we are our our platform is the cluster not the host so it should be fairly easy but that is what we really would love to see contributions but you are also welcome to contribute to code doing testing or just use it so join us on github at cubert we don't have a slack channel yet but we own freenote as well on irc good old irc and the cubert channel on the freenote network so there is one more question that marcello is asking is how can I look at the logs between VM orchestration and kubernetes cluster in a kubernetes space they have to catch all the logs what do you do the basic question so that is a good question so what you do is so we try to integrate nicely here so we provide the logs from the pods so if you are interested in what the controller did to schedule the node then you can really take a look I am doing it now you can take a look like with any other component like with any other pod at the logs of the relevant component you want to debug so we are emitting actually we are emitting VM events on the cluster for state changes and we try to emit valuable logs for every component so you can debug it on the node level as well so you can say kubernetes of get pods to get the right so if you want to debug the node level for a specific node because you know that a specific node that issues then you can take the handler and take a look at them there and the same is true for for libred there you get all the libred debug outputs so it is fairly straightforward I think my mind is a little bit blown because it is like I always with kubernetes as everyone knows I always associate everything with containers and this is a huge leap forward and I have talked with the clear container spokes a bit too in the past and this is just the merging of two great technologies in one cluster management approach so I think this is just pretty useful stuff is this pardon I am at red hat and I am asking this question and it is an external facing thing but is this mostly red hatters working on kubernetes at the moment or are there other folks from the kubernetes world working with you guys so it emerged from red hat because we did not see that these needs were addressed elsewhere what we see now is so first we also we see participation from open stack which is nice so we are classically from the over side but open stack also has an interest in it we also see that users come from different distribution so we especially see that bugs are filed by users from the outside but the development so far is is a red hat but I would really love to see that we see more contributions from others one of the other things on the kubernetes community meetings have you done one of their like the intro of every I think it starts in three minutes is the next one they have little demo times where you can demo your new stuff have you done one of those yet oh we are we want to the demo I just showed is really hot it was effective it's just possible to do that since last week so it's really to demo it before was really cumbersome so doing the demo with minicube really is a leap forward for us because we can easily show it and that's actually the opportunity that we want to step up to other forums to show what you can do let's get you on there because that will get you a lot of eyeballs from around the kubernetes world this is great work thank you itmar and fabian for doing this today from the audience it's a really good question so thank you for coming in and asking them and we'll have you on again in the next iteration and hopefully we can get you to show it with mini shift or maybe even open shift live at some point too that would be awesome I'm not to put you on the spot but that would be really really cool and this is very interesting to me because I think all this mixed workloads is definitely something people are looking for as it's obvious from the audience that we gathered here today so hopefully we can get you some feedback and more eyeballs and more resources working with you on the project thanks again for coming and we'll talk to you all soon