 Hi, my name is Petr, welcome you on this afternoon talk. And I'm actually honored that you arrived after lunch. So I promise I'm mixing light and allow you to digest peacefully. So, but first things first, I need to know that who is here. Please, if you have any experience with virtualization, raise your hands. Awesome, I didn't expect that. But it'll get more tricky. If you do have any Kubernetes experience, please raise your hand. That's more or less the same. And the last question, have you ever developed extension, the feature extension for Kubernetes? Raise your hand. Yeah, that's what I expected. And do you plan to develop extension for Kubernetes and are you here to learn? Cool, so I'm glad I matched the audience. Let me get started. If you're worried about me, I'm senior developer at Red Hat. I'm cloud enthusiast and always been ever since AVAWS started. But after that time, it grew out of me and I have no idea what's going on there. I work on Kuvert, which is the virtualization platform for Kubernetes. And I love talking and you will learn that soon. For those of you who are in a hurry, here's a brief what you will learn. You will, I will explain how we in Kuvert use custom resource definition, how we use them to extend the Kubernetes and how we define virtual machine. I will explain you how we use that to build controllers, to actually create a mechanism to operate the virtual machines. I will tell you not to use aggregated API, we use it. And not that it's not useful, but it can be hard. And I hope I will give you an insight how to think about development for Kubernetes. So let's get started. Just the goal of this talk to make expectations right is to give you the bird's eye view, not the deep knowledge. Because for the deep knowledge, I believe everybody here can go to the source code and go as deep as possible as he wants. And one thing more is that already there was a great talk this morning about writing Kube controls for everyone. If you saw that, you already have the knowledge and the best practices that you can apply with combined with the knowledge from this talk. So let me start with the definition of Kuvert taken out from the website itself. Hey, I see the shock. No, I don't expect you to read that. I will tell you what it is. Kuvert is a powerful virtualization platform built on top of the Kubernetes. This is actually how I view that. And I believe it's quite cool tool to have. And this is a digest. What I would like to tell you is that we are, we the Kuvert are building machine, virtual machine management on Kubernetes. What does that mean is you take Kubernetes, you deploy our extension, and you will handle the virtual machines the same way as you would handle pods, which I believe is quite cool because that allows you to do fascinating stuff. For instance, you can bring with you your legacy. Everybody wants to do that. You can mix and match. You can just port part for the pods for the controls for the, sorry, for the containers. And then what can't be ported just bring a virtual machine. But what's most important to me actually is you can run different operating systems in a container on a Kubernetes because most of the clusters run on Linux. And there are times you want to run Windows. For instance, your wonderful Windows XP with a .NET 2.0 with web server. There are quite a few applications for that. I helped develop a few of them. So believe me, there are there. So since everybody here is known with the virtualization, let me just bring back the main four components I've used that are important for build the virtualization. You need to have store because you would like to store the virtual machine, the information for that, the state, the metadata, the stuff you would always want to persist. You would like to have an API because you want to talk to that. You would like to build the UI because user usually don't want to use Bash to communicate. Sorry, use terminal to communicate with the management platform. You want to have scheduler because there ought to be something that knows where to place the virtual machine. And you have to have a handler because last thing you wanna do is run your virtual machines by hand. So now let me ask you a question. How would you build that? You have an idea? This is how we build it on Kubernetes, of course. So this is a brief overview of Kubernetes. Just quick recap, we have a QCTL which is the front end from the command line that you talk to the API, right? Then you have the API server which is the main component you can even talk to directly. Then you have the scheduler which is really the important thing that you want to have even for the virtualization. That is the stuff that sits there, lifts them, and waits for unscheduled objects, figure out where to put them, and then just assign them the node. Then you have controller that actually moves the things registered for the Kubernetes, like ports, replica sets, and deployments. And then you have the store, you have the HCD, where objects are stored are persistent. And then you have the Kubelite which is a controller that sits on each node and waits if there in a store is an object assigned for this node. And then, of course, lastly, there is your application running there. And this is how we extend it, the Kubernetes to be able to run virtual machines. Don't think of it that deeply because I will try to explain each of the component in a later on. So we have Kubernetes and we benefit it greatly because we have the store, we have HCD. We can place the virtual machine inside and we can persist it. We have Kube API, which is a huge deal because we don't have to build that. And we can extend it with our own API. We have wonderful Kube scheduler which, again, we don't have to do anything. We just describe our workload and the Kube scheduler figured it out and it just plays it on the right place if he knows the specifications of the node. And then we have the extensions of the Kubernetes that allows us to build the handler to handle the virtual machines. So please, let's build together KubeVirt from scratch. We will start by placing virtual machine in a container. It will probably look like something like that if you are brave enough. You just run QEMU with your Fedora, place some arguments and run it. It will run. It will be single process. It will, it can utilize POD network if you use the same interface as in the POD. Then you will talk to the internet and internet will talk to you. But there are some drawbacks. There are drawbacks like command arguments because if you want to do multiple workloads, you want to change architectures. You want to basically customize the VM, plug in additional devices. You want to extend the commands and you don't want to remember them or build them. You don't want to build command builder. And then, of course, if you want to select, if you want emulation or if you want accelerated KVM which can be done because KVM can be passed to the container by host pass through. But more importantly, you can describe your virtual machine by writing the POD specification. Only POD specification. This is a huge deal. But what's next? Well, how to overcome the issue with I mentioned? Well, you will probably end up with leveraging the PODs and the fact that they can host multiple containers. That will be, you can probably place one Libert container and second virtual machine. They will talk on the same network and Libert will manage the virtual machine for you. And as a benefit, you will just place a XML file descriptively specifying your virtual machine and it's done. But that's not enough for us for Qvert because we wanted our gateway. Gateway inside the virtual machine. So for us, we add up a virtual launcher. This is not especially a Qvert extension, Kubernetes extension, but it is a way how we can communicate with the virtual machine. And that allows us to run the virtual machine in the cluster as a normal POD. And this is truly what we do because in Qvert, virtual machines are packed inside PODs. You can specify a POD specification and a Libert XML, put it together, send it to virtual machine and it will run. So at this point, we have a cluster in a virtual machine just sitting there. The positive things, you can describe it with a POD specification, the negative thing, you have to write a POD specification and you usually don't have to do that. You don't want to do that. So we have a VM, what's next? What would you do? Probably teach Kubernetes to know about virtual machines. And then the simplest thing to do is to leverage custom resource definitions which are the simplest way to define a new kind, a new object that will be stored in the Kubernetes. The specification you see, ah, it can be read. You can see on the left. It's basically what we have for the virtual machines. You see, it's very simple. You just define the virtual machines, qvid.io, you tell that the version is v1, alpha 2 and you basically specify the names and that's it. What you gain from that, if you put that in the Kubernetes, you will get an API. API automatically created for you with endpoints. That will allow you a basic create, read, update and delete. And this is again a huge deal because you don't have to implement that. You will gain a validation, which is native in the, which is nowadays native in the extensions. You will get open API if you specify the validation and you can even attach your own validation. And then you get an option to register a finalizer because for us, Kubernetes, the finalizers are very important. Imagine that you are running a virtual machine and then you want to delete the object, the virtual machine from the cluster. If you just do that without the finalizer, what will happen? Well, a problem because virtual machine is not finished properly, so you can lose a data. With the finalizer, it will hold the deletion until the virtual machine will flag, okay, I'm ready, delete me. And we use that. And this is how the endpoint will look like. This is automatically generated again. It's very simple. You will have the namespace, you will have the version, and then you have the name for the virtual machines. For each virtual machine, the dots represent virtual machine one and virtual machine two. So that's cool. You have customer source definition. We use that. And you register a new spec file that you've defined your VM. And now your cluster know about the virtual machine. But that's not enough for us because if we just use a virtual machine, then we would not have the persistence. And if you're building virtual machine, virtual management platform, you need to have persistence. So we fold it by splitting it into three pieces. We have virtual machine instance. This is our epithermal, basically mortal virtual machine. If you create this object, the virtual machine will be created. And if you delete it, the virtual machine will die. Then we have virtual machine, which is our immortal virtual machine. The one that you create and define how the virtual machine is also look like, how the virtual machine stores the configuration, metadata and basically every possible aspect of the virtual machine you want to purchase. And it will survive restarts and deletes of the virtual machine instance. And then we have replica sets, virtual machine instance replica sets. This is actually quite useful object because it allows us to scale the same virtual machine instances, just in the same way as the replica set scale pops in a Kubernetes cluster. Maybe you ask why we need that? Well, again, back to my previous example of Windows machine. You have a Windows machine with a web server running .NET and you want to scale it up. You can do it with that. So at this point, Kubernetes now understands the virtual machine. But tell me, you've written a code for users. Can something go wrong? You just let users put there something. I believe it can. So now we have to teach the Kubernetes to validate because if the hooks are there, they are not used by default. So with the custom resource definition, you specify the open API structure of your request. For us in QBird, this is done automatically because we generate that from source code. And I urge you to do the same because the last thing you as the developers want to do is have a different source code and the API description. I've been there, I've done that and I failed terribly because we just went away and we cannot figure out what's wrong. And then if you put the open API inside, it is validated automatically just before it's stored in HCD. And if it fails, it will fail with an error so you know what's wrong. And then if you want to do any expanded, any extension, you want, for instance, to validate if you put a data volume and the data volume is specified properly. And not that it's written properly, but if it's specified properly. Or you want some device to be set with defaults. For us, these defaults are set up the random device. Users just want to place random device inside but then there are some defaults that you want to set. Like for instance, maximum number of requests allowed to do, to put your machine to ask. User doesn't care about that, but you have to set them. So this is done with the defaulting. And these two are done in the, in the cube here by dynamic admission control which is a new, I believe from Kubernetes 1.7 and up. You can register your validating webhooks and mutating webhooks. And again, quite simple. By the way, on each slide, there are links down there that points to the right page of documentation because when I was learning this stuff, I could not orient myself with so much documentation. So I'm trying to save you some time and point you to the right documentation. So at this point, we have a cube weird registered with validators and defaulters that actually can talk and verify the virtual machine. But what about if we want to perform any custom action on the virtual machine? Because Kubernetes is a declarative way of specifying your object. That means you set up your specification, you put it up in a cluster and then you wait until the stuff happens. This is a reactive nature and this is how it's written. But for some time to time, you would like to register a some handler that is performed now, right here, right now. This can be done in Kubernetes and you can use aggregated API for that. It's a standalone API that you register to the API itself and register the handlers you want to do. For instance, for us, it's a virtual machine slash restart because we want to do that right here, right now. We don't want to wait at some point to Kubernetes decide, okay, now it's time and I will restart that. As I said, there are some points that you want to do right here right now and allow some custom actions each restarting or gathering the metrics. And this is how it would look like. It's the exact same previous API endpoint with a restart in the end and that restart will trigger your custom endpoint. And the obvious question is should you use it? And the answer to that is usually no because most of the time when you're extending Kubernetes, you will be just plainly want the kind and some handler for the kind and no custom actions. But if you definitely need it, please save yourself time and use AP server builder, which is part of Kubernetes in Kubeiter and it's quite good. We didn't use it originally and it can because it does not exist, but it exists now and it will save you a huge deal of time because writing API aggregated server, there's a lot of boilerplate that's a huge place of time. So now we have a qubit core with handling the virtual machine from in to out from in. But tell me, what's missing? Am I missing something? We have the virtual machine there just sitting at CD. Well, of course, storage network. This is done by Kubernetes. We use the infrastructure in the Kubernetes and for the storage we use present volume claims that will allow us to just direct this present volume in. But now the VMs is there and it actually does not do anything because we did not implement any controller and in Kubeiter we have quite few controllers. So I don't know if you know that, so I will repeat that. What is controller? Control for me is a state keeper. Here it watches a state in a store. It monitors for the state and if the desired state is not reached, he will pull the strings and push the levers to make the state happen. Eventually, it will not be done automatically right here right now, eventually. So, but besides for that, for me it's actually quite simple concept. You just watch the CD with Informer, I get to that. You register your endpoints, the read, update, delete and create endpoints. Once the endpoints are read, you do your validation or whatever you need and if you're done, you just place an item in the queue. And if there are any items in the queue, there is a control loop called basically event handler that will take stuff out of a queue and work on that. Now you might wonder if I'm just monitoring the state of the Kubernetes object, why I don't do that myself? Why just I don't in the control loop ignore the Informer and crew handlers and monitor the state? Well, then do that because you can dose the at CD. It's not designed for that. Imagine that your controller will just do a few requests per second and then you scale up the controller and there will be for instance, hundreds controllers doing few requests per second and you have only one at CD there. You will dose it and you will probably clock the internals of Kubernetes. So use Informer. Informer is a smart way to communicate with the at CD because what it does, it basically synchronizes the upstream, the at CD with the downstream your local cache and it does it in a smart way because it shares the objects with all the registered informers and only when there is a change, it will inform you about the change. So the amount of information is minimal. And it does that, inform you about the change by calling the handler. So please use Informers. Again, use the QBird as a reference point because we have them and you can use them. And even if you want to do any advanced stuff, you can override Informer and use it to monitor any other stuff you want. What we do in QBird, we monitor virtual machines state on the hard drive for the file, for specific file changes. If the file changes, we mark change and call register the event. But I know this is quite abstract. So let's go to concrete example. And the concrete example is how actually the virtual machine is created. You place a virtual machine inside the at CD via API or with Qubectl, doesn't matter. Then if the virtual machine is set to be running, a virtual machine controller will notice that up. He will pick up the virtual machine and transform it to virtual machine instance, copy the state and place it in the virtual machine instance and place the virtual machine instance back to at CD. And that's done. He does not do anything more. Then in the at CD is a new object. It's the virtual machine instance. So a new controller is pinked. It's a VMI, virtual machine instance controller. Now he knows there is an object and I need to do something about that. So he take away those objects and transfer and create a pot. Remember I told you that we use pots as a virtual machine box. So he creates that pot with the right hardware request and right specification and place the pot back in the at CD. So now there's a pot in at CD. Can you guess what's next? Well, now scheduler will take its part because scheduler will see a pot without a sign note. He decides where the note, where it should be and place the pot in a note. And then finally, Q-Blood will notice. Take the pot and run it on its note. And ta-da, we have a virtual machine running on this note. But this is mostly complete. I will return that a little on. So now, as I said, we have a virtual controller, we have controllers in a Q-Bird and we have quite few of them. I already told you about two of them. We have three more. There's a replica set controller which manages scaling of your VMI. We have a note controller which is actually important for if you want to check out if your note is unresponsive. If it's unresponsive, he marked the controller mark that note as unresponsive and then we know we need to move the virtual machine somewhere else. And this is done by a migration controller. We have very basic support for migration right now but it is something there. So these are our controllers placed in the Kubernetes. And if you remember the original picture, there is something missing. Can you see that? Yeah, sorry. And I will turn the lights. I forgot about one important thing. Please, don't write controllers from scratch unless you totally have to because, again, there's a lot of boilerplate played the code that has to be written. And there are better ways you can use that. If you want to have fine grain control, fine, write it. But if you only have a simple controller, just use boilerplate code. There are two of them that are quite good. First one is operator SDK by Koros. The second one is by Kubernetes QBuilder. Both of them provide you roughly the same tools. They will basically enable you to build CRDs, custom resource definition, and controllers for them. You're depending on what your use case is. The operator SDK is a little bit more powerful because it was originally designed to be operator framework to handle deployments on Kubernetes, which, by the way, is, again, just a custom resource definition with controller that knows what to do with the custom resource definition. It's nothing else. The last thing that was missing was the virtual handler, right? I'm glad you asked. The virtual handler is our backdoor to the virtual machine. And it's our way, our actually to communicate because on the virtual machine, we occupy, originally we occupy the network. So there is no other way for us to talk to the virtual machine. And therefore, we have a build handler that open up on the node. He sits on each node because it's a demon set. He opened up a socket. And each virtual machine running on that node register for the socket and open up the remote procedure call. And while that, we can start, stop, restart, update, and get information from that virtual machine. Remember when I showed you the picture about creating the virtual machine? This is how the LividXML gets in because the QLED creates an empty box. It will allocate the resources available for the virtual machine, but nothing's there. But once the pot is running, then the visual handler kicks in and plays a LividXML inside. So this is additional way, how you can think about extending Kubernetes. Just think out of a box and figure out that you would probably need controller to do most of the stuff. So now we have the full picture of Hubert Core. Quite cool, right? But I'm reaching the end here. So in summary, what I wanted you to take away is that if you want to add additional kind to your virtual, to your Kubernetes, just use custom resource definition. It's way simpler than aggregated API. To build your controllers that handle the resources, please do use the SDKs available, either operator SDK or either QBuilder. And if you absolutely have to build aggregated API, then again, use APBuilder boilerplate. And for thinking how the Kubernetes actually work, I want you to really think about how the things are happening inside of the Kubernetes. Because inside of the Kubernetes, nothing happens instantaneously. You will create an object and it at some point will be modified or deleted or just updated. And it can be now or it can be in 10 minutes. And if you start thinking about the events and that the actions are not taken immediately, you realize that you have to be as stateless as possible. So when you start building your controllers, please remember be stateless. That's the main point I learned when starting developing for Kubernetes. So well, for me, that's about it. Unless there's one more thing I want to share with you that we don't use in QBuilder, but you can use and build an awesome stuff. And it's virtual Cubelet. And I want to share with you that because it's quite powerful and it allows you to extend Kubernetes in an unthinkable way. What it is, it is a masquerade. It is basically a mock implementation of Cubelet that can run anywhere you want. And what it does is that masquerades the API for the Cubelet. And with the masquerade, you can attach any action you want. More practically, you can register Jenkins as a note. So then you can write a custom resource that will create Jenkins jobs purely on the Kubernetes. Or more practically, you can extend Alibaba Cloud, Azure, or AWS with their container instances to be endless note and offload your containers or any workload there. It is originally written by Microsoft as extension for the Azure cloud. But fortunately, they've written it in a way that it can be extended for anything you want. So just take a look at that and I hope you will be happy using it if you want. And my last thing is if you want to extend Kubernetes, take a look at QBuild because we already had a few limitations on the design and we have to compromise and we have to figure things out how it would work. Because we were out there with the early versions of Kubernetes and we were building on them. So we figured out quite a few solutions that would probably unlocks you and make you think speed, build things at speed. So that's it. Would you have any questions? Yes, what you said was quotas. Yeah, you're asking that if I set any quotas on the host, if they will be transposed to the pods? Well, the thing is if you set the quotas inside of the Kubernetes, then it doesn't do anything on top of that. So it will be handled by scheduler inside of the Kubernetes. So he'll know that he already reached the quota and he will not schedule something there. But if you are setting the quotas on a bare metal, we have no way to know that, okay? Question, okay? Quimo, because I wanted to show you something to scare you. Ah, Libert, yeah, because the thing is, the Libert provides you the abstraction on top of the Quimo. And it can be cumbersome to build full virtualization on the Quimo. You have to build, you have to basically make all switches right. You have to put hot plugs, you have to check out the whole devices. And the Libert actually abstracts that. And on top of that, he handles the monitoring of the status. So just add the Libert, just don't go directly to Quimo. So it simplifies our life. Currently we support only Quimo and there have been no thoughts about upgrading to any other hypervisor yet. Okay, let me repeat the question for the recording. You're asking if we support resource allocations because the virtual machine can grow, right? We support that. You have basically two options. You either doesn't specify that and then you have to live with what you were given. Or you can set resource limits. You can say in the domain specification of the virtual machine, you set your bottom limit and you set the maximum limit, which is your boundaries you want to operate. And then we transpose that to post-specification and Kubernetes has the primitives to work with that. Does it answer your question? Okay, the thing is we are not hiding the resources. We, if you specify the request you want for the hardware, they are directly transposed for the post-specification. So the Kubernetes know exactly what are your hardware requirements. And then again, it's the virtual machine inside. It's just a process. So the Kubernetes knows how high consumption is inside of the node. So he can decide if he can put it there or not. So we are not hiding anything. At the early stages of project, we decided if it can be implemented for pods, just do it for pods and don't do it for virtual machines. Because in the long run, it simplifies our life and does not have, does not add additional complexity for us to think, okay, we have it, but how do we transfer to the Kubernetes? You will not have them. Because you already said, I want two CPUs. I want three gigabytes of RAM. You will just not have more. And for what you would do in, let's say in open virtualization, you can add a hotspot and memory for that. But we don't support that at the current situation. Both of them. You can specify both of them. So you can say this is my bottom limit I have to run and this is the resources, the upper limit. Any other questions? So if not, then thank you for your attention and I give you back few minutes of your time. Thank you very much.