 So, hi. This talk is going to be about monitoring converging infrastructure containing and running together containers and virtual machines together. The main focus here will be about metric collection because I can't cover all of it. So, let's start. My name is Yannir Quinn. I'm working in Red Hat since 2016 and I'm a part of the Acuber team. So, we'll briefly talk about Kubernetes and metrics collection in Kubernetes. The main sources, the key sources for metrics collection. We'll see how it comes together with Formiteus to provide us a full monitoring pipeline solution. Having covered that, we want to throw in virtual machines into the mix. So, we'll briefly look what Qvert is that brings virtual machines on top of Kubernetes. See from past experience what we have collected for virtual machine and matrixes and see how it all comes together eventually. So, we have Kubernetes as a start as our container-centric platform orchestrator that orchestrates computational nodes and pods, network, storage database, all of components that you would like to monitor in your system. And the second building block here is Prometheus. Prometheus was originated in SoundCloud in 2012. Now, it's an open-source community. It's a monitoring and alert team system. It's a multi-dimensional and label-based model. It's based on a time series data which is comprised of key value pairs. It has a very strong query language, PROMQL, that analyzes that time series data. And you can see it really integrates in a natively almost way with Kubernetes. Some of the Kubernetes components already have integrated with Formiteus. A quick view on the model. So, in the middle, we have the Prometheus server that scrapes and pulls data through HTTP endpoints into its server, collects and aggregates that data. On that data, you can perform a PROMQL series in order to analyze it and move it on, aggregate the data, and even create the alert team. Alert team are pushed to the alert manager and to its clients. And you can also use Grafana or other API clients to visualize that data. But again, our main focus here is metrics collection. So, because Prometheus needs as much as auto-discovery as it can get, it needs to scrape data, pull data from metrics endpoints, HTTP endpoints. For that, it's very easy to integrate with Kubernetes since it exposes these endpoints on APIs, on HTTP. As I said, some of the metrics are already exposed through the metrics endpoint, and some will need to be added and collected. So, I'll go over two or three sources of metrics in Kubernetes and the Kubernetes architecture. The first one of them is the host metrics, Kubernetes nodes. So, Kubernetes nodes didn't have natively exposing metrics endpoints. For that, they had to write an exporter. An exporter is Prometheus exporter is a service that collects all of your data from your node, for example, or any other data you need to expose from your application, et cetera, exposes them on the Prometheus endpoint. So, Prometheus can come, discover it, and scrape them out. So, node metrics are scraped from Proc, and you get all your familiar host statistics such as Node CPU. Node CPU, for example, has like 15 dimensions of metrics type for CPU, like idle, IO8, but if you want to analyze, for example, how much time the CPU has spent on a node or per CPU core, you can perform a Promet QL query and remove these and get your Node CPU statistics. There are, of course, memory, memory available. You can also perform queries against the total memory left. There are more advanced metrics than only Node statistic. If you look at it in a Kubernetes kind of way, for that, we have another component called CubeStateMetrics, which looks at it from the Kubernetes way and also takes into account memory limits and CPU limits. Moving on. So, in our host, we have containers, and containers' metrics are exposed through Cadvisor. Cadvisor provides you data about containers' runtimes such as Docker or Cryo. It is embedded in the Cubelet. So, Cubelet natively exposes an endpoint to slash metrics, which is a port for Prometheus, so we don't need to add additional exporters to collect that data. So, you analyze these metrics and resource users' usage and expose them on the metrics. These are what you also call core metrics for each container. And, again, much like Node, just in a container view, you have your original, your CPU metrics and memory. Actually, this matrix is very advanced. In the nano-signal, it gives you actually the exact memory usage. You have left, taken into consideration all other metrics you have under your container. And, of course, memory transmission. Another part, let's look a bit on the orchestration parts. So, I'll talk about only API server here. So, because the API server is the front-end for Kubernetes cluster and gets all the calls and relays them to ETCD and controller, it's an important component to monitor. For example, you want to create a pod or run a deployment. You want to see, and it's all done through REST API calls. So, you want to see the request rates and latency. How long did it take for me to complete a request to create such a deployment, for example, or how long did it take, even counting the amount of that. And let's say you want to create a thousand deployments on your cluster. When you create a deployment, it goes on to work use for controllers. It accumulates in cache the request for creating objects or deployments. And if your queue gets accumulated and gets stacked, you want to detect that bottleneck. So, it's a very important aspect to monitor. Because Kubernetes is based on goals, so you get out of the box, goal and statistics, garbage collection, memories, threads, et cetera. And you also monitor the server's process itself. So, that was only one component. Of course, you have metrics collection for scheduler, the Kubernetes controller, ETCD, but I won't get into those. Now, there's an add-on called kubestate metrics. This add-on you need to deploy as an instance, only one instance in your Kubernetes cluster and it relates more to not health or utilization. It focuses on counts and status. So, for example, you want to check what status is your pod in in terms of scheduling or health, not health, the phase it's in. You have the concept of CPU limits or memory limits. We won't allow a pod to run containers without any boundaries. We don't want to exhaust all of them. So, I want to check statistics, how much memory you have been according to your limits or provide even more advanced queries on these statistics. There are a thousand of examples in kubestate metrics. I'll talk about, for example, replica sets. You have defined a replica set. You want to run a number of deployments on your cluster and you want to make sure that these number of deployments are running in your cluster. So, you want to see if they are not available, how many are there in the cluster and so on. So, that was very briefly about Kubernetes components and sources for metrics collection. And now, we want to add virtual machines into the mix. So, kubert. kubert comes to close the gap between virtual machines and containers under the same hood, under Conveld's infrastructure. We want to run also virtual machines, also containers in the same platform. And we ask ourselves, why? So, this is a great allegory. Containers are considered to be lightweight. They're fast to bring up. They have native performance. But virtual machines, with their bulky, heavy utility, how well built, are not going to disappear. We will have them in the next 5 or 10 years at least and we'll still want to maintain them. You can look at an example for mainframes. It's taken a long time to phase out mainframes. They still even exist. So, what can I do with it? First of all, we want to bring the legacy. We want to run it under the same hood. We don't want to maintain two infrastructures. So, we're going to mix and match and add virtualization on top of Kubernetes, thus also getting all the Kubernetes advantages, such as scheduling, the storage, networking. We have some addition and fixes to manage to do that. We also, in the use of Kubert and running virtual machines on the Kubernetes cluster, we can also enable, and run any OS we want to run on top of Kubernetes. So, this is Kubernetes and we added to the mix some virtual machine components. Again, I won't go too deep into the components. So, on the left, you can see the node. In the node, we have a pod which runs a VM. Virtual machines run on pods and you have also side containers on that pod to enable virtual machines such as LiveVirt. You also have key components, very resembling Kubernetes components, like VIRT API, same as API Server to create your virtual machine CRDs, pass them on to UTCD, having the VIRT controller identifying that, creating a pod, running a virtual machine on a pod and also scheduling it to a node. Once it's scheduled to a node, the virtual machine gets the node assigned and then all the handling of the VM life cycles moves to the virtual handler. Again, on top of the fork. So, of course, I added something new. I have monitoring on Kubernetes. I want to also monitor virtual machines. So, I want to figure out first what to collect. So, a quick example from pure virtualization, Overt. Overt uses the virtual machine to collect it to make tricks. It has powerful plugins, a lot of plugins you can add to it and run it. It gets you host statistics, database statistics, engine stats and more important for us right now it's virtual machine statistics. So, for virtual machine statistics, we have a VIRT plugin. VIRT plugin communicates with LiveVirt and LiveVirt gives us the information about the virtual machine. So, of course, you will introduce on here some new aspects, not necessarily all will be monitored and adapted to Kubernetes, but virtual CPU, how much time my guest was running on a CPU. Memory, some specific related memory matrix like ballooning or overcommitment, memory management matrix, disk reads and writes, flashes for disk, and also network aspects. Some of them might merge from the benefit of Kubernetes but we might want to adjust them to virtual machines but some will need to add. One strong example I can give you that Kubernetes has a scheduler, a very strong scheduler but it doesn't have 80 scheduling policies we have in virtualization and we'll probably add these policies also to the scheduler. So, we want to see why our cluster behaves in such scheduling matter like let's say high availability, HA reservation and we also want to monitor that. Maybe we'll decide in the future to change the scheduling policy. So, it's a very important aspect. So, we're consisting of similarities and adding virtual machine statistics. So, coming back again to Prometheus and Kubernetes, we had our collection, node exporters, cube components, cube set metrics and of course, we're missing the collection for virtual machine. Virtual machines and Kubernetes just came in so we need to write a custom exporter very much like node exporter. So, for that we need Prometheus exporters. Writing an exporter could be done in Prometheus client library. For us it was very easy, it's mainly in Go, so it's very native to write it. So, that exporter will collect your data from the app matrix. It will expose it on matrix endpoint from Kubernetes, slash metrics and it will be available to scraped by Prometheus server. So, we'll add custom exporters for our virtual machines and we'll now start pooling other components. So, we have the Kubernetes components and now we talked about virtual machine matrix. We talked about components for orchestrating the virtual machine such as the VIRT API, the VIRT controller. We have also processes that running inside a pod as I said, for enabling virtualization such as library and QMU. I want to complete the picture in a small note about the whole cluster. In order to deploy Prometheus servers on a Kubernetes cluster, we can use very easily Prometheus operator. It has two main functionalities. It's first of all to deploy and manage the Prometheus server. You get also alert manager. You can get also node exporters or other sponsors you may define and also important to watch your components on the cluster. So, we added virtual machine and in order to watch them, we have a Prometheus operator, we have an entity called a service monitor. You define the port, you define the objects you want to monitor. You add the corresponding labels for that matter and if you now create new CRDs, it detects that and starts scraping these metrics also for virtual machines in our case. So, QVIT metrics. So, I'm taking you back. So, we looked at metrics we collect for Kubernetes. Some came from the Qubelet. Some came from node and components. ETCD, scheduler and controllers. And now adding to our mix. We add the VR controller and we have the VRT API and the VRT handler. And I told you it's very resembling the Kubernetes components. So, if we look back at the API server, we talked about it's communicating with controller walk use and performance. It has native statistic about Golang and it also handles the REST API requests and measures them. So, very similar. We have the VRT, VRT controller walk use. We want to go over that and get statistics about that. We have Q latencies, work duration, number of time, also the caches that comes from Gol, REST API calls. Again, very similar to the API server. And now I want to focus specifically on virtual machine statistics. So, the previous were general components in a Kubernetes cluster. Here you drill down into the virtual machine statistics and matrixes. So, two major parts. Virtual machine matrix, which we saw how and what we collect in OVIRT, in a pure virtualization environment. And processes. So, for process service, I'll say it's more for the point, the developer utility less for real cluster management and observability. And these processes, for example, are also scraped from PROC like another exporter and we get metrics about them. So, some important information about VM metrics. We can see the four pillars of metrics we collected in OVIRT. And usually these pillars are repeating themselves in all of the metrics. These are not the only metrics we'll collect, but these are the most prominent examples. So, for example, VM CPU or CPU time. I want to see my guest CPU how much time they use. Memory, for example, you want to create a metric that is composed for several matrixes. How much memory we have on set. How much actual ballooning was left for the VM itself. RSS, the amount of it. All compiling and coming down to completing a collect matrix. And also about network. On the bottom, you can see the pod info matrix, which I talked about processes themselves. And you can see a full form of metric, how it looks in a Prometheus exposed format. You have the name of the metric himself, itself, which domain or for our matter, what VM does it monitor. You can add a host or other labels. For that matter also, we added the process name and the type of matrix. This applies to all metrics time, all metrics time, time series, format we want to show here. So this is an example of how it will look on a Grafana dashboard. Of course, you can also use SkiBana for that matter or other tools for a very simple example for a VM running on a pod is CPU usage and memory usage. Okay, so in conclusion, that was a very fast talk about some concepts, but the main take here is that we have Kubernetes and Prometheus. We have this whole platform and framework and solution existing. We have very easy extensibility for that. So for us, by adding a new add-on, Kuvert and virtualization into the mix, monitoring that was kind of native and very supportable. And for us, we got a complete monitoring solution and it's continuing to grow as we speak for a convergent infrastructure. But it shouldn't be just only for virtual machine. Everybody can take it for himself and if he adds new components on top of Kubernetes or add-ons or side application, we can monitor that using the concepts of Prometheus exporters and other many notions.