 Hello everyone and welcome to Kubernetes on edge day. The next talk is ours. Obviously that's why we are here. Kubernetes Hunger Games distro performance in the edge. My name is Saiyam Bhattak and I'm a field CTO at SIVO, Reimagining the Broken Cloud. And I'm joined by Vivalat Millisarj. This is a great site to be presenting today. All right. So of course, we know that we are at the edge conference. And then when we talk about standard Kubernetes, while it is great to also work on the edge, a lot of times when you're working in resource constrained environments, your standard Kubernetes distribution can be very complex to set up and also might not work well for edge environments where you're often constrained by the resource and the bandwidth often. So that's why we have a look at the different types of communities distributions. So what exactly are communities distributions? So these are essentially just distributions for your standard communities packets together by different companies. These could be enterprise or even open source. And they do serve different purposes as we kind of explore in more. Now, of course, the major differences that we look at are managed as well as open source. So when it comes to open source distributions, these are often led by communities. And these are free to use with open source licenses that anyone can leverage. So these are great if your team is wanting to self host these distributions and self manage them. But of course that comes with a caveat that often the team will be responsible for upgrading the version of communities in these distributions and will be responsible for setting up all different things like networking that will be supported by these distributions. Whereas with managed communities distributions, these are often designed for more large enterprises because of course these distributions are managed by the providers like 3k or AWS AKS clusters and the responsibility of like managing and upgrading the communities versions. All those are taken care by these companies that are running these managed communities distributions. But of course that comes with the caveat that these can be sometimes be slow in order to upgrade them because that will depend on the company which is running these distributed communities platforms. But today's session is going to be more focused on leveraging open source communities distributions. We'll be taking a look at K3S, K0S and microkits as an example to kind of showcase how you can leverage these distributions for edge workloads. But of course one of the questions that everyone might ask is that why do we need so many distributions? So the simple answer is that there are a number of different use cases for which you might leverage all of these different distributions. Some of them might be focused on running on edge devices. So that's what we are going to be exploring today with the likes of microkits, K3S and K0S and all of them serve different purposes. Of course at the core they are all running communities but they will have support for different CNIs or different storage mechanisms and depending on your use case you might end up using one of them. Of course you'll also have the capability to choose between open source and managed communities distributions. So let's start by first taking a look at K3S. So K3S is an open source community distribution created by SUS labs and it serves as a single binary. Now if you take a look at the architecture of K3S you are primarily having the server node and your agent nodes and both of them kind of work on providing different capabilities. So out of the box it does come with all the core components that you will expect in any community distribution. Now when it comes to the server node that's mainly taking care of your of your communities API controller and all of the scheduler whereas the agent will take care of your things like your cube proxy and of course all of the container CNI related things of course like with the controller with K3 we are using Flannel as our default CNI and then continuity as our default container and time. So that's the first and of course like K3S is really small. So perhaps in comparison that we're running today it has the smallest binary size as compared to even K0S and microkates and that makes it really great for edge workloads. The next one is microkates. This is run by canonical and served as so as compared to K3S and K0S it's not a single binary. In fact like you have to install it via snap and again but the great thing about microkates is that it's actually the fastest single node cluster that you can run in comparison to any other distribution that is out there and then finally we are going to be talking about K0S. I think it's one of the the most newest community distributions that is out there. Similar to K3S it also serves as a single binary that you can install and it is about three to four times larger in terms of binary sizes compared to K3S but it does come with more security add-ons or security benefits as compared to the other community distributions that we have covered today because it's fully compliant with the and we are kind of also explore why is it better. So we'll be exploring those in the latest sites but it also does come with standard distributions like support for rule R back and then open ID and support for being able to actually connect GPUs with NVIDIA graphics as well. So that's a quick overlook on these three different types of distributions that we're going to be taking a look and this is a kind of a feature comparison between the different capabilities distributions. Of course you can take a look at for your reference especially because of the CPU architecture right because when we're comparing different types of community distributions for edge workloads you end up working with a diverse set of device architectures. So choosing the right kind of community distribution for your edge workload depending on the type of device architecture you're working with is really important. And here are some other comparisons primarily with respect to the minimum CPU RAM. Again this information will be useful depending on what kind of use case you're going to be running. Now with the next section same will kind of walk you through our test benchmark around a setup that we used to test the performance and what all different things we compared for our test setup. So over to you Sam. Okay so by now you should have an understanding or at least a brief understanding of what K0S is K3S is and microkits all are the Kubernetes distributions certified CNCF out of which K3S is a CNCF sandbox project the only distribution to be a sandbox project. So in the benchmarks that we are running disclaimer is that this is running on a single node which is one CPU 2 gigs of RAM 20 gigs of storage lesser than Raspberry Pi 4 specs running Ubuntu 20.04 each of them has the base installed like there are different servers obviously different nodes with the same specs but each of them has very minimal single node Kubernetes deployment for microkits K0S and K3S all three of them. And we have choose the benchmark operator. So if you see there is a benchmark operator I'll put the repo as well. So we have run a few tests. The first one is the IPERF test. So in this if you see the YAML file this particular YAML file. So first is the operator deployment. So there is a benchmark operator that is getting deployed and then there are series of tests that can be run against the Kubernetes clusters. So one of the first test that is run is the IPERF testing where it actually creates a client in the server sends the request and it's basically to measure the network throughput and this is how the logs will come out. So it creates a job. It gives you the logs and it gives you certain values. One of the thing that it gives is the average bit rate like you can calculate the average bit rate that refers to the average amount of data transmitted over the network in the given period and then the stability of the network because that's pretty important. This is how the performance benchmark looks like. So the green with the dot is K3S the plain line without anything is microkates and the green with the arrowheads is the K0S. So that's how the benchmarks looks like. What it depicts is the outcome. It's there on the right hand side still the same. So average bit rate for K3S is one gig per second. Then the variability is highly high availability with the spikes up to 2.4, 2.74 coming to microkates. It comes down to 559 and then for K0S it's similar to what microkates is. The next is the sys bench. So again the same operator we are using a different custom resource. So for this particular thing it's basically a sys bench where we are measuring CPU and file IO. CPU is measured how quickly the system can compute prime numbers in this particular case which is the direct indicator of CPU performance. On the file IO side it's using the MTDIR because we haven't specified any volume over here, volume provision, the storage class. So it measures the performance of the system's disk by executing random read write operations. I mean that's how usually the file IO performance would be calculated. But we'll also do a specific, we have also done a specific FIO test using a specific storage class which is same across all the clusters. But for this one this is the sys bench one. So here you have again when it gets completed you get CPU speeds, you get general stats, you get the latency and all these it depicts the events per second. So in this particular case you have the 5535 events per second latency per request 1.79 that is mentioned over there in the average and then general stats is the total time CPU test was approximately 10 seconds. Now again in the same logs this is for file IO that was because we did two. So it created 128 files of this much size that was used and the file operation you can see the reads 92.77 per second the writes 61.84 per second and the sinks then the throughputs and then the latency. So and also the total time. So you also get to see the total time for the number of events that were there. This is how the results look like. Sorry. This is how the results look like. So you can see the CPU performance. This is the k3s one. This is the k0s one and the micro k8s one. So k3s one is faster in the CPU one but for the files you can see k0s overall the file read write operation k0s is exceeding k3s. Yeah. So k3s demonstrates the best CPU performance with the high speed, high write speed indicating robust processing and data handling k0s offers best overall file IO operation in terms of reads, writes, sinks per second micro k8s is a bit on the lower side. So whenever you are choosing a distribution this if this read write operation matters to you or the CPU cycles matter to you then probably this benchmark would help in that. The next benchmark is obviously the stress because it's important to put stress on the clusters. So this particular thing is putting the stress on a cluster with CPU memory and virtual memory running in parallel for 30 seconds and using one single instance because we have it and we have also defined the limits request the limits as well so that it doesn't cross the particular limits because we only had two CPUs for this particular node. So we don't want to cross that so that it crashes. And this is how the results look like for the stress performance comparison in terms of so you can see the k3s is the the light green one with the big one and then is the k0s and the small one becomes the micro k8s. And CPU performance so k3s demonstrates the highest CPU performance with 528 and k0s363 followed by that is micro k8s with 231 and then coming to the virtual memory performance k3s again leads significantly you can see the numbers then k0s and then again on the lower side is micro k8s and then the last section is of the memory copy again k3s all performs the others with 90 then you have 60 still fairly comparable of k0s and micro k8s but k3s is a bit on the higher side. So k3s performs better across the cluster suggesting it may be best for the resource intensive apps k0s is moderate so usual general apps and this particular stress test that we did on the current setup with disclaimer that I mentioned micro k8s is with the lowest performance. The next one is not using the same operator so this is a new tool that is being installed on the Kubernetes cluster. So first thing that has been done is installation so you have three clusters the next thing that is being done is adding the local path provisioner as the storage class now with k3s if you have seen the one of the first slides where Shiva was showing the feature comparison so it had k3s already comes with local path provisioner on the other since we want to keep we wanted to keep everything same so we added local path provisioner as the storage class for micro k8s and local path provisioner as the storage class for k0s after that we had kbench so kbench is a small repository under longhorn under the longhorn which is again a big project for the storage and it defines it calculates three things IOPS bandwidth and latency like how many individual read write operations basically higher would be the better number here then the bandwidth total amount of data that can be read write to from the storage system again this one is higher the better last one is latency delay before the transfer begins this one is lower the better so like this we have done the benchmark on all the three again and again the logs this is how when it finishes you get the logs from the benchmark summary you have the IOPS it also do random read write and then sequential read write so you have random measures and then you have sequential same for the bandwidth same for the latency this is how your benchmarks look like for the IOPS part so in this particular thing IOPS read k0s is greater than k3s then micro k8s in the IOPS write micro k8s is greater than the k0s and then the k3s coming to our results for the bandwidth micro k8s read bandwidth is for micro k8s is better than k3s okay i don't know why i put two but yeah it's better than k3s and then k0s and then bandwidth for write k3s is greater than the micro k8s and then k0s the last one is the latency so remember the latency lower is better so although k3s is greater than k0s than micro k8s but micro k8s been over here because in latency lower is better and both for read and write now another thing that that i just did randomly was the free hyphen m command after installing all the three interestingly micro k8s has the lowest footprint in terms of the memory consumption just as a plain cluster like just if you install micro k8s on a same spec vm and then you install k0s and k3s so slightly lower micro k8s and then k0s and k3s is on the higher side now this is all good these are still bigger devices like Raspberry Pi 4s are coming up with you know 8 gigs of RAM these are not now smaller devices even the nucs you still can have big clusters um on these devices even you actually can do a simple cube adm kind of cluster on these devices but what about the industrial iot devices industrial iot devices are even much smaller they don't have uh gigabytes of RAM so they cannot afford even the 500 um RAM consumption that i showed in the in the one of the previous slides so there is one new thing which is out there i thought i would mention it in this particular session as well which is k2d it is not a distro so it is not a distribution it is just uh you can call it a translator for the kubernetes apis to the docker commands so it's no operator there no kubernetes components um for example there is no operator support no rback support no service account support um it is it's just uh docker with the kubernetes api translator so what it can do is it it just takes 20 megabytes of overhead for a particular node and it actually behaves as a node so you can either use direct docker commands on that particular node or you can communicate with the api translator provided by k2d to translate your specs from the pod to the workloads that gets deployed as containers on this particular node where you have k2d it's it's kind of interesting because it's not a distro it's not something everything that you can do on a kubernetes cluster you will be able to do it but it still serves a kind of purpose on very small devices um and it gives you the feel of kubernetes another thing i wanted to mention over here is talos um people do count it in the kubernetes distributions but it's basically a kubernetes native operating system so for example people have been deploying you know they're highly available kubernetes clusters on the operating systems which are way older and not meant for kubernetes because kubernetes was not invented by that time so andrew one of my friends at sidero labs around 2015-16 time frame he decided to kind of write an operating system natively just to run kubernetes from scratch so this is actually an os written from scratch highly secure widely used in production uh it has no ssh the same design principles of you write a custom resource file the yaml file spec and that can be translated towards whatever you want to do and how you want to spin up your kubernetes cluster so all api driven image based no ssh every user is rback based and atomic upgrades in place so if it's upgraded it's done not like some of the binaries are upgraded it fails in the middle and upgrades are an issue uh some of the features declarating yaml file you can choose your cni apply yaml um most like 90 of the directories are read only directories there are specific directories for specific use cases for example your storage class and stuff where you have ride only access to certain drive drives and also you have extensions for example you need a different container runtime all together like g visor or you want to run web assembly workloads um so you can have an extension for that i created a vasame edge extension um so that can be done via the extensions but as a os it's you can not do much these are the set of 12 i think 12 binaries one of the latest blogs from them 12 binaries just needed to run kubernetes because you don't want all the stuff in the operating system which is there to run kubernetes coming to the final piece uh cube edge um you want to take so of course today we kind of covered what are the different type of distributions and comparing the performance but of course we also have cube edge as a project which not necessarily a distribution but it comes integrated with kubernetes to help you to adopt a lot of the functionalities that you'll get uh instead of a native implementation for running it on uh edge workloads so i'll definitely recommend you to also take a look a lot of times you might want to see like running more kubernetes or edge friendly distribution versus also running cube edge but of course uh just one final slide around um when to use what so this is the speaker opinion that you wanted to share today in terms of like simplicity of usage uh we recommend micro kits just because the ability to very easily like do upgrades and also like it gets additional support for gpus and all uh then binary size as we mentioned like k3 is probably one of the smallest ones out there because of its smallest binary size uh in terms of dashboards you get built-in dashboards for micro kits and k0s but not for k3s so if uh and for example for default storage uh k3s comes with default storage but it's not supported directly for uh k0s and micro kits uh and i think in terms of security so i think uh k0s does uh rarely win over here because of course it comes with 100-person FIPS compliance as well and finally uh machine learning because we know like today for edge computing machine learning is very huge while uh all different kind of kubernetes distributions uh can work well you might want to consider the ones that have native support for gpu so for example k0s k3s and micro kits all might have some sort of support for uh for gpus but of course when it comes to like micro kits it has inbuilt support for nvidia drivers uh for you know uh dynamic resource allocation but also uh it has really nice support for charmed kubeflow so if you're running machine learning orchestration then both k3s with its service machine learning workloads capability and with uh for you know like for running kubeflow uh as well as with micro kits with the special version of kubeflow that's charmed kubeflow that's usually a bit smaller as compared to your regular in a kubeflow that you might ship uh that might be a great workflow for you to run so if your if any of your companies is running machine learning workloads then that's a great way to kind of compare between these three and then use but finally uh thanks for uh you know what my presentation just one thing want to add quickly add over the security aspect um like talos is the by far best in terms of kubernetes distribution like not the distribution that you can call a distribution and a native os but yeah talos kubernetes clusters would be best for the securities k0s has very interesting thing which is control plane isolation so it gives you a managed kubernetes field where you just do kubectl get nodes and you do not get the control plane visible over there so you can have only the visible nodes would be where you actually run your workload so it gives you the feel of uh managed distros where you usually do not see the control plane uh nodes um also yeah do share your feedback uh for this particular session um and so today we we covered the kubernetes distributions microkates k0s k3s the theory part what are they why distribution exists uh then some of the managed kubernetes distros as well and then we ran the benchmarks which are the benchmark operator by cloud bulldozer i think um benchmark yes cloud bulldozer they have this particular um repository called benchmark operator it has wide variety of benchmarks that you can do um for all the things so we did stress and we did iper we also did uh one of the k bench for fio testing uh all the fancy graphs and then um some of the conclusions so yeah hope hope this helps and yeah i can also show you how they were done uh yeah this one yeah i'm not doing any demo actually so you can just see all the kubernetes version were on 1.28 so it's very difficult to move the mouse around like this so for example here as well you can see kubernetes 1.2a.7 so this one is the k0s this is k3s and uh this one is the micro k it's all the same clusters uh similar vms and the benchmarks that have been done uh results can vary across different environments and stuff would be happy to chat if you are willing to provide some of your inputs on where we can test these how we can you know uh benchmark in a different way as well thank you so much and thanks for coming to our talk thank you