 Hey folks, welcome. Come on in. Happy Friday. We got one more minute. Okay, two o'clock exactly. Welcome to Sharpen the Edge with K3S and Containerize Operating Systems. I am Ray Lajano. I work for SUSE professionally, but I'm also an upstream Kubernetes contributor. I am the SIGDocs co-chair, so I help maintain Kubernetes.io with the upstream documentation. I've been a member of seven release teams. Some of the highlights was I was the release lead for 1.23, I was the emergency advisor for 1.25 that removed pod security policies. I'm also a sub-project lead for SIG Security. I helped run the external security audit that was published on Wednesday, so please take a look at that. I have links here and these slides are updated and uploaded as well. But today we're going to start off talking about what is the Edge? The Edge, there's many definitions for it. It could be a browser, it could be a YouTube guitarist as well, and if you ask multiple organizations, you will get multiple answers. A grocery store company might consider each location as an edge site, but a bank may not consider each branch location an edge site, but only ATM machines not in a branch as an edge site. We actually asked folks and organizations what they define as the Edge. We got several answers from that survey. It could be any remote site or device that processes data and connected by a network. Some said it's any place that's not part of the organization's core network or DC or cloud, or it could be any remote device that can sense, infer, and act by itself. And there are many more different answers as well. So, but when I actually googled what is Edge competing, there's also multiple answers, but there is a trend. I'm not going to go through the list, but there's four common traits. One, it's a remote location away from the DC. There's some sort of compute and processing involved, and there's a network connection. The Linux foundation has a state of the edge at LFedge.org. Please take a look at that. It's a vendor neutral platform for research on edge competing. So, there's challenges to Edge no matter what or how people define Edge's locations. One is resource. So, there's resource constraints. So, you might have small devices. This may not be totally true for every organization, like if an organization actually ships out hardware to host the center on. That wouldn't be a very resource constraint, but it might be. There's physical restraints. So, there might be difficult locations or there might be hundreds or thousands of devices, and it's difficult to have onsite technicians, or it's difficult to send onsite technicians to those locations. Sometimes, and so with these resource constraints and these physical constraints, we need flexibility. Some of the other challenges is provisioning and onboarding. How do we manage dozens, hundreds, and thousands of devices? So, this is a challenge of many assets. How do we do life cycles things? How do we patch upgrade the OS? How do we upgrade Kubernetes on these devices? It's easy to do on a dozen. It might be easy to do on a few dozen, but when you get to hundreds and even thousands, that becomes very difficult. How do we design this for scale? It could be a small scale, like a local Edge network, or it could be a large scale. It could be global. So, why are we talking about Edge at KubeCon and CloudNativeCon? Because, folks, we are bringing CloudNative principles to the Edge because of their benefits. We can plug in a device and turn it on, and everything should be automated to onboard. So, ideally, we could take machines and turn them into Kubernetes clusters, have them be consistent to Kubernetes clusters, the versions we want as well. How do we manage the OS life cycle in a declarative way? And we could also tie into our automated pipelines, GitOps to deploy our applications across these hundreds or thousands of devices that become hundreds and thousands of clusters. Lastly, we're bringing DevOps and containers and Kubernetes to the Edge. So, this exposes the need for operations and developer automation to be streamlined. So, now we have this Edge puzzle and how to be solved this Edge puzzle. We need to look at the Kubernetes distro for resource constrained environments. We should also take a look at the OS for resource constrained environments. How do we have a minimal OS? At this point, I'll point out later on that everything besides the application is overhead, including the operating system. How do we gain operational efficiency? How do you remove the need of sending on-site technicians? And how do we support existing Edge infrastructure with Cognitive? So, at this point on, we're going to take a look at each piece of this Edge puzzle to solve this Edge issue. First, we're going to take a look at the Kubernetes distro. So, K3S is a CMCIP project. It's in the sandbox maturity level. It was donated by Rancher Labs, I believe in 2020 or as accepted in 2020. There are other Edge flavors of Kubernetes as well. It's not just K3S. Do you want to point that out? There's Kube Edge. There's MicroKates. But for this talk, since this is also a CMCIP project, we're going to focus on K3S. And K3S is purposely built for resource constrained environments. So, you need one CPU and 500 megs of memory. So, K3S is a fully CMCIP certified Kubernetes distribution. It does not need HCD, but what we like to say is batteries included. So, you don't need to install a container runtime. It already installs container deep and run scene. If you ever used Kube ADM recently with Docker Shim removed, you've had to configure the config.toml. So, it gets kind of more complicated now when you use upstream Kubernetes. K3S includes a CNI, but you can use any CNI, Flannel, the default CNI. Of course, it comes with Core DNS, metric server as well. It also comes with a local storage provider. If you need service load balancing, it comes with a project called Clip for LB, which is for service load balancing. So, it opens low ports on nodes and routes traffic back to the service and responds to load balancing service requests. And also, because it uses Flannel, there's also a network policy controller. So, if you do use a CNI that does employ network policies, you would just have to disable the network policy controller. It also includes some host utilities like IP tables as well. And you don't need to use SCD. It comes with kind, which is an SCD shim. So, by default, K3S uses SQLite, but you can also use Postgres, MySQL, NATStressstream, and of course, SCD as well. And there's a one-line install. So, I like to use this one-liner as well. If you go to the Quickstart guide, it gives you just a curl command, but I like to add in that K3S keep config mode, so you can do the keep config without using the K3S binary you like. So, this is just the architecture. I'm not going to go into great detail with the architecture for K3S. In K3S, we talk of the K3S server as like the control, as like the control plane components of Kubernetes. And the K3S agents is like the worker nodes of Kubernetes as well. So, now, we talked about a Kubernetes distribution for the edge. So, we need an operating system for the edge. So, let's take a look at containerized operating systems. And we already have containerized operating system images already, like Tumbleweed, SusanLinux Enterprise, Ubuntu, Alpine, but we can't really use these because we need something bootable in bare metal and IoT devices and virtual machines. So, how do we make a bootable container operating system? And like I said before, if we consider at the edge devices or at the edge level, we only care about the application and everything below the application is overhead, including the operating system. So, how do we make a bootable containerized OS? So, we have a COS toolkit, which we have renamed from COS toolkit to the Elemental toolkit. And this toolkit includes the runtime and build time framework. So, you could boot containers, in virtual machines, in cloud and the hyperscalers, bare metal as well. And we'll take a semi-deep look into this as well. It's based on LUITS, which is a container-based package manager as well and has some utilities, like creating ISOs from a container image. And this Elemental toolkit also includes a kernel and C-groups as well. And also, all the common drivers you need for your OS. The Elemental toolkit is actually part of the Elemental software stack. And the Elemental software stack, we have it which enables centralized, cloud-native OS management with Kubernetes. But for the scope of this talk, we're strictly going to stay at the Elemental toolkit level. So, we're going to create a bootable OS container image. We're going to run it on bare metal, cloud, and virtual machines. Like I said before, we're going to use the Elemental toolkit. The great thing with the Elemental toolkit is that we can also now embed K3S as well. The Elemental toolkit, I can mention, can be used to create ISOs, raw images, OVAs for vSphere, cloud images with Packer, iPixie, Vagrant, Kikau. You can also customize it with clouding it as well. It does A-B upgrades. It's immutable as well. And like I mentioned, you can embed K3S. We're going to take a look a little bit more deeply on how we do that. It also includes Grub as well. It's for the bootloader. It uses Dracut to create the initial image used by the kernel to preload the block device modules so we can access that root file system. So this is just an example if you want to do customizations. If you have cloud init files and specific directories, they can be applied in five different phases, boot, network, init, ram, FS, and recon cell. And you could also just use one file for all the different phases. But for this talk, we're going to focus on just building the container and also building the ISO. And there are some vanilla images you see here on the left side. We have three that's offered, actually four, technically four, but we're going to focus on three. We have a color scheme to them. So we call green for open SUSE, blue for Fedora, and orange for Ubuntu. Don't know why it's not red for Fedora, but it is blue for Fedora. And like I said, for the rest of the talk, we're going to focus on building the container, the bootable container OS, and then building the ISO. So using the elemental tool kit, which uses LUIT, to make a bootable containerized operating system container image, we use a Docker file. So this is an example of that Docker file. And this is all in the documentation as well. But I just highlighted a few things in green here. The rest of this that's not green is just setting some default layouts. What's highlighted in green is just the base container image that we're going to use. So we're going to use a SUSE Linux Enterprise base container image. This could be open SUSE tumbleweed. This could be Ubuntu. This could be Fedora. Also, we're setting an argument to install K3S. That's what you see here in the two lines of green. In the other line of green here, we're setting up the K3S, we're installing K3S server as well. It also creates a K3S agent, which is the worker node and variety of K3S. So in this Docker file, we're setting some default layouts and installing a specific version of K3S. In this case, it's 124.10. And at the end of the Docker file, we're going to copy over custom CloudInit file. This CloudInit file is going to be very large, so it's going to take a few slides, but I'm not going to go in great detail of the CloudInit file. I'm just going to highlight a few things in the CloudInit file. So in the CloudInit file here, where it's besides just setting up this container OS image here, we're setting up a user array and it's setting up and attaching an SSH key as well for login, setting up a default password, which is optional. This is more of the CloudInit file, just setting up this bootable container operating system. This example here, in this CloudInit file, it's just starting up K3S when the VM or Edge IoT device or bare metal machine starts up, it's going to start up K3S. We have repositories, the AMOLED file as well, just to set up the repositories. So now we have this Docker file that we've set up, CloudInit file as well for any customizations. So we're going to create this container image of a bootable containerized OS, and we're simply going to use Docker build. So we're going to use Docker build with the Docker file, give it with the tag mylinxderative 1.2. So now we have a container image and it is a bootable containerized OS. So with this container image, we could use binary within it to actually create an ISO of this, the same bootable containerized OS. So to do that, we do need an ISO YAML as well. We're just going to give it a few metadata for the ISO. And to do that, we just do a Docker run of the same container image that we created using the mylinxderative with the tag 1.2. We're using the LuetMakeISO binary to create an ISO. So in our directory now, we should have, after this process is done, we have an ISO. So this, just to summarize the provisioning of this, so now we've built an onboarding image. We could deploy it in a few ways. And this way, we have an ISO that we could onboard to a machine. It could also mention that you could use KUKAO, you could use OVA, cloud images as well. So the remote images boots the onboarding image in the matter of what flavor you'd like. And the cluster is now created. And the cluster is ready. So the idea of taking an edge device, it could be on a factory floor, or it could be a small nut or a raspberry pi, or it could be a small machine to have it baked with an ISO, or if it's a VM and OVA, you would have it with that bootable container image OS with an embedded K3S. And you have a Kubernetes cluster. So now you have a cluster. How do we do life cycle things with that cluster? How do we do patching? How do we do upgrades? And we want to keep things Kubernetes native. So how about we have a controller with privileges so that it can run node upgrades. So we have what's called a system update controller, which is a Kubernetes controller that can upgrade the node. And we've done a CNCF webinar on the system upgrade controller already. There's a link in the slide deck for that as well. And the way this controller works, like I said, is a very privileged container. It actually runs as a job to pod to container, has access to very specific namespaces to host IPC, net PID, has a links capability for capsis boots, has access to the root file system as well. And what this introduces, it introduces a CRD called plan. And this controller manages plans and it selects nodes to run upgrade jobs by using on a plan, which defines which nodes are eligible. But in edge use cases, it's probably going to be all those nodes. And it's going to do the upgrade. It's going to select nodes with a label selector. And when the job has run to completion, the controller will label the node on which it ran the plan that it's done its upgrade. So this is an example of a plan. And in this version, which I've highlighted, was the upgrade, just a simple few lines here just to upgrade using the next, the bootable container OS image that we've created in the previous steps with the my Linux derivative tag 1.2. So we can upgrade in a few ways with our edge devices. We could do it automatically or programmatically with this system upgrade controller. We could also do it via ISO. And we could also do it with the elemental CLI, which I didn't talk about, but it's not going to be in scope of this talk. So on installation, there's two image file states that's in COS states. One, it's used where elemental boots from and one is reserved for in case that we need to fall back. And so this upgrade mechanism, it helps to prepare, install, and a new element or a new system when we've upgraded. So this management OS management workflow works like, so now we've created a new OS, a bootable container OS image. It's push, we push it to your registry. The OS image aspect is updated for that target cluster. That's that plan, that plan manifest now. You can also operate it with GitOps. So you could have that plan manifest in a Git repository, like you can use Argo, he's fleet, he's flux. You could deploy that plan manifest into your cluster. I didn't do want to make a note when you do use the elemental toolkit and you do embed K3S. It does include the system update controller already installed for you. So that using GitOps or revise using GitOps, you would deploy that new plan to upgrade your system using system update controller. That's going to start the rolling upgrade of your edge cluster or of your edge device. Then it's going to create those alternate partitions on the nodes and update it with a new image. It's going to restart that node and your Kubernetes cluster state is preserved on a persisted partition. And your cluster is ready. So now we've gone through the Kubernetes distribution for the edge, the OS or how to make a containerized OS and how to manage life cycle of that containerized OS. But now we need to talk about edge infrastructure. So edge infrastructure means that there could be devices that applications on Kubernetes need to access. So sometimes summer applications need to access cameras, GPUs, sensors on cities. These are what we call leaf devices. So to do that, we actually use another CNCF project, project ACRI. And ACRI lets you expose those devices such as the IP, like the IP cameras, sensors as resources in the Kubernetes cluster. So simply put, ACRI finds those devices and you could use it. So solving the edge puzzle. Now we have this stack where we have a layer. It could be the hardware layer where we have the bare battle machine or edge IoT device or it could be a virtual machine as well. We are on top of that. We have the OS layer where the elemental based Linux derivative and using components like the elemental toolkits, which we use to create this bootable container OS image. And we use a system update controller to manage the life cycle of patching and upgrades of both the OS and Kubernetes itself. We have a way to manage the life cycle of this device and the Kubernetes layer with K3S. And this could be in any edge sites across the globe. And I want to thank you very much. That's the end of my talk. I'm here for any questions. I have a few minutes before I actually have a back to back talks. So feel free if you have any questions. Come on up. There's a mic. I think there's mics here. Is it working? Yes. Yeah. Okay. My name is Cesar. I'm from Wayfair, the German branch of Wayfair. But I'm using K3S mostly for my personal computing and for small projects. It was kind of, yeah, everything is great about K3S. I love it. And even for small setups that are set to be growing in the future. It's good to use K3S as a first stage. And I wonder if there is a somewhat better control of firewall. Because when I've exposed one of the node ports, it got assigned to the 0000 address, which is like broadcast. Can I somewhat have fine grain control over firewall? You can, but can you repeat what was assigned to the loopback? But you can, but it's going to be more of the OS level layer. I mean, in terms of K3S itself, since it's CNSF certified Kubernetes, you would do firewall things like Kubernetes. So it will be network policies. It would just be more very strict network policies. It might be a different CNIs that you would use as well that would control, but so there is no command line flag that I can pass over environment viable to the binary itself. In terms of what do you want to do with a? Just limit the node port exposure to some local IP address. Oh, yeah, you can set the node port. You can do that with the node ports as well, but it's, yeah, there is a flag. I just don't know what that flag is. The important thing is to know that there is a flag. Yes. Yeah. So thanks a lot. Okay. Hi, thank you for presentation. I have a two quick question. You mentioned Docker. Can we use podman build that instead of Docker to build the same sort of thing? Great question. Right now I just Docker, but we are moving towards using podman using nerdy CTL. Yeah, there's for some, there's, there's reasons under the covers, but right now for elemental toolkits is it is just Docker right now, but we do plan very, very soon actually to be using a podman or nerdy CTL and for that to be, you know, to not matter which, which, which container build, builder are going to use. Yeah. And my second question is sort of the same. You mentioned cloud init. Will you support ignition files and maybe immutable OSCs in the future? Maybe in the future, but right now just hot in it. Okay. Thank you for your answers. Hello. Thank you for your talk. Nice work. The question is you're creating a bootable container OS and you're concerned moving it into ISO file and then boot this ISO. My question is, can we have instead of one container having several container and having a several bootable, you know, OS that at the same time, they spin off together. So it is possible right now with the case. Yes, absolutely. So with, with elemental toolkits, the OCI registry becomes this source of truth for your, of your OS versions. So you could have multiple versions. You could see where you could fall back. You could do a rollback using the prior version in case, you know, they're the patch didn't really, there's something wrong. There's a regression in the next, in the next container image level. So yes, you could do multiple versions as well. There is some get ops tools. I know fleet does this. You could target specific clusters. So what I would do with, I would use a system upgrade controller to target or and use get ops tool, Argo, anything fleet to, to, to target specific clusters with the, with the plan manifest with your chart, with your upgraded OS tag. And you could manage upgrades. You could just target only in like clusters in this region will be upgraded clusters and this type of building will be upgraded. This cluster with this type of devices attached to a leaf device will be upgraded. So you could have different, you could target those separately at scale with using those get ops tools as well. So thank you. Thank you. Hi, I have a more of a technical question. One thing about containers is that they're using the hosts kernel and kernel version. But in this case, we don't have hosts that we're basically building the image, which is going to be the host. Where is the kernel being taken from? Is it? The elemental toolkit includes, includes the, includes the kernel. So if we go to the, the GitHub repo for elemental toolkit on your packages, there's a slash kernel. Oh, all right. Because there might be a case like for instance, if somebody was going to go with the image for sentos, whatever reason, the sentence is using pretty old version of, of a Linux kernel. I think it's like three 10 with a lot of patches. So is it also modifiable to certain limit? Cause like, like I mentioned, there's only like four flavors of Linux distros that you could actually use elemental toolkit now. So like Ubuntu, Fedora, open Sousa, and we have compatibility with those kernels that's packaged and came from the little wets, but it's also now an elemental toolkit slash packages slash, slash kernel. So all right. Thank you. Welcome. Hi. Thank you very much. I was just wondering if nothing would prevent us from provisioning K3S agents like you showed, right? Yeah. Yeah. You could do agents as well. And, and then that example that's in the slides is often the documentation as well. It does, it does a slash agent. And so I was just wondering how you would provision like, for example, the Kubernetes join token, which is kind of well in K3S. Yeah. So there's an argument so you could add on to where the server, the K3S server is and the token as well. So you could, you could add those into the arguments of the Docker file. So you would basically have the secret embedded into the image, right? Yeah. So it would be a sensitive, sensitive image. Yes. That's, that's the, so you would have to pay like a custom derivative for those agents, typically for edge devices. A lot of them are going to be single node clusters and a lot of them are going to be embedded SQLite. So it's going to be a little bit different, not your data center, you know, HA Kubernetes cluster with at least three, it's going to be a different deployment of it. So there won't be, there may not be that many use cases of it, but in that case, yeah, it's going to be, um, yeah, there's, or it could have a way to inject it into that Docker file or before, before it does, before it does that build. Um, or I'm wondering also if you could just add in the cloud in it as well. I'm sure there's some kind of customizable with how it's to have a very specific cloud in it while, uh, in it as well. Okay. Thank you. Thank you. All right. Thank you very much. I got to run to my next talk. So