 Hi, welcome to the Kubernetes Edge conference. This is the session on taming unbounded resources with node feature discovery. My name is Mark Abrams, and I'm a field engineer and edge specialist at SUSE, formerly with Rancher. So first of all, I want to take a minute to say that edge means different things to different stakeholders. For some, the edge is the edge of the cloud, and it functions like the cloud, but it's closer to where the data is generated. In this presentation, I'm actually talking about compute at the edge of the network. So Kubernetes at the edge of the network, not the edge of the cloud. These are the leaf nodes, so the leaf clusters. For example, we'll see these edge use cases in factories, in retail scenarios, in energy, in the automotive and aerospace industries. We'll see it in smart cities, and there's lots of other industries. With that, let's take a look at node feature discovery. Node feature discovery is really, it's simply a fancy labeler. You can use it to label Kubernetes nodes. So why do I need a labeler? I think the number one benefit is for the ease of just mapping workloads to resources. Here is a five-year-old worksheet, a kindergarten worksheet, finding the resources. That's what this node feature discovery is doing for us. In addition, I would say that with massive numbers of devices, we must automate. It's not an option. We have to automate. And then, of course, the resource types that we're targeting don't fall neatly into the CPU and RAM category. And additionally, we have the challenge that the resource capabilities at the edge are unbounded. What do I mean by unbounded? Well, because these unique resources are not, they're not infinite. So I say unbounded because there's really no limit to what device might be attached to your nodes. With each individual use case, there's like a finite set of resources. But overall, for the whole global edge, it's unbounded. So anyways, with the fact that you have a finite set in each use case is good because now we can use labels to classify and give logical boundaries relevant to each individual enterprise. Without node feature discovery and massive numbers of clusters, it could be like playing a game of Go Fish. I don't know if you know this game, but basically you take turns with your opponents asking if they have the resource you need. I say to my opponent, do you have any threes? They don't have any in their hands, so they say Go Fish. It's a guessing game. And eventually you can use cache and history to make more accurate guesses. So I have a sense of what my opponent has. Obviously in the real world, we shouldn't guess which devices have the right resources. So node feature discovery is going to take the guesswork out of it for us. I'm going to demonstrate the power of node feature discovery, but first I want to show my cluster architecture for the demo. This is a Turing Pi version 1 board. It has seven Raspberry Pi compute modules. Each module is a Raspberry Pi CM3 plus with 1 gig of RAM and four CPU per device. On device node 7, I run the master and the remaining six devices are workers. Additionally, this is not terribly relevant to the demo, but I'm sure somebody's going to notice. The node numbers and the host names don't match up with the device numbers in the diagram. So here you can see that the host name Turing Pi 0, 2 is actually on device slot number 3. So in addition to having the seven modules, that's the compute. Let's look at the attached devices and peripherals. So these are the things that I attached to my edge nodes. Here I have a bunch of USB ports. There are eight ports, but they only map to four of the seven compute modules. So not every compute module has access to USB, and each pair of USB actually only maps to one of the compute modules. And for example, here is the compute module by device slot number 2, which has access to the set of USB connectors shown. In addition, there is an I2C hub on the Turing Pi. Now in this case, every compute module has access to this I2C hub. And out of the box, it's designed so that the hub exposes a real-time clock, the ethernet hub for the devices, and the power management for all seven of the devices. Of course, you have to enable I2C in your operating system. So you need to load the kernel modules to run I2C. And then in addition, each of these devices has 40 pins of GPIO. That's pretty standard Raspberry Pi thing. But it's unique to each device. So you can see here, I've knocked out the number 5 and the GPIO pins for number 5 are in the red border. So I'd like to show you now how we can use Node Feature Discovery to access these three different resource types. So let's go to the demo. So the first thing I'm going to do, I have Kubernetes running on the nodes, as I said, on the system. And so the first thing I'll just show you, is the six nodes where the workers run. And then I wrote this little bash script that will just grab the labels using kubectl, dumps the labels for each node into a directory. So TuringPi01.lables will have the labels from TuringPi01. I'm going to show you the labels on TuringPi01. And you can see that these are just standard labels. I haven't installed Node Feature Discovery yet. So in a moment, we'll do that. So let's clear it and then let's install Node Feature Discovery. So I'll install it by installing a couple of config files. These config files are, there's a master and a worker. I simply downloaded these. I did modify these. Well, yeah, I changed these to use an image container that I created for the ARM system. I couldn't get their ARM containers working, so I rolled my own. But basically, these are preset ready to go. So I've installed Node Feature Discovery. And now let's take a look at the labels on TuringPi01 again. So here they are after Node Feature Discovery. Great. So you can see that I now have labels that have this domain prefix. And then I've got stuff for CPU, for kernel, for storage, for system. This is all just defaults. I didn't change anything here. This is just what Node Feature Discovery does out of the box. And you can see there's even one for USB. This is actually the Ethernet adapter for the devices. OK, so we've now got Node Feature Discovery installed. We can see it's working. And what I want to do is actually modify the config map. So the Node Feature Discovery is managed by a configuration file which tells NFD worker which types of resources to look for, to find. So I'll go into this config file. And I will uncomment out the sources section. And then I'll find the USB section. I will uncomment out the USB section and open up the device class whitelist. I will enter just one device class and that is class 0, 3. That is a human interface device. I'll make it so that we can see the class, the vendor, and the device on the label that gets applied. And I'll save this. And Kubernetes is going to basically now label the nodes that have a human interface device. Now I have plugged in a keyboard, so a human interface device that will show up on one of the USB ports. I just modified my script here. Well, I'm using the script to get the labels. But then I want to show you every node. And we're just going to grep for USB. We're not going to look at everything. I'll just show you where the labels are for USB now. So here you can see that USB on Turing Pi 5, node 5, has this USB with a human interface device label. So now I'm going to move the USB unplug it and plug it into a different port. And so what's going to happen when I do that is that it's going to re-label. It's going to remove the label from the one that no longer has it. And then it adds it to the one that has it. So you can see here it's moved. Originally it was on five. And now it's on three. And then it's just by unplugging and plugging. So hot swap capability with node feature discovery, figuring out which devices exist on which nodes. So that's the USB. Let's take a look at I2C. For I2C, I'm going to use a kernel module. So again, I'm editing the configuration file. I'm going to look for the custom section. I will uncomment custom. And then I'll add this block of text, or this block that I call it I2C power management. That's a name I've chosen. I'm going to match on loaded kernel modules. And this is a feature of node feature discovery that allows me to select loaded kernel modules, I2C dev, and this other one. If both of those exist, it will label for I2C. So we're going to get the labels again and we'll pick up which ones have I2C on them. I'm going to take a look this time at, you can see I'm grepping for both USB and I2C. So we're going to look at all the nodes and just ignore everything except for I2C and USB. And what we should see is that every node has I2C because as I said, every node is connected to that bus. And so every node has those kernel modules and therefore the labels get applied everywhere. Plus, we still have on node number three, we've still got that USB hub, that USB human interface device. Great. Okay, last one. So this one, we're going to do GPIO. And for this, I'm going to log in to the first node. So I'm going to log into Turing Pi 01. And the first thing I want to do is create a directory. There is a directory that gets mapped to my container. And it's the node feature discovery worker container. This directory hosts any executable, it could be any executable or a shell script. And essentially the output of whatever runs in that directory needs to be a list of labels. So what I'm going to do is create just a short list of labels in a shell script. Those labels will then get applied to this node. Now, I happen to know that each of these nodes has GPIO, but I'm only labeling Turing Pi 01 as having GPIO. And I'm saying that on pin number 40, it's an input pin. And you can imagine that there might be other labels I want to apply for specific needs for each node. So here's the file. I will take this file, move it into this directory that I created. Node feature discovery will notice the change in the directory. It will reload it, its configuration file, and it will begin to label the node that has GPIO. So I'm just changing this script to be executable. We'll exit out of Turing Pi 01. Node feature discovery is already doing its thing. So now we'll get the labels again. This time after I get the labels, I'm going to take a look at the list of all nodes. We'll look at all nodes. We will exclusively look at USB I2C and also GPIO. Whoops, let's add GPIO there as well. So there we go, USB I2C GPIO. And what you can see here is that I now have on Turing Pi 01. I have my labels for GPIO. I have my I2C label, and then on node three, I have my USB and my I2C label. Everything's got I2C and we're good to go. So that's it for my demo. Very simply, the node feature discovery worker is a demon set and it scans the nodes for capabilities based on configuration. It reports what it finds to the node feature discovery master. The master communicates with the Kubernetes API and is the component that actually does the labeling of the nodes. From a human perspective, the operations team of course needs to know their devices and the features that they want to expose. They need to create a configuration to expose those resources. Node feature discovery does the rest as I just showed you. How do we install it? I installed it using Kubernetes resource configs. You could install it as a Helm shark. You can install it as an operator. And when the underlying config map changes, the worker reloads it. The config map is that default config map that was commented out that gets installed automatically with node feature discovery. There's a healthy bit of documentation on how to use that configuration file. And I'll share a link with you at the end on where you can go look at it. This is a snippet of the config map that I used. On the right side, you can see at the bottom, my USB device class 03, right? That was the human interface device. This is pretty simple. I run node feature discovery with my unique configuration and labeling happens. I did find that there is a non-trivial issue for use at the edge. At the edge, we're managing not one single massive cluster of nodes, like what I showed you. That's actually, that might be a cluster at the edge, but we're really looking at a multitude of smaller clusters. We're also often adding, removing, and changing the resources on these clusters over time. So how can NFD be managed across hundreds of thousands of clusters? To answer that question, let's look at some architectures as we consider the problem. So this is a typical data center and cloud deployment of Kubernetes. It's pretty much app lifecycle focused with one Kubernetes cluster per development environment. Red lines in this image represent the cluster boundaries. So you can see I have four clusters and my little Turing Pi cluster represents something akin to just one of these environments. So I showed how to manage the differences on six nodes with changes to a single node feature discovery configuration. But in fact, my edge looks more like this, where I have lots of single node clusters, some multi-node clusters. Each cluster is production. At the edge, it's all production. Also, in most of the edge scenarios that I've come across, we're talking easily 10 or 100 times the number of clusters shown here. I have accounted, I have 32 clusters here. So should we play GoFish with 320 or 32,000 clusters? Now, I don't think we wanna play GoFish, right? We wanna use node feature discovery. How can we do it with this many clusters given how node feature discovery works? I think that GitOps is a solution for this problem. One of these three tools, I'm really only familiar with Fleet. However, I know that Argo, CD and Flux are also being used as GitOps tooling. There are others, Azure, DevOps, and there's a bunch of options. We need some outside tooling to manage large numbers of clusters. In Fleet, it's possible to manage large numbers of clusters and target the workloads, but also the custom configurations because these are really just Kubernetes objects. And so we can use Fleet to actually manage those configurations. Fleet has been tested with a million clusters, for example. Now, despite the large scale, even though it can handle that large scale, Fleet doesn't have some idea of an auto cluster labeler at this time. So that's a feature that I will be requesting. The concept of cluster is, it's a custom resource definition in Fleet because it's managing multiple clusters, right? But it's not a concept in Kubernetes. Now, fortunately, there is a Kubernetes special interest group called Cluster API. And I think that the solution they're working on there may hold the answer to how we can get our DevOps tools to have a standardized model for cluster labeling based on node labels downstream. This is what I think the GitOps pipeline would look like. So assuming we can get this working, simply install node feature discovery into the pipeline itself, make sure your config is set up, either do that as part of the install or of course you can do it afterwards as well and you may end up doing it multiple times. That is probably the, aside from upgrading node feature discovery, the config file is the changeable component. I would then configure my tool, my case Fleet, to automatically update node labels. And eventually I need to notify developers of what node labels are gonna be available and what resources they map to. Rinse and repeat as resources change, we repeat steps two through four as needed. So that's a bit of hypothetical GitOps. I did wanna also talk about security. With node feature discovery, it is currently possible to configure TLS between the node feature discovery services between the master and the worker. So that's out of the box. That's a nice security enhancement that I have not yet enabled. As I was doing this, I believe that allowing that local user specific capability where I put a script into a directory and let it execute that directory, I believe there's some risk associated with that because it will execute anything in that directory. Now it only executes it in the container. However, these containers are somewhat privileged because they're accessing resources. They're at least need read access to the resources. I'm sure there's a way to lock them down in Kubernetes that would need to be managed in order for this to be more secure. In addition, it's possible probably to reduce the attack surface by limiting what features are usable in the configuration file. In that config file, there is a section called sources and we can do things like specify just USB, just custom, leave out local or only include local on the machines, the nodes that need it, that sort of thing. So that's a little bit about security. This is a summary of what we just went through. Labels are applied to nodes in a cluster. I think GitOps will be useful in labeling large numbers of clusters. Features are unlocked using a configuration file. Node feature discovery has a limited number of out-of-the-box featured types, which does include a kernel module discovery. However, if there are types that aren't matched, you can use the local type when there's no other default available. At some point, it might make sense to use Kubernetes device plugins instead of the local feature type that will depend on what's available and what, you know, if there is an existing device plugin and it's not something that can be discovered with node feature discovery, it may make sense to use the device plugin instead. That has to be seen on a case-by-case basis. Some more summary items. I didn't show this, but there is a mount in the container for uniquely named configuration files instead of the default NFT worker comp. So I was modifying the default configuration every time. It is possible to take snippets of that, put them in uniquely named files and provide a configuration through this other directory. The nice thing about this, I think, is that it provides a capability that I think vendors should be promoting, right? If I have a device, I should be enabling node feature discovery for those devices. I know that currently NVIDIA, for example, well, they don't, I'm sure the config file is available, but essentially their GPU operator uses node feature discovery to label nodes. It would be as simple as finding that configuration that they use and you could label your own GPU nodes without the GPU operator. There is some value to the GPU operator, so I'm not saying don't use that, but it's one of the things that I think we should be looking at is getting vendors to support this. Also, in the beginning, I said NFT is just a labeler. It's more than that. It's a dynamic labeler. It's a configurable labeler. NFT is very powerful and especially at the edge. The last thing I wanna get to is problems, the pitfalls. So I did discover that when a kernel module is loaded, it doesn't mean that there's a device on the other end. I have, for example, a node that has the I2C capability, but when I physically unplug the device, the I2C slave that's on the other end, it's not usable, right? So just loading the module alone isn't significant. It's not enough to ensure that the feature exists. There may be some symbiotic relationship between using the local feature to automate labeling and being able to use Kubernetes to discover if the physical feature is actually there, like some sort of a scout or a test of whether that physical item is there. And if it is at another label through the local capability or use health checks, we have to look at how can we ensure that a device that can be plugged and unplugged is actually there just because the kernel or the device driver exists doesn't mean it's actually there. What else? Removable devices. So hot plugging works, but it's not immediate. In the demo I showed you, I did it. It looked like it was happening right away. There was some time lag between when I unplugged it and replugged it a fair amount of time. Node feature discovery needs time to go through the Kubernetes controller process of checking is the state what I think it should be? No, it's not, change it. Once it changes the labels, there's going to also be another process where Kubernetes says, oh, this workload can't run on this node, it doesn't match up. I need to remove it and find a new resource that it can, I need to reschedule it somewhere else. So these are things to watch out for as you're using node feature discovery. As I said, I would share with you some links. These first two links are the node feature discovery page. There's a ton there, I just called out these two links to get started and the features page, which has all the list of features and how you can figure that file. I used defined class codes at usb.org to find my human interface device class and some of the functionality that I use there. I used K3S as my Kubernetes distribution and you can find this entire presentation and the work that I've done here at maker slash nfd-demo. My name is Mark Abrams. I contribute to open source and you can too. Thank you.