 Today, I'm going to be talking to you guys about advanced networking with Luigi and host Bumber. My name is Arjun Bindur, and I'm a software engineer at Platform 9 working primarily on the Kubernetes and the networking stack. So you might be wondering, advanced networking, what does that mean? As you know, normally with Kubernetes, you have one cluster wide network, you generally have one CNI, all the pods connect to it. It might not fit all your networking use cases. For example, for virtual network functions, for virtualization, connecting to legacy, legacy, and networking networks in your data center. So we're here to help you with all of that. I'll be talking about two different operators that we have developed. As I did earlier, they're called Luigi and host Bumber. So first, Luigi is basically a cluster add-ons operator for advanced networking plugins. You've probably seen different types of add-on operators under different projects. But we decided to develop our own, because they didn't all fit our use cases. And they were missing certain plugins, like SRI, OBE, and IPAM drivers, for example. So we just decided to have our own one that's focused on these advanced networking plugins with the goal of deploying multis, which we'll almost always need, along with all the other CNIs and plugins that you need to get set up. So for example, obviously, you probably heard about this project from Multis, which is the first plugin that you'll need to support any multiple networks in Kubernetes. If you want to go from one CNI more than any, you will always need Multis. Luigi also supports SRI, OBE, and the device plugins needed to support SRI, OBE. It supports a particular IPAM plugin called whereabouts, which will provide your pods with IP addresses in the absence of a DHCP server. It'll install OpenV switch, as well as CNI for you. It'll install Node Feature Discovery for you, which can be a handy tool. And Host Humber, which is the other operator that I just mentioned, which will help you actually configure the nodes that you need to use all these plugins. You can't just install the plugins and then start using them. You probably have to prep the nodes. And you're wondering how to do that. So Host Humber is our second operator that I'll be talking about later today. And this is used to actually configure the nodes for these advanced networking use cases. For example, creating the SRIV virtual functions. So you suppose you've installed Multis and you've installed the SRIV CNI device plugin. You still need to go on each node, load the virtual function drivers, create the virtual functions, verify that info. So Host Humber is kind of here to help you do all of that. It can also do various other things like preview line interfaces, configure OBS. So you can view the routing tables on all your nodes. And this allows you to do it basically from a Kubernetes custom resource. It's operator without having to go on each node and configure it. Just provide the Kubernetes kind of centric way of configuring your nodes. So with that, I'm gonna be running through a demo shortly. Hi guys, I am here to talk about a new operator called Luigi, as well as a new operator called Host Humber. And I'll explain what those are. So Luigi is a new Kubernetes operator and its goal is to basically deploy and manage advanced networking plugins and secondary CNIs. As you know, the default Kubernetes networking model can be sometimes restrictive. You have just one primary default CNI, a cluster-wide IP pool and network. This can be a bit too restrictive for NFV or virtualization Kuber use cases. And then secondly, there's a lot of these discrete plugins. You might have heard of Maltis, you might have heard of SRIOV, you probably heard of OpenVSwitch. You're wondering, how do I install all of these? How do I configure my hosts to be able to use them? How do I get my IP addresses assigned on these other networks? How do I install an IPAM plugin? Does the DCP work? So Luigi and Host Humber are basically here to kind of help you out in this regard and simplify the deployment process. The first is that this requires an already working KS cluster. So if you need a working Kubernetes cluster, it should work on any kind of deployment. It's agnostic to which vendor you have Kubernetes from, but you just need some kind of working K cluster with core DNS and a primary CNI up. In my example cluster here, I have a Kubernetes cluster that's running Calico. And as you can see, it's fairly simple. I don't have any deployments on, nothing else really, mainly just Calico right now. And that's it. So to deploy it, so first let me actually get into what Luigi is, and what plugins it supports. So the first is, you probably heard of it, this all is always required. If you wanna add multiple CNIs, it also supports SRIOV, the device plugin, OpenV switch, it installs actual switch name in itself along with the CLI tools. It also installs the OBS CNI plugin. It'll install a IPAM driver called whereabouts, which you may have heard of out. It also supports node feature. Discovery. And the one that also we'll talk about later is called host plumber. This is for now is located within the same repo in platform and Luigi slash host plumber. It's another operator, independent operator actually, but it's mainly an operator that allows you to configure your nodes, networking prerequisites, also view your nodes networking state. And this is mainly to get your nodes set up and working with SRIOV, OpenV switch, Mac VLAN where all of these plugins, right? Because you might need to load a certain SRIOV drivers, configure the VSI interfaces, create VLAN interfaces, create OBS bridges, view the routing IP information of your nodes, so forth. So unless you have your own automation in way to prep your nodes and bootstrap your nodes, you can use host plumber, which is a plugin that's deployed via Luigi. So I'm not gonna go too much in a detail here, but Luigi is basically like another cluster add-ons operator, but specialized for these use cases. And we will add more plugins as we see fit. This is just the initial, what's supported. So for now it'll take like you can choose whether what namespace to deploy a plugin in, what the image override is. So right now Luigi will deploy a stable release version of each of these CNIs. Suppose you're doing in a dev test environment where you have a custom bug fixer, you're working on the actual CNIs. You can also override the image as pulled, you can set the image pull policy. And then if you're hosting in a custom registry rather than pulling from like say public Docker Hub or GCR, you can also specify a private registry base. So let's get into the first example here of how to actually deploy all these plugins. Here's a very comprehensive example here. So with Luigi, there's one CRD and it's just called network plugins. And so you can define for now the, you can set that by the private registry base, we're not using one right here. So we have the plugins that are supported. And I'm here, I'm saying to deploy, host number with the default options. I'm not setting any of these configurations here, deploying node feature discovery, deploying Maltis, deploying SRIV and OBS with the default options. Whereabouts on the other hand, as you can see has some configuration overrides. I'm not gonna again detail the specifics on these. These are all specific to the actual plugin. So if you're wondering what the IP reconciler is, for example, in whereabouts, you'd like to actually go to the whereabouts GitHub and see what this feature does. Otherwise, like I said, it's best if you don't know what you're doing. You can leave all these blank like this and Luigi will deploy with what is a good set of defaults. This is just to show what some advanced customizations may look like. Here's another one that you may want to actually do. By default, it's gonna deploy these in the default namespace. Here, I'm just overriding them and I'm saying deploy these in the cube system namespace. It depends on what namespace you want to deploy. Similarly, Luigi will take care of upgrading these plugins for you. Each version of Luigi, when you upgrade Luigi operator, if it's managing any of these plugins like Maltis or whereabouts or SIOB, it will then in turn upgrade these plugins to a new release. So let's go to my cluster here where I have, like I said earlier, I have Calico. I've already deployed Luigi and it's just running right now. It's just a controller. It's a deployment of one replica. I deployed it and you can deploy it either way. From the sample I have here, just kubectl apply Luigi plugins operator in the samples folder at our repo here. So if you go to samples, this is the one you want. Just get the raw version of this and kubectl apply. So just as a quick glance, you can see it'll define namespace for itself. This is the CRD for the plugins resource that I just described and this will show kind of what fields are actually supported if you want to get technical. And these other are back resources to create for yourself. Here's the main deployment spec. As I mentioned, just a simple deployment of one replica. So going into that, I've already deployed it, but I've already deployed it. And yeah, so let me just get into deploying the plugins. So I have a, I have a sample CRD here. I think it's the same one from the example on Github. For now I'm deploying actually going to deploy all these plugins right here. Actually first, I'm not, I'm going to show you how to use it. I'm not going to deploy SIOV because I need to create the virtual functions first and then I shall deploy SIOV. So for now, let's comment SIOV out. And I'm also going to comment OBS out just because I'm not going to demonstrate OBS in this demo but do know that if you do, if you want to deploy OBS like I said earlier, this would deploy OpenV switch itself as well as a CNI. So we'll deploy both of those. Everything you need to basically use OBS. For now these should be fine. I'm going to deploy all these. Basically I'm going to deploy whereabouts, multis, no feature discovery, and host pump. To save that, let's watch our deployments and make sure that all the plugins that we deployed are coming up. So we can see, oh, before we get, just to show I have three nodes in my cluster. I have a master node and I have two workers where SIOV has been already been enabled in the BIOS. So I have three host pump running one on each node. I have a multis that's running on each node. I have a whereabouts that's running on each node. And we have also deployed the node feature discovery deployment as well. So now that these are all running, the next question might be, you know, how do I actually use it, right? We just have these plugins running. That's the first step before you can actually use them. So now we come to a host pumper, right? This kind of helps you actually configure and prep your nodes for all these use cases like OBS, MacDlan, SRI, OV. It defines two new CRDs called host network template and host network. And these are basically as you can guess, used to either configure host networking or you can view their state. So here's just an overview if you know what all it can do. You can configure VS, configure VLAN interfaces, configure OBS bridges and attach interfaces, configure and view their IP addresses and routing tables of your nodes and more plan for the future such as, you know, enabling to CTL configuration, bonding interfaces. The ultimate goal is, you know, this will kind of be a operator that's used to configure networking for any Kubernetes management. How to use this? Here's a comprehensive example that kind of shows a lot of what it can do. So you just want to create a host network template object, give it a name. You can use node selectors to select what nodes it will target. In this case, you can see that I am applying a template and it's going to apply it to every node that has the SRIV label as true and the foo equals bar label as true. So where did this first label come from? Well, this came from automatically node feature discovery. And that's why this plugin is included. This plugin, if you automatically deploy it, it'll automatically discover node capabilities, you know, from SRIV to CPU flags and append them as labels to every node. So for example, if we did a cube cuddle, here's one of my SRV capable nodes, you'll see if we scroll all the way up to the top. We have a lot of labels here called feature.node. That lists a lot of flags here. And this is one that, and this is all automatically added. The ones that, you know, we are interested in are these SRIV ones. And, you know, the reason for that is, is we're obviously configuring certain networking configuration. Not all our nodes may support this topology. In my example, my master node is not as very capable. So, you know, we don't want to configure anything and have it error out. By default, it will go and apply this configuration globally to every node in the cluster if you don't have any node structures. So let's look at actually what this is doing. It's configuring, this does a lot of things, right? So the first is it goes on the NIC EN02. It'll create four virtual functions and configure them to use the I40 EVF driver. This is the kernel SRIV driver. If you're using DPDK, for example, you would use the VFIO PCI driver. Then this is going on the interface EN02 is configuring Jumbo frames for an M2 of 9000 and it's creating three VLAN interfaces, 1000, 1001, 1002. This is needed because, you know, some CNIs don't support VLAN tagging of their own. Mac VLAN, for example, being one example. So this will kind of allow you to automatically go and create VLAN interfaces on everything. Then here we have an example OBS config where we create a bridge named OBSBR01 and this is, again, possible because we deployed OBS using Luigi. So this is actually going to create the bridge on each node for you and it'll attach EN02.1000, which we just created up here as the NIC assigned to this bridge. So this is obviously a very comprehensive example. We're kind of combining everything into one, we're using node selectors. You can split this up in as many templates as you want. So, you know, here's, I'm just going to scan through this, but you know, here's an example where we've split it up and just SRV config and we have a different template for each particular NIC, right? We have a template for configuring eight VFs just on the F1 NIC here, another template on just F0 NIC here, that'll create four VFs. This one uses the IHTB driver, this one will use the DPDK driver. There are many ways to configure SRIOV. You don't have to use it by the name of the NIC. So for example, if your OSs are different in your interface naming scheme, you can also assign them by vendor and device ID. So in this case, you know, it'll go and select every Intel NIC because that's the vendor ID for Intel and every NIC that matches this 1528 model number and go and create 32 virtual functions. And as you can see again, we have no node selectors here. So this is, you know, going to create every node in your cluster and apply this config. You can also configure by PCI address where, you know, instead of naming NIC or the vendor ID, you know the PCI address of each NIC. And this is just another way to do it. In my case, for example, you know, this should be, you know, one, this would be, you know, two. And so I would be applying this to two to separate NICs to create 30 virtual functions. You can play around with these, which, you know, it really depends on how you want to apply the config and what your naming scheme is across your nodes. The interface config is, allows a way to configure IP addresses on nodes, B line interfaces. For now, as I said earlier, we might add support for configuring bonding and other, any other Ethernet level and IP level configuration here. But for now, you know, IP address configuration only makes sense if you're using a target of particular node. Otherwise, you know, you will run IP conflicts if you apply this to the whole cluster. But, you know, this is the way to configure, say secondary IP addresses on nodes or set an interface MQ. The VLAN interfaces I mentioned is one of the more useful features here. Just as I mentioned, because some CNS may not support VLAN tagging. So here I'm creating actually four VLAN interfaces on two separate NICs. I've created three VLAN interfaces on E&O2 and I've created one on E&O1. And apply this, this will again apply to every node that's only SRIB capable. And then finally here we have an OVS config. As I mentioned earlier, you know, this is used to basically create a bridge and attach any interface on the node to that bridge. So now I'm going to talk about the, so now actually let's go and configure the nodes, right? So let's say I have, so as I mentioned, I have a two worker nodes here that are SRIB capable. I'm not going to be demoing OVS. So for now let me, I'm just going to remove the OVS config. And let's see, I might know the NICs where I had SRIB enabled are E&O2. I'm going to create eight virtual functions. I'm going to be using the kernel driver I40 EBS. And let's just go and apply this template. So now the hostfumber operator is basically going to, you know, reconcile that CRD that we just applied. It's going to go until every daemon set that I showed you on each, that's running on each node here. The hostfumber daemon set to actually go and apply that config. So now if we look at our master node, remember, there's nothing applied. And right, that's because we had that node selector here that said only SRIB capable. And this is not an SRIB enabled node. So, you know, we don't see any B-line interfaces and we don't see any SRIB virtual functions created here. Let's go to our other two nodes here. So if we do, actually you can see I had virtual functions here from a previous run, only four, but you notice that I configured eight virtual functions. So what we should see now is that I reconfigured the node to use eight virtual functions. Yep. So now we have eight virtual functions here, number zero through seven that are present under the ENO2 device. I have three B-line interfaces on my node here. And I have on both these nodes that I applied to. If you're looking here for eight VFs and eight VFs on ENO2 and to re-rate, that's because this is the template that I applied, you know, only SRIB capable nodes, eight VFs on ENO2 create three V-line interfaces. And as I'll show you, I'll be using the V-line interfaces for my Mac V-line network. And I'll be using the eight VFs obviously for an SRIB based network. Great. So now how do I actually, you know, so now we've used Luigi to deploy all these plugins, the host number, multis, whereabouts, the IP reconciler is actually part of whereabouts. We've deployed also node feature discovery to automatically label our nodes. So we have all kind of, and we've also gone and configured our nodes to actually be able to use these features. So how to actually create a pod. If you go to our GitHub, we have some samples that might help you actually, you know, create the multis networks, as well as create that sample pods to use these networks. And the samples that I just deployed are in the top level samples folder. This is again to deploy the Luigi operator itself. And then here are some examples to actually go and configure your network. These would obviously, you know, change it to your host spec. But now let's navigate into our, first let's do a Mac BLAN based network, right? So here we have a definition for a network. So again, these are all, this is going to be specific to the CNI that you're using or the IPAM that you're using. In my example, we're using whereabouts and using Mac BLAN. And so I'm saying, you know, create a Mac BLAN network on this BLAN interface that I just created. And we're going to be using whereabouts. And here's my subnet definition. So it's just a simple slash 24 network going out this interface. So let's go and create this Mac BLAN network. So I've checked out the repo, obviously. So create that network. It's not going to do anything yet in the pod. So, you know, the point, this is just deploying the actual multis network. So now let's go and create, let's go create a stateful set, just because that shows more of it. I'll show you what a pod spec would look like as well. Here's a pod. As you know, this is, this is anything related to Luigi. This is standard of multis annotation where you create a multis network. And then you just specify the annotation for the name of the network here. So my network was actually called whereabouts conf in this example here. And so my pod is just going to refer to that network as what network to attach it to. And this is just a simple alpine pod. I think fancy. So if you have a pod spec, all you literally need to do is just add your annotation here. And this is just a simple example of what network or networks, you want to attach it to. So let me go and. Here's a stateful set example that I had that will create, you know, a stateful set, which is consists of three replicas here. And this is nothing fancy. It's just a staple set of three replicas. It's just a simple line with some anti affinity parameters here. And I was talking about the network annotation, as you can see, this is part of the pod specs. You don't want to add it to the metadata for the deployment replica set, Damon said, or staple set, whatever you have, you want to add it under the spec for the pod itself. So it's one level lower. And this is, you know, identical to the pod spec that I had earlier, same network. And that says an annotation that says what networks I want to attach it to. So now we did. It's in the default namespace. So we have a sample pod running. It got assigned to one of our nodes. If we describe this pod. Cuttle. You see some logs here and obviously multis attached the two interfaces to it. And the reason for that is, is, and with multis, if you weren't aware, the primary CNI, whichever one you're using will always be present in your pods. There's really no way around that. So it's kind of like a default interface that appears in every pod that you have. And that'll kind of be like the eight zero. Whereas any secondary networks you attach will be present as the remaining interfaces. So even though my pod only had one network annotation, it came up by default on my Calico network. This is my Calico subnet. And it also came up on my Mac VLAN network. So it actually has two interfaces. So if we actually were to go into this pod, you can see it has two interfaces and actually got two IP addresses. So eight to zero is it's Calico interface. And net one is the Mac VLAN interface on the multis network. So got an IP of 32. Now let me show an example for SRV. So again, as I mentioned, we have samples here. So just navigate to the SRV folder here. The first thing is this is, you need a critic config map. And this isn't a requirement from coming from Luigi, your host fumber. This is just something that's required by the device plugin. And it basically tells the device plugin, you know, you, we want to create eight virtual functions previously on our hosts, right? This tells the device plugin now what resources and virtual functions to watch and then tell Kubernetes that are usable. So here, if we go to our example here, it's the same example. So, you know, this is just saying, create a resource name called Intel SRV kernel one and watch, you know, two virtual functions to zero to eight. Any of them using the I 40 EVF driver. You can obviously leave this blank and I'll watch all virtual functions on, you know, two, but this kind of gives you a way to say, if you want to reserve the virtual functions for other purposes and only have Kubernetes manage some, you can do it. If you want to know more, you can actually go to the device plugin GitHub and see the full list of configuration options and selectors that you can have. They get a lot more involved and complicated than this, but this is just a very simple example to show how to do it. Okay. So now, as I mentioned earlier, we, we, you know, create our SRV config map. We created the multis network or our SRV network. And as mentioned shown earlier, you know, we use host pumper to actually create the virtual functions and assign the I 40 EVF driver to these nodes. But as you may remember, we didn't actually deploy the SRV plugin yet. We had to configure the virtual functions first. And that was only so the deployment device plugin could detect those upon startup. Let's go and enable the device plugin. And this kind of shows also how to use Luigi to enable or delete a plugin. So just uncommented out, we're going to be deploying SRV with the default parameters. And then just, we will just reapply or replace the network plugins, a CRD. If you want to delete a plugin, say we've deployed a plugin that's managed by Luigi and you want to delete it or deploy yourself, you would have to uncomment the particular plugin. So if you want to, let's say you use host pumper to deploy it and configure nodes and you don't need it anymore. You could just comment that out and reapply the template. So I'm going to keep OBS configured, unconfigured because I'm not demoing OBS right now, but we've done that. Okay, so now we've deployed, reapplied the Luigi network plugins CRD with OBS shown. Here's the plugins now for reference and if we see, great. So now we have both the CNI running as well as the device plugin for SRV running. The CNI is actually, you know, what's going to be creating or doing the plumbing of assigning a virtual function to a container. We're trying to result to Kubelet and the device plugin is more to monitor what virtual function resources are created and allocated and assigned to pods and this will be used more for scheduling purposes. But the SRV section here will deploy both of these in container. So now we have that. Now we have all the missing pieces before we can deploy the actual SRV pod. So now we go here. We already did created these two resources first. Let's go and create the pod. As I mentioned earlier, we have two pod definitions here. One of them has basically this is similar to Mac vLan. You have an annotation for which kernel, which SRV network you want. Here was an example where I have a static MAC address. This is just to show the format for how the network annotation, the format, when you want to add a static MAC. So let me create these four pods that I have here. Three is also using a static MAC and four, pod four does not use static MAC. So let's create all of these. Okay. So now we've deployed these four pods and we can see that they all deployed here. Remember the sample pod one was my Mac vLan pod that I showed earlier. And these are the four SRV pods that I deployed. You won't see the SRV or Mac via a network annotation and cube cuddle get pods out for, for example, this is only going to show the Calico primary CNI configuration. This is kind of outside. All the multis networks are kind of outside of Kubernetes. They're not really, the API servers aren't really aware of them besides, you know, the network annotation. But if we inspect this pod again now, we'll see that similar to the MAC vLan network, we attached two interfaces to it and think some oldest. We have a zero from Calico and we have a net one from our SRV network. We'll see a similar annotation, a network status applied to the pod describing, you know, in detail the PCI address, the Mac IP network name. And similarly, if you, you know, exec into the bash prompt of this pod, you'll see an eight zero on a Calico network and net one from the SRV network. And, you know, everything should work. So now we have a pod that's using SRV and it's assigning the virtual function with this PCI address. So finally, I just had one quick thing, more to think to show. I talked about the host network CRD. So up to this point, we use Luigi and host Humber and the net host network template to basically configure our nose with the plugins and everything required to deploy a pod on these multis networks. So now how do we actually want to view the networking state, right? We may want to verify some stuff. We are not able to SH on each node to like inspect this detailed information. So how do we manage the networking state of each node? We have another resource here called just host network, not post network template. And this is not really meant to be used by you, the user host from the daemon site is going to manage it. And so if we do kubectl get daemon, get host networks, you'll see we have one created for each node in our cluster. And the name of the resource is going to be the node. So if we inspect the one for the master node. So the master node, there's not going to be a lot of information. As I mentioned, we don't have any workload scheduled on it. We didn't create any VLAN interfaces. We didn't create SRV on it because it's on the NICS. So you can see these are all the NICS or all the interfaces on the host. SRV enabled is false. And this just lists all kind of information about it, like Mac, MTU, PC address. So I have four NICS actually on this, but SRV has not been enabled. You can view the whole routing table of the system here. IPv4 and IPv6. So this is just the equivalent of doing IP route or IPv6 route on the node. Now let's view a node that we actually created workloads on and configured SRV on. So if we go here, let's look at this one. I always forget to do OEM. Great. So this is actually one for SRV nodes where we created SRV on. So you can see there's quite a bit more information here. If you want to parse through it, you'll see for example, SRV enabled is true on this particular device at this PCI address. And this is a PCI address for ENO2 physical function here. So you can see the physical function driver, the name, PCI address. And because we've enabled SRV, you can see detailed virtual function information for all of them. So how many virtual functions are created. You can see the PCI and MAC addresses for each of these virtual functions along with other, you know, link layer information such as these. VLAN zero, you know, just means it's untagged. It hasn't been assigned. But as you remember for our other SRV network here, we had assigned it as VLAN 1000. Right. So now you can see from just looking at this output, I know that it happened to assign virtual function number six. It's one of the virtual functions that was assigned because we see that it has a VLAN tag of 1000 in this particular PCI address. Here's some more routing tables obviously a lot bigger now because we have more workloads on this. So there's a calico route used by the Calico CNI for each of these local workloads. So it's just a bit more verbose right here. Yep. So I think that concludes the demo here for how to use Luigi and host fumber. If you want, I suggest you, you know, you can poke around at these repos, deploy Luigi yourself. And then use that to deploy host fumber and all these other plugins play around. Yep.