 Hello, good afternoon, everyone. My name is Miguel Duarte. I'm here with my colleague, Daniel Meliado. We both work for Red Hat. I work in the OpenShift virtualization networking team. And he works for OpenShift Monitoring. So we're here to present a talk titled May Day, CNI Overboard. This will make sense in a few minutes, I hope. So first thing we'll do is explain a little bit what is the current state of the CNI project and what actually led us to care about this. Then we'll introduce CNI and MULTUS for us to get a total understanding of the lingo and be able to properly specify the problem. From there, we'll enter a new Kubernetes enhancement proposal about multi-networking and go to the upcoming CNI release. Well, this is not an upcoming CNI release because right now it's just like a set of requests for enhancements, like there's just a list of issues. That is something that is on scope to be worked on CNI 2.0, but it's very important. And then we'll finalize with what we've learned and where we see this going forward. So the first thing we should spend a little time to discuss is the relationship between CNI and Kubernetes. So for that first thing is Kubernetes networking model is something extremely simple because it basically just says that every workload, every pod gets a single interface in it with an IP address and every pod independently of where it is scheduled will be able to communicate with any other pod in the system. Through that single IP on that interface. That interface is created and configured by CNI which stands for container networking interface. Now, you look at this like that, you get the impression that these two things are actually bound together in a way. The reality is this thing in the middle actually does not exist. And what you do have is that Kubernetes understands something called CRI which is the runtime interface. And this thing actually speaks to CNI. What this means is that there is no way for CNI to actually communicate to Kubernetes in any sort of way nor Kubernetes knows anything about CNI. It really does not know, it even exists. On top of that, another thing that is missing from here is so let's say that your workload for whatever reason it actually requires more than one interface. You have one, Kubernetes gives you that, it manages it, but it just gives you one. What if you need more than one? So for that, well we didn't create but there's a project called Multis that is responsible for that. It's value proposition is that enables the pod to have multiple networking interfaces. On top of that, this Multis thing actually understands Kubernetes. It speaks its API and it actually speaks also with the CNI API which we'll see later on exactly what it is. But yeah, so we have Multis that is responsible for these two things. It speaks Kubernetes and it grants a potability of having multiple interfaces. These multiple interfaces might not be only a virtualized interface. Sometimes you actually need to tap into like a physical host interface like for instance, SRIOV, you might need that. For that, you need to add more stuff into the picture. Like you need to use a device plugin, you need to use a thing, SRIOV network operator which requires Multis by the way to be able to get your pod, your workload, an SRIOV interface. Like the picture is getting like more and more complex depending on your use case. Like the more things you need, the worse the picture becomes and the complexity increases. And this is what we have today. Like all these things. And there are certain, there are new initiatives coming up. Like for instance, right now, this DRA, Dynamic Resource Assignment, this thing got merged on, I think it's alpha for Kubernetes 1.26 or 27. And now what we have is a new emerging community enhancement proposal, no Kubernetes enhancement proposal for native multi-networking. And this is what we'll focus on later on this presentation. Set all this. So I'll hand this over to... Can you hear me just fine? Okay. So, Mika introduced us to a little bit about what's CNI, what's current status, what's evolution, but I just wanted to go ahead and go a little bit more on that. So first of all, CNI, it's basically, everybody thinks that it's basically Kubernetes networking. So we've got all the plugins, but there's so much more to that. So I would know if you have ever wondered why if this is Kubernetes, why a CNI plugin doesn't have, let's say, a native config. In case you don't know, it configs it on JSON. So you may be wondering why JSON and not Jamel and why not the CRD, and even more, why there's no demon. So currently, this is just a quick overview about what's going on in a Kubernetes node when we are just having all the components over there. So currently, a CNI plugin, in the end, it's just a binary, which speaks JSON, and it's a binary that is being run by the Qubelet or a container runtime, CRI, or what Mika was saying before. What do we want out of that? In the end, we just want a network name space with an interface or more, but we'll get to that. But in case you're familiar with any and other projects of visualization infrastructure whatsoever, for instance, let's compare that to OpenStack. In case you're familiar with OpenStack, such as, I know a lot of people here are, you can just get a VM with whatever amount of subnet interfaces you'd like to, but you can't do that natively in Kubernetes. So yes, as we're calling that, this is a binary, which is installed by a demon set. So that means that you got a copy of the binary in every node of your Kubernetes cluster. In the same way, it has a CNI playing path, which stands for, okay, where the hell is my config? So when you get the binary, it's gonna be looking for a config file, a JSON file with the same name as the binary itself with the full and we'll be seeing some examples later on. What is the CNI spec? So so far, you know that CNI, okay, is it the plugin? It is a protocol, it's an API. So in the end, it says a specification and it allows you to use four primitives that are CNI add, del, seg, and version. And even more, out of those two, out of those four, most of the CNI plugins that you'll see around only implements two of those, which is CNI add and del. That means, okay, give me a port, okay, delete that. And there's some complexity out of that because, as I was mentioning before, this is not at Emon, so all the implementation gets down to the CNI plugin writer. So if you wanna have a controller out of that, okay, go ahead and implement that. The CNI spec, it's totally okay and it's all the expects to read some JSON config file, some MV bars from the system and it will just execute the binary and give you a result. So I think it's worth just taking a quick look at how about this. So you got a couple of environment bars here, CNI command. Okay, what the hell am I running? I'm speaking about like this CNI add and del commands and then I need to basically, which name or name space I'm gonna be using, which interface name, recall, there's gonna be only one, just like the highlanders, but there's more than that. And then the continuity and if you see again here, I can point out, but it's over here. Like this is just some kind of JSON in which you have the CNI version. Depending on the version that you're using, you may have some limitations, but I won't get there unless somebody specifically asked for that. And you get your CNI plugin name and the type and then you can do whatever you want to. Once it's there, it's gonna be executed and it'll get you in and out. If you have been to some other sessions, you may have been so hearing about the prep results and that's because you can also do CNI plugin training which means that you can put several of those in a pipeline and basically, if a thing goes well, you get an index code, make a zero, everything goes well, okay, go on. Then goes multis. What's the problem here? I've been speaking about that we only got one single interface per pot. That's it in acute limitation especially if you are working with some telco environments because the most basic telco workload is gonna be a firewall. And how do you do a firewall if you only got one single interface with lag? So multis is a project which aim to be some kind of meta plugin and it handles, well, first of all, it handles several interfaces over there and it's gonna be, first of all, allowing you to use the CRD. Okay, hey, finally this starts to look like some Kubernetes native but don't get me wrong. In the end, this CRD, it uses an object on Kubernetes called network attachment definition but hey, it's gonna be just a wrapper for this old style JSON. This is something that we would like to change in the upcoming CNI 2.0 but yeah, I'll get that for later. So for now, what I want you for you if you get anything out of this is that, first of all, Kubernetes native only allows you a single interface per pot and if you use multis, you got several. So this is a quick assemble of these. So you got them, let's use Flannel but I don't really care about this and which plugin we are using. You can use any plugin for the community. Flannel, open Kubernetes, whatever you'd like to. But most of the using multis and you get another, if you see here, another net CRD interface, net one, net n, so you have several of those and those we can start doing something which is pretty much more interesting. So what are the pros about the current approach in CNI? So first of all, it's super simple. Basically, even if I stated that there are four primitives, basically I don't give a damn about anything but two. So I want to create a port, I want to delete a port and well, second version are interesting, are cool. So you know which version I'm using about the plugin but if you see, let's go and save some fences in our plugin if I recall correctly. So if you go and get blame, you may get me wrong but Cilium doesn't even have CNI to take implement it. So it's just nothing fancy. But again, so this is no demon. So any developer should have to reinvent the wheel and create their own reconciled loop for any CNI plugin. Which kind of sucks because why would you like to reinvent the wheel for CNI plugin? And also for the same reason, it's life cycle somehow limited because it doesn't give you an act. It just gets you a result. So what happens if you fail for deleting something? You may be leaking resources and you may even not know in that unless you implement your own solution of your CNI plugin. And also CNI to be honest, although I think it's a super cool project. The community is getting somehow smaller which is really bad so feel free to go join up and contribute if you want to. We are totally happy to get patches for requests, docs, comments, whatever. So really please do so. Take a look at CNI.dev. The thing is that its docs could be really enhanced. Again, pull request accepted. And also there's pod minus also somehow replaced and that with some newer implementation. And as you were seeing in the original diagrams that Miguel was showing, it's not really Kubernetes native. You may say, why? But you are saying that it's a CNI, container native network interface and what? But this is because originally CNI was meant to be used with Rocket, which some of you may know. It was a container interface, we're able to some of the other implementations that we have. So it's like it goes a long way backwards and it's not really that native. Why haven't you just made a great attack to Jamel and proper CRDs? Well, it just evolved that way. So we plan to change that. We are coming a new aspect in it to all. Miguel thinks that it's super simple. What happened here? Okay. Thank you. So what's the deal here? As I was saying, this is super simple. Just four different primitives. It's so simple to do a CNI plugin and a scale. There's even some examples on that. If you go to make it half and you say the KubeCon CNI, you can just see that high road. And example, CNI plugin for the KubeCon in Valencia and it does nothing but just fork it and it'll allow you to get a fully working CNI plugin for your own. But it also has some limitations and in the end is just a two trick, let's say pony, which is add, del, and that's it. So far, and now I'm gonna be getting back to Miguel because I want for him to highlight you a little bit of some things about theirs. Currently it's a cap, which is a Kubernetes Enhancement Proposal, which isn't going about okay. Let's make this really Kubernetes native and how we would like to use CRDs, we would like to use objects. There's a caveat to this though because the current proposal aims to be implementation agnostic. So that means it could evolve in a way that would totally ignore the current CNI ecosystem, let's say. We would like, at least CNI maintainers and CNI developers would like to avoid that because we would like to see that it's backwards compatible and it works fully. So CNI gets evolved rather than substitute. And also this may lead to, if some of you guys are working on support, if this goes on, go like, trying to do some migration path anyway. So I'll let Miguel go ahead and explain you a little bit about this proposal but we'll get back to you later with some Q&A. All right, okay, next slide. So I'm going to explain the current state of this Kubernetes Enhancement Proposal for multi-networking. So the first thing about it, it's being split into three different proposals, actually. Like the first focuses exclusively on use cases. It's a quite thorough list of use cases that actually takes into account lots of things that we think are missing from the original specification. Like it has as use cases stuff like hardware devices. It's mentioned there, a hot plug into the pod. Basically, this means that you'd have to react and actually introduce a new interface to a running pod. This is something that is not very Kubernetes but there are use cases that require it. Then the second one is about defining the API. So it's like the one we are currently on. And finally, it would be a final cap that would basically introduce these code changes. As Anil said before, this will be implementation agnostic. So every time in these discussions that you ask for something specific, like you want to, for instance, plug some runtime information from the, from a pod. So a pod requests a particular IP or a pod requests like a particular MAC address. So the reply usually is that is implementation agnostic. Like we might not care about that and we are providing one single implementation and it seems to be a little bit cloud focused. But on the good side, if you look at this, there are a lot of things you'll get for free. Like you'll get a Kubernetes native way to interact with the network plugin. You'll get a Kubernetes, an entry Kubernetes native way to have multiple interfaces on your workloads. This means like all the entire ecosystem that I've shown in the beginning with multis and all that, like half of that is actually not required. You have stuff like dynamic interfaces are featured in the community enhancement proposal. Hardware-based devices are also mentioned as an objective which pretty much will simplify the original diagram by a lot. Like half of those balls will disappear, which will simplify by a lot the complexity of the solution. And finally, you get a native integration with things like network policies and services which are of course also part and native to Kubernetes. And I would just like for you to take a look at, so here in the left, this is what the multi-CRD looks like. So you just, you just give it a YAML with a packed JSON thing in it where you can put pretty much everything you want. So if you forget to put a comma here, this thing will not work. Like it's JSON packed within a YAML. It really looks bad. It's error prone and it's hard to get right. While to the right, what you actually have is YAML, simple. Like it's very easy to understand and obviously to get right. Now, another thing that happens nowadays with CNI is whenever you have to address one of its shortcomings, like for instance, reconciler resource. Like if you are an IPM plugin, you manage IP addresses, you need to reconcile those IP addresses. This is done on a case by case, this is done case by case. So every plugin that you have will need to be able to do that. If you want to have dynamic interfaces, you will need to find another way for that. So for instance, what we did was to create a controller that looks at the annotations of the pod, season annotation change. It computes like the delta between the interfaces it has now, the interfaces that you want to have and will then like either add or remove interfaces. So we had a new controller, we had to redesign multis in order to be able to receive more inputs. So it's like a huge amount of work whenever you need to CNI to do something it was not built to do. Again, the one trick pony knows one thing. This one actually knows two, but that's what it does. For instance, for Slack, there's no way for you to do this like natively. And there are people that are after these things. Now, not everything is bad. You have the upcoming CNI 2.0, like as I said in the beginning, all you have is like a list of issues, but you can opinionate on them. You can say, what's your opinion? You can try to raise their priority, but there are lots of things that are being considered right now that will make this easier and better. So it is considering demonization. So instead of it being like a binary file just on the host file system that will be invoked, it will be run in a pod that is managed by a daemon set. That's a lot better. It helps you to add another lifecycle methods. It will be easier to deploy, and of course, it's Kubernetes native. You'll get an enhanced lifecycle, so there's a particular issue for this, and the most important thing here is they're planning on adding a garbage collection verb. So instead of you having to write a controller to reconcile your IP addresses, for instance, you'll add a verb in CNI that will actually do that for you. So it will simplify your life quite a lot. Again, this is just like an idea right now, and it's being planned. Another thing, plug-in events. So this is actually a way for CNI to feed back information into the inter-kubernetes. So for instance, let's say that CNI actually sees that your pod got a new IP because of Slack. So it will introduce that new IP address to Kubelet. And finally, also device interactions. Okay, as conclusions, what we have right now is that this multi-networking enhancement proposal is very strong in terms of use cases. It basically addresses all the current limitations of CNI, at least as we see it, and it kind of addresses the entire multi-feature set. So argument devices, it speaks, it is native to Kubernetes, and it allows you to add more than one interface. Come on, its name, it's multi-network. All of this put together makes that, well, CNI 2.0 better be good, it has one chance to do things right, or it will actually probably become extinct, or at least that's what we are concerned about. And well, you can help, like you could give feedback to the existing issues, like if you think that you as a CNI developer could get something, like your life could be easier in any way, just comment, like ask for it, and it will probably be taken into account, quite for sure. And yeah, we have no doubt that this is where we are, like Kubernetes users will go on and live a long, happy life, but CNI is freezing in the water, and probably, it will probably die. The fun thing is that probably there's room for everyone, and they can still fit together and find a way to be happy. And yeah, thank you. Any questions? Oh, that's very important. So there's a weekly meeting about by weekly, so like twice per month of this multi-network community and Kubernetes enhancement proposal, and all the caps are online, you can look at them, comment. So, welcome to that. Yeah, if you have any questions, like now is your time, and if not, well, thank you for your time. You got a question about that. Oh, sorry. Yeah. It sounded a bit like there is no reality. Totally true. It's not Kubernetes 2.0. That? Potential migration path or river migration, if you start to go wrong, it sounds like there's so much stuff that can explode and things like this can get accepted and we can't meet the new one every night now. You know, this is a huge test, so what is yours then? Like, oh, I need to repeat the question. Well, it was very, very, very long. And I'm really sorry, I would really hope that we could kind of make this a little bit more interactive, because there's one thing that I notice that if you ask me that question is because I did not explain this properly. I'm not saying that this KEP has anything to do with CNI 2.0. Those are two different things. CNI 2.0 is on one track. It is trying to do something. And at the same time, this thing is trying to do kind of what these guys are also doing, but in a native way, and they're trying to specify it. So, both of these entities are trying to address the shortcomings of what you have right now, right? And one of them is native, and the other one is like an improvement over what you have nowadays. So, would you like to rephrase your question with this in mind? I'm not saying that the KEP is about CNI 2.0, because they're really not. Yeah, the kind of the, so the question is if we are scared of the multi-net KEP as someone that looks like we are pro CNI 2.0. So, the thing is, I think there's room for everyone involved. Like, first off, I really think like all the advantages that you get on this KEP, like those are real, and that's the real way to do things forward. Now, what we want to have is a CNI 2.0 way of keep being relevant. We still think it can still be relevant even if this thing enters, because like part of its problem is also part of this multi-net KEP power is also its weakness or the other way around, like its implementations agnostic. Like it can be just a wrapper over the existing CNI, or it can be something totally different and replace it or ignore it somehow. You can just combine them, which is be a wrapper or combine what there's plenty of ways this could go. Okay, so now. Second? At least we are talking and going. Wow. So, the question is, this multi-net KEP, it sounds like it could it be ambitious enough and traumatic enough to the existing code base that you would actually need to bump a major release of Kubernetes? And the question is, I really don't know. It might, I don't, in the meetings we've been, that was never addressed that I remember. No, not at all. Not at all. Again, like the, as far as understood, like the way that you configure the default network, actually not I think a little bit about, the answer is no, like because Kubernetes only cares about one interface, right, that thing will be configured the same way. This thing will just give you the ability of adding more things to it, so you can preserve exactly what you have today. I guess that eventually they will, instead of how they will adopt the common or the pod default network as one of these multi-networks. I guess that's the overall direction they're going. And once that happens, it might make sense to do what you're saying. Plugins. So, yeah. So, the question is, if in this KEP, does it still makes, will the hardware devices still be, will it still use the device plugin to actually grant the pod access to the physical network device? So, we really don't know, but because that is only really listed on the, on the use cases. But I do think like the device plugin still has a role to play in it. Probably the thing that Maltis does nowadays of acting like a man in the middle to kind of instruct the CNI plugin of which device it allocated. Well, probably this implementation agnostic thing will have to do something quite similar, but we really do not know. It's a use case only. It's listed as a use case. So, it'll probably address it when, by whom. We don't know. All right, so, well, thanks for your time.