 Hi folks, we'll give it just another minute or so as people log in. But welcome, so please stand by. Okay, we're going to go ahead and kick things off. Welcome to today's episode of the latest LF networking webinar. Today we're going to be speaking with some folks from the phyto community. And the topic is Calico VPP Kubernetes networking with boosters. Before we go into introductions of our speakers, just a couple of housekeeping items. Attendees will be muted during the presentation. However, if you'd like to ask a question, there is a little Q&A window towards the bottom right of your screen. Feel free to click on that and type in questions at any time during the presentation. Our speakers may be answering some of those via type type to response in that Q&A window, but we are holding some time at the end for open questions. So just know that if we don't get to your question during the presentation, we will do our best to address it towards the end. Additionally, a recording of this webinar will be available in the next few days. Anyone who registered for the session will automatically get an email with a link to the recording so you can go back and rewatch it. It'll also be available on YouTube and you can access it from the LF networking website. So without further ado, our speakers today are Chris Topkins and Nathan Scribzak. So I'm going to hand it over to Nathan to do a quick intro of himself and then he'll hand it over to Chris. Thanks Jill. So nice to meet you. I'm Nathan. I'm a software engineer at Cisco working in the FD.io project. I'm a contributor on the Calico VPP integration and on a personal side, I'm a biking and hiking enthusiast here, as you can see in the photo. And a personal note, I have a French accent, at least I assume despite the name. And I'll hand it over to Chris. Yeah, hi there. My name's Chris Tomkins. I'm a developer advocate at Tigera. I was formerly a Calico user at the company I worked for before Tigera really enjoyed working with the product. So so I joined the team. I tried to update this slide to make it interesting for people who seem all the wants. So today's obsession I bought a globe and I'm very excited about it. I'm also never without music. So I'm listening to Van call the friendly fires at the moment. I'm always looking from a technical point of view to learn and share. So you'll see at the end how to get in touch with us but I love to challenge my knowledge and learn more. So let's connect. So I'm taking lead on the first part of this just to talk a little bit about what Calico is before we move into VPP specifically. Project Calico is a network open source networking and network security solution for containers, virtual machines and host based workloads. It is developed and maintained by the project Calico community. A great way to implement best practice kubernetes security across public and private cloud across many kinds of environments, and it offers exceptional performance proven real world scalability and interoperability. It's been running on production clusters for many years so it's you know reliable production hydrogen setup and it has full support for kubernetes network policy. It has an active in contributor community and you might have noticed I left the top left one until last. The reason for that is because it's related to what we'll talk mostly about today about data plane support. So, Project Calico has three data planes today in, in production, production ready data planes which are the epf data plane the standard Linux and the windows data plane. But today we're going to talk about a fourth which is intact preview which is VPP. But before we do. I wanted to just talk about a few software concepts that are important to understand which will help the context of the conversation. The first is modularity. So on the right, we've got a diagram that most network engineers will be familiar with which is the SI model. And you can see that it looks something like a brick wall and the reason it looks that way is because when we build software. We build in modular blocks using a concept known as modularity. It allows us to keep the abstraction of each of the functionality of each block separate to those above and below it and that makes the software easier to maintain and has other benefits. So sometimes this works very well. So if we look at the OSI model layer one at the bottom of the physical layer is a huge success in terms of being modular in relation to the others. By that I mean that we don't really think very much about what the physical transport medium of our internet connectivity is, you know, you can use fiber or copper or even the air for wifi. We just don't have to think about it and it has no bearing on the higher layers. So it's a success in the sense that layer one is isolated and entirely modular from from all the other layers. Failure where the concept applied but it wasn't implemented quite so well is layer three. So, again, network engineers will probably know that layer three is where IPv4 is implemented or IP is implemented. And you'll know the story of how we are still struggling to move away from IPv4. And that's happening, because we need to move away from IPv4 due to address exhaustion, but it's becoming incredibly hard to do so and that's because layer three is tightly coupled to the blocks above and below it so it's not as modular as it should have been. So I would say that layer three has overlap with layer two in the sense of broadcast behavior, it has overlap with layer seven in terms of browser behavior, it has overlap with layer seven again in terms of name resolution. So basically, layer three is not as modular as it should be. So this concept is important modularity. And the next important concept is software coupling. There are two bits of example software here. One is a pizza store system, and you can see that between the two parts of the pizza pizza store software, you can see that there's a strictly defined software interface. You can think of it as two departments in business that have to talk to each other through that interface, and by having that strictly defined interface. It makes it very easy for us to fix troubleshoot and replace any block of software without affecting those around it. So that's called loose coupling. And on the right we have tight cut them, which is where the two parts of the software stack are basically merged into one, and it's very hard to edit one part of that software without having undesirable effects on the others. Now so I've gone through those quite quickly because we have more interesting things to talk about but modularity and coupling are both key software concepts. Now in network devices, what we're seeing in the diagram here on the right hand side. Is a single network node that network nodes could represent a switch a router a virtual machine hypervisor a container host, but at the end of the day it's a network node. Network nodes tend to have an architecture of a control plane and a data plane. So this is a modular software with a defined loosely coupled interface between the modules. Well, the job of the control plane is to wash will go into that a little more in a moment but it's to get an overview of the network. And the red arrows represent the actual user traffic by which we talk about emails or cat videos on YouTube content or whatever is passed between the data plane. The control plane doesn't have any direct visibility of the actual network traffic. Instead it exchanges network state, and that's represented by the green arrows. So if we elaborate a little bit more on the role of the control plane inside the network node, the control plane is responsible for figuring out a high level complex consensus of the network. So that might mean for example the consensus of sophisticated routing protocols like BGP or SPF for ISIS. It is necessarily complex software. There's a lot going on. And so as a result, it's typically implemented on a general purpose CPU because a general purpose CPU is needed to to be able to carry out that that kind of complex software functionality. So the CPU in fact is something a lot dissimilar to what's in what's in your alone laptop. The data plane in contrast is responsible for processing the actual transit traffic that moves through the node. As a result, it has to forward an awful lot of traffic, very fast. And if possible it should use whatever the hardware acceleration features are of the node. On whether this is an expensive switch, it might be a hardware ASIC, or it might be kernel functionality or other software functionality or acceleration in the NIC. The data plane's job, since it's designed for just forwarding traffic, its job is to be the simplest possible implementation of the required packet forwarding features. So maybe you have a requirement for cryptography in your data plane. Well, then that's a required feature. So it needs to have the simplest possible limitation. But it could be that your data plane requirement is very, very simple. There's a really good example of this which if I have more time I'd go into more detail but MPLS is a great example of this in the sense that IP forwarding has a certain amount of necessary complexity. There's a certain amount of variable length submit masks and so on, but MPLS which is hugely successful in service provider networks. Sidesteps a lot of that functionality and reduces the necessary feature set down to a very, very simple minimal feature set. And as a result MPLS was hugely successful. So this is a good example of how data planes can be designed for different roles. There's no one right solution. Calico was designed with a four tier modular wall from day one. I was really excited when I joined the team to realize that this this thinking had been there since day one, and those four tiers were the data model, the Felix calculation graph, the data plane driver and the data plane. So I think of the top to the data model and the Felix calculation graph as being analogous to the control plane of the node and the data plane and the data plane driver as being analogous to the to the data plane. Now, each of those is loosely coupled to the ones above and below it. They are very modularized now we'll talk in a minute about why, why that benefits us and then you can probably see where I'm going with this but VPP is one of the different options that we support and we're able to support any data plane that appears that offers, you know, great features like VPP does. So, in the case of Calico's data plane depends upon what what do those four components actually do. Well the data model defines the actual custom resources used to enable Calico. So it's the data structures that allow Calico to do the work that it needs to do. So it lives up in the control plane. The Felix calculation graph makes the smart decisions that of what needs to be programmed that need to be programmed into the data plane so we said the data plane offers a certain set of functionality. Well the Felix calculation graph does the complex. Calculations and then programs the data plane to make those forwarding decisions. It does that for a data plane driver which is a simple and minimal driver for programming the data plane so it's like an abstraction layer between the Felix calculation graph and the data plane itself. And the data plane which is responsible for actually forwarding the packets. So we'll look at the diagram first of all. This diagram says data path but you could use it as data plane just the same. In a way this this diagram is great because the Calico cat logo actually sits in the scenario of the control plane. It sits above the various data plane options, but you don't have all three implemented in a single node at a single time you would only have one. The Felix IP tables is our original data plane which is performance and very very well battle tested. The windows house networking service is our different sports windows and Linux ebf data plane is an alternative data plane for Linux that has extra requirements around the kernel version, but also offers better performance than RIP tables Then there's the fourth option which is VPP, which is not yet shown in in this diagram because it's intact preview. But we'll, we'll go into what VPP can offer us in a sec. So, why do we even have a pluggable data plane well it allows us to reuse the control plane code. So, all that complicated sophisticated code that was written for the control plane doesn't need to be torn out and replaced. No matter what data plane. It also allows us to keep the data plane code specialized and minimal so it's easy to audit easy to test and lightweight, and it allows us to target a particular feature set, and it allows us to future proof. So we're able to switch orientation and support particular data planes that our customers would like us to. It makes both us and our customers agile. So, I've already spoiled this slide a bit. I should have remembered it was there but broadly we offer those four data plane choices. So we're pretty much on time so with that background knowledge, thinking about modularity, coupling and data planes and the advantages of supporting multiple data planes. Let's hear Nathan can tell us a lot more about VPP the specific data plane choice. Thanks Chris. I'll try to switch to sharing. I think you need to stop. Yes, I think I need to stop. That should be sharing. Yes, we see it. Okay. Thanks. Thanks Chris. So, now let's take a look at how we can leverage the Calico architecture that Chris described to insert VPP as a data plane for Kubernetes networking. So first, a few words about VPP. It has been presented many times in many presentations. So I won't spend too much time on it here. But in short, it's a user space network data plane, which is highly optimized both for packet processing and at the API level as well. It is also easily extensible through plugins, which is something we use a lot for the Calico integration. If you'd like to learn more about it, there are plenty of presentation and resources available so feel free to go there. Now for the Calico VPP integration, what we do is we use this module that Chris described and we add VPP as a data plane under Calico. So when you deploy Calico VPP on a Kubernetes cluster, you will get one VPP instance on each node that processes the container traffic. It does the routing, of course, but it also implements the Kubernetes specific data plan features such as policies, the service load balancing and also potentially going traffic source netting, IPIP, VXLAN, the tunnels, the IP sector and so on. So all this logic is done in dedicated plugin that are optimized for this use case. So we wanted to make it as easy as possible for users to configure just a switch to flip. So you only need to pass an interface name that VPP will use at its uplink and a driver for consuming it. Actually, you can even leave the driver and VPP will try to find the best driver for the environment. We also configure VPP in a friendly way for container environment. For instance, we use interrupt mode instead of pull mode so that we don't waste CPU cycle busy looping for packets. In many cases, we support running without huge pages as well to limit the number of provisioning step on the servers that that might depend on the driver. But now you might think why do all this? What are the issues that require such performance or specific features that the regular Linux kernel can provide? So we'll describe a few of the ones we are aiming at solving the main three. The first one is internal decryption. So if you want to enable encryption from every node to the other. Second is exposing highly available services in a in an easier fashion. And the third one would be network intensive application like VPN or proxy is being deployed in an instance. So for the first risk is internal decryption. The thing is encrypting traffic is usually required at some level for compliance and security reason. For example, if you want to talk to compliance or PCI DSS. You will require it some sort. And having the infrastructure provide this encryption has really nice properties because it guarantees that regardless of application evolutions. And also maintenance gets easier because if patches have to be applied or security parameters have to be bumped. You only have to do it in one place and you can separate the concerns between the encryption infrastructure and the application. The main issue with that comes from performance as the default Linux implementation usually makes it quite impractical to use in production as it reduces to the for put enough that you it does not really make sense to enable it. But with Calico EP as we expose optimized implementations. We allow this to work at line right and and with a manageable CPU usage. So that enables us to use it in production. And then we also expose both IP sec and we got to address the different requirements application might have. The second use cage is, we target is HA for services. So Kubernetes aims at providing highly available services that are robust to pod or not failure. But this usually gets quite complex in practice because Kubernetes will often drop all connections going through a node when when the node dies, even if the target pod still lives. And there is no guarantee that the node will load balance the same way when it restarts because all you have to synchronize state in a in another way. In Calico VP as we have our custom implementation for the humanities services, we can we chose to expose a maglev enabled load balancer, which allows us to keep connections, even when the load balancing node dies. They are only lost when the handler for this last. And we're starting node with a load balance deterministically without having to share state and services can then be exposed by routing external IPs or service IP directly to the nodes. But doesn't require to fully change for ingress and it allows even allows for a simpler topology. So you just have to be cmp between the nodes to to have it to have it work to have your ingress working. Finally, for the network intensive use case like the VPN proxies CDN or storage. The network stack usually becomes quite quickly a bottleneck as the traditional offloads we have like GSO GRO so packet coalescing cannot be leveraged. So application developers can use SRV to to get around the performance limitations. But then they have to do the addressing scheduling on their own so they don't get the benefits on communities. What we try to do on that use case is offering an high performance alternative so still high performance alternative to the Linux kernel without breaking the benefits that communities brings. So we do this in two ways, either providing your memory memory enable the interfaces members that are exposed by the two pods to to speed up a packet transfer. And we also try to want to expose if it's a stack that that provide the alpha or plus so layer for implantations of protocols, the CPU Dp TLS and quick to applications. And when we put all these use cases together it starts to look like this. It gives the following architecture. So here we have three nodes each running BPP. The ingress of balancing is done in two layer one ECMP in blue and one maglev in green. And there is an accretion layer between the nodes, but showing you this this graph is the graph gets a bit cumbersome so I'll dive quickly into the detail to explain a bit more about it. So let's let's first dive into the VPP logic. So the way it works is we provide an extension to the regular Calico YAML declaration. So the regular Calico agent feel like still runs as a demand set on each node as Chris stated the we keep the same code we keep the same mechanism. In addition to that, we add a VPP demand set so one VPP instance running on each in a pod on each community is not living on alongside. And this is VPP registers itself in the Calico control plane, so that the control plan will program it instead of the Linux or the Windows or EPP Firm. We start VPP with a low footprint configuration by default, no DPDK know which page is an interrupt mode to be as lightweight as possible. And then the VPP plugin implementing the Kubernetes logic do the magic and do the actual networking. So now because VPP is a user space stack, there are several things that will be different between VPP and the data planes. As we can see here on the topology VPP inserts itself between the host network it's in red in the figure. So on startup, it will grab one of the host network interface the one that's specified in the configuration and consume it with the appropriate upland driver. It then restores the host connectivity by creating a tune interface in the host root network namespace that will and then replicates the original upland configuration on that interface. So it restores addresses routes so that things behave similarly to from the Austin point. That's that that's really useful. For example, on environments where you only have one interface and your SSH in through that interface on the machine and you still would like to keep your SSH session after VPP has started. The pods are connected just like the hosts with a tune interface in each of the pods namespace. And the Calico Contra plays running normally on the host and it configures the data function. Once we are using tune interfaces and not VIF. We also don't need to worry about the layer two in the pods, which is a better match for the humanities networking model and we also gain a few bytes by not having the help to add there. Taking a closer look at the packet flow on Calico VPP node, the way application will consume this. The application that runs in the pod uses the regular socket API to send and receive traffic. And the kernel transfer stack then generates the corresponding packets and send them to VPP through the tune interface. And then afterwards VPP forwards these packets to the rest of the network. So the main advantage of this setup is that from the applications and points it's completely unchanged. The pod is also still provided by the kernel, which limits the security constraints we put on VPP. So the way West is really similar to regular Calico. From a higher standpoint, the logical architecture will look like this on a node where you deploy your Calico VPP. So there are two pods on each node. One is running the regular Calico agents. Here are the blue components if you exclude the Kubernetes API. And one running the VPP pod here in white. The second part consists of two containers, one for the control plane and one for the data plane. So we kept them separated because this allows separating the concerns. It keeps the VPP life cycle separating from the agents and that reduces the risk of control plane events impacting the global connectivity. And also you can upgrade both independently if usually data plane upgrades are not that correlated to agent bumps. The state of this project is currently a Calico tech preview and it lives in the Calico GitHub organization. We now support most of Calico features, so most of what the control plane is able to tell us. And we're looking to get feature parity in the near future, hopefully getting generally available sometime soon. So it's an open source project. If you want to contribute, you can find us on GitHub under Project Calico. In the short term to list a few of the upcoming features, we are focusing on the cloud deployments, providing support for operators and public clouds. Actually, operators should be close to being released, easing even more the deployment process and supporting even more public clouds to really allow testing to be quite easy. We're also aiming at adding package interfaces, the Mami file I spoke about earlier, and VPPO stack support should come right after. In the long term, we are aiming at leveraging those integrations, those features in complementary solutions, like Envoy or NSM network service mesh to allow to address even more specific use cases and speed up those specific use cases with the optimizations we provide. So this should give an overview of the architecture of the project, the way it works, and I hope it makes the overview picture I have shown earlier a bit clearer. So now what I propose is diving into the use case we mentioned earlier, so encryption, highly available services, and packet intensive applications. And seeing the way they integrate with this, how we can leverage them in applications, and what are really the performance improvement they allow in terms of them. So I'm going back to internal encryption. So we, as I said earlier, we expose a couple of options, we have support for Wayagard, which is cross compatible with Calico Linux and Calico EVPF, so you could imagine having hybrid clusters talking to each other with Wayagard. We also support IPsec, which is the standard encryption algorithm, and that's also benefited from the most optimization IPP yet because it was developed a bit earlier. IPsec even comes with an asynchronous mode, which allows to distribute the crypto operations of a single tunnel to all the CPUs or a subset of those CPUs that are assigned to IPP. And this allows to reach a much higher throughput than when doing this operation synchronously on the worker, the tunnel, the IPsec tunnel was pinned on to, because RSS pins us on a single worker, spreading the operation really allows to spin things up. So we did some benchmarks to showcase the performance that this can give on a bare metal cluster. The test we did, I'm going to show is, it consists of a simple IPer throughput benchmark between two nodes, with traffic being tunneled with different encryption modes, different ways of doing the encryption. So when using Wayagard, which is available, which is the one being available on the free implementations, the Linux, CPPF and VPP, we didn't test Windows yet, but maybe at some point we will. So the kernel implementation reaches 2.6 gigabits on one tunnel, both for Linux and VPP, and VPP goes a bit faster at 5 gigabits. Actually, performance should be a bit better with VPP as we really recently integrated a crypto backend optimization for Chacha 20, which is the crypto part of Wayagard. And that gives around 30% on specifically the encryption part, so it should be better. We'll update the results afterwards. On IPsec on the other side, it gives better figure, but this would here only with VPP. We're reaching 9.5 gigabits in synchronous mode, and asynchronous will be 14.4 on a single tunnel, which is leveraging multiple calls. We also measure the global CPU consumption on both nodes during the test, and VPP doesn't consume significantly more CPU to reach that throughput. So this really makes enabling encryption across nodes possible in, let's say, production environment, and still keeping an acceptable throughput. The second use case we are considering is highly available services. So here we took a Kubernetes cluster with three nodes, each running some containers. So in the figure, there are two services with the, with VLC and EngineX running. And let's say we expose those through service IP that we want to have available outside. So doing ingress and exposing services with regular Kubernetes is a bit hard because doing traditional ECMP between nodes usually kills all the connections on the node failure or when adding deleting nodes. If we use a dedicated ingress load balancer, it can create a single point of failure or only some sort of state sharing. So as we have the load balancing code living in the user space plug-in, it allows us to do a bit of magic, and in this case, leverage the maglev load balancing algorithm to ease that. Just a quick recap, if you're not familiar with maglev, it's a stateless load balancing algorithm with some nice properties. It comes from paper that was made by people at Google. And it's, the algorithm is deterministic in that the same flow will always be balanced to the same backend if the state of backend doesn't change. And if we add or delete backends, there is some consistency guarantees that most flow will still be balanced to the same backends. So minimizing the last or rich or full connections when, when events, when rebalancing events happen. So if this work is as follows, we generate a list of buckets in the middle. So usually we take a big prime number and generate a list. And we make the backend successfully choose buckets by order of preferences. So they, they say, oh, I prefer that bucket. And with a round robin, they choose the different buckets. And incoming flows are then consistently asked to a bucket and the bucket leads to the back end that shows it. The, the nice properties with that is that if we add a back end here, back end screen read as back end one and two will still have the same preferred buckets, the number of flow that change back ends will stay low because back ends will mostly get their preferred buckets. And, and so flows mostly won't change on change locations here are actually I didn't draw any flow that change but as I did draw drawing I choose the best possible scenario so but, but it's the general idea. So moving back to what this give for Kubernetes cluster leverage leveraging model GLEV. This allows us to, to directly use ECMP for incoming flows. So we can have a stateless load balancing layer in front of the nodes that will change when the node changes. And when we take an incoming request to a service site going to a service IP. So here we should give a node provided by ECMP here we take the node A. It will enter the PPP the way we saw earlier and my level load balance it to a pod so maybe on the same node maybe on another here it's node C. In the event node disappear. So here node A is gone. It will remove itself from ECMP pool. So we're changing the load balancing decision for the flows. So our flow from the past figure will then reach a different node. But with the consistency properties of Maglev the load balancing decision will be the same. So the flow will still reach the node C. So for the pod serving the connection things will be transparent which won't be the case with traditional load balancing. One of the things I hate a bit in this. So first the set of back end changes as node A was running a pod with an NGNX logo. But this shouldn't affect Maglev as back end changes are mostly consistent. And also here it's a reduction of the number of back ends. So actually their buckets will be spread a bit more but existing flow won't be reshuffled. Second for the return traffic we have to enable direct service return DSR. Because if we do the unnatting in the Maglev so if the return traffic follows the same path that means we have to store state and that we will lose the state upon node deletion. So we have to use that but that's something we also implemented in Calico EVP. And finally the fact that Maglev is stateless also has nice properties in the event of data plan restart. So this can happen if you want to upgrade your CNI or if PPP crashes but we know this never happens for software so that won't happen. But in this case you don't have to restore per flow state. So the load balancing decision will also stay consistent. And the flow will still be low balance to the same direction. And finally the last thing about this is that as this is only an implementation detail because Kubernetes does not say anything about the algorithm being used. This would work in combination with non Maglev nodes provided that you don't ECMP to them. And finally for the last two scales, we wanted to focus on our package oriented applications. So here we have the schematic showing the standard way of consuming packets in container. And finally the nice properties of being transparent to the application. As we stated earlier for endpoint application this can prove performance enough but with kernel optimizations such as GSO and GERO. But when you start processing small packets you might find yourself limited by the network stack. And same thing with compute intensive L4 like TLS or quick crypto or even TCP kernel might be a bottleneck. Another way to address this is to change the way you consume the interface within the container. You can attach the application to the container with AF packet, packet a map or AF XDP. That brings up a bit of performance gains but it's still marginal in terms of terms when building packet oriented applications. As a side note it's AF packet and AF XDP inside the pod not for connecting VVP to the uplink. Both are possible but sure it's inside the application. What we enable with a Kaliq RUP or want to enable because it's still work in progress is requesting an additional packet oriented interface in the container. With simple annotation and for example the destination port and protocol you want to receive on it. And the application can then consume this memory interface, this MAMIF directly in the pod either by running another VPP inside the pod or by leveraging DPDK or libMAMIF directly from the application. So this enables full user space networking so packets will never touch the kernel on the path from the application to VPP. And continuous can even leverage zero copy on the way from VPP to the application. Not the other way around because otherwise we would have to expose VPP memory to the application which is not ideal from a security standpoint. The drawback being that the application now has to be aware from libMAMIF and knowing how to connect to it. Same thing can apply to well for protocol, we expose or want to expose the same mechanism. So VPP has support for TCP, UDP, TRS, DTLS and QWIC. And that's through the host stack. So you just need the same way to with an annotation, tourist quest support in the pod. That will expose a socket like API in the pod that you can consume leveraging the VCR which is a library that you can, a socket like library that you can use in the application. So this enables the same way full user space networking in Kubernetes, so no going through the kernel. Optionally zero copy depending on your implementation. And that way you can leverage VPP's optimized crypto or optimized TCP. But again with the drawback that you have to make your application VCR aware. So that's a bit of the middle ground between getting better performance that you can get with the interface. But the application should know that it runs with optimized interfaces. To recap a bit the different ways of consuming traffic from within the container. In a Calico VPP-enabled canister, I summed here up the pros and cons. So for endpoint application, we don't have really definitive figures yet as this is still work in progress, but the performance gains are still quite substantial. Even though when it comes about crypto, there are a lot of numbers in CISIT, but that are not really Calico VPP numbers yet. I should thank Florin for the work on those optimizations. And concerning packet oriented interfaces. So the MIMIF, the Orgner of magnitude of PPS that you can expect is around 10 millions per core per queue. But again, this is what we're working on on improving it. So we will need to do a bit more testing and optimization to go there. But that's the thing we want to expose. And both have the same drawback of having to do some updates to the application, but I think that's the counterpart of exposing higher performance. At some point, the offload has to be shared by the application. The knowledge of the offload has to be shared by the application. So that's it for this presentation. We have a number of new exciting features on the horizon, including the mag level balancing, the packet oriented interfaces and hopefully soon general availability in Calico. I hope we were going to achieve that soon. So if you'd like to stay up to date on the project, don't hesitate to join the VPP channel on the Calico Slack. The link is there. We publish our release there and then follow the Calico releases. If you'd like to try it out, you can head over to the Calico documentation, which has the setup instructions. And if you have any questions at any point, don't hesitate to ping us on the Slack channel as well. We will try to answer, but you can ask the question right away. Yeah, thanks a lot. Nathan. Yeah. That was great. Thank you. I've answered a couple of questions that were that I knew the answer to, but we have several more in the Q&A. If you have a look with me. Okay. I went straight ahead on the presentation. So I didn't. Oh, that's okay. I picked out a couple anyway, but we have plenty of time now to run over them. Let me find my Q&A. I can read some out to you if that's easier. So the first question that came in is regarding fast API support, 200K updates per second, which VPP data structure is updated at that rate? Is it classifier node? So from what I understood, so if I understand the question correctly, it's how fast VPP can update its data structure. So the way it works, VPP is connected to the agent via a binary API. On some APIs, it can go quite fast up to, I think, way more to 200K per second. I think a million if I recall correctly, but as we leverage different, so many API calls that might get reduced in the case of fast humanities updates. We did actually do a bit of testing on the scaling part of when you want to create a lot of services. And from the testing, the main bottleneck was the SCD slash the Kubernetes logic being able to forward us the updates. So VPP is not the bottleneck yet, but we were definitely looking at optimizing this. And then we upgrade, we update internal structures, data structures that are internal to VPP. So for that, the data plan will use them. Great, thank you. Next question, is it possible to use this as secondary CNI via multis? It is. It is definitely, we haven't tried it. It might lead to some quirks and some tricks to make it work properly because of the way we do swap the interface. We sort of steal the interface from Linux and replace it with a fake one. So depending on the configuration that you have and who is the chicken, who is the egg, you might get into weird situations. But if you have different plain interfaces, I think that won't be any problem. There are definitely ways to make it work. I think that's one way we'd really be interested in you joining us on Slack as well. If you decide to try to do that and share your experiences and it may be that that we can help to unblock you. Yeah, definitely. Also would be really interested in knowing the blockers if there are any on the thing we can do to improve that. Another question, whether VPP batch processing model has an impact on latency? Any benchmarking done with service networking? So batch processing, it has an impact, but it's very, very small. There is a guy in your team who did some measurement with that, but on regular nodes, it's really under the microsecond level. So from more testing, it's barely noticeable. So the fast pass, slow pass going through the NAT and the service logic will be in order of magnitude slower than the batching. So to explain a bit, VPP has this concept of doing vectors of packets, if I understand correctly, the question and processing them all at once. So the first packet of the batch will be processed a bit slower than the last, but that's usually really fast. And the other part of the question was about service networking. They don't use our later moves to answer. We didn't do. I actually, I actually marked it as answered. I think you'll find it in the answer section. I think you, I feel like you answered it really. I think the only, the extra part was, was, have we, have we specifically benchmarked the service networking? So the service IPs. So if by service networking, it's the NAT thing, we did a couple of benchmark on that one. If throughput it does not impact and on PPS, it starts to be a bit of bottleneck at some point. But we are trying to improve that. It plays a role in the, in reducing the 10 million packets per second on the net. So, but they are ongoing improvements. So on most benchmarking, we do an effort on benchmarking, but we also do an effort in improving the benchmark at the same time. Next question, would it be possible to run a VPP based NFB application in a pod with Calico's VPP data plane? In other words, VPP on top of VPP. Definitely that would work. So that, that, that doesn't work in the latest release because we, we still have to add to publish to the support for that. But so, I mean, that, that would definitely work attaching VPP to a tap that would work better with added support for my mute that we are going to going to release. But yeah, there is definitely not nothing that prevents this from running. Maybe so this might bring some concerns on the addressing model and because NFVs are not. And Kubernetes is really made for dealing with endpoint applications in the industry. So this might bring some questions and some things might be done better. So if, if the, the, the person who asked this question, he wants to, to join in and discuss with us will be really interested to find the best way to answer the things that might not be ideally answered by Kubernetes. But, but from a technical standpoint, yes. Okay, just a couple more questions. How does a packet processing workload interact with Calico BGP control plane? Surely it needs to advertise a subnet rather than a host route. So, yeah, that, that starts diving into the how to do addressing and with four packet processing applications. We don't have yet a strong opinion on how to do this. Considering the Calico BGP, we don't, we didn't do any changes to it, which we just integrated it as, as it works with Linux. So we could build something specific there that's dedicated to, to a networking application, but the way we've thought about it for, for, let's say, start up, start up packets processing applications would be to use end caps and to sort of bypass the Kubernetes at first. So you could, you could think about Kubernetes being just the scheduling, starting pods and giving them memory interfaces with addresses. And then on top of that, you could end cap state packet and build your own, your own network. But you could also integrate with the BGP and but that would require a bit more specifications. So it's still something we are thinking about. So happy to discuss also. Okay. And I think we might just have one more question here. Does Calico make use of G go VPP bindings for interacting with VPP. It does. It does. It does use go VPP under the hood. So yeah, we, we use, we use this and we have built a small, small interaction layer between go VPP and our integration to ease the updates when the API change and we play with specific features. But under the hood is that yes. Okay. Well, I think that covers all of our questions. So unless there's anything else I think we can wrap just a couple minutes early. Thank you everyone for joining. Thank you to our presenters and we will see you on another episode of the LF networking webinar series. Thank you very much. Thank you very much for everybody who joined in the lesson. Thanks a lot for LF hand to you. All right, bye bye. Bye.