 Thank you everybody for joining us today and welcome to today's CNTF webinar, Calico Networking with EBPF. I'm Taylor Wagoner, the CNTF Operations Analyst and I'll be moderating today's webinar. We would like to welcome our presenters today, Chris Hodge, Developer Advocate for Project Calico, and Sean Cranton, Core Developer for Project Calico. Before we get going, I'd like to go over a few housekeeping items. During the webinar, you are not able to talk as an attendee. There is a Q&A box at the bottom of your screen. So please feel free to drop your questions in there rather than the chat window. And we'll get to as many of those as we can at the end. This is an official CNTF webinar and as such is subject to the CNTF Code of Conduct. So please do not add anything to the chat or to the Q&A. That would be in violation of our Code of Conduct. So basically, please be respectful of all of your fellow participants and presenters. Please note that the recording and slides for today's webinar will be posted to the CNTF webinar page this afternoon at cncf.io slash webinars. With that, I'll hand it over to Sean and Chris for today's presentation. Hi there, good day everybody and thanks for joining us. Today we're going to be talking about Calico Networking with EBPF. And we're going to cover a lot during the presentation. It's a pretty exciting feature that we just unveiled a few days ago with the latest Calico 3.13 release. And so we're pretty excited to be talking about it and to share the what's in the lies of EBPF networking with you. So if we go on to the next slide and speaking with today is going to be Sean Crampton who is one of the core developers on this feature. And so he has a tremendous amount of knowledge about the implementation and how it works and the benefits from it. And so we're going to have a little bit of a Q&A session with him so that he can talk about why we implemented this and how it works. And so welcome Sean. For the first question, what prompted the team to add another data plane to Calico? So from the very beginning, we designed Calico with kind of pluggable data planes in mind. Our general kind of philosophy on this is like we want to use the best tool for the job. So when we first started with Calico, we went for tried and true technologies. We went for standard Linux networking with IP tables. We use BGP in our networking side because that's kind of proven at internet scale. And that's kind of great for a lot of use cases. And as I say, it's great if you want to stick with tried and true technologies and you're fairly conservative. Then along came Windows and we added a second data plane. So we've got our second pluggable data plane, Windows H&S based and that's in our commercial offering on top of Calico. But one of the things is the best tool for the job is not a static thing and there's been a lot of innovation going on in the kernel, the Linux kernel in this case. One of the things that they've been adding is this EBPF technology and it's kind of reached the point where it's starting to be useful. It's starting to be deployed and we think it's worth investing in now where before we stuck with the more conservative choices. On the other side, there's a lot of enterprises who are out there who are not going to want to run the latest kernels with the bleeding edge technology in it. And for those we have the tried and true technology that we had there already. And yeah, we just want to future proof Calico and make sure that we're ready as the technology develops and we're still using the best tool for the job. So EBPF is a new technology that's inside of the Linux kernel but, you know, being new there may be people who haven't heard about it or, you know, know the power that it has. So what exactly is EBPF? Right, yeah, so, so I think I can go over that. So, EBPF, it's a technology that lets you run kind of little mini programs inside the kernel and they run when things happen in the kernel so it's kind of event driven. There's been a bit of hype around it. But the big idea is that you can write a program, compile it to EBPF bytecode. So it's a virtual machine, a bit like Java virtual machine or something like that if you're if you're familiar with that. And then you load it into the kernel. You don't need to give it all kinds of permissions. So when you load it into the kernel, the kernel verifies that that code is safe. So it verifies that it can't go off and access arbitrary kernel memory or like panic your kernel or like blow something up. The EBPF program is is kind of verified to make sure that it can only do a certain range of things and it's verified to make sure it can't tight loop or kind of run forever. It has to has to terminate and it's kind of limited in that way. But another aspect is it's verified to make sure it can only call certain helper functions. So your your BPF code can do calculations. But if it wants to kind of affect the environment or affect the world around it has to go through a helper. Those are all policed. And so it's quite a lockdown environment, but it runs all the way in the kernel where it can run very, very fast. It's all kind of compiled down to real machine code when you load it in. And it doesn't have to jump between user mode and kernel mode to do things that it can it can run right at full speed inside the kernel in a kind of very streamlined way. I talked about compilation and the technology we're using to kind of build our BPF programs. We're writing them in C and then compiling them down with the Clang compiler. So Clang now has a back end that targets EBPF and generates these bytecode programs. So the big big idea is we can run little mini programs inside the kernel. They can run fast and they can do interesting things that like make different decisions than what the kernel might do on its own and bypass things that the kernel might might do that we're taking taking a longer time. So that's that's that's that sounds really cool. So if you have these EBPF programs what what sort of things can you do with them like what what's the power that it unlocks for for for developers and users. Yeah, so it's, it's, it's quite a broad set of set of features. So the the original purpose of of BPF like before it was EBPF which is extended BPF I remember correctly that there was BPF which was was designed for filtering packets and it's used by TCP dump if you've ever used that program. So TCP dump loads in a program that says give me these packets and they get sent up to user space that was the original original idea behind it I think. But then once that technology was in there and the virtual machine was in there got extended and expanded with with additional features and the modern variant. It has become quite powerful. One, one use case in there is is security so it got some use for set comp filters so those allow you to police various things various operations that user space programs can do with the kernel so they don't make a syscall into the kernel and you can police what files that syscall is allowed to access or what you know what the parameters to that are allowed to be. And your BPF program would run on the data that was passed to the kernel and you could decide is this okay or not. Another aspect is logging and tracing. So there's a very flexible set of hooks inside the kernel where you can basically hook a very large percentage of the functions in the kernel and run a little bit of code that kind of snoops on what that function is doing. It can't really affect what the function is doing but it can snoop on it and record some information that you can pick up later. So you can you could record how long that function took or you could record. Like what the arguments to the function where people are using that for all kinds of exciting things, but it's kind of not not exactly what we're doing. And the the sort of next class of things is is is kind of the meat of what we're doing. So network routing and packet filtering. So there's some some very like specific hooks in the kernel that are designed especially for doing networking operations. So we have hooks where where packets are being processed and we get handed the packet data so we can inspect it and decide what to do with it. And some very some very powerful helper functions in those those hooks that allow us to do quite powerful things. So yeah, it's a very broad technology and there's lots of exciting things going on in the ecosystem where focused on that networking side of it. Okay, so you see you have this really powerful technology at your hands and you also have an existing data plane that is very highly regarded and provides all sorts of features. So when you have these two and you're bringing them together how do you figure out what to build and what's your design and development process. So, oops, went too far, I think. So, I guess, we, we didn't just want to kind of, I guess, jump on the bandwagon and slap the BPF on them on the, the product, we actually went back to back to back to basics and kind of evaluated a whole range of technologies. And so, the various, various ways where we wanted to improve on the, the previous data plane. So we went, we went and we looked at various different types of BPF approach. We looked at whether we could use the XDP hooks which are quite exciting and some, some projects use those so it's a Facebook had a project that was using that. We looked at the BPF hook that can run all the way down in the network card. And that one's, that one's pretty exciting. So we looked at that. We looked at the traffic control BPF hooks so the TC layer, if you're familiar with those tools. We looked at getting involved in socket processing as well. We also compared that to NF tables with so the current, the current, you know, standard Linux data plane for Calico uses IP tables and NF tables is kind of the version two of IP tables. So we, we investigated that. And we also looked at some different networking approaches as well so IP VLAN versus our current current approach which is using VET devices. We built a bunch of prototypes and we spent a few months digging into the performance of those, those different prototypes. And we found that we found some surprises there so NF tables so IP tables version two is generally quite a bit slower than IP tables which were disappointing. I think they, they, you know, it's a kind of version one of the version two and they're still working on the performance. So the main driver for building NF tables wasn't really performance it was, it was the API complexity and so on and they wanted to improve that so they still got some work to do on the performance there. We looked at XDP because it has this exciting like hardware offload capability of it can run all the way in the network card if, if supported. But we found that had a lot of a lot of corner cases that come with it so if you have exactly the right hardware and you're doing exactly what it's designed for then you can you can push millions of packets per second and it's it's fantastic. But if you're, if you're needing to inter work with the various other things that we were using so veth devices for the pod networking and and the like then we were fine. We found that that was, it came out slower or it had it had issues where you had mismatches between the type of XDP that was supported on different devices and things like that. And that that made that prototype very hard to get going so we ended up settling on the the TC BPF hooks which seem to be the most the most mature. So those allow us to run BPF programs on packets quite early in the pipeline so the packet arrives and a buffer gets allocated for it and then our program gets to run. And we get to inspect the packet and decide what we do with it. They're also quite powerful so we can, we can do things like redirect the packet if we want to bypass a bunch of the kernels processing. We can have a packet come in and redirect it out of another interface if we've, if we've kind of figured out that we have a fast path for that particular packet. So a lot of this is about finding fast paths where the, where the majority of packets can take some kind of fast path but then we fall back to the normal kernel processing. If, if that doesn't work for the particular packet. Yeah, so I think, yeah, we spent spent a few months building towards having a having a sort of consistent design. And IPv lands, another surprise that we had there was IPv lands were way slower than we thought they were going to be like we thought that was an interesting technology to check out for for pod networking. But we tested it on like a kernel in the for 15 sort of range. And it turned out they've been a massive performance regression and it was really, really slow, and that wasn't fixed until a really recent kernel so we, we got what we got put off that one and then, but maybe we'll look into it again later when now it's back to performing reasonably. Okay, so, so you spent a couple months considering the options considering performance requirements. And, and I imagine that, you know, if you put that much work into it, that, you know, it's not just like, you know, complete line for line replacement for the existing implementation so so exactly what are the differences, you know, from from the from the ebp f implementation and the traditional Linux networking implementation. Yeah, so like, first of all, I think the key thing to notice, we're only for the same user experience so you define your, your policy in the same way with what we've kept the same rooted approach to networking so we use the Linux routing table. Still, if you're, if you're using Calico's like networking, like Calico's various networking modes like BGP or IP IP or VX LAN, they work the same way with rooted root, rooting the traffic around. The scalability and the kind of calculation engine is all the, all the same with, with the new data plane or that's that's where we want to be for, for GA we haven't, haven't tested to sort of full scale yet I would say just just a little caveat. I just keep a lot of that the same, but basically I sort of alluded to it before but we're, we're looking for fast paths where we can pick up a packet early, we can spot that this packet is destined for a particular pod or this packet is going to a service or, or whatever it is, and we can say rather than sending it through layer after layer of IP tables and the, and the normal Linux stack, we'll just take that packet and we'll, we'll fast path it to some, some other destination so if it's a local pod packet we'll check, have we, have we already done a policy calculation to verify that this flow is, is good to go with a networking policy. And if it is then we'll, we'll just redirect that packet directly to the VET and we'll skip a whole bunch of processing in the middle so that, that's the kind of the goal here is to sort of maintain the outward appearance, speed up things with fast paths, and then also to try and improve some of the behavior that would, that was suboptimal as well so One of the things that we found fairly early on is when we combine some of Calico's like Calico's like unique features like we support host endpoint policy and things like that, where you can secure your host, I think calling that micro segmentation people are that, that term, where you can secure your host as well as your workloads your pods and Kubernetes. When you combine those two, we were kind of pushed in a particular direction with our, with our design, and that led us to think that we to need to take over some responsibility from cube proxy as well. So we actually, we're sort of forced into implementing a re implementation of cube proxy, and then we thought how can we improve on that so we've managed to improve on some of the aspects and some of the pain points of cube proxy, particularly when you put it with network policy, which have tripped up users in the past. So, so we've, we've tried to keep the, the sort of good things about Calico's data plane, where we've, where we've taken over from cube proxy we tried to improve on on the behavior there. And yeah, the remaining point to say is we do need quite a new kernel to do some of the things that we're doing so we've targeted kernel 5.3 in this, in this release. Okay, so you have a newer kernel, you have a virtual machine that is helping you prioritize the fast path, and you've rewritten cube proxy. So with all of this work behind you. What improvements does it actually bring like are you are you seeing performance improvements are you you know what are the what are the benefits of ebpf over the the traditional data plane. Yeah, just, just to flag it when you say we re rewritten cube proxy I think we we've tried to keep as much common code with the proxy as possible so we really hate like re implementing something we much prefer to kind of stay stay compatible with with other things but it's that combination of needing to support features like host endpoints that are not actually implemented yet in the in the in the release that we have yet but but they're required by the design we sort of we sort of got forced into it but then we thought if we're going to do it. Let's let's do what we do we can do. Yeah. So it's less a re implementation and more of a custom optimization based on the the the changes that are needed. Yeah, exactly. We've kind of got a different data plane for cube proxy but the key property like calculation is shared code with, with, okay. All right so it's so it's still so it's still okay that's a that's that's that that's great to know. So what improvements does it all bring like what's the what's the final outcome and result of this. Yeah, so one of the one of the improvements is throughput and so we did some testing on a pair of physical machines in our lab with back to back 40 gig links. This, this is just fairly simple testing we use the tool called Q perf which is mostly single threaded and it's it's you know somewhat optimized but it's not like it's not totally pushing a limit so it's a but we found that when you have these optimizations in and you have the have the fast paths in there. We're getting higher throughput with a single threaded test like Q perf. And if we measure the CPU usage per packet where we're using less CPU and quite significantly less in when you're dealing with small packets. So if you're, if you're in a data center and you're doing mostly East West traffic and your MTU is a nice, you know, 9000 or something like that. So you can push really big packets, then the difference is fairly small. So that's the, that's the graphs on the, on the right of the, of the picture here so empty you, we were testing with eight, eight nine four zero because that's, that's the empty of our hardware. You, you get a small improvement, but if you if you're pushing internet size packets so you know, small 15, 1500 by packets then the reduced CPU per packet and bypassing a bunch of that, that logic does give you a higher throughput, at least with a single threaded test, but whether it's single threaded or not, you're seeing lower CPU usage for your, your networking stack which is, which is a win for, for any pretty much any workload I think. So that's one advantage. So the other one with our cube proxy implementation. So there's three lines here. What we're measuring here is the TCP handshake time to a to a service. So if you, if you run like curl to curl an engine x pod that's, I think that's how we collected these numbers. It's a simple curl test, but you tell it to you, you pass the verbose argument and tell it to log out all the detailed connection statistics. So the first thing that happens in a TCP connection is you have one packet there and one packet back and then an acknowledgement that like forms the TCP handshake. The, the TCP handshake pays quite a high price for when you're doing load balancing because when you, when you send that first packet, it has to do a load balancing calculation to figure out which backend do I send this to. So it stores that in, in a connection tracking table so the next packet doesn't pay much of a price. So what we're trying to measure here is that first packet latency, or the handshake latency so packet in each direction. And for IP tables cube proxy. What we see is that as the number of services goes up, then the latency goes up linearly and that's just due to the implementation of IP tables so you just have a linear set of rules that say, if it's this service then do this, if it's this other service then do this if it's this service do that. And the more services you have the more rules you have to traverse to find the service that you're looking for. So that the longer it takes that first packet subsequent packets flow really fast on on all the data planes because the connection tracking kicks in. One of the things that the cube proxy IPvS mode was designed to address is, is that issue where you have more services means more latency. And we do pretty well on this too so we've in this in this particular test come out slightly ahead of even IPvS cube proxy where, you know we're shaving fractions of a millisecond off that. And if you have lots of services then IPvS or or BPF data plane away better than then IP tables cube proxy. That said, the top of this graph is 1.5 milliseconds. So, if you have 10,000 services you're paying a millisecond cost but your latency across the internet might be 100 milliseconds so depending on your use case you may or may notice this kind of thing but it but it is a nice a nice win that we get that get that number as low as possible. Okay. So, yeah, that's that one. Another another nice feature is the, the way we update the data plane for BPF has the, it can be more efficient than then IP tables and IPvS so just the kernel API is that we use to update IP tables IPvS and BPF are all quite different. And the IP tables cube proxy uses quite a lot of CPU, like measuring across the whole host so including like the kernel CPU and the IP tables program itself. It uses quite a lot of CPU to do its updates. So, in this test, we set up 5000 services and then we churn the service added and removed the service I think so each time the service gets added and removed it causes causes some CPU usage in the control plane and the data plane. When you add that all up IP tables cube proxy and IPvS cube proxy use quite a bit of CPU, whereas updating the BPF data structures in the kernel is quite a bit more efficient. We can do very precise little delta updates to the, to the, the information in the kernel, the BPF programs are working with. So we, we're kind of very low and it's sort of in the noise versus the CPU usage. Yeah, the smoothing of that CPU spike is quite noticeable. It's it's it's something there. Yeah. So I think it came out even a little bit better than we were expecting on this one. I was expecting IPvS to be pretty low in the noise as well. I think there might still be a few inefficiencies in the implementation and in cube in Kubernetes, because IPvS has quite a quite a good API to the kernel. I think maybe it's doing a little bit more work than it needs to do. But our data plane's not. Okay. So that next thing. So, there are a couple of pain points in like service load balancing and and so on with with cube proxy, particularly when you bring in policy. So one of those is that when a packet comes in from outside the cluster, if it's going to a node port or a service cluster IP, you often lose the source IP of the traffic. So that was one thing that we wanted to address. That losing the source IP is bad for your logs, but it's also bad for policy because, you know, what's the first thing people want to do with policy on their engine X, you know, part they want to lock it down so only these IPs can access it. And then the standard data plane like to do that with with Calico you have to, you have to get into having host policy and you have to get that traffic before, before it reaches cube proxy and IP tables and everything. So we kind of have support for that. But wouldn't it be nice if the source IP was preserved all the way to the pod. It wasn't one thing that we wanted to do. And wouldn't it be nice if we did our best to remove any extra network hops to speed up the speed up the traffic on these paths. So we were able to do that so you have on the left you have the kind of cube proxy data plane where an external client coming in goes to it's accessing a node port. It arrives on one host. That's not the right host that the host that has the backing part is a different host. So it has to send that packet off. And then the packet has to come the response packet has to come all the way back to that first host and then and then be redirected back out again. And with BPF we can kind of do more clever like customized things. So this is the kind of the aspect of BPF where we can make different decisions to the kernel. So with BPF we can send that packet and and we have a few ways of sending it right now we're using a tunnel so we send that packet on to the right host, and then that host can can send that packet and send it to the relevant pod. And if your network supports it we can send the response all the way back to the external client from that second host. So instead of having four hops through the network, you've only got three hops and we could out that return and back to the original host. So that that improves your latency and it reduces your CPU because you're not spending two amounts of CPU on that first host you're only spending one so that's a that's a nice improvement. When we look at the performance site, we were able to measure this so we again measured the connect time to a service. In this case it was an engine export again so we were doing curl and we were getting a trivial response back just the hello world page. So the time to get the hello world page like the total time with IP tables cube proxy was 1.5 milliseconds and we've managed to get that down to one millisecond with the direct return approach there. Okay. That's pretty exciting. So, how can we try it out. I, you know, what's what's the best way to get started, you know, playing with this and, and, and, you know, kind of using it for yourself. Yeah, so a key thing to stress is, this is a technology preview at this stage so it takes a while to like really hard in the data plane and put it through its places. So we don't want people using this in production. But if you do want to try it out. We've got a link to the documentation there and there's a how to guide that will tell you what sort of system to set it up on. As I said there's there's a requirement for a new kernel. And we did our testing on particular platforms so we're steering people to the same platforms that we use for now. And then you'll hopefully have the best experience if you if you want to try this out. So yeah, that's, that's how you try that. Great. So, so what's next like what's what what what are what are the communities plans for, you know, the continued development of this of this new feature. Yeah, so the first thing to do is like we want to drive up the like the quality and the robustness so we have a few like known gaps including some security gaps so like really don't use it in production. And a few a few ref edges kinds of things. Right now it relies on like having a fixed MTU we need to get rid of that. We haven't implemented IPv6 support yet. But yeah, we're moving towards our GA release and looking at knocking off those, those ref edges implementing the missing missing features. We did want to cover a pretty broad base in this first tech preview because one of the dangers with releasing this kind of thing is if you optimize the hot path but you haven't implemented enough of the rest of it then you might, you might have something that only works for that one case. So we've tried to implement quite a broad sway the Calico features. But there's a there's a few things that we didn't get to in this in this tech preview release so that's where we are. GA release, hopefully coming soon and we welcome your feedback that you know I'm sure with with something of this size, there's bound to be bugs and corner cases that we haven't hit in our testing yet. And we're as I say we're beefing up our testing and improving that as we go towards GA so do if you if you hit something funny or you have like a packet that's unexpectedly slow or gets dropped or whatever then reach out to us for we really want to know about any any corner cases that we've missed. Apart from that, we'll be bringing this through into Calico enterprise at some point. So the, the enterprise version of Calico that requires more features to be implemented in the data plane to support the enterprise features as well. Okay, cool. So, well thanks Sean for answering the questions and talking about both EBPF and and Calico's data plane implementation of it. We're going to get to the questions that people have asked in just a moment. But before we do that project. I want to remind everyone that project Calico is open source. And there's a there's a large community of people who are who are using it and there are tons of ways that you can get engaged. If we don't answer your questions here and you want to talk with us about how you get started with it how you use it. You know, I, as a developer advocate I'm here to help people, you know, get involved in the community. So we have a Twitter channel. This is all open source over on GitHub. You can talk to us directly on, you know, the rest of the community on the slack channel and we've also launched a new a new discourse for for people that you know for community members to ask questions on and help each other out with. So with that, we'll move on to the questions. Our first one is from Adam that's Dunstan who asks who says who asks IP tables has a high level of visibility for troubleshooting and as well understood while moving to this include new tools to be able to debug these bike, these new bike code programs and the data path. So that's something we're very conscious of. So, as we've gone along we've had to build tools to to help us debug the debug the data plane. I don't think we've packaged those up yet but we have them, we have them kind of ready to go. They allow you to dump the state of the data plane out. You can also turn on really detailed debug logging in the for the BPF programs. So, once you turn that on, basically you can get information about every decision that's made about every packet. And we'll be working on that to. I think the main problem we have with that at the moment is it's a fire hose. So I think we'll be working on that to limit it down so you can say log these packets for me but but don't log every packet. I think maybe we need some BPF on our BPF to control which filters you get. Okay. Another question, which which which I think we're reiterates a point that we covered earlier is Calico replacing cube proxy. It's, it's, and if I if I understand that correctly that we are that the project Calico is not replacing cube proxy but using the existing code and extending it. Okay, so we've, we've kind of imported the cube proxy business logic, and we've implemented an alternative proxy implementation. So, our proxy or implementation programs our data plane instead of IP tables and just in the same way that there's an IP tables one and there's an IPBS one. We wondered about whether we could do it as something we could upstream. One of the things that's quite difficult with BPF programs at the moment is attaching multiple to one interface and making them all work together. So we didn't have a good, a good first idea for how to how to split up the cube proxy function from the Calico function have them work together nicely. So the load balancing function is integrated into our BPF programs and then we have this small shim that programs the programs are data plane from that. Okay. So Christopher Luciano asks, there was talk of the IP tables maintainers creating a new BP filter to handle lots of the problems with scalability and the current implementation. Does anyone know what the status of this is and what it changed things with the Calico implementation. So I, I investigated BP filter as one of the, one of the things that was out there at the start. As far as I could tell, it was still in the prototype stage. I believe that the companies that were behind it may have some private demos where they've done more. But I think maybe BP filters a little bit of a misnomer as well. So the actual technology they put in the kernel was a way for the kernel to call out to a user space program to do some work on behalf of the kernel. But there was a huge amount of work that still needed to be done in the kernel in order to to come close to replacing IP tables with BPF in that way. So they needed, you know, a dozen extra BPF hooks in the, in the places where IP tables hooks into. And then a huge amount of work in, in the user space portion as well to implement the, the BPF compilation engine and so on. So the project was kind of a prototype stage and I don't think it's matured yet. I'm also, I think, I think there was some tension in the kernel maintainers like there were some people very pro it and then the IP tables developers wanted to do NF tables instead and that was their answer so I don't know how the politics plays out in the kernel but there were, I got the impression there was some, some politics and some difference of opinion, but, but I'm not, not sure. Okay. We're going back to some some performance questions from Daniel verus verus Ami. Can you provide more details about CPU usage and and M PPS. And how does it compare with DP DK. I don't think we've done the next level of detail on on those numbers. So I, I would expect DP DK still to be faster, but DP DK is very, like, it requires you to re architect your program in to use DP DK and use all that user space stuff. So DP DK gets an even earlier in the stack. It's more like, more like XDP but it, it requires a lot more, a lot more integration and your application. So we're really going for a general purpose data plane for Kubernetes so DP DK is not really the right technology for us to use because it needs to be implemented integrated into individual programs. Okay. So how, how does the, how does BPF play on on a locked kernel like like one that Red Hat would provide. Um, so there are, there's only one or two kind of things that are locked down with BPF right now. So you can lock down the BPF system call completely and stop it from operating. At least outside of the host, the host namespace, which is where Calico runs. Basically, you need to allow, you need to allow Calico to do its BPF operations if you want to use our BPF data plane. So it's just part and parcel of using this feature. It's the usual, usual answer with with a lockdown kernel like if we're using a feature that you've locked down then we can't use it so you can reconfigure your kernel not to not to lock that down. Or you can use one of our other data plane modes if if you want it locked down and it's not right for you. Um, another another question from, from, from Daniel. And this, this, this comes back to kind of the extended features of EBPF. Are we able to have improved telemetry capabilities with EBPF without performance penalty. So that's, that's certainly one of the one of the hopes that we'll, we'll be able to add using using this new technology. So BPF it's one of the things that BPF is designed to do well is to efficiently update, you know, counters and send trace events up to up to user space for us to collect that kind of thing. So certainly, certainly one of the, one of the possibilities for this. Yes. Okay. A question about is EBPF the default or do I need Calico to activate it. So the answer to that question is, is it's a tech preview and it's not enabled by default, but, but we have documentation that that guides you through enabling it on a newer kernel. Yeah, yeah, exactly. So it's a configuration parameter. We've got a dedicated manifest that sets all of the sets all the configuration parameters for you. So rather than rather than having to set a few configuration parameters to enable it and make sure that, you know, MTU lines up and all that kind of stuff we just, we've put a manifest on our docs website that you can apply instead of the normal calico.yaml you would apply the calico BPF.yaml. And then it will go in BPF mode. I think that the main issue is if your kernel isn't new enough, then it will try to start up the BPF data plane and fail to do so if if you tried to apply it on an older kernel. Okay. Yeah. Yeah. I don't know if I understand the question entirely so help me help me clarify if I if I get this wrong. But it was said that the MTU size is fixed in calico. Do we not keep the size X minus 20 to 50 to keep the MTU size and so. Yeah, just in this in this particular tech preview release the the MTU that the data plane expects needs to be at least like 1460. I think we did a survey of the clouds. And I think I think the Google cloud has the lower the lower MTU so we picked the right value for that. But you can't change it to a lower value if you need to. I'm just wondering how I had some slides that had like 9000 MTU on them. The empty MTU and the kernel is quite a complicated concept. So the MTU on your like ETH zero device your kind of main networking device on the house can be higher than the device and the kernel has offload mechanisms where the the MTU that you're setting for calico. Is not actually relevant on most packet paths so the it will actually send a bigger packet and then the network card will chunk it up for the or Linux will chunk it up at the very last second. Before it leaves the house so empty use quite a complicated topic. But we've we've configured a sensible default but the the main problem is you can't change it to anything smaller if you need if you need to have it smaller on your particular hardware in BPF mode. It the the conflict parameters just not implemented yet. Okay. And the, and the last question that we have is can you detail how are you able to maintain latency with EBPF. Yeah, so the the main way that we get lower, lower latency is to skip big parts of what the kernel would normally do for for packets that where that's appropriate so pod to pod traffic and that kind of thing so we skip a bunch of the process thing we skip IP tables we skip the we do the routing calculation using a BPF helper and so on. So a bunch of that gets skipped. And we've optimized our BPF programs so that for the, for the mainline case so after your initial handshake when we do all the policy calculation subsequent packets get handled with the minimum number of BPF map lookups so a map lookup is how how a BPF program can store data and look it up later. So we minimize the number of map lookups which are the most expensive thing that a BPF program does. So, yeah, that's those are those are the things that we've done really tried to keep the BPF program as efficient as we can. And skip anything that we can. The main impact of that is if you have your own IP tables rules they may not be honored if you have BPF mode on. So, you need to be aware that that, you know, that's a price you're paying to you get the more efficient data path, but we're more efficient because we're skipping some some operations so Hey, Sean, thank you so much for taking the time to talk about this and and tell tell us about EBPF and Calico's implementation of it. I also want to thank the CNCF for hosting these webinars and giving us the chance to talk about some of the, you know, the exciting work that's happening within the project. I also want to thank every one of the attendees who came and listened it's it's to me it's it's personally exciting to see the level of interest in this new implementation. And, and the project, and everyone is part of the project Calico community and we want to encourage you to ask questions and stay involved project Calico.org on Twitter project Calico through the project Calico GitHub repositories, as well as on slack and on on social networks. So thank you everyone. And we also have a list of references at the end of the slides that talk a little bit more about EBPF. So if you go and you download the slides you should be able to access those also. So, thank you everybody and have have a fantastic week. It's a great webinar or minor that will have the recording and slides posted on CNCF.io slash webinars later today so you can find the slides there. Thanks again and hope to see everybody at a future webinar. Have a great day everybody.