 at Red Hat, mostly kernel and a lot of eBPF work recently. I have a few notes up front. I'm super happy to answer questions, but please keep them until the end and we can have also lively discussion about that. One thing, like the second thing, I designed a talk for an audience that is kind and knowledgeable in eBPF already. So it helps if you've already written eBPF, but hopefully you can follow along if you didn't so far. The reason for that is that my goal of the talk is to show you new ways how you can improve your eBPF programs so we can switch from writing small toy programs to writing normal eBPF applications, like bigger ones where you can do more complex stuff. Like I said, I'm working on networking mostly, so my examples might be slightly screwed towards networking, but that doesn't mean that they don't apply to tracing use cases. It's just my examples usually come from networking. All right, so let's get started. I want to cover basically four topics today. The first thing is modern eBPF. What do I mean when I talk about modern eBPF? The second thing is I want to show you how you can compose your eBPF programs, like the small programs you have, into bigger applications. There are different ways to do that and I want to show you three of them basically. The third thing I want to show is testing. So with growing complexity in the applications, how do we test them? How do we go ahead and make sure they do what they are supposed to do? And the last one is I want to cover eBPF helpers and kernel functions. So those are basically the APIs that you can use from your program and I want to show you how you can navigate that space and more easily so you know what functions, what APIs are there and which ones you can use. So let's talk about modern eBPF. What do I mean with that? What do I talk about when I talk about modern eBPF? I want to illustrate that with a few examples. I have brought some code snippets here. All of these code snippets are from the samples in the Linux kernel tree. It's not important what they do right now but I want to illustrate a few things with them. The first thing I want to illustrate is eBPF a while ago didn't have any loops. You could not write loops that you are used to in any normal programming language was not possible in eBPF. The only exception to that was you can ask your compiler to unroll the loop. So basically copy the content like the body of your loop one after another and run that instead. That of course increases the size of your code and it's not applicable to each and every loop. It has to have an upper bound that you know at compile time already. Next thing that was not really possible was calling functions. So you could not just call a function like you would in any other language. It was just not there. The only thing you could do is again inline all the functions. So basically copy the contents of the function into your main program. So you have like a constant flow of execution or you could use so-called take calls which basically means that the end of your eBPF program you jump to another eBPF program and execute that instead but you will never get back to your old or previous function. So it's not really a function call. It's a way to combine programs but not really a way to call functions. So let's see what improved in terms of loops. eBPF nowadays can have loops. Like you can write loops in your eBPF program. The first addition that we have there were bounded loops. So basically loops that have a fixed upper bound as well. So the verifier can really check does this loop end at some point of time so you can be sure that it doesn't prevent a kernel from continuing. With that it was not necessary anymore to unroll the loop. Your compiler might still do it but it's not necessary anymore. And it covered some use cases but not all of them. So we later on had further additions to the eBPF or the eBPF program environment that were the loop functions how I call them. So basically the first one that I wanted to use is eBPF loop. That's a helper function that you can use in your programs and what it does it takes basically a number of iterations and it runs your like another function like a callback function that number of times. And the number doesn't need to be fixed or constant or anything like that. It's checked dynamically at runtime. So at runtime you can pass it a number that you got I don't know from a network packet for example and run the loop this and that many times. Like with that you can probably support all the use cases you have for loops, right? You can do everything you want with it. And then there are other functions that might simplify your life if you want to do something different. So there is eBPF for each map element and like the name already implies what it does is it gives you a way to iterate over the contents of maps with keys and values. So all the eBPF maps you can iterate over the contents basically. It's also something that was not really doable before and at least not easily. And now you have like a very useful helper function to do that and like a more convenient way for that. Let's talk about function calls as well. So eBPF now as well supports function calls something that was not possible before. And if you now want to write eBPF functions you write just a normal function like you would a normal C code and the compiler translates it to real function calls. No inlining anymore, no nothing. Your compiler as well can still inline it but it doesn't have to and you can really have function calls. And with that you get all the benefits of calling functions. You have better modularity. You have like your code size goes down if you're calling like, if you beforehand had an inline function that you would call from many other places your code grows in size. Now if you have real function calls like it's reduced in size again. And one nice thing about it is that every function is treated by the verifier as its own program. So the verifier is that thing in the kernel that basically checks your programs if they are safe to run in the kernel and it checks each function as its own program. That means that all the limits that the verifier has apply to one program or to one function at a time. So for example, the complexity limit applies to your main program first and then it applies to the function you call second. So it's, you can write more complex applications because you don't run into the limit of the verifier that easily anymore if you break down your application into smaller functions. So those things that I just said like the function calls and the loops that's just a few examples for what improved in the EVPF territory. So things that you can start using today. Now that we can split up our execution into like different functions, let's talk about how we can compose all these functions back together into like big application or larger application. I wanna introduce three different or like maybe two and a half different ideas for how you can do that. And the first thing I wanna talk about is how can I do, like how can I compose programs together? Like how can I combine them at build time? And it's right now we have a super simple linker in BPF tool. So that's something that was added some point of like a while ago. You can now call BPF tool and link together BPF object files like you would have done with your normal C object files in the past. It's really doesn't like the linker is not that complex like it is like a normal linker for C user space programs but you can really do the same basic thing with it. You can have your code in one file. You can declare your functions like functions that you wanna call in a header file. You can write the code for that in a different file, compile these things separately and later on link them together to ship one big binary. But you have your code split over different files so you can organize it wisely and so on. That makes it like I said easier to structure your code into something that you can maintain more easily. Something that is very important of your application grows in size. And of course with that you can build stuff like static libraries for example. So you can for example have a team that maintains all the parsing functions for your network packets. Like I said, I'm mostly networking focused. So networking, parsing network packets is something that we do a lot and you need to do often and you basically the code is the same all the time because the network packets look the same all the time and you can go somewhere and build like you can build your library, build your parsing functions into that and then we use them where needed. Like basically make an object file of that and link that into your application. So that's a use case for like these linkers. Another thing is imagine you don't have all the object files available at build time. Imagine you want to link stuff at load time or combine programs at load time. The BPF tool functionality I just described is based on the BPF. The BPF is that user space library that's there for basically all your BPF related wishes and it also exposes the linker functionality I just described to you as a program. So you can programmatically link BPF objects at load time or just before you load the program into the kernel you can just link it together and then load the result into the kernel. One thing that I could imagine you could build with it is you could for example, have your application built already and allow users to provide you with like their code. Imagine you have like a big networking application and you have one place where you could offer the user or the users of your program to collect statistics for example and they can provide you with an BPF object containing the program to generate the statistics and that they can extract from your program and you can link that object into your application at load time so the user can really make use of the BPF functionality still after you have attached your application. So imagine something like a plugin system in easy terms, right? The next thing I want to talk about is a bit more, to be honest, it's called every place. So imagine you have a program that you've already attached that you've already loaded into the kernel and you want to change parts of that program. So there's a functionality called every place. It's available by assist called it's available from the BPF as well. And it allows you to like swap out functions or sub programs of your application that are already there. So you can even if the thing is already attached at runtime just replace a function of it and basically put in your new code. It's a super powerful concept like you can do a lot of different things. I've seen people doing interesting things with that but you have to be kind of careful because it's not as usable as other features from BPF. The first or the main restriction is you cannot use it recursively. So in your whole call stack, like in your whole function stack you can only have one function that got every place at some point of time. You cannot have multiple of them. So as soon as you have used it once in your stack you should not use it somewhere else in the stack. You can still replace that same function again and again but you cannot replace something up in the stack or lower in the stack. And that combined with another kind of feature. So this functionality every place is often used in infrastructure like LibXDP for example uses that to attach multiple XDP programs to one network interface. That's something that the kernel doesn't provide but LibXDP allows you to do that. And that's based on the LibX, on the every place functionality for example. So if you are writing XDP programs you can probably not ever use every place because LibXDP already does that thing. So every place is something you want to use in your infrastructure maybe if you're building infrastructure for the BPF ecosystem that's an interesting thing to use and to know about but it might be not that useful in your applications. I wanted to show it anyways because it's super powerful and if you're reaching like the limits of what you can do with the other method I showed before maybe every place is the thing you need to really get forward. So be sure it is there but be careful when you're using it. Next up, when we combine programs at runtime, low time, whatever so we're building more complex applications out of our programs. We want to make sure that those applications do what they are supposed to do. So let's jump to testing. There is a way that probably a lot of you already know if you're using EBPF or if you're developing an EBPF and that's you're basically running your full application in some environment to test it, right? So in networking it's usually a combination of network namespaces, virtual ethernet pairs, shell scripts, bridges and so on and you basically set up a network within like a virtual network within your computer to have different namespaces and you run your application inside that, yeah inside that those namespaces basically. That's cool thing like I've written here it's used in self-test, it's used in demos. I've seen it used for example as well. So it's a pretty simple thing actually, right? You can just set up your network and just run your application within that network that you set up. Kinda difficult is how do you observe if your application is really doing the right thing, right? So you cannot test your application directly but what you're doing in networking for example you're usually just sending traffic through application and see if it does the thing you expect and if I don't know the traffic is dropped in between do you know why it dropped? You don't know, it's just not there anymore and then you start looking where's the issue coming from, is it coming from my application? Is it coming from, I don't know some setup issues is it if I run this same script on another system is everything in the same place that I expect it to be all these different issues that you get if you're just basically running testing on your main machine. So it gets kinda hard to observe what is going wrong what goes wrong and it's also kinda brittle sometimes it's like you have a lot of race conditions there if you set up network interfaces what does the system do with them and so on. And another thing is that if you do stuff like that you need to change the system of your developers, right? And I can say for myself I don't like people adding network interfaces to my systems I have like they're using IP spaces I might be using them as well and it's always causes some issues. So I want to show you one other technique that you can use to debug or to test DBPF programs that doesn't require all these things and that's called BPF test run or BPF proc test run. So that's a way where you load your BPF objects just your BPF objects not the full application but really just the kernel parts or the BPF parts of it. You load these objects like you would in your program as well into the kernel then you know already those programs pass the verifier and then you basically run that program without attaching it to the real thing so you don't attach it to your network interface or to your syscall but you just run it with a defined context that you give it like you pass a buffer to it with a context and you run the program with that. So you can think of it like unit testing for your BPF program you take the BPF program give it a defined input and run it and observe the output. That's of course super useful for network programs where the context is super simple it's usually just a network packet. It can be used for most network program types. It can be used for others as well and at least for syscalls it can be used for trace points it can be used and the full list is in the link and the slides are shared in the schedule so you can get the slides from there and click the link if you want to. I want to quickly give you some hints on how to use it because the documentation as some parts of the BPF documentation is pretty sparse but my hint is take a look at struct BPF test run ops in the BPF for like to get started so you can really see all the options that you have for running programs in that testing environment and the core idea of it is you build a packet in like user space memory, you allocate some memory you write your packet data into it craft the network packet as you would do it otherwise like it's just basically you place your packet data there or your context if you're speaking about syscalls then you pass all these like the full buffer you pass the full buffer to the kernel to that BPF test run function and the kernel then continues to execute the BPF program in the kernel so it's not like running in user space or anything it's really running in the kernel on your buffer if you ask it to it can run the program repeatedly so that's interesting for example for benchmarking reasons if you want to run multiple times and after the execution the kernel returns the result of the execution so it returns for example the modified network packet if you're speaking about that it returns the usual return value of your program so you know what would have happened if that thing was like being run in the kernel at like a normal hook and for benchmarking reasons it returns the average run time as well so it's a pretty nice thing that you can use for unit testing like I said for benchmarking as well especially for networking where you have like programs that are usually time sensitive so you want to process the packets quickly so that's why you can use it for benchmarking as well so next I want to talk about the eBPF helpers and kernel functions those are basically all the APIs that you can use from your application it's basically a set of functions that you can call and for you as a developer it doesn't make a difference if you're calling a BPF helper or the kernel function or kfunk that's not different it looks the same it's just a function call the difference for you as a developer is that BPF helpers are part of the stable user space API of the Linux kernel so there is a man page documenting them and that's part of the stable API so that will not change in a backwards incompatible way so your programs using BPF helpers will always like if they run on a current kernel they will run on a new kernel as well on the other hand we have kfunks and they are not necessarily stable there are some lifecycle guarantees attached to them like they should follow some rules and those rules roughly boil down to we don't want to change them without reasonable justification and we don't want to remove them without reasonable justification and there should be an application period if we want to remove them but it's not guaranteed so if we see there is an issue with those functions and it's reasonably bad that issue then we can just remove the function or the kfunk so if you want to use kfunks it's a bit more of testing you should put into place when new kernel versions are released or at least check if the functions are still there and if they still work in the same way sometimes these helpers like these kfunks are also explicitly unstable so there are functions that are explicitly unstable one example for that is currently connection like contract access you can access the contact table from your BPF program and that thing is currently explicitly label is unstable so you well be careful when using it that's the main takeaway from that the list of helpers and the list of kernel functions is basically growing with every release so it's hard to like take a snapshot in time and say like this is what you can do because it changes like from release to release there are new functions added new functionality that you can call from your BPF programs and I want to quickly show you how you could navigate that space so how you could find your way around and how you can find out which functions you can call and which ones are available so for BPF helpers it's kinda easy so there is the original man page that's called BPF helpers and you can just type man BPF helpers and you could get a long list of functions that you can use and I invite you to just step through that once and see what's available the functions are somewhat descriptive like the names are somewhat descriptive so you can at least guess which area of functions are available I would not want to guarantee that everything is documented in that man page so there is this header file that I linked here or that I showed the path here you can go to that header file and take a look so all the functions must be in there that all the BPF helpers must be in there that you can use for kernel functions or K-funcs it's not that easy there is no man page documenting all of them so in the end it boils down to you need to look at the kernel source if you want to know which kernel functions or K-funcs you can use some of them are documented in the normal kernel documentation but not all of them so a lot of them are basically only viewable if you look at the kernel source but one nice thing is all of these K-funcs should be marked with underscore underscore BPF K-func so if you just grab in the kernel source for that you should get a long list of functions that you can use and it should be almost complete like there shouldn't be any other functions that you missed that way right, so as a summary what I want to show you with that talk is that basically the BPF development environment got a lot better in the last years a lot easier, a lot more comfortable and so you can really build more complex programs more easily and one particular nice thing about everything that I show today is these features are not yeah these are not from like yesterday's kernel they are not super bleeding edge but they are present in kernel for a while already so if you have reasonably recent distribution or Linux distribution you can use these features today in your program like if you target something that is somewhat current you can use it today and you don't have to wait for the next two years for those functions to arrive and for those features to arrive and yeah with that I want to thank you for your attention and if you have any questions please go ahead so this is happening in a runtime right and previously you mentioned that now at the modern BPF you don't get unbounded by the loops so you can have unbounded loops so is it correct to say that every place can have now that I mean it is having a runtime so you don't need an upper bound and lower bound so you can have a number of loops yet unbounded by the I operations is it so the question is basically if we can kind of escape with every place and the loops like the unbounded loops if we can escape kind of the limitation that it has to stop at some point of time no not necessarily so everything still passes through the verifier and it's not really unbounded so let's take the BPF loop function for example that's also like it's a helper function as well so that's code that is not within your BPF program and you don't control it but the code that iteratively calls your callback function is part of the kernel so you pass a number of iterations to that function and then the kernel makes sure to call the next function a number of times and only that number of times and there is an upper bound to that so it's not unbounded in that way and no matter if you every place anything after that that doesn't change anything yeah, go ahead K funks is not documented I mean it's like Linux is a huge code base is it an open issue that people are documenting K funks because so what I can add there it's not that I'm not documented at all it's just not like the documentation is not exposed like you don't see like it's not in the normal kernel documentation for example that's a rendered online but if you search for K funks like the BPFK funk thing in the kernel then usually these functions have some kind of annotations on top of them saying like this function does this and that those are the parameters so it's not completely undocumented but it's not openly documented in a like online rendered way any other questions watching cat reviews okay thank you no no no oh oh oh one hand go ahead from a user perspective what's in there that wasn't possible before so he was asking for one sentence basically what you can do now what was not possible for yeah or an example for that um so for me the most interesting things are um let's say these composability things so you can um build more complex applications and really you're not bound to the strict rule build my program first and then I load into the kernel anymore so you can really change at runtime what is going on like in the most extreme case and what is possible for that is for example what I told beforehand the libxdp so the library that's used for interaction with xdp programs they found a way to attach multiple xdp programs to one interface using every place and beforehand there was a one-to-one mapping like your network interface could have had one xdp program you could run it but that's it and they made it possible that multiple programs can attach different programs to the same interface that's for example something that is enabled by these new functionalities