 My name is Donald Hunter. I'm from the emerging tech team in Red Hat My colleague Sanjeev couldn't unfortunately make it for personal reasons, so I'll take this one on my own And so let's get started The this is the eBPF 201 and the goal of this talk really is just to Well assume that everybody knows what eBPF is so we're not going to do the eBPF 101 but we're going to provide some Guidelines I guess for how to get started in in larger project development with eBPF and Really the goal here is to target people that are not kernel developers people like yourselves in the room who are kubernetes developers primarily I guess and Maybe somewhat unfamiliar with some of the Linux kernel development practices So we're slightly past the eBPF 101 as I said And we're going to focus on how we might tackle writing maintainable portable programs that can be installed into the kernel And this is kind of based on our experiences of learning as newbies as we went along So hopefully what I present here will be helpful to anybody that's starting on a eBPF programming journey So just before I get started can I get a show of hands of people that know what eBPF is? Okay, most of you that's good and a show of hands of people that have actually programmed some eBPF code Yep, a considerably smaller number. That's I guess to be expected Have people installed eBPF programs and use them for observability in their clusters. Yep a reasonable number Okay, so that kind of level sets what the audience expectations might be from the from the talk Okay, so I'm going to recap a little bit on eBPF Not right from the beginnings, but just modern eBPF as it stands today Give you an overview of what I think are the kind of applications that can benefit from using eBPF I'll give you a bit of an overview of the technology and the tools available in modern eBPF and then For the second half the presentation go through some what I would call describe development best practices And then at the end of that we'll talk about a few Kubernetes specifics Okay, so just a technology introduction first So eBPF is if it for anybody that's not familiar with it It's a sandbox virtual machine environment inside the Linux kernel Which has a verifier that checks whether or not your program will safely run to completion inside the kernel without causing any hiccups So if your program doesn't pass the verifier it doesn't even get installed into the kernel if your program does pass the verifier Then it can be attached to one of many hooks inside the kernel to execute And the familiar ones are the ones shown up in the top left-hand side, which is a you probably can't see the details but it's a network packet flow diagram and There are various hook points in the network packet flow where BPF programs can be attached and traditional BPF was used for packet filtering to just actually just you know Do packet dumps from the kernel? So all you were doing in BPF program was was matching packet headers and then dumping packets But modern BPF programs are considerably more powerful and can be attached at various hook points. So psyllium for example is a CNI that uses BPF hook points at most of the C group SKB hooks to run code to accelerate the data plane for for kubernetes networking There are a whole bunch of other hook points You've got k probes and you probes shown in the center here and an example of a simple program being attached to Userspace program to count the number of times a function gets called and report on it And then there are also many kernel trace points which are easily instrumentable with BPF programs as well So most of the trace points and new probes and things like that are useful for observability But there are other use cases that come to the fore as well, and I'll just talk about them a little bit BPF programs have access to a set of helper functions in the kernel that let them do various things a set of maps for sharing data between different BPF programs and with userspace programs and More modern BPF has access to k funks and k funks are kind of a new development It's been enabled by a new feature within the BPF subsystem called Kory, which I'll talk a little bit about later And I would say that k funks supersede helpers Helpers are kind of like the way things used to get done in BPF code and k funks are the new way of doing things So you'll see more k funks become available But there's an essential set of BPF helpers that aren't going anywhere. They aren't going away They aren't getting deprecated and we'll talk a little bit about that later as well Okay, so from a use cases perspective There's a number of things you can do. There's there's the obvious things which is observability a lot of people in the room have used BPF based observability tools You can observe macro things and micro things you can do very very detailed instrumentation Or you can gather stats about application execution times and things like that The other obvious use case is extending the Linux networking stack with with BPF programs a la salium and other Network acceleration tools, but there's a whole bunch of other potential use cases such as Implementing hot fixes and workarounds or mitigating against external attacks and we're not just talking about networking attacks They may be API attacks or you know things that you're protecting against Containers or something like that from from being able to do to your kernel and The interesting thing about all of these use cases is they can be applied at runtime There's no kernel reboots you install a program and it starts executing as soon as you attach it so you can you can upgrade Your mitigations your protections your networking features without ever having to have a kernel reboot So you can imagine longer term more kernel features might get developed as BPF programs because the management lifecycle the day to operational Tools that you have available to you are much more flexible than with having to do kernel installs and system reboots Okay, so just where are we on the technology maturity map so Classic BPF back in what I'm calling phase zero was the early technology which was only for doing packet filtering for for doing TCP dump and that sort of thing and then Somewhat about maybe eight ten years ago. I can't remember exactly BPF in VM was extended to allow much larger programs and to be usable for many more things in the kernel And those were the early days of BPF where the toolchain was very weak the verifier had had lots of issues and bugs and Developing BPF programs was generally quite hard and probably actually harder than like kernel development We're now in the phase where we're in modern BPF as I would call it and there are a lot of new tools Which actually make the the developer experience more pleasant? and We're getting to the stage where BPF development is Something that more than just people that are traditional kernel developers can control the embark on There's no reason why anybody that saw a developer in this room can't think about starting to develop BPF programs as part of as part of their application development process Modern BPF is built on some foundations, which I'll go through in a moment but just listing them off here BTF which is the BPF type format and Technology called Corey which stands for compile once run anywhere and so we're really getting to the point where there's new innovations happening to make the user experience better and There's more richness and features being made available to BPF programs We're getting to the point and it's you know in in the future's roadmap where you'll be able to do dynamic memory Allocation within the kernel you'll be able to use at well I think as of today you're able to use spin locks and things like that for doing a lock Protection around data structure modification in the kernel. So the features available to you are getting incredibly powerful okay, so Just from the top here modern BPF see as I'm calling it It's like restricted C, but it's not hugely restricted anymore You get to do on bounded for loops for example, you get to call other functions in BPF It's also got some interesting extensions which are enabled by the BPF type format and Corey such as there are macros for relocating Access to kernel data structures, so you can be portable across more than a single kernel version and The next one down here is BTF, which is the BPF type format and within every kernel now going forwards from the five five dot X series and the the 418 kernel that ships with rel 8 The type information of the kernel is actually provided supplied by the kernel So you can use a command line tool such as BPF tool to interrogate the type information of the running kernel and then when BPF programs adapt to the running kernel, so you've got complete portability across Across kernel versions, so early BPF programs. You couldn't do this You had to recompile your code for every single target kernel version which really really limited the portability and Deployability of BPF programs So modern BPF is stepped past that and gives you true flexibility The the technology Corey Which is implemented in Lib BPF in user space takes a BPF program and relocates data structure accesses to the actual fields within a within a kernel data structure And I'll show you an example of that later Which means that you don't need to worry about mapping having the exact kernel headers for the kernel you're going to target at deployment time gives you a bit more flexibility and Improves the the developer experience considerably and deployment and day-to-operations experience as well And finally the last technology piece that enables modern BPF is is Lib BPF Which is the the user space library that provides all of the Program relocation program deployment into the kernel And the user space library functions for for accessing BPF maps and and other things about your BPF program So these four things together give us much better ergonomics the improve type safety the improve portability They reduce boilerplate and they give us much much better developer tooling So to the point that there's good support for BPF programs in in C in Golang in rust And I'll talk a little bit about those when when I talk about tool stacks there that are available to you It's fair to say that there are still quite a few rough edges the biggest issue is Verifier getting your program past the verifier is always a challenge and The kind of error messages you get from the verifier are also a challenge. So There's a lot. There's a lot can still be developed and improved from a developer UX perspective in getting programs that Give you feedback on why they are failing verifier checks and why they can't install into the kernel Okay, so just a little bit about API stability and what what what you want to try and aim for from a BPF program perspective For anybody that's not if not familiar with kernel development and maybe not familiar with the term the kernel you API it might come as a surprise just how How how things change inside the kernel and yet outside the kernel everything is is fairly static So from a from a kubernetes background where you go through an alpha and beta cycle before making version 1.0 CRDs or or resources And then deprecating the alphas and betas and so on the kernel models a bit different the kernel External API is once they've been published once they stay the same. They never get deprecated. They very very rarely ever get get Marked as for end-of-life They are they're maintained as is forever Internally everything changes well not all the time But you know a developer a kernel developer feels free to change an internal API is at will So if you're writing a BPF program that wants to access the kernel internally then there's shifting APIs That you need to be aware of but your external API is Going to be the same How the how the the user space accesses your programs and accesses the kernel is unchanging So what this means is that there's a set of things within the kernel that you Probably want to rely on from a BPF program perspective because they are part of the the user API That you can guarantee you're not going to change but new innovation within the kernels going to happen with these things that are called k funks and Kernel developers have said we're not making these part of you API. These are going to be part of the unstable kernel interface Which seems a bit of a challenge because if they want us? Developers to write BPF programs that use these k funks, but they're not giving any guarantees about their stability Then we're basically being told use an unstable interface, which none of us really want to do So it will be an interesting journey to see how this unfolds as kernel developers say k funks are unstable and the Consumer community wants them to be stable But they're basically saying they will make the best effort to maintain things and if you shout loud enough and you're using things Then they will be generally kept stable for you K funks are basically the way into the kernel to access things to modify things and and get features so for example the most recent features that were made available to BPF programs as k funks is Connection tracking in TCP connection tracking. So if you want to Write networking features in BPF programs, then you can use the underlying contract subsystem from within the kernel You don't have to write your own in BPF code, which is clearly a benefit okay, so Going to step forward into The architecture of a running BPF program now this is looking at things from the perspective of C software stack where We're using Lib BPF and more specifically we're using a thing called BPF skeleton and this is a set of tooling that Is built around Lib BPF and BPF tool on the command line where I can write some BPF code and then I can ask BPF tool to generate me a skeleton C program and the bit skeleton C program has a set of utility functions to load my BPF program into the kernel to access any maps that my BPF program Wants to use so I essentially get The the skeleton of the user space program I need to write with the BPF program embedded within the user space program So the talk to the diagram that's what you see you see a user space application that is linked with Lib BPF That has a BPF program embedded inside it at one BPF dot object is the BPF program When my app runs it uses Lib BPF to install the app into the kernel And that's when the BPF Lib BPF library Uses the kernel type information to relocate my program to match the data structure fields that exist in the running kernel So I can run I can install the same program on a 5.2 kernel or a 5.14 kernel or a 6.3 kernel and Lib BPF will relocate my program to whatever might have changed in the data structures within the kernel The BPF subsystem in the kernel provides various functions including maps and These are things like hash maps array maps bloom filters Least recently used maps and things like that my BPF program gets access to them and the user space application gets access to them via Lib BPF so my Kernel program can for example in this example here. I've got a network data path at the bottom my program is attached to The TC ingress hook in the network data path and I might just be doing some metrics counting in my program and what my program Registered at the TC hook does is just as it sees packets It can do some packet header inspection decide what types of packets they are and then right into these maps I've counted another packet of a certain Say it's a TCP packet destined for a specific port or something like that and then the user space application can read The metrics from these maps and report it to Prometheus or wherever your reporting application is in user space So that that's the kind of like the simplest example of a BPF program you could write If you're doing more packet processing then specifically in the networking data path Hooks like the TC hook you get access to the full packet You can rewrite the packet on its way through so you can change packet headers You can change packet contents you can encapsulate you can de-capsulate So you can actually build pieces of an entire networking stack Now it's important to say that the BPF program is Run to completion event driven So the the hooks that I was describing on the previous diagram the TC hooks or whatever the trace point hooks Trigger programs to run as if they're event driven So if you want to build a more complex application likely you're going to be triggering event driven pieces of BPF code at multiple points and the state of Where the application is running is maintained in BPF maps So you may have multiple pieces of BPF code at different hooks all reading and routing from the same shared maps which is their shared state and The BPF programs of course get access to other kernel subsystems via either helpers that existed or k-functs that those subsystems expose so that gives you a kind of fairly large overview of the architecture of a running BPF program so from a team development model perspective you want to enable more than one person to work on on your BPF program and You need to think about the kernel versions and Supportability that you want to support I've got a diagram a little later on about the kind of challenges that you may have with kernel version support and how you might want to baseline and how you might want to tackle portability challenges that come come along but it's important to point out that Containerization does not isolate you from the kernel version So you're going to be using bare metal or or a VM to provide you with the kernel version against which you want to develop So if you've got if you've got teams with different dev systems, then you need to baseline across those you need to Decide what your minimum kernel version you want to support is you need to think about What features that you need from those kernels and verify that the BPF data structures and all of the other things are available in the kernel releases you want to support So yeah eliminating challenge dependencies on specific dev versions is key here Okay, so just going through some more kind of specific recommendations for for for BPF projects Firstly like I'm saying plan for target platform versions So the challenge here is that BPF development is a has been a moving target for the last several years, so the types of maps you have the types of BPF hooks you have and the types of Support functions you have vary quite what quite widely across kernel releases and of course as development goes on that this is not going to Stabilize anytime soon So if for example you want to you want to support rel 8 and you want to support You know Amazon Linux for AWS or whatever and you need to support multiple architectures and You need to think about well, what's my base release? So is it a 418 kernel which has had a lot of things back ported from the 5.x kernels? Can I baseline on 5.14? You know those are the kind of questions you need to ask yourself and if you need to support a very broad range of Linux releases then you've got to be ready for features not being present or the Modern features not being present in all the kernel versions that you want to be able to support and We'll touch a little bit about that a little later on okay, so Programming stack perspective you've actually got quite a lot of choice these days for a long time All you could do was program BPF in C And program your user space in C. It's fair to say that with LibBPF the richest Set of features is still probably provided for C but LibBPF has bindings to to go with them with LibBPF go and LibBPF RS for Rust but then there are also native platform libraries like Sillium EBPF Which has maybe not quite the full feature set that is provided by LibBPF But sufficient for for many Kubernetes applications and on the Rust side There's a Rust native library called IA and IA lets you write user space and kernel space code both in Rust and Makes it very easy to write an application that passes data between kernel space BPF program and and user space So for for some developers IA might be an excellent choice but for for a lot of things you may have no choice but to use a LibBPF especially if you're using some of the more modern hooks and some of the more modern features that maybe haven't made it into the the native libraries From a from a kubernetes perspective, I imagine a lot of people will want to use Sillium EBPF Even if it only has a subset of the features, but of course you could write you could write Helper Programs alongside your go code that are written in C and you are written using LibBPF go So that you can get the full feature set of the LibBPF library for program loading And there are other alternatives coming coming along as well, which we'll talk about a little later Okay, so the the next thing is how do you use portable type definitions? So you may have heard people talk about using BPF tool to export VM Linux from the kernel and then doing everything developing against VM Linux But of course that means that you're exposing your application to the entire kernel Internal API surface which is an unstable thing. So you could end up accidentally using things You don't plan to use but also it's fairly unwieldy. It's like several megabytes of header file So instead with Cori you get to write your own version of kernel data structures with only the fields You need in them and the example here is in my program I want to access the struct SK buff to get at the at the the packet contents from a TC hook and The only two fields I need from the struct are data and Len So I write a version of struct SK buff that just contains data and Len and when I install my BPF program the LibBPF Relocates those two field offsets to match into the actual struct SK buff, which is huge and the two fields are separated quite Significantly in the data structure and I don't need to care when I access my code I just use the right accessors and everything works just fine So I mentioned if you if you have to support multiple kernel versions and the kernels don't always have the features You want then there's a set of tools available to you to do kernel version probing and feature probing and you can do feature probing Probing all the way down to the individual struct field level if you need to but the two examples I've given here are checking the kernel version from within your BPF program and checking whether a feature such as Lightweight tunnels is available from within BPF as well And so you can then start to enable features to enrich your program on more modern kernels But then fall back to behaviors that maybe don't have those modern features on older kernels and then From from a Kubernetes perspective specifically There are a number of operational challenges First off is that BPF programs can only be installed if you've got cat BPF and sometimes you need several other elevated privileges as well And you don't really want to be giving too many containers full privileged access because that's right Kind of defeating your your security goals in your in your Kubernetes cluster It's also difficult to handle multiple BPF programs There has been mentioned in previous Conferences that for example Sillium getting broken by data dog or data dog getting broken by Sillium or whatever because they're competing for the same Same hook points within the kernel and so that's a challenge how to handle multiple BPF programs for different application purposes all running on top of the same kernel and Then there's just visibility and debugging related problems where you don't know why something's working or not working the way you expect Because you've got no visibility of what programs may be running within the kernel below your below your Kubernetes deployment So I think that this is where you can consider new BPF tools that are getting developed There's a tool called BPF D that is being developed by the emerging tech team at Red Hat Which aims to tackle some of these problems and the way BPF D Approaches this is to say well We're going to be a system level demon that installs BPF programs into the kernel on your behalf And it's going to have a gRPC API so that you can you can provide you know an authenticated API endpoint with which your your Kubernetes application can ask for an application to be installed or can interact with the BPF demon to query data from maps from your BPF program and The goal is for BPF D is also to integrate well into kubernetes and use the kubernetes operational model as you'd expect. So the first thing here is Packaging up BPF programs themselves within OCI images so that you can use you can install BPF programs into Docker registries or container image registries and you get the The versioning that comes with that you get The authentication and so on that you'd expect of registries for download and then declaratively saying what programs you want installed in your kubernetes cluster with Node selectors and attach point selectors and things like that So you no longer need to programmatically have your programs loaded, which is great for observability programs and could be incredibly useful for Infrastructure programs such as networking programs as well and how this looks from an operational perspective your applications Don't need elevated privileges. The only thing on each node in your cluster that needs elevated privileges would be the BPF D and BPF D is the only thing that would need to Make sys calls into the kernel and would need to access maps for which it might need elevated privileges And then all of the user space applications can interact with the the demon either declaratively Using the BPF operator in in kubernetes or using the gRPC API is provided by BPF D so that this is a in development project at red hat, which I think we'd like more people to get involved with and I think it's going to really Enable more use cases for BPF in in kubernetes clusters So finally just a quick recap on some of the things I've said in terms of best practices so your goal when you're writing BPF programs is to Have a clean stable interface between your BPF program and user space and Have a stable interface between your BPF program and the kernel as well And so interfacing with the kernel you're using the coree technology and you're writing a minimal API surface against the kernel which the current which gets relocated to the running kernel and its data fields when you install you use macros within your BPF program to ensure that The kernel knows when you're making API accesses that need relocated And then with user space the interesting challenge with BPF program is that you want to share some data structures between your kernel and user space so you need to be very very careful that you use data types that that that can That you have header files for in user space and that have meaning in the kernel And so typically that means that you can't use you must not use Kernel internal types when you're writing something that's to be shared with user space So you may be familiar with some kernel internal types such as u16 u32 Well, there are versions of those that are available for for sharing with user space that are in the system headers typically when you've got the the kernel headers installed on your platform and between within your own BPF program you've got strongly typed data sets to work with as well Now it's fair to say there's still a lot of tribal knowledge in in BPF development, but there are a number of good links and I when you download the presentation after after the talk You'll you'll be able to access all of these links to to the documentation that's available There's some tutorial documentation available in in eBPF foundation The kernel documentation is steadily getting more complete and in fact a good call to action is for anybody that is familiar with kernel development You can you can contribute to documenting BPF program types There are good man pages for BPF helpers to my knowledge. There are no man pages for any of the BPF K funks yet And then there are good library docs for all of the user space libraries that you might want to use LibBPF, LibXDP, IEA and Celium eBPF and Then there are reference blogs for some of the stuff. I've talked about in the presentation today such as BPF core e the BPF core reference guide and then there are good examples to work with such as LibBPF bootstrap and and the Practical BPF examples repo that's provided by the XDP project But like I said, there's still a lot of tribal knowledge So anybody that is an expert, please step up and help fill the gaps in the tribal knowledge and Make BPF on ramp easier for everyone. Thank you