 Hello, everyone. You could all hear me. Okay Okay So, yeah, so the title of my talk is portable BPF with Cora II. I Am on the open source engineering team at Aqua I'm from Brooklyn, New York. Here's my pronouns. You might see me on the ebpf slack with this little crab guy So here's an outline of my talk followed by the slide. We're going to be talking about my experience as the One of the maintainers of Tracy We'll talk about what Tracy is our goal for distributing it and how Cora II helps us achieve that goal And we're going to look towards the future of next steps in ebpf development especially as it relates to portability and distribution and how Cora II fits into that so The primary project I work on is Tracy. It's a ebpf project that That is It handles runtime detection and forensics using ebpf and the way that works is it uses hundreds of different events Things like system calls trace points Network TC hooks LSM hooks all things that you've learned about earlier today And if you're looking for a simple tool to use to gain a ton of visibility into your system Tracy is the kind of Drop-in tool that that can let you do that And it also lets you apply policy on top of that And in this example we could see an exec VE system call that's being traced So we're we're specifying exec VE to be traced and we're we have all this information about it So we have the actual command line that caused this exact VE to trigger We have the timestamp. We have Process ID parent process ID all of this additional context. So it's a lot of Reading of memory in the kernel. That's essentially the primary thing that Tracy does so the problem that we've observed while developing Tracy is that From kernel version to kernel version there is There becomes differences in Data structures as we're as we're reading them So in this example here, we could see that there is a different field that we're reading in the task struct For kernels before 419 and after that we're reading something different So our initial approach to handling these differences is to Have at runtime this check to see What kernel version we're dealing with and read from it accordingly? This of course has flaws One being that sometimes fields are renamed or completely removed So we could run into errors where fields don't exist. So the code won't compile properly Against the headers that we're compiling with Also, it's just a real pain in the butt to deal with when we want to Support all these different distributions and all the versions of those distributions and if you're not paying attention to every mailing list and You know looking at every diff and every patch. It's hard to keep track of all the all the differences in versions Not to mention if we have somebody that's running a custom kernel You know, I'm not going to have a check saying like if we're running on grants custom gen 2 kernel Then read from this field. Oh So this is one approach that we've taken and clearly has it has its flaws In addition to that what we were doing is doing the same thing that BCC does which is build on the Kernel that we're deploying our project to so as we deploy or as a user picks up our picks up Tracy and they run it for the first time it would look for their kernel headers build against those And then and then run from there and cache it somewhere But this is of course an issue because you know, we would get bug reports about Tracy not being able to be built because Let's say the kernel headers are missing or you know, perhaps we just Aren't keeping track of all the different kernel header versions for every major distro like we would get you know We don't want to have to support scientific Linux sex or something like that like something very specific So what do we want we want to be able to write BPF code Without worrying about kernel version Differences we want to be able to distribute the BPF object where we can just host a single BPF object have people download it and have Tracy Run it without any issue We don't want to have to rely on a special kernel feature to do that And for the most part we don't need our users to know about that So when do we want it? Well, we have it now so This is what CRE compile once run everywhere enables us to do so CRE is A concept that's enabled by Libbpf And these are essentially exactly what it gives us so Right so compile once run everywhere or CRE is a concept enabled by Libbpf I There are helper functions within Libbpf that will take into account kernel debug information to try and relocate kernel fields as they have changed from kernel version to version and Just to clarify that Libbpf is a user space library That is in the kernel Repository And it's used for interacting with the BPF subsystem so Loading BPF programs and attaching them to the various hooks. So let's go into how this is actually working so The first thing I want to talk about another another acronym for us to to learn about is BTF so BPF BTF stands for BPF type format. It's a special a simple space efficient Debugging symbol format. So if you've ever worked with dwarf, it's basically it just a compressed paired down version of that So if you think about it The Linux kernel is compiled program just like any other And it has all the symbol information or you can have all the symbol information for it but since you know the binary itself is is massive all the debug information is even bigger or makes it even bigger so BTF because it's Compressed and because you know engineers have found a way to get all that same information in much less space enables us to efficiently Do all these relocations and by relocations, I mean, you know if we were to Let's say read a specific field in a test struct on a particular version and that changes in another version BTF enables us or enables really LibBPF to read that debug information See where that field has changed like the actual memory offset Find it for us and and read from there instead of where we initially specified it to be so here's a diagram that will Hopefully help explain that so Here you see we have on this this right box or this box on the right Is Tracy itself? So we have the user space side We have LibBPF that it's using to interact with the BPF programs and the BPF system We have the Tracy BPF object itself and then we have the kernel BTF the BTF for the kernel it's running on That BTF can be found either in Slash sis kernel BTF VM Linux so VM Linux is a special file. It's the Compiled kernel stuffed inside of an elf binary That will have a Dot BTF section if the kernel was compiled with a specific Configuration option config debug info BTF Otherwise LibBPF also has an option for us to specify an External BTF file and that brings us to another cool project that my team has worked on called BTF hub so if you are trying to support in your BPF project if you're trying to support a Particular distribution or a particular kernel where this can this kernel compile config option hasn't been specified You normally wouldn't be able to run a program that has CR e Enabled so you would have to go back to that previous solution that I that I outlined that we were previously working with But instead you can pre-compile on your own the The actual kernel for that version include that debug information extract that BTF information and Supply it as an external file Which is what we've done in this GitHub repository where you could just instead of doing that compilation yourself Download this these BTF files and load it with your BPF program So now we want to talk about code changes that we had to make to Tracy And really this is more like code simplifications. It really Honestly cleaned up our code base The first thing is VM Linux H. So in the same way that we have that VM Linux file, which is the compiled kernel VM Linux H is another generated file from that So it goes into that debug information with all the different types And it produces a header file that has all of their type definitions So if you take a quick look at this you could see the definition for underscore underscore U32 so if Your BPF program is relying on that data type You can import it and use it accordingly using VM Linux H. I wrote a blog post about this file There's there's more to it that is very cool. You could check it out the URL there on my personal blog So you could see here the changes that we made in in our imports. We actually are currently Currently have both of these we have a compile time Variable that that we define but we're moving towards being able to remove this This this left side here But on the left we have all these user APIs and internal headers that we're importing Getting those definitions of the data types that we're trying to read from in kernel on the right You see that we have we're just importing VM Linux H as well as this Additional file that I want to talk about So that's a missing definitions file So while we do have the convenience of being able to just import a single header file the VM Linux H to get all of our type definitions That does lack all of the macros and all the functions that we otherwise would be using in Those kernel headers so macros certainly aren't put in the type information or aren't put in the debug information And functions aren't aren't included either. So I had a conversation on the mailing list about the best way of handling it and right now the best solution that anyone was able to Give me advice for is to just redefine it. So theoretically if there was a macro that Changes from version to version we would have to be manually updating that So we still have we still have work to do there and Essentially the last change is just using the correct header file. I'm sorry using the correct helper functions So you could see here. We have this internal macro in Tracy called read kern where we zero out memory before Doing a kernel read And the only difference here is that instead of using BPF probe read we're using BPF core read The arguments are exactly the same. It works in the same way. It's just under the hood using BPF And here's exactly that in action reading a mountain in space ID So the most exciting way that we've been able to change Tracy in the real purpose of this talk has been distribution. So Previously if we Wanted to publish binaries of Tracy Where we want to support all those distributions that I had on that slide earlier We would have to say, you know, here is Tracy for Ubuntu 2004 or here is Tracy for 2004 or Or 2104 or for specific kernel versions Now what we can do is we compile a single BPF object that has btf enabled and We embed it in our user space go program. So at runtime when we start up Tracy it unloads that from memory and Loads it and you know, we can compile that go program statically So it's just a single artifact that where that we can ship and run on any system So this is you know Again this the same slide, but all these simple changes enable enable us to not get bug reports about Tracy having garbled info on a particular distribution Or someone's custom gentoo based kernel and certainly not on production sent to west servers We now have the confidence to distribute Tracy to wide array of environments and can focus on developing the great new features that we want to add and less on the changes in different kernel versions and More broadly, I would say that this is the future of BPF development that this isn't going to be Something that people are going to ask like is is this BPF project see already enabled, you know as new kernels are Or major distribution kernels are shipping with Btf enabled by default You won't really have to worry about about that at all so you can have your BPF program written in confidence to Not have to worry about if CRE is enabled or not And for those custom kernels and those older kernels, that's why we have btf hub so I actually want to show a demo of exactly this Going to quickly mirror displays make it easier so Here on this Linux VM. I have a Sorry, is that is that big enough? Okay, so here on this Linux VM I'm running Fedora 34 this has a 5.12 kernel and I'm just going to run a version of Tracy so you could actually See what it does just Look at the first 20 lines. So Here with this debug flag turned on we could see that I unpacked the CRE BPF object into memory And I'm loading that and and running with that and this is actually a static binary And you could see that it's running properly. We have All these different commands that are being run we have the events that are actually triggering These to be printed. So this is an LSM hook of a file being opened This is a proc mem info. We have all the arguments that are being Passed back into user space and being interpreted correctly meaning that it's the We're relying on the proper definitions and if we're not relying on the proper definitions CRE is handling it for us to to find the Where they've been relocated to? so Here I also have another VM and this is running Ubuntu. I believe it's 21 oh four This is running a 5.11 kernel I'll run the same command and yeah, look at that. I have a security file open for starting it as well and It did the same exact thing where it unpacked the embedded BPF object from memory And loaded that into the kernel and ran it so Again all the arguments are properly properly displayed and the kicker to all of this is that if we look at the Shaw It's exactly the same file. That's where you're supposed to all clap and go Okay, so right so that's what CR CRE gives us Next steps things that we want to work on Working upstream to find a better solution that than that missing definitions header. So, you know, one of the suggestions was having a like a in claim to have a Like built-in attribute for checking if a type is already defined so that we can still import the headers that we want but Not worry about them having a collision with the names in VM Linux.h A better verifier output. So that's certainly not something that's related to CRE or even LibBPF That's across the board, but we certainly ran into issues where we had verifier output that we just could not make sense of during this switch documentation so a Recent effort that I've been working on is launching the LibBPF read the doc site. So if you haven't seen that It is a full API Documentation for LibBPF you can go check it out and contribute. It's sort of if you're familiar with go and go doc It's exactly that so as we document more of the API that documentation will get better And certainly we're going to keep working on on BTF hub. So go check out that repo as well And and that's my talk. Thank you very much