 All right. Welcome, everyone. Surprisingly, we're going to talk about eBPF today. Specifically, we want to mention how we actually build a program that uses Rust in the kernel space and go in the user space. So let's first start with the interactions. Yeah, so hi. I'm Dave. I work at Red Hat as a software engineer. My background is in software-defined networking and most recently, eBPF and a lot of Rust. You know, it's quite often that you can forget to update little bits of your code and then spend hours debugging something which was really trivial, which was, oh, actually, I'm sending the wrong thing on the perfect band. It will nag at me and tell me, Dave, you're not allowed to do that. And that's a lot of what I was able to offer for you if you choose to use that in the kernel space. All right. Let's also talk about how actually Go helps with the, OK, we actually heard a lot of nice things about Rust, but how actually Go comes in the equation. First of all, writing Go applications are super easy. So you can actually get faster. And you also have the ability to produce static binaries. And we will see in a minute that you can actually embed the eBPF programs in the binaries themselves as well. But for me, I guess, the most important part, like a pro for Go communities, there's a lot of development going on around CNCF. And we have a lot of packages that we can just utilize. In our case, it's gone again. We are talking to the container runtimes. We are talking to the Kubernetes API. For all those work, we can actually find some SDKs to just embed in our user space code and discover the C groups rather easily, and then attach our profiler programs. I will keep doing this. All right. Can you progress on? I forget what we're going to talk about here now. Ah, right. Yeah, how it actually works. Yeah, now that's a question. So the way that we're actually able to write our Rust or Rust BPF programs is all really thanks to LLVM and the work of Alessandro Decina, who is my co-conspirator on IEA. He was the one that actually added support for BPF as a target in Rust C. So Rust knows how to compile down to BPF byte code, which is amazing. And we're just building on top of that using Rust. So we were able to use Rust macro system to make things really easy from a developer standpoint. You just annotate your code and say, hey, this is an XDP program. And it will keep you on the right path to writing that. OK, in this section, I'm going to talk about LibBPF Go, which is the library that we use in the user space. Thanks to IEA being super nice and compatible with the LibBPF ABI, we can just use the LibBPF Go loader in user space and just around the programs. And there are certain advantages to this. We can just study the link, the LibBPF. And we just have core support for our user space program as well. And if you want to change any implementation, it's kind of, in Parca and in my company, we are true believers of open standards and LibBPF kind of an open standard for us. So keeping in place gives us the flexibility to just jump between, for example, we just rewrite our whole our LibBPF programs in Rust, and it just worked. So yes, I'm just skipping forward. Sorry for the agony. So in our example, I was going to just demonstrate what we do actually with this program, which in the user space, we just discover the C groups and then we just attach the BPM programs to those C groups. And we collect data and aggregate them into the EBPF maps. And in the user space, we just convert them into the PEPRO format, which is another open source format. And we send them over to the search layer. So the program is actually rather a simple one because most of the APIs are just provided with the EBPF and LibBPF. We just get the stack IDs and the stack traces from the kernel, and then we have a map that actually counts the unique stacks and those aggregated values then sent to the user space. So I will just try to walk through if this continues to work on the Rust program. So Rust programs are for the EBPF, they are rather simple. So they don't have any standard library or they don't have an entry point, like a main program. So your program starts with important couple of libraries from Rust create that IO provides, which is specifically for the kernel space. If you are using an old kernel, you need to actually put a license section in there. Otherwise, compiler will just rant about it. But it's not the case anymore. I guess if you are using something recent, 5.2 plus, I guess. And to be fair, this is not indicative of the nice sort of API that we like to provide you in IEA. It is a hack to work around all kernels. So don't read into this one too much. And then we just create an EBPF map to actually count the stack traces. And this is a struct representation of a key that we put in a hash map. And in that hash map, the important part, I guess, in this section of the program is having a map macro where you can actually define the EBPF maps. And having this key, it needs to be represented as a C struct since we are just also sending this thing to the user space. And we need to be sure that there is nothing rather than the data itself, like padding or what not. Yeah, and type safety. Look, so here is our map. And it's a hash map of stack count keys to use 64s. So we can't make any mistakes with that, right? Exactly, yeah. So what else? This is how we actually attach to an event. In this case, it's a perf event. And this is another feature for Rust. We can use actually results for that. This is a rather common pattern in IOPrograms. We have an event handler, but that event handler actually calling an helper function which returns an result. And we can just in a high level handle all the events basically rather successful or failure. Yeah, so that's important because there's a lot of fallible operations that can happen inside an EBPF program. But we can't panic inside EBPF because we're not allowed to. It's not a permitted instruction. So this actually gives us a relatively nice way to handle errors. And we also have a logging library as well, which is kind of like BPF TracePrintK on steroids. So you can actually leave it there in production as well for your logging. And that will help try and debug some of these errors if and when they do occur. And this is the actual program. We actually update the maps by also getting the stacks from the kernel itself. And then we just count them and update the maps. I will start being a little fast here. I don't want to take so much of your time, which we already had. So you can also panic in the EBPF program. So you need to add a line like that to actually handle the panics. This is literally just making the Rust compiler happy that you actually can't panic. So we just have to have it there otherwise Rust gets upset. Exactly. So this is actually how we built. We still need to have a nightly tool set to make BPF linker. And there's already a task in there for you to just generate that program. And let's quickly check out the Go program that we have. Like after you compiled that EBPF program, Go has a facility to just embed them into the binaries. You just say that, OK, this is where that our object file actually stays and just embed that to the library. We actually mentioned that in the Rust as well. Like we need to match the keys so that we can read them in the user space as well. In this case, kind of the memory layout is the same with the C program. And the Go program. And we can just deserialize it in the user space. This is how actually you use the BPF. Go to load that module and the object. And then this is actually how you start listening to perf events. And then you bring everything together. And you attach your program to that specific perf event. And the data starts flowing. This is you need to just keep a reference to the maps that you have. And you can just get that maps using the LibBPF Go. And then there is another kind of nice API that LibBPF exposes. You can read them in batch. You can do your magic in your user space. This is just a snippet to how to build it. LibBPF Go user program. Slides are available. You can check it. The whole program is in a single PR. You can actually see the difference how we get from the C program to a Rust program, even with how we actually updated our build tools, whatnot, like GitHub actions. Just check this PR out if you want to go down that path. Cool. So what is next? The whole idea behind this, we are planning to write rather complicated profilers that runs on the Linux kernel to just support, for example, dynamic languages or the VMs like Java. That's why we just go down this path. At this point, we didn't have a lot of C programs written for BPPF, so that's also helped us. We are planning to add them in Parker soon. For the IABids, do you want to? Yeah, lots of work to do in IA. Plenty of program types, still not implemented. It's an easy thing to do. We're just looking for users. So if you're interested, come and grab me later, and I'll be happy to point you in the right direction. And most importantly, we really want to pitch both of our projects. We need help. We need contributors. If you're interested, just go and find us. We have links everywhere. You can park as a Discord channel, IA as a Discord channel. We are rather active. You can just come and ask for help, and we can actually help you to get into the IABPF land as well. So thank you for listening, and thank you for bearing the agony with us. It was rather such a mess. Sorry for that. And these are the links. Yeah, hit them also. Quick shout-out to the IABPF and Aqua Security. They really helped us as well. Yeah, they are also looking for contributors. So yeah, please. Thanks. All right, yes, well done for dealing with all those technical issues. I see one question already. We'll just take a few questions. So first of all, thank you for your patience in dealing with all the issues. Really appreciated the talk. When writing IABPF in Rust, you have to make a lot of unsafe calls. When you call unsafe, you turn off a lot of the compiler protections, borrower checking, et cetera. What are some ideas or plans that you have to move away from that so you can actually gain the full power of the Rust compiler? That's a really good question. So a lot of it, we're trying to abstract away inside the library itself. So you, as a user, don't have to do the unsafe work. We want that for you. So you just call hash map insert, and all of the unsafe stuff happens in the background. We can't get away from it, because ultimately, it's a foreign function interface, and changing stuff into real pointers to pass into a CE program is inherently unsafe, and working with real pointers is unsafe. So yeah, the real way is just abstract that away from users. That's the plan. More questions? Anyone? Yes, I see another one. So first of all, thank you for the great presentation. Enjoyed it. The question is, what is the performance overhead against writing in C? In a kernel, of course. None? Not that I've observed anyway. I mean, ultimately, it compiles down to BPF instructions, and the program that you would write in C is basically the same as Rust. There's a couple of things around some of the LLVM stuff, like memset and memcopy, which we're just working through. But for the most part, they're identical performance-wise. All right, one more question if there is one? If there is one, am I missing anybody? All right, so let's give one more giant round of applause for all those technical difficulties.