 Good morning. Good afternoon. Good evening. Wherever you're hailing from welcome to another Stack Rocks community office hours. This is episode two. We're gonna be talking about ebpf 101 today And I'm joined by the one and only Robbie Cochran. How are you Robbie? I'm great Glad to be here. You want to introduce yourself to the audience just to let them know who you are what you do around these parts? Yeah, absolutely. So I am an engineer at Red Hat formerly was an engineer at Stack Rocks And I've been working in Kubernetes security for a few years now and in particular, I've taken a focus and interest in Kubernetes security and ebpf together and Yeah, cool. So we have A doc for some questions in it that I'm trying to pull up First question. What are the use cases for ebpf? So ebpf you can think about ebpf is a better way to extend the kernel And the use cases are pretty wide open Observability security Networking features Ultimately, you can think about, you know, I like to use the analogy and it's I know this has been used before from the perspective of web browser without JavaScript is kind of like a Linux kernel without ebpf and you if you if you thought if you think about all the things you can build with JavaScript in a web browser and The the way that Linux kernel development used to go is that you would have to if you wanted some new feature in the Linux kernel you'd have to Go talk to kernel maintainers or write your own kernel module and what ebpf provides is a way to get data from the kernel the Linux kernel as well as Take actions on events and that is a seem may seem like kind of a Not that big of a deal, but it allows for a whole nother layer of tools and techniques and things to be built on top of The kernel and another way to think about the use cases of ebpf are, you know I don't know your persona or you know, what is your role and you know, what problem you're trying to solve The the kind of the first way and I'm gonna give a demo in a few minutes That I think about it is in terms of just debugging So I want to know is this you know, like think about a breakpoint in your code I want to know what's happening when this function within the kernels being called or Or you know, how many times it's being called that sort of thing so performance counters things like that There have been subsystems built into the Linux kernel, but ultimately you run into, you know, well This feature isn't available in the subsystem like the perf subsystem the audit subsystem These things are not extensible and what ebpf allows you to do is attach small programs on these events and then report data back into User space so you can have a user space program that understands what's going on in addition you can actually write new networking stacks within the kernel in the ebpf and then if you need to update them or change them or configure them You're able to do that nice So, you know, if I were a new person Using ebpf inside my cluster like what would that look like from a pod perspective? Yeah, so there's there's several ways. So let's say we have a kates cluster up and running an openshift cluster and we want to Understand what's going on within the cluster. So we have a performance issue For example, and maybe it's related to disk activity. You could use ebpf scripts and tools and pieced together kind of An idea of what's going on by just extracting the data and getting performance metrics. So this is no observable from the observability standpoint Another use case is if you want to actually Use ebpf to help with security. So visibility Less less about getting sampling and aggregate information from the kernel but more I want to see an audit of all of these events and Have it correlated with additional pieces of data like the container ID or the network connection that a process made or something like that And so that's that's one way to get use ebpf in your cluster. I think from the individual perspective The debugging is kind of the debugging and performance analysis is the first entry point It you know, it actually requires just like building a large web application on JavaScript a lot of effort Building a platform that's built on ebpf is you know a significant effort And so there's a lot of existing platforms and tools and things that are that are available so for example the Stack Rocks product now ACS is uses ebpf to extract process Executions as well as network connections from the kernel and what's great about ebpf is it allows us to operate on any kernel version kind of going back to that challenge of You know any recent kernel version That challenge of you know We're not able to add stack rock specific features into the Linux kernel. We don't want the customer to have to have a special kernel We can just insert an ebpf program within the kernel and then get this data out That sounds way easier than running custom kernels everywhere Yeah, totally um, yeah So like I think of the ebpf and I think of like tracing Kind of the same vein right you said debugging right like there's nothing better than running the s trace on a process, right? Really seeing where it's breaking right does ebpf give us More insight less insight. I mean, what is that? What is it like on yeah, so s s trace if for folks aren't familiar s trace is kind of an old-school system call tracing approach and it uses It's kind of ping-ponged back in between user space and kernel space So you you can print every system call that an application is running and that's super useful for determining You know what actually is happening when I'm reading a file or you know and so The the problem is that s trace has a huge performance over it. Oh, yeah It's it's essentially, you know adding an extra round trip between user space and kernel space the Cool thing about ebpf is that you can you know, of course ebpf can introduce overhead because you're doing work You know some sort of additional compute within the kernel, but you can be selective about it You can say, you know, I want to record all the system calls and then report them out. That's a good thing So Yeah, you could easily build an s trace on top of ebpf And getting those system calls like very telling very helpful How would you install it right like how would you get ebpf in your cluster? so I will let me do a demo and Let me show you guys share my screen and Before I do that, I'm gonna go through just a little intro on some slides here Is that visible Chris, yes Cool so kind of as I was saying, you know We have the Linux kernel which is hard to update and then we have our application stack on the right That's just a set of processes and you know These could be running within containers and our cluster but ultimately the processes are sending information on the network reading and writing files or launching other processes and you know since the The UNIX API the post-ex API's were created. It's it's made a It's been a stable place to run programs on Linux. So you can run exec you can run open It doesn't matter which version of Linux you're on or you know, which flavor However, within the Linux kernel, there's all these different subsystems the memory manager process scheduler disk network If you want to get data about those internal components, it's difficult. So you can either write a kernel module or Actually submit a patch to the to the kernel maintainers and that takes a lot of effort So we have this stable API for our user space programs But we don't necessarily have a good way to extend the kernel about, you know, a lot of effort so what eBPF let's us do is add hooks within the Linux kernel and You know, whatever we can add the hooks at the syscall level So that's thinking about strace where we want to see all the interactions between a given process and the kernel via syscalls or we can attach eBPF Programs to any any function really within the Linux kernel. Of course The syscall layer is going to be more stable So I think the you know the easiest way to get up and running if you have a Kubernetes cluster And run some eBPF programs is to use a really nice tool called kubectl trace And that's what I was going to give you the demo of here. So this is a available here at this github URL and the prerequisites are really just you need a Kubernetes cluster up and running So let's see kind of share my terminal so I have a cluster a kates cluster running and I will show the No, it's so let's see we've got Make sure we can connect to the cluster there we go. So we got three nodes in this cluster. In fact, let's look at what what the host OS is So these are running Ubuntu 20.4 Linux kernel 5.4. So that's You know, you're normally when you have a kates cluster, you're not really thinking about the node level nodes are cattle they come and go and What eBPF allows us to do is Kind of get into the the Linux kernel on the host. So we're kind of getting underneath our containers and seeing what's happening Right now. This is a completely empty cluster. So there's no pods. Of course if I Check the cube system namespace which we'll see the We'll see that there are Here we go. So we've got the you know the base Containers that are actually running in Kubernetes. So even though there's no applications running in this cluster. There's still stuff going on So what we can do is use kubectl trace and kubectl trace is a kubectl plugin There is Essentially it allows us to still use a kubectl command, but then it calls out to another process And so we can run kubectl. We can install kubectl trace using crew which is a kubectl plugin manager or just downloading the binaries directly and kubectl trace allows us to run BPF programs and particularly a particular type of BPF program called a BPF trace program Which is a higher level scripting language that's built on top of BPF And as I said, this is I think the easiest way to get started with running EBPF in your in your case clusters. So what we can do is let's let's see what Processes are running on a particular node. So of course I could exec into the node and do look at you know There's a lot of different ways to get that information, but let's see how we would do that with EBPF So I'm going to get nodes again and then I'm going to run a BPF trace program Let's start with exec So what this is going to do is it's going to run a kates pod on this node G4x2 is the the particular node and it's going to run this execs new program So we start that it creates a pod and This pod has to be configured a certain way so we can actually look at it and let's Describe this pod actually Edit it so we can see all the different configurations So it's key about this is it's running a container that is actually Just to cuddle trace BPF trace and it the other kind of key thing here is that it's running In with admin privileges so the security context here is privileged and if we're going to execute BPF programs within a container we need to be privileged because The BPF system call Which is you know, we're monitoring system calls or observing system calls. We also use a system call to insert the the program It requires admin privileges So I've run this trace program now. I want to see what What what what's actually running So we can tail a the pod itself, which is actually going to be just printing the data it's getting from the BPF program and so this is just tracking the It's adding a BPF program to the exec system call Looking at the arguments and sending them back to user space And so from a single container we can see all of the process executions that are happening within on the host system within each container and on the host system itself And so this is actually kind of a cool way to see what's you know, what's going on what kind of activities happening on a on a Kate's cluster. There's a lot of just, you know, Version checks and sanity checks some health checks that sort of thing That's cool. Yeah, so let's so we ran this program called exec snoop. So let's look at that program And this is kind of when I when I was saying earlier about how you know one of the first places I think people you end up using BPF is to diagnose performance or Other kind of issues in their cluster. So, you know, you might want to see what's running in a pod or that sort of thing And this BPF program is very simple. I mean, here's the here's the code right here. It attaches a The hook that we're attaching to is the sys call for exact the and we are just Combining the arguments and then printing them out. So an exact system call has a path associated with it and we're also printing the the process ID Okay. These programs can be run using on any Linux system. I think all distributions have these tools available in their package repositories. This is I'm talking like, if you're if you're not on a Kubernetes customers on the Linux box, you'll install BCC tools or you'll install app get BCC tools. You'll actually be provided with a lot of these programs. So, yeah, so there are this is just exact. We could also look at networking need to meet this trace and look at you in hygiene your wonderful pristine cluster. I know right this cluster has Has never seen any any high CPU all the CPUs are just ready to go. So what I'm going to do here is Let's see what we wanted to Actually, sir, what do you want to see Chris? Well, what do you got. I mean, well, you get there's a directory of BPF trace programs. I'm curious what's in there. Yeah. So this is a this is the kind of default programs. And of course, all of these are, you know, I would say not it's not too difficult, not too much effort to extend them for what you need. And this the snoop programs are particularly interesting. And you can imagine there's a couple different models of getting data out of the the kernel. You can just stream all the events or you can also aggregate within the kernel and then So let's do just Let's do connections. So yeah, yeah, what's something that's actually going to give us a bunch of results. So So we'll see let's get the logs for that. Now this is probably going to be really active. There's going to be a lot of connections and this will show us the Just stream this into a Okay, so this is just a snapshot of connections that were being streamed Via this BPF program and we're getting the the PID the command of the program. So this is the command is going to be, you know, stored in the kernel and just reading that data on the source address the destination address and so we can see all of the Connections that are being made in which program is making those connections. Let's say we there's a, you know, curl that's probably a health check just checking for readiness. We've got kublitz talking to the API server, most likely and so on. Very handy. So, yeah, you talk about performance metrics right like And can you show us anything around that. Sure. Yeah, let's take a look at the performance of a trace. That'd be super meta. Yeah, you know, and I think it's that's worth mentioning. You know, what's the overhead of running eBPF anything based on eBPF within your infrastructure and Like, I think s trace is a nice example because I, you know, we were describing s trace for every system call you're adding an additional roundtrip because it has to go in and out of the kernel and So you want to make sure that whatever events you're attaching to aren't too frequent. And that the data you're extracting is reasonable. Of course, you know, you could you could attach to every function in the kernel and just completely, you know, doster system, or you can have, you know, kind of targeted Events that you want to get information about. And there are a lot of ways to Aggregate data so eBPF provides different techniques for for getting data out of the kernel one one approaches histograms and maps and so on. You know, I don't know if there's a good way to do that with These programs, oftentimes you kind of want to like, you know, build a flame graph or some kind of More visual view of this data. This is kind of the raw data coming out. But let's say Just for example, you might want to see like, I think what's really nice is looking at TCP Issues like drops or, you know, TCP lifecycle that can be very helpful for networking. Oh, yeah, debugging everything. Yeah. Yeah. So question came up in chat. Where did these programs come from? Yeah, that's a great question. So these are BCC programs. So let me go back to my slides and I'll kind of give you a better idea of The ecosystem of eBPF. Okay, cool. So let me actually go to switch down a few slides. Okay, so this is a, this is from the eBPF.io website. This is a nice, excellent resource for eBPF and it's Recently forming, you know, kind of an open source consortium of developers and companies and everyone that's interested in eBPF. And these, what I just showed you with kubectl trace is A BCC program. Those are BCC programs. So this is the logo for BCC. And these are available on, you know, get up so you can download these. And of course, there's our, there's a lot of Oops. So yeah, there's a, I was going to try to show you the website with all this. So yeah, BCC tools are what those scripts were and BPF trace programs. And then we were running those using the kubectl trace. It's a lot of BPF and trace and tools and that sort of thing. You know, I think that it can be overwhelming when you're looking at all the different ways that BPF can interact with all these different kind of subsystems. The key thing is that, you know, the use cases of networking security and observability and tracing are kind of where you want to start with. I would say BCC tools are ideal for observability and tracing. And these are historically based on the detrace approach. Nice. Solaris. Yeah. So is Iovizer BCC the right repeater to get these from? Yeah. Yeah. Cool. I'll drop that chat folks. Yeah. Yeah. The BCC compiler collection or BPF compiler collection is what these are toolkits and wrappers for writing eBPF scripts. You still need to write in kind of a pseudo C and then the BPF trace programs which I showed you guys are built on top of that. Cool. Cube Cuddle Trace, this was from chat. Cube Cuddle Trace makes things easier. How do we use it to generate flying graphs? So earlier. Yeah, I think that the best way probably the one of the most full featured toolkits for generating data like data, like, that is the vector subsystem or vector system from a performance monitoring system that's Netflix has open sourced. And also, you know, these, you can generate flame graphs in different ways. You can just have, you know, text histograms, but I think the vector. Performance management system is ideal for flame graphs. Okay. Cool. For folks. Drop them a link. I'm going to see if. I see if web pages right now. So what has been done to harden the eBPF subsystem, right? Like, how do I know this isn't. A back door. Right. Like, like, is this something I need to watch? Right. Like are people making the. Yeah. Yeah. So, um, you know, the. Kind of going back to the, you know, what are we going to do with this. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. So you know, the. Kind of going back to the, you know, what are the alternatives to running, you know, this, this container that is inserting these bpf programs is running and is privileged. So, um, of course, you want to make sure that your user space code that's managing and running your edbf programs is, is secure and managed properly. The, uh, um, Verifier. you know, there's kind of two angles to that question. So one is, what's this, you know, is this gonna crash my kernel? Is this gonna crash my host? And then the other angles, you know, is this, is this secure? And I would say that, you know, there have been some, there haven't been an EVPF attack, but there has been some data leaks and things that enabled with EVPF. But really the, any user of EVPF has super user privileges. And so that's something to watch out for. Now, of course, the verifier essentially, that your program must pass is built into the Linux kernel. And it will, the verifier will ensure that the code terminates and isn't gonna get into any infinite loops. In fact, unbounded loops aren't allowed in BPF programs. The BPF instruction set is actually quite, quite constrained in what you can do. And that actually, that helps a lot with security. Null dereferences, a lot of the kind of, you know, caveats that come with writing in C that you would have if you were writing a kernel module are not there with BPF because it's got more safety features and it's a more constrained language. Okay. So very tightly scoped, that's good. Very, you know, sanitized for lack of a better term. Yeah. And I would say that, you know, if you're using an BPF, the, you know, it's essentially a sandbox within the kernel. And so you get the security guarantees of that. Nice. Awesome. So what about, are there any downsides to the BPF, right? Like we talked about overhead a little bit. What other downsides are there potentially? You know, I think that there really are no downsides because it's, and, you know, of course that's a very blanket statement. But, you know, maybe what are the, you know, what are the issues that it can bring? I guess it's maybe a different way to think about it. And so the, I think one challenge is making sure that you're running on a kernel that has, you know you really need to be on a more recent kernel. There's been a lot of effort and changes that have been added to the links from recently. And so if you're running on in the cloud you're going to be using a more recent kernel. But if you're, you know if you're in an air gapped environment or running older distribution you're not going to have all the most recent BPF features. RHEL, Red Hat has backported a lot of BPF to older versions of RHEL. And so usually I think if you're on RHEL you're going to have all the features you need. But making sure that your host can support EVPF is key. The other challenge is, so I mentioned that we're essentially extending the Linux kernel. And in order to do that you need to know what is the layout of a particular struct that I want to get data from. And across Linux kernel versions that those struct layouts will change. And so in the kubectl example demo I just showed the what actually is happening is we need the Linux kernel headers in order to run the BPF program. So when we compiled that script into BPF bytecode before it's inserted into the kernel the headers are needed. And that can be very challenging because especially if you're trying to support you're developing software that you want to run on a variety of different Linux kernels. And coming from someone who's worked on a security product that you want to run on a lot of Linux kernels that's a challenge. And that's something that the community the BPF community is very well aware of and they've been adding features to export the struct formats and the data types within the kernel so that you can just subscribe to that. It's called BTF or BPF typed format. And there's another initiative called that uses BPF called core or compile once run everywhere. And that's kind of the ideal in-state for these tools is that you're able to write a BPF program and run it anywhere. But right now that can be challenging because only the most like recent kernels support that. Depending on if you're on a cloud if you're in the cloud it's gonna be supported but otherwise maybe not. That's just something I'll have to look into, yeah. You know, the other challenge I would say with BPF is it's, you know, still relatively new and you're running a code in the kernel. And even though this is like a great tool to get information out of the kernel and you said something earlier about profiling the meta of BPF that can be difficult. You know, anytime you're adding overhead within the kernel if your application's slowing down maybe that can be difficult to diagnose. There are tools and techniques being kind of developed for performance counters and measuring. Okay, I wanna see, you know, how much time is being taken every time. For that example, I was only attaching to the exec system call or the TCP connect system call. But if you've got a lot of places that you're attaching you're gonna wanna profile these and see make sure that especially in production if you're running a BPF in production. Nice, good call. Is there anything else you wanna just show off with the BPF, is there anything you think people's minds would just be blown by it? Yeah, let me, let me go back to this slide here. And I think that, so we have this bottom layer of the Linux kernel. And you know, this is the, I would consider this slide from the kind of the roadmap of the BPF. There's, you know, BPF has been added to Linux kernel. Microsoft has been adding an BPF layer to Windows. Yep. There are several SDKs for writing BPF. Because not only do you have to write BPF code you also need the framework to help you inject the program and kind of manage it and get the data out of the kernel. And so there's Python and Go and C++ and Rust libraries for doing so. And then on top of those are these tools and platforms that are being built. And I think that's probably the most interesting thing going on with BPF is, you know, it's just like it's good to know about assembly code but no one's actually gonna be writing assembly code usually. And so it's really cool to see what people are building on top of the BPF from the perspective of security. The coolest thing is to be able to have a way to not only get data from the kernel but actually take action within the BPF. And so that's where I think when you start writing when people start writing and this is already begun writing BPF programs that actually can take action within the kernel then there's a lot of really interesting tools and platforms that can be built. So from the perspective of security there is a new Linux security module based on BPF called, I'm not sure if I get this right but kernel security, it's KSRI. Oh, okay. Kernel or maybe I got the SNR slot. Kernel Runtime Security Instrumentation. There you go. And what that allows you to do is actually have checks. So, you know, is this process authorized to take this action? And you can just deny it. And so those types of rules can be updated and tied into the higher application or authorization frameworks. And what's great about that is it's, you know anytime you're taking action within the kernel it's performant because it's not requiring you to go back and forth between user space and kernel space. The other place that's really interesting is just observability. So there's a lot of observability frameworks Pixie is one observability framework. And I think in general if you're doing any kind of application tracing or monitoring, it's gonna ultimately be based on EBBF. And I haven't really specified this or kind of gone into detail about it but not only are you able to trace events in kernel space, you can also do that in user space. So you can attach to essentially use EBBF to check break points in user space or to look at events in user space. And basically you can imagine that just opens up a whole world of observability and just flame graphs are kind of like the beginning of all the different ways you can like dive down into what's happening. So security and actually doing enforcement that's one area that's I think super cool. The observability frameworks that are being built on top of EBBF. And then finally service meshes and networking. So Cilium and Cilium is a Kate's CNI plugin. So it's an EBBF based networking plugin for Kubernetes. And I'm not an expert on Cilium but my kind of my understanding of what the benefits it provides is again, you're able to not have to use IP tables to manage the connections between deployments or even between clusters. This can all be configured within EBBF. And there's a component of the Linux kernel called XDP, the Express Data Path. And that is used often by hardware manufacturers to tie as a way to integrate their NICs with the Linux kernel. But then also you can have essentially a software layer built in there. And so that's what Cilium is. A kind of software EBBF based networking plugin. K-Tran or K-A-T-R-A-N is a load bouncer based on EBBF. And even Calico has added EBBF features. So I think moving forward, if you're running a service mesh, it's also gonna kind of probably be built on EBBF. Yeah, it'll have some EBBF components to it, it sounds like for sure. So quasi-question slash you might not know the answer to, but wouldn't EBBF effectively introduce a stable kernel interface and permit proprietary programs for the kernel? Yes, so like, yeah, I'm sure that's a contentious thing. So... So I'm actually not sure I completely understand what the questioner is asking in terms of where they think, where they're going with that. Of course, if you are writing a BPF program and it's inserted into the kernel, there actually is a GPL license check in the verifier. And so, as I'm not a lawyer, but you still have to, if you're gonna be interacting with a GPL API within the kernel, then your BPF code needs to be open source as well. So, I think that's an interesting idea. I'm thinking from the perspective of hardware, like a device driver or something like that. I would say that this is already feasible with kernel modules and other things. The kernel is already extensible. This is just an easier and safer way to do it. I think safer is kind of the, right, like... Yeah. Easiness aside, this is just safer to do than like building kernel modules left and right. That's right, yeah. Yeah, and to be honest with you, I don't know where eighth doctor was going with the question, but if that didn't answer your question, eighth doctor, please let us know. Khaled Raid says, the most interesting thing to me is how you can control network traffic on layer seven with EVPF. That is super awesome. The fact there's a load balancer that uses EVPF find the scenes. I don't know how you could extract any more performance out of the kernel than that. I mean, yeah, I've never used Patron before, but it's gotta be fast and performant in all kinds of different ways. Yeah, absolutely. I mean, that is, you know, essentially you're giving these kind of core data paths context. And you know, user space can provide the context through EVPF, and then you're able to build these performant networking tools. You know, whether you're doing packet inspection or, you know, yeah. Taking action on things, you know, I haven't seen anyone do like an IDS and EVPF, but those types of things are... That would be cool. Yeah. Yeah. You know, I think, and maybe going back to the challenges, you are limited to how complex these programs can be. I'm not sure off the top of my head, how many, there's an instruction limit, how many instructions can be in the program. And so, you know, I think when you're thinking EVPF, you're thinking of small snippets and like, you know, you're building together, you know, for Cilium, Falco, these projects have large EVPF programs, but they're really pushing the limits of the size of what you can do. So, you can't build, you know, you can't have a machine learning engine running in your kernel and using EVPF. Right. So, you are limited to also the just, the limitations of the EVPF verifier, as I was saying earlier about, you know, bounded loops and what memory you can access and that sort of thing. Yeah, so. Yeah. Detective Conan Kudo, he basically, the issue here is that proprietary programs for the kernel that are tied to specific kernel versions bring us back to the problem that prevent people from upgrading kernels in the first place. I don't think EVPF does that here. You have that instruction set limit, it doesn't live in the kernel, it calls the kernel, right? Like so, it's one of those things where it's like, if people are concerned about this causing you to like stick to a certain kernel version, I don't think that's the case. This is going to be persistent and kernels going forward. You're able to run the same, you know, traces as before. Right, yeah, I mean it's important to, we're not hot patching the kernel. We're kind of, it's event driven hooks that we're taking action when that event happens. And so that does have certain kind of limitations. You can't necessarily, you know, of like change code paths or that sort of thing. You're adding on additional, you're extracting data or, now for example, if you do the thing, so I described from the perspective of attaching to system calls, but psyllium, which is attaching to the XDP subsystem, you're extending that. So rather than going back and forth between user space, you're going between your EBBF program and that subsystem within the kernel, if that makes sense. And the same with Linux security modules, you know, you're not necessarily, you need to have some code within the kernel that allows you to then do those actions. And so I think it does from, you know, from the perspective of Linux kernel features being added, knowing that you can build a subsystem with the idea of users using EBBF to then make it what they need to be. You know, it's like to extend it the way they would like. You know, the Linux kernel, if you're previously without EBBF, you had to be very opinionated on what things were, how things were gonna operate. But now it can have a, it can have the framework and then users can write their security rules or audit rules, for example, for the kernel security instrumentation, runtime instrumentation, and, you know, use them as they see fit. Yeah. Cool. So we've got about 15 minutes left. Is there anything that we haven't touched on that you want to? We talked about the cons. We talked about, oh, what has been done to harden the EBBF subsystem? We touched on that a little bit. Yeah. So, you know, I- Instruction limits and stuff like that. Instruction call limits, the, you know, I think from the, from the beginning, you know, EBBF is one of those acronyms that doesn't really have any meaning anymore because it kind of, we're talking about the whole EBBF subsystem and all these platforms and things when we say EBBF, but ultimately it's extended Berkeley packet filters. And so Berkeley packet filters and IP tables have been around for a while. And so, you know, it's, I think it's, it is secured through the simplicity in that, in that you can, you need to have a constrained programming language or instruction sets. And once you have that, then it allows for the verifier and static analysis to be much more effective because we're not exploring, you know, a million different paths. And all the different paths in your EBBF program are essentially walked to look for null dereferences to make sure that you're, you know, it's, there's no way that you're gonna dereference a null pointer. And also the kernel provides a lot of EBBF helper functions and that's usually what's used to get data. So you're using the helper functions to then interact with the rest of the kernel within EBBF. That's cool. Okay. Yeah. Awesome. Folks have already asked about Cilium. I've dropped the link to the Cilium page for that. So I would definitely want to mention that the EBBF.io organization just, actually, I think it just finished like an hour ago. Had a summit yesterday and today is the second summit they had it last year for the first time. And for people that want to know more particularly about, you know, the lower level details of EBBF, but also just different use cases. Definitely check out the EBBF summit. It's free. All the videos are online. That sort of thing. There's a lot more information there. And it's... Find a link. Go ahead. It's, yeah. And, you know, there's an active community and kind of like I said before, that, you know, depending on what persona or what role you have in your organization, EBBF may, your interaction with EBBF may be either debugging or you're just using a platform that happens to be built on EBBF. Right. Yeah, that's a good point. The, I'm looking at some of this schedule. It's intense. But, you know, typical suspects, Liz Rice, Yana Dugan, all the fun people to have at the sale. Sable. Yeah. All the low level folks I know that know everything about SysCalls are speaking at EBBF summit. So that's cool. So we also have, we're a diamond sponsor for the Cloud Native Security Conference coming up at CubeCon. Are you part of that or? I'm not playing in any way. I'm not playing on presenting, but yeah. Yeah. I'm sure I will be involved one way or another. Yeah. There's just to read a little bit. There's a hosted capture the flag learning experience that will run in parallel to the event for attendees to learn firsthand practical attack defenses and techniques. Yeah, I think those capture the flags and those hands-on labs and demos are such awesome ways to learn. Yeah, the capture flag demo that was done during QCon EU was just fascinating to me. And like the second we found the problem because we were running out of time and I say we because it felt like we were all working on it together in the live stream. It was just like, oh, there's the problem. That's the naughty bits right there. Let's get rid of those and off we went. But yeah, it was a nice. Yeah, you have a lot of those aha moments. Yeah, like you can really get an eye-opening kind of mind-blowing experience from some of those sometimes. Because yeah, you can easily stick something on a Lexbox and make it do bad things. Yeah, and I think there's something really exciting and cool to be able to get information from the Linux kernel and get into these low-level details. You really kind of opens up your eyes into like what's going on. So yeah, those are great learning tools. Absolutely. Anything else for the audience? Do you think they should know? So I can share some resources. Yeah, that'd be great. So, you know, I've kind of... Slide her up, folks. I'll give you a link. Yeah, no need to get your pin and paper out for this one. I kind of broke down some of these resources based on, you know, if you're going to want to do EVPF development. I think EVPF development is going to be, you know, most likely associated with some sort of platform, right? You're working using EVPF within Cilium or that sort of thing. But for scripting and debugging, and that sort of thing, the KubeCuttle Trace is a great place to start. And then there are books. Brendan Gregg is the resident expert at Netflix on EVPF. He's got a great book on performance and observability with EVPF. And yeah, there is a lot of platforms. Red Hat Advanced Cluster Security for Kubernetes. We use EVPF, Falco, Aqua has some EVPF tools, Cilium and so on. Yeah, now there's a wealth of information available online. And like I said before, I think it's it can be really overwhelming when you're getting started, like looking at these things and, you know, just to start with something simple, like BPF Trace. And I think that's a good, good starting point. Yeah, looking at questions here. Are there any chances Red Hat will become a member of the EVPF Foundation? I do not know, I have no idea. Have you seen any discussion on additional stack unwind support for EVPF? No, I haven't. But that is always a pain point when looking at, you know, performance counters and things like that where you want to be able to, you know, diagnose issues. I know that Pixie has done some work on performance, you know, kind of continual performance monitoring with EVPF. And I believe they've opened, I don't think they've yet open sourced there. It's they have a community edition, but they have some interesting blogs about performance or profiling with EVPF. I can share that after the talk, yeah. Yeah, that'd be great. Yeah, that's so, you know, Conan Kudo says, I hope they don't do too much more for EVPF if the goal is to not be turned complete, then it needs to be fairly restrictive. I mean, do you know of anything on the horizon for EVPF that would give folks reason for concern? So no, I mean, I think the language is going to stay relatively stable. The instruction said, I mean, there are, I think there's been recent work to add atomics and also additional like Linux features to get data out of the kernel. And so there's several different ways to, you know, you can insert a BPF program just using the BPF system call, but then to get data out, the program can write to a Perf buffer or also there's BPF ring buffers which have more performant characteristics. And so I think really the language, the instruction set will stay relatively stable, but then it's all of these kind of tools and other ways to interact with it that are going to change. That makes sense. Yeah, and grow. Awesome. Well, cool. I don't have anything else for us unless you do or the audience might toss in something here at the last second, who knows. But this has been awesome. This has been very insightful. I'm glad you came on today. Thank you so much. Thank you to the audience for your questions and participation. This was awesome. What's next? I mean, you know, where do we go from here? And, you know, I think you're just gonna see more products and tools powered by UBPF, you know, that's really... I dropped the projects link and I noticed that like some of those are newer projects that have come along in the past few months. So I expect to see a lot going on in this space as more people become aware of it, right? Like having the second to VPF summit. Yeah. That's a big deal. And it just really fits well within like, you know, the microservices architecture and like where you have, we have all these ephemeral containers and clusters in our environments and you need a kind of a way to slice through, you know, not horizontally, but vertically. What's the user space program doing? What's the kernel doing? And UBPF is kind of like the best way to do that right now and so the, I think, like I said, there's gonna be more and more things built on in VPF. Awesome. Well, Robbie, it was great having you on. Thank you. Yeah, thanks for having me, Chris. Hey, no problem. And we do these, how, I forget how often we do these. Hang on a second, let me double-check. Please, like, obviously please come back for the next one, but the next one is, I forget. I believe these are monthly. So confirm, not, that is incorrect. But so here's a better idea. Subscribe to our calendar and you will get all the fun links to all the programs that are coming up on the show or the channel. And yeah, highly encourage you to do that so you can keep up to date with all the latest and greatest cloud native fun things we're doing here at Red Hat. Yeah, so Robbie, without further ado, thank you very much and y'all stay safe out there. See you soon.