 So I'm happy to be here today to talk about BPF Trace and Pixie. BPF Trace and Pixie are both open source projects. We love them both. I'm from Pixie, so my omid, my colleague Yashong is sitting over there. We worked on this stuff together. And so we wanted to present today how we kind of brought these two open source projects together to make them better together. So just starting off real time debugging of Kubernetes is challenging. We all know that, right? Debugging, we get frustrated when things go wrong, and we need to have the right tools available to debug things on the fly. And so for us, we're always looking at how we can make observability easier, right? How can we do that? And EVPF has really been a game changer in that space. You're bringing the ability to essentially monitor parts of your application without requiring any instrumentation, without requiring any redeployments, no code modifications. It's been fantastic. And we really love that. And I mean, the proof is in the pudding. We're really seeing a lot of different EVPF-based observability projects take off. And it's fantastic to see. Like, we're really thrilled about all the different tools that are out there in the ecosystem now. It's just great to have all the support. So as I said, I'm going to talk about, in this talk, two particular tools. One has the open source BPF Trace project. The other one was Pixie. And then how we bring them together to make something even better, right? So first, we're going to start with BPF Trace. So let's meet BPF Trace in case you haven't met it before. I assume a lot of you have, because it's a fairly popular project. And I'm going to keep this brief. But BPF Trace, it's a high-level tracing language for EVPF. It makes it really easy to write EVPF programs, right? So as a very simple example, if you wanted to trace every open syscall in your system, on your host, it's essentially a one-liner. I've broken it down to three. But this is from BPF Trace's own docs, right? So you just kind of specify. You say, the trace point, syscall, sysenter open. You say, I want to trace the open calls. And then you say what you want to do whenever that event occurs in the system. And so we're going to say here printf. And then we're going to print out the command that triggered that syscall. And we're going to also capture the argument to get the file name of what file was being opened. And if you run this on your terminal, it'll start printing out every single syscall open that you see. So really powerful. There's great community support for BPF Trace. So if you go to BPF Trace's repo, you'll see that they have a bunch of built-in tools already. Like, these are BPF Trace scripts that have already been written for you. They're at your fingertips. We counted there was 35. There's all sorts of stuff from what's happening at the IO level to TCP, to like umkills, like tracing all the umkills that happen in your system. So anything you can kind of think of, it's kind of there's something out there to do to kind of monitor that with BPF Trace. And so it's, again, really powerful stuff. The other thing about BPF Trace is it's really useful because you can write your own, right? It was so easy to write. It's a high-level, like writing. We do a lot of writing BPF, like low-level BPF code. It's hard, right? But when you write with BPF Trace, it's so much easier so you can get so much more done quickly. So we love BPF Trace. If you want to be a BPF user, it's a must-know tool. So we met BPF Trace. Now let's meet Pixie. So Pixie, what is Pixie? At a high-level, Pixie is a platform for real-time debugging of Kubernetes systems. And it's based on, it's based on EBPF. What you get out of the box with Pixie is you get kind of basic stats, like CPU, networking, IO metrics, all those sort of table stakes stuff. But in addition, using EBPF, we trace all the messages going through your system. So whenever one pod talks to another pod, since, for example, an HTTP message, we'll trace that. And we'll pull it into Pixie so that you can kind of query against all the messages that were in your system and figure out what's going on. In addition to that, we have CPU profiling, so the flame graphs to kind of help you find bottlenecks in your system. Our kind of primary characteristics are we really want to keep things zero instrumentation. That's where the EBPF part comes in. Also, Pixie has a distributed architecture, so we'll take a look at that in a second. And there's essentially in-memory data tables that store all the trace data. And then finally, there's a scriptable interface that's based on Python slash Pandas to let you query all the data that's being collected so you can kind of real-time debug what's going on with your system. From an architecture perspective, the way that Pixie works is that there's a module called the Pixie Edge module, and that gets deployed. There's one copy of that per node in your Kubernetes cluster. And the job of the Pixie Edge module is foremost to monitor all the stuff that's happening in the node. So it's monitoring all the pods that are in the node. If they're, again, sending messages to each other, it's going to trace them. For all the pods that are running, it's going to be collecting profiles on them so that you can identify performance bottlenecks. The components that collects all the data and does all the EVPF magic is the data collector, which is called Sterling. As it collects the data, it pushes into essentially an in-memory database. And it's holding that in memory. And then you, as the user up top, you're interfacing with Pixie either at the UI level or with the command line. You're running queries, and it's going to dynamically come down. The query is going to come down. Hit the Pixie Edge module, get the information that you actually need, and bring it back to you in the UI, and present whatever you wanted to kind of real-time debug. Drilling a little bit deeper into that Sterling component, the data collector, the architecture internally looks as follows. So we have Sterling. Internally, we have these different components, these different extensible components that are delivering the different sorts of data that you're looking for. So we have one that's collecting kind of your basic process stats, CPU, memory, networking, I.O., all the basic stuff. Then we have the protocol tracer, which is collecting the traces. That's EVPF-based. We have the application CPU profiler, EVPF-based. And then BPF traces, the next one over. And that's the one we're going to focus more on in this talk as we integrate the two BPF trace and Pixie together. Just because I want to give you a feeling of what it looks like in the end, with Pixie, when you kind of go in, because we're tracing everything, you can run queries and see all the traffic in your system. So you can see, for example, there was an HTTP request to request pass addresses. There was a street, White Lease Road. That's the JSON payload of the HTTP message. We captured all that stuff. And based on that raw data, we can take Pixie and kind of create automatic service maps so you know who's talking to who, how much traffic was going through, whether there's bottlenecks, how much request per second you're getting, things like of that nature. And then we also said, out of the box, you're going to get these profiles. And you're looking for wide bars here. If you see a wide bar, that means your application is spending a lot of time in one particular function, and that might be an area of concern or somewhere to optimize. So we've met BPF Trace. We've met Pixie. Now let's bring them, put them together. So when we were working on stuff, we were asking ourselves, wouldn't it be great if we could bring the power of BPF Trace and integrate it into Pixie and kind of get the best of both worlds? And our thinking at the outset was, well, BPF Trace provides the ability to write EPF code really easily and get into the nitty gritty of what's happening into your system really easily. So we want to leverage that part. And there's already a kind of large collection of community scripts that we would be able to access if we were able to do that. So it would be really cool to see that. On the other hand, Pixie provides kind of the easy deployment to Kubernetes. It's managing all of that. And it has all the data collection from the entire cluster, all the nodes in your cluster. And it's putting all that data into structured tables so that you can query it. So that query engine side, Pixie could provide the visualizations, all that stuff we could get from Pixie. So that's how we wanted to marry the two together. So how do we do it? So the idea was that we would want to slightly modify BPF Trace so that instead of outputting events to the screen, which was what it normally does, when you write a printf in BPF Trace, it'll just print it out to standard out, we want those events to kind of get sent to tables within Pixie so that we can query them. And so we focused on those BPF Trace scripts that exist today that primarily output things based on printf. And we said, OK, we'll take Pixie Sterling. The user somehow sends in a BPF Trace script. We'll get that. We will send it to BPF Trace to compile it and then deploy it to the kernel. And then as the events are coming back from the kernel, they go back to BPF Trace. And we'll kind of intercept the printf callbacks so that they never go to standard out. They come back to Pixie. And we're not really just parsing kind of like a string. We're really making this in a structured format. And we'll see that in a second. And so the data then can go back into Pixie. And then we'll take it, put it on our output tables, and make it available for querying. So conceptually, like I mentioned, we want to capture the data that's coming out from these printfs in BPF Trace. So let's look at an example. So we have a printf here. It's a simple one. There says time and count and command. So there's three arguments to this printf. And so when you have this sort of statement in a BPF Trace script that Pixie is taking in, what we're going to infer from this is that there should be three columns. There were three arguments to this printf, so we should create a table with three columns. And then if you put the optional kind of labels in front of the format specifiers, then we'll automatically also label the columns for you. So we'll say the first column was a time, the second column was a count, and the third column was a command. And every time the printf triggers, these events will get sent to Pixie and we'll populate the table that was automatically curated. So what does that mean in practice? So here's a simplified version of an existing community script. This is a script that collects or traces TCP drops in your system. It's a little bit simplified, so you can take a look at the real one at the link below. But what happens in the original script is it actually has essentially three printfs. It prints the time and then it has two printfs. But we want to put this in a structured format. So to get this into Pixie, the only change that we need to make is that we need to kind of change that printf into one line and that's telling Pixie there's one event that occurred and here are all the columns that need to be populated within the table. And so if we take this script and then plug it into Pixie, we'll take a look at that in a moment. It'll deploy this and automatically collect all the data. For those folks who are familiar with BPF Trace and are thinking to yourselves, what about those aggregation style scripts? So BPF Trace has scripts where when you run them, they just silently are running, they're not outputting anything and then you hit Control C and it's been aggregating a whole bunch of data and it dumps it out at the end. So that doesn't really naturally fit kind of with the time series-based approach that Pixie takes. And so you'll need to do a little bit more kind of working with that script if you want to get it into Pixie because you'll probably want to go in and output individual events rather than aggregating data within Pixie. Okay, now let's put an example together. Pixie has this notion of a pixel script. It is what drives the kind of UI within Pixie and it's where we can do our querying within Pixie. And so if we want to get a BPF script into Pixie, we have to put it first into a pixel script. So there's really three components to this, to this example. Here we have a trace point. We want to trace all the exec calls. So we see at the top the BPF Trace script that we want to use and that's kind of embedded in our pixel script. It's just put there into a string and we're saying this is what we want. If this was a community script, it's essentially what we copy over from the community. And then we have the second section which is telling Pixie how to deploy this BPF Trace script. So we're saying, absurd trace point, absurd is terminology used in databases which has insert this trace point if it doesn't already exist, if it does already exist, update it. So it's a combination of update and insert. And we say, give this probe a name, what we're about to deploy, give this entity a name, specify the output table name where the results will be put out into. And the third argument is the BPF Trace program that powers it all. And then the final argument is a duration. We say once you deploy this, leave it there for, in this case, we're saying five minutes and just keep collecting data for the next five minutes. During that entire five minute window, we can go query it. We just want some cleanup so it just doesn't linger on forever. So you could make that much larger too. You could say leave it in for a few hours. That second section is what actually deploys your embedded BPF Trace script into Pixie. And then the last part is kind of the query part. So we'll say DF data frame, that stands for data frame. So we get to say get a data frame from the table and we're gonna specify the BPF Trace table that we just created where the results are going out to. And then we're gonna say, for example, this is an exec tracer, so it's searching for all the exec calls. So maybe in this case, we're saying, hey, look for anything that has sudo in it. Maybe we're trying to monitor dynamically for anything with sudo, any sudo command. So we'll catch those. And then the last line says, display the results onto the screen. All right, so you could change it. You could filter however you want. That part at the bottom is kind of dynamically running when you run your query. You could look for any sorts of commands, any sorts of exec calls, something you're interested in. And that's the part of Pixi's dynamic ability, right? So we have a dynamic component in the BPF Trace part, and then we have a dynamic component in the query itself. All right, so with that, gonna switch over to a demo. See if I can do this. Ooh, too large. Still too large, hold on. So if you come to Pixi, this is the kind of main page that you arrive at, right? So right at the top, you get a service map. This is not the BPF Trace fed information. This is just kind of what you see out at the outset. You see who's communicating with who, the traffic. You can scroll down and drill down into different namespaces. You can look at the different nodes. You can get a whole bunch of different information, but we are most interested today about the BPF Trace integration. Pixi also has a notion of scripts. So that's kind of what I showed on the previous slide. At the top left here, you can choose different scripts for different things that you wanna see. For example, if you do PX HTTP data, it changes the view. Now, this is just kind of raw data of showing all the HTTP traffic in your cluster. But we're not interested in that one, so if you start typing in BPF Trace, it'll start autocompleting, and you'll see we've actually taken a bunch of community BPF Trace scripts from the BPF Trace project, and already ported them over to Pixi. So, for example, if I run exec snoop, that's similar to the example I showed before. So if I hit run, what's happening right now is it's actually compiling the BPF Trace program dynamically and deploying it across all the nodes in the cluster. So that's happening actively. It takes a few seconds, and then you're live, right? And we can see now across the entire cluster, all the nodes of the cluster, and see what exec calls are being made. We can see what commands are being run, so we see a lot of run C stuff. We see Docker, Docker init, whole bunch of commands. The first argument that's being passed in is available as well. And so we get all this information, and that's the power of BPF, right? This is what we love about BPF Trace. It's like these one-liners are very few lines of code, and BPF is like taking care of all this, all this for you and getting a wealth of information. And so then also in the script, so the pixie side, we can say, maybe we want to filter for something, maybe we're interested in all the Docker commands that are being run, and so we update that, run again, and it didn't actually have to redeploy the second time, it's already deployed, so it went much quicker, and now we see only commands that are, so the query actually went, so when I hit run, it actually queried through the distributed system, went down to the nodes, and collected all the commands in the in-memory database in the tables that run, have the Docker command running, and it just pulled those ones up back to the UI, and so now we see just those Docker commands running. And now it'll show one other one. So we have another, let's go back to BPF Trace here, this is another community script, so we have TCP drops, so we've pulled this one in, again, you're gonna run it, so you run it, this one's gonna take again a few seconds as it compiles the BPF Trace program, deploys it across the cluster, so we give it a few seconds, deploying, deploying, deploying, okay, and here we are, and now this one ran, and so this one actually shows another kind of cool part of the Pixie part, so what happened here is the BPF Trace probe that goes throughout your cluster, and it starts monitoring for all the TCP drops in your cluster, right? It's doing that, and we can collect the raw data, but with Pixie, you can also kind of build these visualizations on top of it, and so we've taken all that information and kind of plotted it, and so you can see where all the TCP drops visually are between the different pods or processes within the connections within your cluster, and so you can see if we drill in here, right? You can kind of see the different things and where the drops are, and if you hover over a particular connection, you'll see how many drops actually happened on that connection, and whether it should be a concern or not, so if you're trying to troubleshoot, you know, TCP sort of related issues, this could be very useful for you. Now, you might look at this and say, whoa, whoa, whoa, this is so much data, how can I make any sense of this, and so I'll show off one other feature here, right? Okay, so what I did is actually pulled up the Pixel script that's driving this whole page here, and it's the source code, you can actually just pull it up live in Pixie, and we can go through here and I can say, you can actually see what's powering it. You can see the embedded BPF trace script that we got from the community, you can see there, you know, you can actually even see the copyright, Dale Hamel wrote this initially, right? And then we go down and we have the kind of pixel part of it, which is taking that BPF trace script, deploying it, and augmenting it, and we have a line here that says, set a threshold to display, like we wanna say how many drops, right? So maybe we wanna change our query now, because we're hunting for something. And so let's say I'm only interested if there's been maybe a thousand drops or more, just to kind of declutter this, so I can change it on the fly, run again, I think there was connectivity issues I saw and noticed before. Let's try Zergen. Yeah, I'm disconnected right now. I was having wifi issues earlier, so it could be that. So, but the point was that, you know, you can change the kind of script on the fly and kind of what would have happened is a visualization, you would have just seen those kind of connections where there's a higher amount of TCP drops, and you can kind of debug in real time in this way because you can pull up the editor of the script query for what you need, but the power of all the data is coming from BPF Trace, so you get kind of the best of both worlds. So that's it for the demo. Let me switch back. Yeah, I'm just having overall, that's what's wrong. I'm having wifi issues right now. One second. I will try to tether. Let me change my wifi. And I know everyone's hungry and wants to go to lunch, so I will wrap up soon. There it goes. Okay, and so kind of to kind of bring this all together, the state of this project, we've got kind of the basic stuff working as you kind of saw in the demos. You get a lot of powerful stuff out of the box right there, but there is future work. And so right now we only support K-Probes and TracePoints, and in the future, we're, you know, looking at U-Probes, U-Probes' future work. And then also kind of we have this restriction, because we force everything into a structured table, you can't have these random printfs all over the place. We say, you can have multiple printfs, but they should all output to a single table, and so they should all have the same number of columns, right? But these are kind of future work, things that we can change, help better support the aggregation style scripts that I was mentioning. So all these sort of things we're going to be looking into. Of course, contributions are welcome on that, so if anyone wants to participate, we're looking for people to contribute. But just to wrap this all up, so BPF Trace plus Pixie, what we're providing here is, it's again just to recap, dynamic observability on Kubernetes. We got these two powerful tools, put them together, and you got something on superpowers, right? And so we let, you know, we leverage the high level language of BPF Trace, really makes it easy to deploy BPF programs out into your cluster, into your Kubernetes cluster, and then we use the power of Pixie's querying visualization to bring everything together for a kind of seamless debugging, real-time debugging experience. With that, we have our GitHub page, website, contributions are welcome, and I'll open it up for questions. Don't have this mic? Yes, this mic is on. Thank you very much, Omid. Do we have some questions? Yes. Great talk. A question around exporting to a hotel, in a hotel format. I think I've read somewhere that that's on your roadmap to support, actually, exports, so that we could leverage some of our regular tooling for dashboarding instead of using the Pixie interface. Correct. So hotel, there's both hotel ingests and kind of data out. We have recently added for Pixie a plugin component so that you can ingest data in using hotel format. Export will be coming as well. Thank you. And thank you for creating or being a part of Pixie. We love it. Security concerns with giving users or the ability to inject BPF code across a broad number of machines. Yeah, anytime. I think it was mentioned before, too, like with BPF, anytime that question comes up of security and it is a legitimate concern. Even before the BPF trace stuff, just like with everything else that we do, like tracing all the traffic, it's like a question like, can we limit that? And so we have some security modes to obfuscate the payloads of HTTP traffic of other fields when you want to be in the secure mode. And so for BPF trace as well, well, we haven't done anything specific there. It's something that we could also enable. Any more questions? OK, so let's give Amid one last round.