 So about BPF guidelines, on EBPF-IO we have projects, which is a list of projects in the BPF ecosystem that meet stated requirements. That's great, but we have wrestled with the idea of how do we also steer people to the easiest projects to start with, giving people from different disciplines the best experience with EBPF. And so the idea was, apart from the EBPF-IO projects, we also have an EBPF-IO slash guidelines to show best practice projects and steps, tailored for end-users, observability, security, networking, application, accelerators and so on. I'll talk about what I've got for observability so you can see the idea, but I will need your help to fill out the other things. So we've got security, networking, performance accelerators, probably other use cases as well. So this is just a draft. So to start with observability, familiarize yourself with traditional observability tools so you're not reinventing the wheel. One thing we tend to see is people will do submissions where they say, here's a new BPF tool, and it's like there's already another tool for that. So we want to get people to get up to speed to where the actual gaps are. We can also list recommended requirements to get people onto a BTF version of Linux early on. So when you're enubing, you're getting started, yes, you need to be on something good. For observability, the best thing for people to do is to install BCC and then try running the libBPF tools and find some performance wins. And maybe I'll word this to be more clear. A problem I'm trying to avoid people doing is I get emails from newbies who have been playing around with BPF for a month. They've done nothing with it. They're still trying to write BCC tools. And it's like, please use the tools first. Tell management at your company we just found performance wins, EBPF is great. Before they get tired of you working on it with no deliverables and yank your funding. So I really want people to just use the tools to start with, get some performance wins, tell the rest of the company, and then spend time developing it, writing tools. So I get that in early. Then for for, install BPF and try out one-liners. And at this point people can try doing custom one-liners and their own programs. I tell people use workload generators to sanity check your output. I always try to do 23,000 events, like write my NC program that does 23,000 events. And then I do my observability tool and it should match. The observability tool should say, yes, you did 23,000 events. Sometimes it's like kernel noise because I'm doing like a syscall or something. And so there'll be extras from the system. But if the output says 23,143, it's like, well, I'm in the right bullpock. This is a good point to check it with the user space because once we had an intern, I asked them, count the number of executions happening on the system with the BPF program. They wrote a counter and they were incrementing it. Of course, this was not an atomic counter and they were like, it's not matching. And then here's a simple thing that you think should have worked with BPF and it didn't work because of concurrency there. So yeah, that's a good point. I see it all the time when you're writing tools because you find the fast path quickly, but you don't find the slow path to code. And so you're writing some TCP tool and it's like, hey, look, it works, I think. And then in production, when there's like buffer bloat and retransmits and UDP, the numbers just don't work. So that's why we've gotta come up with really good suite of workload generators to test out the different code paths, make sure they add up. For front end UIs, I would tell people to build upon BPF PC tool output, which is horrible, but like scraping the output via stat. But that's easy, BPF Trace has minus F JSON, so you can do BPF Trace and then feed that straight into UIs. That's what we're doing, what Netflix is doing. And so, yep. And if you make it to six, I'm still learning Windows. If you make it to six, if you've tried all of these steps, then absolutely go code your own stuff. Go code your own LibBPF tools in BTF and Core and try not to use the Python stuff. Go through the contributing scripts checklist. So that's the idea, it's to get people to not shoot themselves in the foot. Get them on the, like using the, think about this in other contexts, security networking and application accelerators. They need to understand the problem space first. Like don't do an application accelerator when it's already got built in. They need to do the problem space first, they need to meet the requirements, try to give them some early wins so that management and the company likes their project. And then give them an easy path into developing BPF. Any comments on the observability one? I'm still tweaking it, editing it, but. I think even before observability we need some guidelines on how do you develop and how do you start looking at this. So ideally we'll have some documentation on eBPF.io various program types. But I feel that self-test is a good place to start on, like here's this, you could use the BPF skeleton to generate or get yourself started. What is the command line to use to compile a BPF program? All of this is just a big barrier to entry for somebody beginning with BPF currently. Okay, KP says people should be doing self-tests and looking at how to start programming. This is exactly why I'm writing guidelines, because that is not how you should start doing observability. Not observability, but BPF development. Absolutely, yeah, so we have BPF development, right? Okay, got it, okay. The problem with observability is that people start in doing that. And this is why they send me an email after a month, and they've accomplished nothing, but learned to hate the BPF verifier. And then I tell them, why didn't you just run some tools? So BPF, so we absolutely can have guidelines here too, that should be self-tests. I'll just fill it out later. So what are the top things? Not just kernel development, right? Like when you're starting to write BPF programs, there are some guidelines around that. For example, don't attach to certain events in the kernel, because they're going to overwhelm you, right? Just have a general knowledge of shed switch attachment of a BPF program is going to be bad, right? And then how do you begin writing a BPF program? Look at the self-test and try to see what is already implemented, and try to look for patterns that you might find useful. In the absence of documentation, which ideally should be there, so, like. Right, so how to begin, and like you check for existence of docs, and then look at self-tests. But people also probably start looking at, okay, my applications that they have, they are based on, isn't that runtime or isn't that language? And then they start looking for the library, and then they look at, where can I start with that library to get some stuff done, right? This is for LibBPF, right? Yes, that would be for LibBPF. Yes, yeah, you're right. In terms of the compilation and set up self-test, it's like the most complicated BPF application. Like, its make file is like nightmare to understand. So I think if you are starting with LibBPF, LibBPF puts trap is probably way easier to start. We actually have a CMake, like a helper file or whatever, the terminology, I'm not sure. It would be nice if someone actually can pick it up and upstream it, and like provide it as like a standard thingy, because it has a logic of like how to find LibBPF, like was overrides for like using it as static, non-static, and all that stuff. So Daniel Xu, who was contributing to BPF, tries a lot. Like he wrote it as like a first version, but we never got to contributing it upstream, whatever that means in CMake world. I think that would help like anyone who's familiar with like CMake, C++ developers like not, because a lot of people think like the even simple CMake file is incomprehensible, so like CMake might make it a bit easier, especially if it has some like common logic of where to find the BPF and all that stuff. But maintainers will review contributions, you know. So I want to farm for ideas from the other, just we can bounce around. Let's see what ideas we've got. For security, I want to add a step about studying time of use attacks. I think the first thing to say for security is not the use case perspective, but just configuring your system rightly, correctly to use BPF, and this is disable unprivileged BPF. It's disabled by default, right, that is good, but don't fall into the trap of enabling it yet. Okay, so BPF, unprivileged, disabled by default. But if you want people to be successful, right? So one problem I see is some startups are taking my exec snoop TCP life and open snoop and saying there's an intrusion detection tool. And I did not write them as security tools. I know that you can defeat them and make them print out wrong things. They have observability tools, they have a different set of requirements. This is why I need to do guidelines and say, these have to change. In fact, there's a Netflix engineer in the bill who has rewritten my exec snoop to be secure. And so hopefully he'll publish it or get it in BCC and I'll use that as a use case and say, he's an example of taking the observability tool and then making it secure where it's far more complicated to maintain and work on, but at least it's got, and higher overhead. So I don't even want to use it as an observer, as a performance engineer. But if that's your requirement, then if security is your requirement, then that's the path you take. There is a talk that we did at CNCFEBPF Day where we showed that if you hook at the wrong level, an attacker could trick you into showing the wrong address for an IP. And then if you hook at the right place, then you don't have the talk to raise there. So maybe that talk could be a recommendation on the parallels of this. Yeah, I can link to it. So what was that? There was CNCFEBPF Day. I'll find the name of it. I'd also like to study opportunities, study events to trace. You've got LSM hooks. I've also got my B-Sides SF talk 2017, where myself and a former Netflix security engineer, we went around the kernel and said, here's all the cool play. Because you can trace UID calls, you can trace TCP open, you can trace a lot of common parts. Signed programs. At some point, well, when I had signed programs there, once we have something to actually say. Yep. And I think regarding the six point, maybe you have it in the XXNOOP as well. But like some of the bigger projects that are out there, they were using, they were probing the user memory. And that's, of course, racy. So like they were attaching to the wrong hooks and using the wrong methods. And it's been there for a long time. And they got a CV on it. But it would be something that people should have considered much earlier. Yeah, I think that's something that Nabil has done with his. For the comm, iterating problems we see. Like I've seen people use the comm, like it's some sort of thing that is actually, right? It's like. Yeah, I mean, I use it all the time. It's convenient. So like. For observability. But for something that you trust, it's dubious at best. And to explain this, for observability, I will, on the entry to syscall, I'll take the pointer to user space, structs, cache it. And then like some time later, I will then print it out. But like there's this huge window of time where people can change it. So yeah, because I just didn't write the code to be for that sort of requirements. I didn't care. It was for you. Yeah. I mean, observability tools need to be, need to not impact the system. That's the number one priority. Like don't, don't have too high of an overhead. And then, then it's about being as accurate as possible. So where it's security tools, it's about being as accurate as possible. And then not having the highest overhead. So there's some ideas for security, networking. Familiarize yourself with XTP, Cilium, Catran. What else? IPv6, UDP, quick. What's quick called now, HTTP3? Right, HTTP3. Anyway, familiarize yourself with the current state of the art. I can feel that, that stuff. What would you want? So understand the problem space, have the basic requirements. So for the first step is like, how do people get quick wins? So like, you're at a company, you've heard, you've heard that EBPF networking is cool. What's the quickest thing you can do? Is it like install Cilium? Or is it, I mean, if that's the case, that's the case. Like that's going to be it, right? I mean, like, so if you want to start one layer lower and you want to use, for example, TC, then I would say use the LibBPF TC libraries because they're much easier to use than the comment line, for example, and it will do all the things in the right way instead of making mistakes that. LibBPF TC libraries? Probably the same for XTP as well, right? We also have, I think it would be a good starting point, at least, to experiment with that. For networking designed, right? There's a number of things that you can do in networking that you could, in theory, do at different layers, right? So the first big design question is, what's the right layer to do your thing at, right? And so at least having an understanding of the relationship between different program types and attach types to say, OK, you want to do a thing, here's the menu, how do you know what's the right thing for you is actually an important design step to do early on, right? Or your performance is going to be a fairly bad. Maybe you're related, I feel like building a very basic DOS type thing at XTP is just a good way to get a sense for how BPF, what kind of interface it has, the packet, like how do you deal with this stuff, even if this has been done better by other tools already, but it gives you experience with it. Right, but if you're going to do something like, oh, I want to scan the payload inside encrypted, quick, or whatever, are they doing it at the XTP later, is probably a bad place to start. On the BPF summit, there was a talk by Liz, which does a very basic XTP program, and that was quite useful, if somebody doesn't know where to start. So again, experience of TC and Q disks, that's familiarizing itself with the problem space, like look what I did, that's the Q disk, there's already one for that. I think another perspective could be different layer, XTP is the bottom one, but we also have per LatinM space program, per C group kind of program, and then also per socket kind of BPF program. So it's a different scope of running the program. Right. I think. Yeah, it's part of designing options, and it's also understand scope. So it was C group, socket, device, XTP is the scope. We can keep adding to this stuff. I think one thing we should probably do is like to document all these with the use cases. When would you use these, for example? Because I think newbies, they have a really hard time to get into this without understanding or reading the kernel code, where you can attach your programs to. I mean, with the TC and XTP, it's maybe easy to understand, but with the other attachment points, maybe it's less obvious, for sure. So you've made a good point, and what I should really do, because we're running out of time, is to do the meta version so that I can go back, not that meta, the other meta. OK, so meta. So it's all the different things you've got to familiarize yourself with. It is the problem space, including other solutions. It is the use cases. It is the grid, like technologies. What else was there? Yeah, and actually, what makes sense for you? Then it was requirements. Then it was basically get a quick win. And then it was get into the recommended programming path from easy to complicated, like it's the easy one, the hard one, the hard one. So that's what I've got to make sure is in each of them so that as people work through it, they're getting the best value. Could we get you to write a new book? Next book is going to be much shorter. I'd love to do a 250 page. I see all these people publishing very thin books. Why can't I publish a thin book? Why do my books always too long? Because it's all this work, copy, edit, and publishing. I'd love to do a 250 page book. Maybe it will be EBPF on Windows, 250 pages, EBF tools. Looking through the window glass of EBPF, something like that. I think it's never happened for Windows. What's that? It's never. Small book never happened for Windows. It never happened for Windows. Anyone have any other ideas you want me to put up here and discuss before we break? I have a quick question for Martin, though. With TCP-BPF, can we easily show that auto-performs other TCP algorithms with a small setup? You're asking the congestion control algorithm, right? Right. What about that, sir? To easily show why you want TCP-BPF, that should auto-perform other alternatives? Yeah, I think it's like you want to write TCP, but it's not necessary, it's better than the latest kernel one. It's the way that BPF gives you easier to deploy or upgrade and change the algorithm. So I think that is the best advantage. And that will, that shorter turnaround time to productionize your experiment is, give you a better chance to improve the performance than the kernel one, I think. It's not necessary. The congestion control algorithm will, we think BPF will magically make the throughput better. It's the quick turnaround time that BPF gives you to deploy the experiment in BPF that will make you improve the congestion control faster than writing it in the latest kernel way. Yeah, I think that could be a very good selling point to try BPF in networking. Yeah, yeah, yeah. And we are also doing that. We are experimenting a lot. So I was just talking to Daniel. So we were, I'm hoping now we can present some data in LPC, this LPC or something. So I really like this crowd-sourced guidelines session, Brendan, that is a good idea to get this moving forward. I wonder if we could spend some time tomorrow doing the, in a fancy way, call a technical roadmap in a simple terms, like the wish list of things they want to do in BPF, fill out the stuff and then have everybody look at it, right? That's a great idea, yeah. I was also thinking, given you doing the guidelines, this should also be pointers where, I mean, sometimes I get emails where people asking where to start, and it would be good if they want to go into a BPF development where they can help to have pointers in the different areas, right? I mean, that's what this is, right, different areas? Or you mean sub-areas? I mean, like for contributing to also the project or the kernel or areas where they can help. But I think this ties into the roadmap that we've been talking about. I think we should do a session on this tomorrow. Yeah, let's do a wish list. We should have a contribution guide. Yeah, from time to time, we've had a wish list on BCC and BPF Trace. And Alexi's done kernel BPF wish lists before. But yeah, to link to them from here so that if people don't know what to do, they can start with one of my bugs. So that'd be nice. Where is the existing wish list for the kernel? I haven't seen one for years. Okay. I think we maybe had something like this six years ago, I don't know, but I think what would help is not specific items, but maybe on a high level, like a roadmap where we want to be strategically and people can dig into that. Not fix this null pointer exception. Yeah, no, that doesn't make sense. So we have a similar half started document right there. Let's just fill it out live here. Okay, it's three o'clock. I think it's, do we have a break in the schedule? All right, thank you very much. Thank you.