 You can do that currently as well though, like you can hook on the VPF fiscal, the VPF probe, SM hook, and then you can like dump the old instruction logic to the user space and ship it off the machine, right, so you can already do that. Yeah, I mean, that could sort of be one of the stuff that's in the best execution of it, where Netflix is missing in the fiscal side, so... We have a lot of stuff that can be post-analyzed. But you tried it at night. Yeah, Netflix is a good source of data. But my point was the missing bits of the new fiscal bits. Yeah. Because we can observe on the banks what the best theory is. So, when you say printed enforcement, this is not a VLSN book here. It's like when you constructed a malicious trace, like you open the socket, you did this, you did this, and then after maybe some offline processing as well, you said, okay, this is bad, get it out. Either that, or six secrets of the VPF program. You don't need to say, like, close the socket. Okay. You're like, I want to construct an RST packet on the socket, because it's just RST. Okay, yeah, that's a good idea. Close the socket. And then, like, sending signals is not really possible, I think Alex had mentioned it. Yeah, I'm just trying to think of whatever enforcement stuff should be over that. Cool, I think the general topic of enforcement here first. But this becomes more, I think people become, start becoming more aware of this when you start writing telemetry. Like, there is an open source telemetry implementation and an LSM implementation. You start realizing, ah, these are the bad patterns we need to watch out for. And then you start doing enforcement decisions there. So, there's a circular, we need to see that conversation by starting these two areas, at least in the open source land. So, like, the audit bit is interesting. We discussed at Lumber's, audit has some, like, hooks in the kernel, which are just very hard to parse all that information. But we sort of found that these hooks are really useful, at really useful points in the kernel where the LSM hooks are not there. So, typically, you will have, like, in the LSM hook, you have very minimal data, like, that doesn't have the right context to be logged. And after the action actually has happened, this is when you want to do the logging. So, there are different places where enforcement needs to take place and where auditing needs to take place as well. So, I think the audit surface of EPF currently, like, say, you need its trace points, but you can, sometimes, you need to trace in the middle of a function, which you currently do. So, some sort of, like, what we talked about back then, where we didn't, like, get into it. Look at how you can extend audit with EPF. So, get the un-flexible side of audit out and use it, like, hook BPF programs to audit hooks. That's the idea, right? We dropped this idea. There's no one else within this. We dropped this idea because we weren't able to work around it, right? No. But it was quite a long work run. Like, if I imagine somebody is trying to... They shouldn't need to invest, like, three months crafting an audit hook, which is just available to Linux Audit in general, right? Like, that is a... So, do you propose anything specifically or, like, just a wish list? Wish list, yeah. Wish list is to extent, like, be able to prepare BPF programs on an audit event. That's the right context and be able to block that into the ring. That would be in detail, right? So, like, what is the answer? It's QD, right? Yeah, yeah. Oh, it's nothing. It's QD, so... There are two different kinds of questions. I just don't see you speak, right? So, like... So, I imagine we need to get from, like, from the... Get rid of the audit users, right? But, like, just trigger BPF programs on, like, audit... I just don't want any audit users to face any audit policy language. I want to configure what events I want to lock by, like, my own... set of BPF audit programs. I want all of my security, policy, telemetry, implementation to live in different BPF. I don't want to mix them up. And there are some cases where you're actually required to, like, have audit enabled. Now, that requirement then turns into, like, you configuring this audit policy language. I think BPF should sit in the middle there, and, like, they do IP tables, so people who want to write audit policy use BPF at the back end, right? And then people like me who want to advance stuff with that, they can use BPF programs... Politically, for the chaplain, right? So, you're going to... On the online patch? Yeah, it won't be okay. I mean, this is a road, this is a bit... This is, like, for more miserable stuff, right? We're not discussing policy stuff yet. I'm just saying, like, art is down. It's just, like, you don't see anyone working on it. We will... Maybe somebody will look at the road map and say, yeah, that's a good idea for me to work on. They'll take it on. So, I'm just writing it there anyway. So, the next thing... This is we alluded to this yesterday as well, right? The long-term future of unrelated BPF. This is, from a security perspective, like, it's kind of sad that Hardware bugs came along and it, like, pushed BPF into the root territory, whereas it would be really useful for unprevisualistic BPF. So, what do we propose here? Like, are there some concrete steps we could take or is this a further topic? And the BSE work with Hardware people? So, the only discussions that I've been in, like, with Microsoft's trade people, is the... which we don't allow unprejudged BPF at all, right? Like, we've implemented that in Windows and currently we're planning to do it, but the discussion comes up and says, okay, if you could somehow run BPF programs in a different address space. So, there might be some scenario where you care about hyper, right? That you might be able to run in a separate address space and then it might be okay. Like, pretend you're actually running in a separate VM or something equivalent to that and you can't move inside channel packs because you're not sharing chain, you'll catch on to whatever else. So, you might be able to do something in the same process if it's sufficiently separated, and what sufficiently is, is probably a research problem. Right. So, we don't have a use case to allow the unprejudged BPF programs, but if we did, this is probably what would be required, but right now, we're working on it until we actually have a use case for the products. So, will this run in some way until it's being complaints? Yeah, that is a very big question for us. A container is the one of the attention that comes to the use case we could type in in the doctor's web. I'll put something under the cross-platform stuff, which is absolutely a little bit with the different address space for BPF programs, that is, there are some of those other runtimes. It's not the Windows or Linux runtime, the ones on the slides that are all for the user-man BPF programs attaching to some other runtime up there and say user-man, demon, whatever. And so, for that, it's kind of a different address space and so there's a little bit of overlap relationship kind of discussions. They're not quite the same thing, but there may be some techniques that are common in those two areas. Okay. So, container observability to most, I think it's a good one. Auditability of BPF events is, I cope one level higher than the auditability stuff. This is what Jason was mentioning. There needs to be a reference implementation of the app. We tried doing something with an intern, but it didn't really go anywhere. Maybe we could just take off that effort and have somebody take this beyond some of the ring buffer stuff we've been implementing for cross-sectoration logging. Somebody should be able to take that and deploy it on a machine and see some event coming there, right? And this is where the audit stuff could come. So, a project that enables BPF for telemetry would be nice to have open. But there is already system channels or whatever, like Microsoft Oaks has developed. But there are some different ways they're not just, for example, they don't do PTF there. Maybe we could influence, like, such projects to use PTF or have a BPF-based implementation of these things. Sure. There's things you can do better. I think where Kevin and folks were starting is they wanted to also work for kernels that didn't have PTF yet. True. As they were saying, once that went first, it would work for all of them. And then, if you need PTF-specific stuff to light up extra functionality or better performance or whatever it is. I'm going to move this forward. Yes, sir. I don't know. The previous wording of that, I had a different idea in my mind. So, I'll leave it. Okay. Okay. No, it's just fine. I'll just... This is done. We may need to extend the next audit to trigger BPF programs to expand the audit surface, right? And we also need a reference implementation in the open source that can show how you can do telemetry using BPF for security purposes, right? I mean, this is... You can do observability already with a lot of tools, but then, friend of me has mentioned that there are talk-through races that can come up, and these observability tools are often not a good surface to do security telemetry there. So... Okay. So, what I have in mind, it's something along the lines of how do you get some sense of the integrity of the kernel in terms of how we get programs without this program is loaded and attached to this point to this thing and getting some sense of of all that actually is happening in the BPF system itself and being able to look at that after that's the sort of idea. I mean, I haven't had... So, basically, what do you mean that you've changed a lot of stuff so then, if you're... you can come into the system afterwards, how do you go back and find out what kind of stuff is happening in that system? Now, if you pre-deploy some tooling short, you can put whatever in place to be, what if you haven't got that tool in the first place? Is there some way of knowing what programs are loaded and at the time, and maybe how they're attached to different things? Some sort of... like, how has the kernel changed its behavior since it moved forward? I mean, the BPF system, at least, that just broke the patch and then just prints it out. It's longer. Like, it zips you. It's a bit processing. You do? Yeah. Do you program the patch? It's one strand. I hear you're saying that this is feasible. Yeah. Like, BPF, when it's loaded, does the program also start when it's loaded? Is it much fun when it's loaded when it's loaded? It's just... you try to, like, come out with your own understanding and see what the sequence is when it's loaded. Great. One thing we're considering is what we discussed with the verifier, a notifier, stuff, right? Like, some events that you've learned that you can understand what the verifier is doing and attach a BPF program to those events. But then there's a circular thing. Like, you know... Yes. That's different. So that's back to the... developer problem, right? Yeah. Yeah. Yeah. It's something that we have on the range of that work. What I'm thinking is effectively important on how we can stop the BPF stuff. Yeah. So, like, the BPF runtime generates a bunch of trace events. And that's what we do. So we don't do everything in terms of what you might want. Well, I think that's something we'll put together right now. But it's not attached, but it will... like, interpret the... Like, load and download programs and actually attach... Great addition to the show. Load and load, attach, detach, and generate operations like that. And that's great. That's an extent, right? At least in a very basic way. Yeah. Load and load, attach, detach, and create that. Now, we already have that. Good interaction. I think we're all good at some work. Yeah. And then... We do this as well. Basically, all of this is called... We should persist in the log or something, which you can go back in history. See what happens in the context. Usually, we get it all from the node, right? We're about to know what the future is like. And then for the... Okay. Does anyone want to add anything in the security section here that we might have missed? So, we did cover it in, like, the long-term future of Unprivileged EBPF, right? Like, how can we... When will processors change enough that we will be confident enough to do that? So, Alex, I mentioned... So, Alex, I mentioned, like, the end-bran stuff from Intel, these sort of things. How can BPF influence that? Like, how can we sort of start a dialogue in that area? Maybe Linux Foundation or BPF Foundation can help us connect there. Thank you. Cool. I think let's move to the next topic and we can revisit if we find something. This is where, like, I would expect you folks to start chiming in more, because I'm not a compiler and library expert, so somebody probably who works in this space will be better, like... So, we just have, like, feature parity between LLVM and GCC back-ends. This was added by somebody. Okay. I think that's a good one. How do we... One thing we alluded to there, right? Like, how do we ensure that these compilers are behaving the same way? So, is it just self-test or do we have, like, a compatibility test suite where you can say that LLVM and Klanger are doing the exact same thing for their BPF back-ends? Yes. Yes. I think this is what we want, right? Like, we should try to make sure that one, given a piece of code, there's one verifier, there's one... Clang generates something, passes verifier, GCC generates something, it doesn't pass the verifier, or the other way around. So, you want consistency between compilers and the verification result, for the verification result. So, here's what you need. Here's how you need to rent your code with these patterns or something. But this is what we also discussed yesterday, as you were mentioning, Mikaela, right? Like, they have, on a ClangPath submission, trigger self-test, and same thing for GCC as well. Now, we also, as a recap, I think we discussed that whether a revert on a submission would be acceptable to Clang people, we don't know. Maybe there is some, like, some community stuff we have to do there. Can you... Yeah, JIT is later. We... Okay. Let's talk about libraries a bit as well, because we've discussed quite some stuff in BSC about libraries, and what sort of libraries we encourage and stuff. Andree has, like, we have LibBPF as the C-loader, then there are a couple of co-libraries, there are some Rust libraries coming up to load DBPF programs. Do we want to give any guidelines? Do you wish less guidelines, whatever, in this space? Guidelines would belong to Brendon's talk, by the way, so we can backfill, but... We don't know... On the other hand, if someone comes up with some genius other way, it actually would be super popular, and you say, like, do you review that? If they come for no, come find a good site. What? If they come for no, if you don't put something down, then it's pretty much guaranteed that you'll have a heterogeneity that can be drawn, like, high up holes in the ground. Can be drawn, like, high up holes in the ground. Yeah, we can't convince them, but giving them a cookbook or guidelines is something better than nothing, because in this space, you can't build that version in order to, say, add some great new features or something like that, as opposed to, oh, I didn't even think about that. Anybody know, like, the example for an section needs to be given a specific code because nobody else can come up with that outside. I think the only way you could enforce something is if you have, like, a test suite and say, well, it should be able to do it. It will see that it can be resolved and if you know how to do it, then congrats, you know. What does it mean to load? Like, why do you say, like, load them, load them? Like, what does it mean? Exactly. What does it mean to load something? Partially, it's not that good, but, like, no, no, no, I mean, it's complicated. Give us a little follow-up back to the technical program. Like, if we just say, okay, this set of self-tests, if they would also go, like, if they also pass on the loader, then the loader can form, or, I mean, so far, we can do anything like that. But this usually, like, you all learn from loaders first, integrate them self-pass. Second, they're, like, more or less compatible. The API, you know. It's always the users, right? And, yeah. So, it's hard to, like, isolate the set of samples. Do you think we should have, like, a self-test? Like, you have LibBPF as a separate repo. Should we have self-tests? They're not self-then, but then there are their tests. Should we have a test repo that is not with the kernel? Who's going to spend their entire time trying to self-test? I think let's, at this particular point, right, we shouldn't think about the who. Because I want somebody to look at this doc, and they're, like, maybe I'm going to do this. I like doing this. So, let's decide the resourcing part afterwards. Like, let's figure out the ideas here. Could we, like, so you mentioned L-format standardization. Is it even possible, or that's never going to be a thing? We could give, like, a criteria of this is, I mean, at least section names. We shouldn't have 10 different ways to name a particular program section, right? Like, this is just, from a usability perspective, if I want to today use my program in Java and then in, like, whatever, Go or something, and then LibBPF, I shouldn't need to change my program to do that. Yeah, wish list. But they do, I mean, they do agree on, like, GCN client, a lot of things, right? Which are, from a user perspective, much easier to implement. Like, for example, the red-polline thunk is the same in GCC and Clang. They could have chosen, no. It implemented differently. No, I agree. But, like, the thunk, the name of the symbol itself, right? They agree on that. So, I mean, I'm saying, like, from a, as somebody who needs to implement it in the kernel, we could have had if GCC, this thunk, this symbol in the table, they do agree on something. They do agree on sometimes the flag names and stuff. That would be perfect, but, like, it makes users' lives easier when they try to use Clang and GCC interchangeably. So, yeah. I think there are basic stuff like naming. Name bike shedding should be eliminated for people focusing on other things, at least. I'm not going to call it LSM slash, I'll call it, like, security mod, or security slash. This sort of, just take it away from the flexibility. Yeah, exactly. Right now, the standardization lives in libBPF, right? Our view of the standard is the code in libBPF. We don't even have it anywhere. So, okay, okay, then we can discuss. We can add it, I move it there. Cool. So, core is something, core, this is not core that... So, the goal behind this, think about it, like, in so many years, when there is multiple ways to load BPF programs, our intention is to make it easier for users to use BPF programs. Right now, there are sysadmins, kernel, people trying to use it. It's going to transcend towards applications, right? Even beyond then, once processor vendors get their stuff together, it is going to move to unprivileged users eventually. So, at that point, you want to have, like, you don't want to have, like, a scattered way of, like, a format standardization. You want to have, like, okay, I wish 10 years back I would have agreed on a format and this wouldn't have scattered so bad. So, ship may have already sailed, but not too late yet, I guess, from my perspective. Cool. So, I think, like, core is the other core, which is this group of people are more interested in here. The verifier, this would be nice because, as like Dave mentioned, that there are other platforms implementing verifier. So, document the verifier behaviors and cross-platform stuff. Yeah. Yeah. I think that the... Thank you for organizing this and being on the committee for organizing it. Okay, we have a break with the VSC quota then. So, that, I believe, more in, like, the tool that's smart enough to do some experimentation, like, first with, you know, it's, like, the fact that, like, an exercise, like, more, more difficult than that. Right? Like, this doesn't really even think that's correct. But it's, like, the pleasure that the verifier can have. Yeah. Yes, but, like, we take T now, which is, like, so easily formalizable. And, like, the best, like, the extent of, like, easily formalizable things in the verifier. And, like, surely we'll verify the state out of, like, 2% of the logic. Right? Right. Yeah. Right, yeah. Well, it was to not have 98. It's a start. And also, like, given the rate of, like, G is the verifier, who's going to keep up with, like, formalizing... Yeah. ...the theory I believe to make that clear? We'll go in there at the end. I mean, really, like, the... What would be even, like, K-1 stop? Like, how much logical force like there are on that? That's important. Is it formalizable? I'm not sure what that means, but... Formalized may, like, you often make, like, you know what that means. I know, like, I'll say, like, I mean, is that formalizable? K-1? Just sure what that means, like, that, like, if he keeps going, if he keeps going, if he's not going where, where he's fired, then, like, he'll go sad, like, how he's having... I mean, I think the first step is just writing it, right? Like, this is what it is doing. And then the next attempt will be to come to formalizing it. Right now, the only place where it exists is the... I agree with you with the rate of the verifier changes, what we write and what we formalize. There'll be, obviously, always a delta, but if there is now... If you can, like, create a feedback loop where you can say, this is how... This particular bit of program sort of represents this formal stipulation of the verifier. You could say whether it breaks that assumption or not. But in the very beginning, just write stuff there. This is what is happening. This is how offsets are handled in the verifier. Somebody who wants to implement a new verifier could read through this, and, at least, this is how the verifier is supposed to behave generally. Annotate the verifier strings itself in the kernel and generate documentation from there. Just wild idea, but, like, no. Like, this is what is happening here, right? Wild ideas, visionary stuff. But it's, like, in a technical roadmap, right? Like, I... Sometimes, some questions we try to answer here are about resourcing, right, and feasibility. On this roadmap, and we'll classify this, right, in terms of various tracks these things belong to. For example, cross-platform, documentation, formalization and standardization and stuff. And then you effectively... This is not a roadmap currently. This is currently a wish list. This needs to be developed into a roadmap. And then you'll have some sort of a feasibility check or annotations around this. Agreed. You remember, right? One of the reasons why we want to do this is, and this is currently not the roadmap. These are notes from various couple of meetings and ideas. This should turn into a roadmap eventually. And some stuff will be punted where, like, folks will be like, hey, this is not a priority for us. We need to... Roadmap will also have, like, feasibility and prioritization attached to it. Currently, it doesn't have any of that. So, yes. Yes. That's actually a very good... It'll give more benefit to the ecosystem for a developer perspective, I think. I would need to use just more to know that. Capture all the arrows. Thanks, Joe. I think it's the goal of the meeting for playing with the local studio, right? I mean, that is still... So, if you don't say that, but, you know, like, you're an engineer to seek all the bills and all that. Like, so, one other thing, like, where we often run into is that we hit the complexity limit of the rarefire, but it's super hard to reason where on the point where things go south, right? So, like, some sort of analysis framework where you can maybe see, or, like, more into the details. Right? Sometimes, like, the... Like, it makes a number of... So, we have a way to map the... But we have a way to map that back, like, way, like, at that time. There's a lot of information about, like, you have one line of T-codes, and, like, you have many instructions spread out in terms of the valid instructions. There are some attempts when you, like, obstump BPF assembly, that would correlate some, like, lines to assembly code as well. So... I mean, there's one thing is, like, better... Sometimes it's just harder. And then, like... I think that, like, people that have been doing this for a long time, like, get a verifier there. And then, I don't know what you... You don't need to even know what it is, right? Like, the object dumps with the assembly by the side, following the lines. Yeah. Like, we could maybe, like, do, like, a tutorial on BPF, like... Here's how I debode this problem, right? I don't think that... That is what people want. I think what it does, like, is pretty straightforward with people who still look graduate. Yeah. Yeah. This is, like, the... You could go to a different room in... Even within, like, Google or somebody. I can, like, no way on another time reading that assembly. So, like, you could go to security folks somewhere. Of course, there's a reverse engineering section where people will be willing to read that. But people don't... Yeah. One thing which I feel this could, like, make user experience better is when you're writing the BPF program, think of an IDE thing, right? And you know patterns. This helper returns an arg pointer to btfid or null, right? And then you're trying to de-reference that pointer. Please add a null check because your verifier will obviously fail here. So there are these obscure verifier errors which advanced-level programmers get into and this needs to dump and all that. But there's some very basic stuff for people who don't do that. It just, like, reduces one iteration. Yeah. So what this goes into is a general linter, then, for... I do remember some Twitter posts around somebody writing some linter somewhere. I don't know what happened to that effort or how far did it go or even are they suggesting the right suggestions in that linter or not. Anything else you folks want to add in this ginormous verifier section? Syscaller does that, yes. What we need is a self-test to, like, BPF code coverage metric, right? You run self-test and you see what portion of the BPF code is actually tested. It'll be fun. Maybe we have a really good number there, right? And this will be, like... I mean, like, this is true for any metric, by the way. Like, you can game it, but if you... If you cover, like, all differences, but not, like, first and third... Yeah, yeah. You can... Yeah. Yeah. People who want to contribute code, they're like, there's some people who want to, like, for this stuff. Can I send a patch? What can I send a patch for, right? This will be a nice... You could actually trigger this on the CI. So when you actually accept a new patch for a helper, you may be able to see that is this self-test that you're adding covering most of your corner cases? Maybe you can say, can you add a case for this? This will help your maintainers will be happy to review patches because they'll have some help doing it. Like, CI is already helping you. It gives you a signal of, this is absolutely unacceptable. And then you'll have more metrics to work on. So code coverage. Let's just add it here. I'm going to split them into categories, and then we go into prioritizations later on in the BSE meetings. We can... We can give... We can assign a... We can say that this will have a higher impact, right? Yeah. So register blinding. Yeah. Tiny URLs. Embed tiny URLs in the kernel verifier message which link to, like, descriptions of what the error is. We do that internally. There's a link that says, like, read, go, slash, or, like, read whatever bugganizer for more information here. So... JIT compilers. This is fun. So one other thing is across platform stuff and calling conventions. This is... This I really care about, actually. Feature parity amongst architectures and probably also then operating systems. ARM, we saw that there was a gaping hole in the whole trampoline stuff which is now getting fixed. I'm very happy that this is going on. But ARM still has multiple things missing. I think, like, there is a... Dispatchers is still unimplemented, I guess, on ARM 64 or... How did some analysis of what is the missing stuff? It was added in February, I think. Like, the end of February. Atomics were added this year. Can you... Okay, actually, this is a good time to switch. So what is the... Where do the maintainers think on, like, so now when you accept patches for the BPFJIT for new features, as the ARMJIT is going to reach almost feature parity with the trampoline stuff going in, will you folks recommend that, hey, you're submitting this feature, can you do it for this stuff or is that too much of a... I think the main problem there is to ask somebody who is not familiar with architecture, MIPS, S390, whatever, to write architecture-specific assembly code. It's very hard. Yeah. We talked about yesterday about ARM-based VMs on, like, on cloud. Is this something that... to enable this stuff from happening more easily? Could we, like, have machines that people can access for different... It can move works, man. Like, we have can move support for S390. Yeah, I think this is not feasible at all. Yeah. I think it's a case by case. At some point, you might say, we're not going to take this feature if you don't know. You don't want to have... You don't want to get into a place where you have a feature that could not be done on a different platform. So this is actually a very good point. Like, the whole trampoline stuff, there was some initial discussions around, like, whether text-poke BP is synchronized... We had the talk yesterday as well, synchronized on all CPUs. So, yeah, just a feasibility check, I guess. Like, can this even be done on architecture... at the architecture maintainer there? Do you think it is possible for this stuff? Yes. We never have? It's done now. No, I agree. I agree. This is... But it's not like a feasibility check in terms of implementation gating, right? You can just check with the maintainer. Do you think... you built enough context? The maintainers have been reviewing architecture-specific code anyways. So you can sort of... It doesn't have to be always accurate. It's not that... It's just your... I know this is not possible on this stuff. So are we doing ourselves a disservice here for that? For the window stuff, how does the JIT work? Do you have a JIT? Yeah. UBPFJIT. I think let's move on to the... Yeah, specification stuff. This is your section. Okay. So architecture specification related conformance test suite that could be shared between platforms. I don't know who wrote this though. Strike it. Yeah, I think everything just hinges now on self-test being... Yes. Which is actually not a bad... Which is not a bad thing to be honest, right? Like... That's a theme that starts to emerge. Yeah. Okay. Yes, networking. Yes. Yeah, let's go to networking completely. If people always come to me or probably you say, do you have to know what the performance is that Intel makes? You know? Or why? No. But it's pretty common, right? I don't just... I think not just Intel do and all that. Yeah, I think that's really important because the show is more serious and also we would maybe have a way to see between kernel versions whether things are actually going in the right direction or whether they're getting worse. I think there's something that... I worked on this quite a while ago like this NIEM group. And they actually funded the foundation from the addition kind of work. So that you had like a neutral place that is doing the networking and you don't have like vendors that are trying to get away from it. And that guy is like... You don't know what they're going to do when they come to do NIEM stuff. So it just looks... I think it looks better if it's like a neutral one. I don't know. Yeah. Yeah, there's two pieces of that. One is who writes the performance benchmarking test and who actually runs it, right? Yeah. And so sometimes those things are done by the same organization. Sometimes the writing test is done by one organization and the execution is done by somebody else like UNH and well for example who runs them for lots of organizations and you can always outsource that part. But yeah, I would figure that out too. Cool. Yeah. I would, yes. Lex is not here. Lex. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. So compliant with what? Compliant with like... Okay. Okay. Yeah. It's a good idea. I think this is we discussed the XTP stuff where you want to talk in particular about feature parity documentation. Okay. Okay. So this is like basically the BSC will provide a lab access to the XTP and the XTP will provide a lab access for some developer who's trying to work on them. Okay. Oh, that's sounds, is anything else we want to talk about on the networking side beyond XTP? So the next are going to ship with boxes that are saying BSC compliance certified. Yeah. Don't mistake it for Bitcoin. So then we have, I think this before we go to developer excellence, there are new use cases coming up like HID for BPF. Do we think of other use cases in the kernel that could benefit from BPF that we haven't touched yet, but we may want to at some point. We talk scheduler is one thing we've talked about, like with the ghost project, right? Like implementing a shed class in BPF. Like the, I don't know. How has probably. I think my take of some new use cases and I think that probably gives some, I don't know, I love the areas where we would think that would be a new use case for the actual, actual concrete things that probably have to come out from production. From users, yeah. Yeah. Like, got it. Let's move to developer excellence quickly because we've touched on this in areas, right? And for one main, one thing that we discussed internally was somebody who's trying. Yes, we are. Yeah. So, yeah, like imagine on the BPF.io website, you have like a program and a program attachment type and a sample that shows up and explains what is happening and what is going on there and what you can, what you can do that. Currently for us, we were even trying to look at SOC ops and all these, at least, especially the networking stuff for people who don't understand that much too well. It was hard to figure out like, how do you even get started, but self-test them, right? But that's not a great place to document things. So, but in the kernel sources, right? Yeah. Yeah. However, we're like, see a sample program. Like what, what can you do with this kind of program? Like when, also a good, good place to know like what works where, right? Have a little bubble that says Linux windows. Okay. I think somebody wrote like BPF standard library, C headers, or an easy to consume code example. So getting started for BPF library side. So it's, I think, I think it's a, it's a good idea. So we put it as a part of the roadmap somewhere in this developer ecosystem side complexity analysis of BPF program. We talked about this, right? So this is, and debugger is something we didn't touch. I've never used a debugger on a BPF program. I just stay right at the code and like try to figure out what went wrong where, but that's also I do it with the kernel as well. But that doesn't mean that this is the way developers should normally operate. So, so currently for me, sometimes what happens is the verifier is going to give me an e-inval, right? Especially with the preload programs where you don't get the verifier trace back. And then I go in, in these e-inval locations where there's not even an error message and I put print case, right? I think that could be done. Yeah. So general development, better messages and better error messages and also debuggers on the BPF side. But you can, so further thing we talked about security policy stuff, right? Where you have callbacks there. Could we just have like callbacks from the verifier that says this is the verifier context that is available currently and inspect that somehow. Just debugging. I guess the answer here is why a program fails to verify. Sometimes you don't get them. Sometimes you get it from an e-inval from like some BPF context access or something and it doesn't have a log there. Yes, I, yeah. Okay. So, like, you know, either the children stuff or the incremental interpreter and then like single step through it and see what is the actual content of the registers and looking at this side by side with the verification law does install all those constraints with new big parts in the verifier but it's maybe more like, for general development. Yeah. Yeah. I usually have to verify when I use this, right? Yeah, ID support is, we talked again about this, some sort of linter, some sort of things that can autocomplete stuff, add section names, like do all that stuff you shouldn't spend time doing, right, figure out, I want to write a program that is modify return or like LSM just generate all of the, that works, for me the main painful point in VS code that I would like to, I use VS code, so I can, so I think the, like, I want to implement an LSM program, right, big signature of the LSM hook, you go to LSM hook, LSM dot S, BPRM check security or like security, I don't get Xhatter, copy that signature from the LSM hook devs dot H or something and then bring it here, that could be effectively autocompleted for, like, from a developer perspective, tracing, what kind of functions can I trace, maybe some sort of information from VM Linux dot, VM Linux dot, hey, you're trying to trace this function, but this is already inlined, so you are going to run into issues, right, like, yeah, yeah, yeah, I mean, so let me just write this down, let's move on to observability so that we can speed the session up a little bit. So, Brendan, probably your section, maybe I added it, I don't know, okay, yeah, I think you said heap tracing yesterday for the Java, Java stuff, right, and you probe speed up, yep. Next LWN article, there should be a new era of trace points, I agree with this, I think this is, is there generally pushback when you add more trace points these days or is there, okay, so I think this is all great, documentation, we've, we had a documentation section, we have another one, yeah, so there is like relationship to tracing mechanisms, I don't know, what would be nice is, okay, I think you have seen that particular post. So, yeah, so is this even, so the one quick question on tooling stuff, so BTF support for inline function tracing, is that even possible? So I see that DWARF has, I mean, maybe I missed, this is a misconception here, but DWARF has some information of like what function got inline where, this is how debugger sort of single step stuff, could that be put into BTF somehow or is it just a lost cause? For kernel probes, somebody from compiler line, okay, okay, you can attach some, the attachment is going to be because there's not going to be a knob, you can patch there, right? Cool, I think this is, just needs more expert exploration, but I also think it's a tough ask, that when you say inline functions are already like these things, but I'm looking for those functions that compiler automatically inlines, maybe at least some information from the, from hint from the kernel that this thing you're trying to probe here, you think it is, like, couldn't... There'll be code movement and it'll be most likely the other stuff in the way. You can know, you can know at what is the first inline instruction, but I don't think like anyone can see that that will be the first one that you could do, for example. What would, yeah, like, if you're taking the, what's in the register, or what they were... Well, but even if you wanted to create a pack of, like, the call of the inline function, I don't think we would know where, like, the first instruction of the inline function. You can look at the code, like, object... I think this particular... No, but I, no, not human part. I think this was, I think there is some information about what code inline where, in dwarf, and maybe... It is, like, not the first one to be, because there's some jump, jump back, and all this stuff. Yeah. Yeah. Makes sense. Okay. So I'll just delete this then. I mean, this is, this is, this is, like, in wishlist and feasibility, if we establish the lack of feasibility then... I mean, unless you're a human, somehow to say, like, I always know that RIS point is something that... I don't know what you do with it. What's that trace of things? Like, I wonder if you're telling us, I think the stack trace, like, maybe something is... Is that cheating? Yeah. Is that cheating? Yeah. Generally in the kernel. Yeah. So I'll move to standardization. This is... One thing in tooling, is it possible for, in the skeleton, would it be, would you accept something like this, like extensibility of PPF tool to generate custom skeleton snippets? Like... So, like, let's say I want to do attach, but in the attach, I want to send an RPC. The use case we have is, like, so imagine we want to use PPF tool, skeletons, right? The underlying code that gets generated, we want to customize some of that so that while we can retain the tooling from, like, an open source implementation to some custom internal stuff, the interface remains the same, but we can provide different snippets that get generated there. Now, we could patch PPF tool on top and do that, but it would be nice to have some extensibility to what skeleton can generate. So, like, templates, right, like a skeleton template of some sort. So imagine, like, when I do a BPF ProG load, right, instead of doing, like, a BPF sys call here, I want to do an RPC. I want to retain the... So, in BP... I want to retain the skeleton API, right, like BPF ProG load, attach whatever, but in the load, I want to do something else, right? I want to do... Let's forget about the RPC, right? I want to do, like, there's a BPF sys call. I also want to increment a counter, running somewhere, or, like, some statistics. Like, let's not use precedents here, right? Like, I mean, go has that, for example. I'm not saying this came from piracy, but, like, there needs to be a balance, right? You can't, like, keep adding stuff. Like, next thing will be, like, why don't we have a callback before some skeleton function for the next one? And then, like, in between, and all that stuff, we can just show it again. We do, like, have a pre-call, and then we give a load, and then we give a gas. But, like, at that point, it's usually very long-scale, right? Like, you have to be careful that it doesn't do anything. So, no, so, I want, like, I want this, for example, for BPF programs that are loaded in the Perf tool, right? For example, you want to use BPF skeleton. Just keep the same interface. You don't want to specify custom stuff there. That uses a default template. And then on, like, in data center deployment, you want to do something else. You want to increase, add some instrumentation there. But you want to keep that interface the same. You want to keep using BPF tool, but just change the template under the hook. From my side, this is, like, a feature request, and it makes the whole, sort of, not keep using BPF, not requiring to patch Perf or not requiring to patch, like, BPF tool, but keep, be more, or, like, anything, anything. I'm not implementing, I'm not giving it the implementation. I'm, like, I'm not even, I just know that there is code gen functions in BPF tool because I saw it two days back. I did not even read that code before. Yeah. The use case is that people want to load BPF programs in Perf, which use, which should use a standard interface, right? But then the same thing when deployed on, in another, you don't want to, like, have something that is checked into open source, like, not, that is using BPF tool. We, yeah, we can maintain our custom patches on top as well. Flexible skeletons, which they are not generally. Add muscles to the skeleton. Make it move a little bit. So it's fine. We can discuss this later. Yeah. OK. So standard, so current instructions, verifier, behavior, elf layout, query format, BPF, standards. That's it. So I think the next steps for us will be, like, we'll discuss, arrange these into, like, mix and match the categories, prioritization, or, like, we impact metrics or desirability metrics, and then publish a more simpler version that is not running node stock. So verifier messages, right? Like somebody, somebody combing through the verifier and, like, printing the stuff, documentation for sure, standardization stuff, your network performance funding. Thank you. Thank you, everyone.