 All right. So I'm Dave Thaler. I work at Microsoft in the group that does both Windows and Linux, and so everything that I've done has been open source, cross-platform, standards, and so on. But I am excited to do my first time here at the Linux-specific conference and I'm presenting from Windows desktop. So there you go. Because this is all about cross-platform, right? We want EBPF to go cross-platform, so. All right. So just some easy background ones to kind of get us back from lunch here. EBPF already has a bunch of runtime platforms. Okay. And so you cross the years. EBPF for Windows is the one that we've been working on, but it's been more than just Linux since maybe 2015, if you look at all the runtimes that have been out there. And so the goal that we want to talk about in this presentation is to get at least the BPF and BPF tool to be cross-platform. Okay. Now, to Brendan's point, yeah, eventually we want a bunch of other tools that sit on top of things like the BPF to also be cross-platform, but you got to start somewhere, right, and move up. This would allow more people to be using the same implementation rather than re-implementing them that we've seen over time. And allow it to be a lot more easy to write applications that are indeed cross-platform. So how do we do that? Well, the usual approach that's been talked about is you need to split the platform-magnestic functionality away from the platform-specific stuff, okay? Non-trivial exercise, and so that's what I'm going to go through. This is the table of contents, the things that I'm going to walk through. And rough for the order of probably a least controversial to more controversial. So probably more discussion towards the end, but feel free to ask questions anytime. But those are the five questions I'm going to go through. So if I look at what's in the BPF and BPF tool right now, okay? There's things that fit into all these different categories here. So the first one is fairly straightforward, which is things that are compiler-specific, independent of what platform you are. Things that are very specific to GCC, okay? So these are things like pragma-GCC, underscore attribute. You have some GCC language extensions that are not part of the C standard. Things like equals curly braces with nothing in the middle. You have things like assumptions that you can take VoidStar and do addition on a VoidStar. That's not in the C standard. And then things like the parentheses around curly braces, okay? All those are GCC-specific extensions, okay? Then there are things that are assumptions about sizes, like is size t the same thing or different than an unsigned log, okay? Answer varies by platform or compiler or whatever, okay? So all those things are kind of littered throughout the code right now because it was intended to only be compiled with one compiler, right? Well, not surprising, okay? And so here's the proposal, sorry for the small print if you're in the back of the room. Adhere to the C standard whenever possible in at least cross-platform code. Or at least features that are supported by all the relevant compilers. If there's something that's not standard but it tends to be supported by all the compilers out there, then we don't care. Avoid unnecessary assumptions about type sizes. That's the bottom one there, the size t and unsigned log. Cross-plat files, here's a principle. The cross-platform files should not hard code anything that's compiler specific pragmas or attributes, right? So you would never hard code things like pragma, GCC, or attribute, whatever, okay? Anything that's compiler specific would move to a separate header file that would be the same name for different compilers, right? So you'd have a compiler.h for GCC. It kind of already is one, it's linux compiler.h, but if you're using GCC on a different runtime, then maybe it shouldn't be called linux compiler, maybe it should be GCC compiler.h, right? As GCC isn't just for Linux, there's other runtimes out there on the older slide there. And so that might be renamed and you'd add things like MSVC compiler and so on would have the compiler specific definitions for those. And then of course in the C file, as you just say, include compiler.h. And let the include paths work out which one it is, okay? And I have a bias to saying let's try to avoid if-defs as much as possible. You can do a lot of stuff with if-defs, I hate if-defs, okay? So in my proposals here, I want to try to propose things that would work with minimizing use of if-defs any place, okay? But maybe they're not unavoidable, I'm just going to try to minimize them, okay? So just for example, right, that's the sort of stuff that you see in there. And then you use libpf deprecated libpf alias, dot, dot, okay? I said to put the easy stuff first, right? Okay. So that takes care of just making stuff compile with other compilers, okay? Second thing is platform-specific code. My platform-specific code, I mean things having nothing to do with ebpf, okay? Just things about the regular platform that are not ebpf-specific. That's issue three is if-three ebpf-specific, okay? So things like Linux limits, you have other things that aren't ebpf. They're things that you also tend to use in addition. Things like rlimit, netlink, and so on. And there's implementations of various functions in libpf or commands in bpf tool that the implementation may vary, but of course you'd still have one, okay? And so I'll give you an example on the next slide. So here's the proposal there, similar to the compiler one, that says, okay, we could put platform-specific includes, like include linux slash limits.h, stick that inside of another header file, and call that platform.h, or pick your favorite name, I just made something up. And so you'd say something like linux platform.h, windows platform.h, and you'd include in that, and then that would do all the other includes and things specific to the platform. And the same thing, you'd use the include path to say which platform is it, while I'm compiling for windows or private. A quick clarifying question. With the GCC and the platform, are you planning to allow then for like GCC compilation for windows as part of this, it should quote work or is that out of scope of this proposal? I don't want to preclude it, but I'm not going to do it. If you'd like to do it, more power to you. Contribution's great for me accepted, but the principle is, I'm not trying to make this proposal here specific to windows, okay? You saw that table of runtimes, the point is, here's what changes would be necessary, I think regardless, you just open it up to say, you need those other runtimes, whether you're doing it for, if I went back there, you're doing it for FreeBSD or Mac OSX or whatever else, you'd want the same things, okay? And so if Mac OSX wants to use GCC, awesome, right? Okay, all right, yeah, so avoiding platform-specific gift devs and doing the same for functions whose implementation varies by platform and C file, so here's an example. So Lawrence had asked this question in Slack on the public channel for ABPF for Windows, which was, have you decided what your API, API, boundary is going to be, is it or is this called the libbpfcapi, okay? And my answer was, the answer is libbpf. Now it turns out there's a lot of things that want to do things like, you know, BPF or SysCalls or whatever that we want to enable porting to Windows, and so this is temporary aside that's going to get to my, to, to motivate when I'm proposing here, so this is not core here, this is just what we did on Windows, is we shimmed the equivalent of Linux SysCalls over the top of libbpf APIs instead of the other way around, okay? So the more typesafe one is actually the more performant one, okay? So why did we do that, okay? Well, because SysCalls doesn't exist on Windows, SysCalls is a Linux specific API, and the Ioc tools on Windows work differently. For example, file descriptors do not exist in the kernel. They exist only in user land, okay? And Windows handles, which is pointer size, is what's used in the kernel for all objects, okay? It has the same kind of meaning as file descriptors do, but you can't fit it into an unsigned N32, okay? So what that means is that things like BPF atter that has, you know, unsigned N32s in a bunch of places, you can't actually fit it there, right? You can't take a BPF atter that's composed by an application that's expected to pass it to the kernel, right? You just, you can't do it. And so a lot of libbpf APIs contain a user space step, okay? And so at the point that we would call into the SysCall, you call into a different API, and so think of it as a taking whatever calls syscall and calling a window-specific thing at that step. And everything else in libbpf tends to wrap those other common functions. It's the lowest layer in libbpf, right? You have the low layer just wrappers around syscalls, and then everything else that kind of marshals things and does everything else. So the takeaway is that anything that directly calls syscall is in fact platform-specific. So what we want to do is take everything that actually calls the syscall call in libbpf and put that into a separate file and say on another platform, it could be syscall or it could be something else at that very lowest layer such that 90% of libbpf is unchanged and 10% of it is platform-specific. I don't know if that's the right number, right? So one question here. I presume we also have some libbpf APIs which take file descriptors, right? Oh, absolutely. That's the whole point. Yeah. So like on Windows, they all take file descriptors, right? It just so happens that at the very bottom layer, right, we would call the syscall, that's when they translate from file descriptors into handles because file descriptors are still common in user land on all platforms basically because it's just part of the BSD API, right? POSIX API uses file descriptors, right? So all these different platforms that support POSIX and BSD APIs, they all have file descriptors. Just on Windows, they don't exist in the kernel, okay? And so you do that mapping right at the point in time of you're making the octal call. And so the equivalent of syscall up on user land takes a file descriptor, converts it to the handle, puts that in the octal and sends that down to the kernel. Okay. All right. So this is the type of thing that whether it is a platform.c or a, you know, something more specific because you probably have one in multiple of those .c files for different types of functionality. Again, the goal is to reuse as much of libbpf as possible and only wrap the syscall itself. All right. So that's the platform-specific code part. Any other questions on that? Because now we get into the hardware-complicated stuff. You thought that was complicated? Yeah. Okay. All right. So now we get into eBPF feature-specific code. Okay? Can I get back to the file descriptors? You're saying that there is file descriptor which is translated while doing iOctal. Who's doing that? The libbpf. Well, yes. No, no, no. In the world, it was out libbpf and it was out bpf. Yeah, yeah. You're saying standard library has this concept of file descriptor. Correct. The C standard library, for example, has it. Right. So like if I compile my application on Windows using the standard library, right? Yep. I still have like the FD. So who does the translation? The C standard library. So it just maintains. The standard C lib. Okay. Yeah. And so the thing that exposes all the APIs that take file descriptors, so like the call to open or fopen, things like fopen takes the file descriptor, that does the mapping inside fopen, for example. Or fclose. All those have a mapping. Yeah. Think of a handle as being a pointer. And a file descriptor as being an index. Right. And in fact, that's actually implementation is just use the index and return the pointer to it. So it's like a one-step operation. Okay. If you do it in the right place, so. Okay. Any other questions? Okay. All right. Now we get onto ebpf-specific things. Okay. There's a bunch of ebpf-specific functionality that varies by platform. There's many ebpf features. You think about like btf, you think about the bpf file system and so on. All of those are things that vary by platform or at least vary over time. So for example, not all possible features come in at the same time for every platform. Okay. Think about what would be the unramp, let's say, you know, macOS, because we went through this with Windows, right? But you know, when the next one comes along, is it a take all or nothing, in which case you get nothing until you're at the end and you got to always play and catch up or whatever. Or you can pick up stuff piecemeal. Well, Linux picked up stuff piecemeal, right? We got Linux versions that didn't have btf, right? That stuff came in later. Okay. And so today libbpf and bpf-tool both have things that make calls into, you know, btf and assume that there exists a bpf file system and so on. And so it's like, well, what do we do with these? So here's my proposal that, because the same thing is true for libbpf. All right. So there's two things we can do. So the first question is, before I can give a proposal, I first have to say, okay, let's say libbpf-api, such as the one there, bpf-object-btf. Okay. What do you do on a platform that doesn't have btf? Okay. Should you have that api be absent from libbpf on that one? In which case, you know, you get a compiler error if you try to compile a program that uses it. Or would it be present and always return failure? In which case you don't get a compiler error, you get a runtime error. Okay. My personal bias is compile time errors are always better than runtime errors. Okay. Because it's predictable. Otherwise you get an indeterminate state that your application hasn't been tested with. But that also either makes assumption that you're compiling library on like specific version of the platform, or you cannot have some features added later, right? So. Yep. I did say these are getting progressively more difficult, right? Yeah. So that's why these are legitimate questions. Well, my point is A is not an option, I think. You think A is not an option. Like BTF example in libbpf, right? Yeah. Like we return null or error or whatever if the kernel doesn't support it. But like api still expresses this concept, right? But if you think about another runtime, if you are compiling a program and linking with libbpf, okay, then is it acceptable to say when compiling for Windows, you're linking with libbpf.lib, right? It's not, you know, .A or something as .lib, right? You're linking with the specific the Windows one, right? And would things be absent in that static lib that you're linking with, or that stub lib that you're linking with? There's no single right answer here, right? So I argue that for such cases that A, for some platforms in some cases, you can make the argument that says A for some cases and B for other cases and let it vary. Yeah. Over here. So is there some minimal subset that everything has to support as a core library for BPF, or is it platform-specific for everything? That is probably a matter of opinion. My personal opinion is, yes, there is. For example, I would say that you're probably not a real runtime if you don't say support the map API. You know, map update, map lookup, map create, that kind of thing. That's an example of something. Things like load, right? You're not really BPF platform if you can't load programs, right? So it probably is a minimal set. I've not tried to enumerate them, but when developing a platform that wasn't Linux, we went through that and said, well, here's a set of features that are incremental and we can't really be called to runtime until you've done, you know, load and unload programs, have hooks and helpers, have maps, and that's kind of the core set. And then you can keep adding stuff from there. But still, even among the things that I just rattled off, there's been different APIs over time, right? There's deprecated ones. There's newer ones. And so are they all part of the minimal subset or maybe not the deprecated ones? So it's a fairly complicated question. Right? Even if you look at APIs and LibBPF to load a program, how many of them are there? How many of them were there in older versions of the kernel? How many of them are deprecated now? And you have some of all those, right? Now, here I've been using LibBPF, but it turns out the exact same questions we can ask about BPF tool commands. You know, BPF tool, you know, BTF, right? If you got a command that's there and you say, you know, help, should you show that command? If it can't do anything on that platform or should you not show it? If you try to type it in, should it give an error or what? And so this is a A or B, right? So again, my personal preference is always for the don't show it if you know what's always going to fail here case. But I'm not adamant about that. I'm just saying once you make this choice, it affects a number of other things on later discussions. But all this is not new to Windows, right? It is not. Like we have the same thing on Linux. Yes. And we sort of have some solutions, whether it's best or not. But like why would we change this for Windows, for example? Well, on Linux, part of it is you compile it along with the kernel. And so it always matches. And you don't care about backwards compatibility, right? No, not at all. No, you do care. Yeah. Like we both compile DPPF separately, and now we compile BPF tool separately. That's good. Yeah. I mean, and also like, for example, in our case, we use the BPF tool from BPF Next, but it also has to run on old 4.9 stuff, right? There you go. So, and I think it's probably the same issue here. It is, yeah. Like once your BPF runtime on Windows gets this capability, it will be supported. Correct. But for all the version, it will be there, but it will just bail out with an error, maybe, or I think it's probably still better than being absent. Okay. So back to Andre, do you think that based on that history, you think that B is the better answer or that you think it's still debatable? I think it has to be B. You think it has to be B? Yeah. Okay. What's that? Presumably, my BPF tools. They won't work in either ARB, right? Well, okay. The functionality is missing that's in your script, right? No, because my script has air checking, right? My script knows what Linux will do, and it will say, oh, I'm going to get this air if BPF doesn't exist, and then I'll do something else, right? But I would have to add something else, presumably to handle this air where it just doesn't exist, right? Like it might be a different kind of failure. Okay. So let me add a test case. It's more seamless to me if they're using the same errors. Okay. So let me test this hypothesis, because it could be that there'd be, because I'll just make my question, is it A for some cases, or B for other cases, or is it always A or is it always B? It's always B, I think. Okay. So I'm saying like, right now I'm in practice, right? Let's say that we have a, and I'm making this up, because there's no such thing right now. It's hypothetical. Let's say we came up with, there was some BPF feature that did not make sense on Linux. Like I said, existence proof, I can't give you a run right now. But hypothetically, if there was, would we say that command goes in there, even though it will never work on Linux? If you make BPF tool cross-platform, yes. Yeah, okay. I'm just testing to see if they give you the same answer. As long as you give me the same answer, then we can go with B. Okay. No, no, because I have a possible answer to that. But again, whether they're a good idea or a bad idea, I don't know. But just in case you went with A, then I had to have something on my slide. So we can skip over that now when I get there. Okay. So. But it could also be the other way around, right? If you have features that are only specific on Windows, you would... Yeah. I mean... So basically, you're writing a system to a platform, right? Yeah. You can decide what's available for that platform hypothetically at compilation time, right? Yeah. You cannot have single BPF tool compiled both for Linux and for Windows, as a single executable. That's why you can make this decision. That's not true for different versions of the platform. Yeah, yeah. So the BPF file system is a fairly Linux-specific, related to maybe Unix-specific concept where everything is a pseudo-file system, right? Where other platforms, that's not the case, right? So if I made up something, let's say, let's pretend that there was something that somebody came up with of using the registry in Windows, which doesn't have an analogy exactly on Linux. You had a BPF thing in the registry, right? And commands and stuff to do that. We'd say, yeah, we put that into there. I can't think of a case where we want to do that yet, but who knows? Maybe somebody will come up with that three peers from now, so... All right. Was there another question, or shall I go on? Just thinking out loud. I mean... Oh, sorry. I was just going to say that we have platform-specificness that we're... When it comes up, we should probably standardize on a format for laying this out. Much like Go has been very, like, strong on their opinion on how you deal with cross-platform. Yep. Yep. And so, somebody that's dealt with that, I would love to collaborate. If we can come up with that, if people are familiar with other standards across there. I've done a bunch of open-source stuff in C, but if you know of precedents in other languages that would be more familiar to people that would more easily map into, say, a wrapper around LibBPF, and that kind of thing. So... All right. And just to finish a thought on top of yours, I mean, maybe, like, LibBPF could have, like, a platform-specific include. If this helper code or so is really useful in the LibBPF context for, let's say, Windows or only Linux, maybe there could be, like, a Linux-specific or Windows-specific include. Like, we... Yep. ...similar what we have with the assembler stuff, just like where you pull in that specific platform. Yep. Hold that thought. Yes. All right. So this will... A lot of this is if you went with A, and so this is the one that I said, you can probably skip a bunch of this. But we could say, should feature-specific functions... Okay. I mean, it's part of the bottom here. Feature-specific functions, so, like, the wrapper around says calls and things. Should they be in a separate file for other functions or surrounded in a feature-specific if-def? Right. So now we get to what you're talking about there. I could live with things like have BTF support. I don't adore it, so I would rather have a solution. It doesn't do that, but I could live with that. Okay. But then you say, okay, what about code? The numerator stuff, this is the one that only is relevant if you were to only selectively display things in help, right? And so this question here would only come up if you said A or A sometimes. Okay. So... But on the bottom here, putting feature-specific code in a separate file per feature or maybe per platform if it varies by platform. That's the bottom one. So I think we can skip the strawman here because this is for A. All good? Skipping it? Okay. All right. Now we get to things that are specific to a program type, slash hook. The set of program types and attached types can vary by platform and version. And there's a bunch of APIs that are very specific to a particular program type, like TC APIs and a text that's inside the BTF tool and that help that references them and so on. Okay. And so you can have some program types. You can imagine program types that show up only on other platforms and are not Linux. You can imagine some program types that might be Linux only. I don't know if there are any, but maybe there are. They're probably are. And then there's ones that show up on some set of platforms and so on. So this help text here, if you want it to be the best help text would actually vary by platform, right? Because this help text assumes that the only ones that are supported are XDP and TC. Okay. And so if you support a third one with, you know, some other platform, then this help text is wrong. Right? Because it's just hard-coded and the question is what do you do because the list of what's supported or at least the maximal set maybe, I don't know, it varies by platform. And so this was also a note about in Windows the list is not fixed at compile time because you can dynamically add and remove them after boot even. So some additions don't require the BTF tool, BTF tool code changes. Like if you were just adding an additional attach type for an existing program type. Then most of the commands that dealt with that program type would still just work. So proposal. Sorry, one question. This list is not fixed at compile time. Yep. How does it work? Like some add-on can like add like extra type of the program dynamically. Yep. Yep. So there's the core execution context and there's other drivers on Windows which would be kind of analogous to a LKM, right? You can be dynamically loaded or unloaded at runtime. As long as they're signed, right? Then they can be loaded or unloaded dynamically and they can register stuff with the core. The core has no knowledge of any program type, has no knowledge of the other ones they're dynamically registered and so it's all runtime introspection. So it's basically at the time that the driver starts up it registers stuff and then it has the complete list. So the BTF runtime itself is just this master index of things with function entry points into other drivers. Now there's a bunch of ones that happen to be in the same driver by coincidence just so you don't have to install multiple drivers, right? Some of the core ones. Okay. But it means that at the time that you're compiling BPF tool or even LibBPF, you can't enumerate them because they would still work at runtime and they would be kind of installed after LibBPF tool was even compiled or LibBPF or BPF tool was compiled. So what this says is that in order to deal with that case where it's dynamic then the list, it's better to not hard-coded in the C file but to put that into a function, okay? That function could have the hard-coded one. So like in LibBPF tool in one case and LibBPF in one case and BPF tool in another case, there are hard-coded lists in there, okay? So if you take those hard-coded lists and you put them in a platform-specific function that's specific to Linux and it returns the table in Linux and on Windows it makes a call and then returns the list then it would work on either of them. So that's the proposal here is to take anything that's a hard-coded list that's in this category here and move it down in the platform-specific file and not in the platform-agnostic code. Integer values are another interesting discussion that I think has happened either on slack of the mailing list I forget which but it was a while back. So I talked about this first part here so we just talked about that. Integer values and by integer values we're talking about a new program type, right? You need another integer value in the new, right? And so those are, of course, centrally coordinated but the actual numbers like the fact that what is five, okay? Well, five could actually vary by platform. Now, whether you want that to happen or not you could have an argument there but right now it is the case that the actual numeric value of, you know, XDP program type actually can vary by platform. They're not all the same number across all those different platforms back there. You can have common source code because now you're writing your source code using the symbol, right? Using BPF XDP or BPF product type XDP and that will work, right? Because the pound define or the enumeration is actually different on different platforms whether it's a number five, a number seven, a number two or whatever. And so typically the integer values you don't hard code them in source code, if you do they're probably not a cross-platform code if you're hard coding the number two, okay? As long as you're using the symbol form then it's fine, source compatible. And the current belief, at least my current belief unless somebody educates me otherwise is that there actually isn't a need to coordinate the actual integer values across all platforms. Let's say you're going to add some window-specific program type and all you want to know is another number, okay? Is there a need to coordinate that number that's only for windows with the Linux community or vice versa, okay? My claim is no, there's actually not a need to go to that extra level of coordination burden, okay? As long as the mapping from symbol to number is in the platform-specific file you only got to coordinate across that community, right? The Linux community maintains the Linux header and the Windows community maintains the Windows header and you might have to agree on the same symbol name, okay? You know, BPFXDP is BPFXDP, right? All caps, right? But whether it's the next number here or the next number there they tend to just be sequential, right? And so if Windows implements earlier one on Linux in a later milestone, it'll have a later number, for example. And the same would be true for, you know, one of the other ones, generic, eBPF and other runtimes on the first slide, so. Alright, so interestingly, LibBPF has a great API that converts a name to a program type but BPF tool hardcodes the reverse mapping. And so my argument here is that for parity we probably want to move that from BPF tool over into LibBPF and have a reverse mapping API. Pretty straightforward, yeah. Okay. Now we get into the difficult part or perhaps more contentious part, okay? And sorry for the small print here. So status quo, in case you're new here LibBPF and BPF tool have their own repositories but they're just mirrors of BPF Next. Nice quote for Alexi here. Non-Linux patches would be awesome to see. Thank you, Alexi. All patches have to go via BPF at Veejer and land via BPF Next trees. From other runtimes perspectives that's actually a huge hurdle. Okay. Your opinions may vary as to whether that's a hurdle you want or a hurdle you don't want, okay? But it is a huge hurdle right now, right? If you think about everyone that's familiar with GitHub and just the way that GitHub works in open source projects. You go back to that first slide, right, back here. They're basically all GitHub projects except for maybe the top one, right? They're all just regular projects in GitHub that are used to common flow. And so saying, okay, hey, all you other five runtimes or whatever, just need to submit stuff, the platform agnostic code, okay? Then that's actually somewhat of a hurdle, okay? Now, anything that's actually a hurdle is overcomemable, right? You can jump hurdles, right? But it does provide an incentive for other runtimes, especially newer ones. I mean newer than Linux or Windows, right, for that matter. It provides them an incentive to either fork or reimplement and I believe that neither of those are good for EVPF as a whole, right? Again, common implementation is better, okay? Now, of course, there's often the question that some people ask is do you really want lots of non-Linux files like MSVC compiler.ah or whatever in the Linux source tree? I would argue probably not, and you'd want to look at the platform agnostic part, okay? Especially since trying to build the non-Linux files may require dependencies on other SDKs or repositories as prerequisites to build and you probably don't want that in your CI CD or the specific to say Linux, for example. So, strongman proposal anyway. Feel free to beat it up. Put files for other runtimes such as Windows in separate repositories. It could even be the repository for that runtime. Linux platform files stay in Linux source tree as is. That's pretty obvious. For now, keep the platform agnostic files in Linux source tree, meaning no change there, but it still leaves an incentive there and that claim is going to cause problems over time and it certainly causes extra hurdles for me personally to jump over, but that's surmountable. It's just a big effort, right? And so we have the existing mirrors to add additional GitHub workflows. So this is for the next slide, which is like CI CD and stuff. How would you actually do that? How would you actually verify the platform agnostic code? Didn't cause a regression in Windows or Linux or whatever else comes along. But the biggest open question is what do we do with platform agnostic files? Longer term, right? This is my last slide here. So if you got comments on this, I'm just giving you a heads up. I got one more slide after this. Any comments? There's things that I would love, but then there's practical reality. What I would love is I would love for things like the LibbPF mirrors to become the authoritative version, because they're just regular GitHub stuff. Just like open SSL and things and then have that be incorporated into Linux. That would be my ideal world. I don't know if I can get there from here, but that would be my ideal world, right? That would be the least hurdle, the least incentive to fork or reimplement as we start seeing more and more BPF runtimes. I would love there to be more and more BPF runtimes. So in general, I think like having or keeping only the Linux platform specific files in the Linux source tree could work, but then you have the issue to keep up with what's being added there. Like if there's new features or new compiler specific things, then you need to always catch up and implement it in that other repository, right? If we keep it out, then I guess like the question is, how do we solve the CI issue? That's the next slide. That's the progressive, very difficult, right? Because now you've got, that's the real problem. How do you solve CI CD, right? So yeah, this might be all, motherhood and apple pie, putting the source tree outside the core agnostic files and something outside of Linux because they're not supposed to be Linux only, right? Then how do you solve CI CD? So here's my initial thoughts and Daniel shared the kind of we also have one CI CD test. My other hope is that Windows relies on the same one. This would allow for portability and mock testing available with Linux hooks on Linux and vice versa. Yep, 100%. So now the real questions are, okay, so if you're going to make a change in platform specific code, so I'm going to make a change in the Linux specific code or a change in the Windows specific code. When do you do CI CD test? When do you do see it build and test? So the proposal here that I had is that the platform specific code stays in the platform, in some platform specific repository, maybe some platforms have multiple repositories, with its own CI CD build and testing. So if you make a change to the Windows one, the Windows CI CD needs to run not the Linux one. So Linux does not build or test Windows. Windows does not build or test Linux. Pick another runtime, does not build or test Linux or Windows. The platform specific code is the easy part, right? The hard part is the platform by not stick code. So let's say I make a change to the core libpf API, and it's supposed to be, because we went with B, it's going to be present on all platforms whether the API works or not. So everybody's got to put it in the same one. So when do you do build and test? How does the CI CD work? So again, you have A or B, and notice I don't have a preference between A and B, because I think it's not an easy answer. Maybe you got a C. That's not a problem. So the first category, A would be, before our core change is merged, that it requires every runtime that's in the supported set to actually sign off that says yes, CI CD passed. So if I make a change to core libpf, Windows has to say it works, and Linux has to say it works. And if there's a third one in the future, from macOS, whatever else, they have to say it works. Or the core change doesn't get merged, at least without explicit knowledge of what fails. Maybe we said there was a bug in that other thing, we're going to merge it anyway, you fix your bug. That's valid. It's actually harder to coordinate if that is in the Linux repository to say we can't merge the BPF next stuff unless Windows has run a CI CD test. That's much harder. And of course, Alexi pointed out that libpf has a much higher rate of changes in the kernel, so this happens a lot. Another alternative might be B, which is you test the Linux runtime, because it's more special than the other ones because it's kind of the core part right now. And other runtimes are tested after a merge. This has a higher risk of regression for the other runtimes and creates an incentive to fork or re-implement. It's certainly more practical to do it in the short term, but it does have this longer-term debt that accrues and incentives that we don't like the incentives. So maybe there's a C, I don't know, but I'm stuck between A and B right now. This is my last slide. So this is where I wanted some discussion. Anybody got any magic answers here for me? So we'll start with our heads in depression at this point. So this kind of... I'm trying to understand or like trying to rethink about the decisions we made in the parties. Like, I feel libpf to me is a riper of Linux BPF APIs that we do. It's not a riper around like a BPF instruction site that we try to put it into an operating system, right? But this feels like we have some instructions the library is around the instruction site. But it's kind of not that way to me, but I don't know whether Andrew, Alex, Daniel, when you think about libpf it sounds different. It's a mix of all of those. All the same library right now. That's part of the problem. I mean, it's true that libpf has like very strong Linux roots, right? Even the FD itself, right? So this factory answer to all this is like no one to do platform agnostics. FD is not Linux specific. FD is part of the POSIX standard. What is Linux specific is that the kernel has FDs. Technically any version of POSIX would do that. I just think like if we really do this, the platform agnostic, like is it really going to be 90% or is it going to end up on like maybe 50 or even lower? Of pure APIs? If we took the libpf logic and see what we really cut out like platform agnostic ones, maybe that's not 90, 80%, but more like 50%. Not all APIs are equally used. The ones that everybody uses and there's this long tail of decreasingly less used ones, right? And so things like prog load and map update and map lookup and the core set and so if you were saying what's the percentage of APIs that are platform agnostic it's a much smaller number than if you said what percentage of use cases are covered by the platform agnostic case. What percentage of applications would it use whatever the percentage of APIs that's cross-platform? You get to a much higher percentage of applications and it's similar to the BPF tools numbers that we saw earlier today. I guess from you. Alexi are you going to say something? Well, it's probably controversial. This whole slide is controversial, I admit that. My comment would be that this whole windows somehow you're painting it that it's you're starting from the premise that it's good to share the code, right? So we kind of try to do this but to me the goal link like Rust they're even bigger delta than like different platforms like at least on windows and Linux we both like using C as common denominator but goal link is completely different language and they have their own like libraries now at least like couple libraries written and go then there are like at least three different libraries BPF like that are written on Rust. So I would say that preventing the fork and trying like really, really hard to prevent the fork it kind of goes against the grain what we already have like the Rust and goal link and all of these libraries they kind of fork the idea and they're being developed in parallel and what we've tried but if you look back and what BPF actually does like 95% of it is parsing L files and goal link libraries are also parsing L files and so on. So what we've tried to do with this skeleton and light skeleton approach is to construct something that the job of BPF of actually like loading the whole like health business will go away. So I think something similar can be done like for windows and goal link and Rust like I'm just trying to put like together the problem of having windows and Linux and C and into kind of the same bucket and then say well everything that the existing BPF does will be like moved like moved away like it's not what we're solving but then on windows you pretty much do the tiny wrappers or whatever window stuff where a slotting and parsing on Linux is not part of what you do. So that's kind of trying to say well let's not do any of this stuff with the BPF itself let's focus the BPF on windows to be it can be, it can live in its own like GitHub reapp and will be only windows stuff and that because it won't be doing any health parsing the only thing it will be like tiny wrappers wrappers on top of like map accessors. So I just make sure I understand what you're saying that you said there's that you believe the future direction is that all the health parsing coach would move out of the BPF into something else. No it stays part of the BPF but you don't need to even compile it on windows like windows doesn't need to Windows uses the same Elffire pressure right it still uses .ofiles that come out of clang and everything else and uses the same .ofiles they're compiled with different headers right but you still have an Elf file at the end of the day that BPF program is just an Elf file and so the same Elf loaded logic and stuff can be used on windows. Sure well yes so in that instance you want to run like BPF tool itself as a windows binary well currently the older skeleton and light skeleton generation dinebind BPF tool it's a combination of both like the BPF and BPF tool that doing this job of generating something that potentially usable by goal and crust or whatever but it's not universal answer of course even now like even in Go people parse BTF and parse Elf Elf parser has its own issues in goal and Rust again another story C++ skeleton C++ libraries and wrappers I'm not saying that this is the answer that I'm like advocating not even strongly it's just like I'm trying to look at the problem whereas like Linux versus Windows it's kind of the same as C versus Go. I agree with that yeah I'd love to be able to solve them both the same way yeah whatever that way is yeah now I agree with the way that Daniel put it that if it relies on the same LibBPF then it allows for greater portability because you have less chance of divergent and having you know prototypes that are different and so on so I the other possibly that one could go is to say if the only thing that was in common across platforms was the .h file okay now that just says now you have to reimplement everything and if you still code or whatever I just don't like that as much right because now you have copying of Elf parsers and different Elf parsers even if they're doing the same thing so I don't like that as much but it's also possible I mean as a temporary state it's not good right but as a temporary state we're currently the windows one is currently operating out of a fork of the main and so whenever main updates it pulls in the fork and it has if depths and stuff there and it's ugly and I'm trying to get rid of it and so this is all what's the right way to get rid of it and we'll go forward with whatever recommendations we can come up with here right right now it's a temporary state and there's already get up issues file to say get rid of it and what do we do so that's what I'm bringing up here and I just want to work together with anybody else that wants to work on you know LibBPF and BPF tool making it cross plat there's people from any other runtimes whether they're Linux or something else on that list or otherwise then what do you do right now on windows what we do for like we don't have LibBPF on windows so like what do you use for applications like building replications so the answer for what we do right now for LibBPF and BPF tool are different so you want the LibBPF answer well like you write like BPF based tool or something not like BPF tool itself like something temporarily okay and this is not a desirable state right just to be clear right I'm not trying to defend this this is current state not saying this is a good idea okay uses the same .h file but a different implementation well like but h file is like one small part of this like the elf parsing like loading the program like doing like core and all that stuff like who's doing that right now on windows there's a library that exposes the LibBPF API that does that some of the code comes from other places like LibBPF or the verifier whatever so yeah so like there was like a talk on the last one on the other hand it is the actual BPF tool it's just a fork of the BPF tool that if defs out the parses don't work on windows right that's not the case currently for LibBPF that's the case we want to get much closer to so we already have a little zoo of different loaders right and like at least like one thing that they kind of agree on is like roughly like the same concept of like the map is the variable programs like functions in sections all this stuff right like this this is general expectation of how like the BPF programming looks like right then each library probably has like tiny incompatibility so like extensions but this overall approach to like variables functions or stuff so I think Lawrence was like really really advocating for having like a common battery of tasks right like I think their idea was to just like make sure that everything is parsable I don't think that's enough like we can start that right maybe that's the way forward thing instead of trying to share the code between LibBPF on like Linux and on Windows maybe we should put our efforts into building something that can be like testing all the different concepts that we added into like the BPF side and maybe write some like minimal applications that have codified behavior right like that you loaded and like if you run it once the map has to have value one two three and stuff similar to what we do in self-test make them maybe not make them specifically cross platform but like use similar idea to have like some some set of more or less automatable integration acceptance let's call them acceptance test yeah I think the goal that we're trying to meet is the goal that I think most of us are trying to meet I won't say all okay so let me just agree here is to allow for easy cross platform application that if I can take an application that's written for one OS or for another one I don't have to write it twice I can write it once it will tend to work okay so how do you do that right if you have the same .h file that says okay here's the contract right that goes part way there but it doesn't enforce that you've actually had here to the semantics right so for that you either need to comment implementation underneath or you need a really good test suite that says we're going to make sure the same test suite exercises it in the same ways and as long as you adhere to all the stuff the test suite tests right then you're good because now your other implementation conforms to the semantics right I think that's where you're going in your in your comment there okay we could go either of those I tend to like sharing code not having re-implementation but since I'm trying to proxy not just for windows but for you know the next one that comes along right that is that I don't have a strong opinion so I could say that either of those would work for us so if we were to take this one step further wouldn't the next thing up be like its own language that wraps arbitrary concepts and then gets the kernel implements the correct translation of that at runtime sounds like a lot of work I love right I look forward to to your contribution I mean that's seems like that's where like if we were to keep going up this chain of like cross-platformness and like cross language I would love to not have to invent new languages or even new APIs and just reuse as much as possible and just sanitize it gets back to you I don't remember if it was somebody's point that that was way back here is that back in the set of goals here that the more developers there are I'm planning on a common platform the faster that one goes instead of fragmenting the developer stuff across two different limitations the same thing and now you only get half the work done because you got to implement twice right so I'm all for I would say I would say just send like your whatever you have is the if death hacks though you don't like them it's understandable that well probably a lot of people will not like them but since we haven't seen like any of that code it's kind of hard to like this specific what whether like A or B is better or that there is an option C my suggestion just send whatever you have whatever hacks you have to the release then let the community see what is even possible maybe the whole thing can be somehow massage into indeed having like single that compiles fine and then we'll deal somehow with CI problem and it will both compile on both Windows and Linux like there are plenty of like none of this is new right so LVM as a project exist on Windows and generate all of this and as far as LVM goes the CI there is done that it runs on all platforms all the time so but it has slightly different merge criteria because LVM can merge patches and anyone was a commit rise can merge the patches and they can revert anyone else patches so somebody was Windows like the Windows for example CI will run later like a day later and say like LVM commit broken and the Windows folks will just revert the patch but LVM on Windows because not the same so we cannot use the same approach here especially because like commits in the kernel tree not easily revertable unlike LVM but we can try to use if you go back to yeah next slide to use the platform agnostic code to use A to have a CI on both Windows and Linux running and then we'll see how much of the problem it would be for majority of the developers because let's say if Windows like keep breaking for every change people will start complaining they will say well like I have no means of testing this on Windows like how I will do it and like CI will become red and we sort of will gravitate towards option B we think today and somebody on Windows like it's all nothing new but we can try to step on the same but we can't do it and it doesn't hurt as much in here another point I wanted to make is it's not only like different platform but thinking about the VPF and VPF tool as cross platform meaning that we would have to be able to build potentially on whatever CI will be running like on Windows sorry let's say Linux but we would build for Windows as a target so it's a different cross platform and same like on XHG6 build for ARM or from XHG6 build for S390 this all thing will go like there are plenty of things to solve that I think is a much harder problem that I would like to not have to solve meaning say build for Windows on Linux or vice versa I would like to not have to solve that but like going back to the mock thing right if we have somehow like some mock API that developers could understand and could even extend or run on Linux but which would where the code would be like the platform agnostic code for the VPF would be using that underneath maybe that would help also for the CI and if something breaks people can look it up and see what is expected there I don't know maybe that's an option I mean GitHub which I mentioned because both Windows all the other run times and the mirror are all in GitHub right now. GitHub you can have CI CD that run the CI CD builds and tests on different platforms all part of the same CI CD we use that on Windows and there's a bunch of other projects that do multiple builds right now so that part I think is solvable because you're building Windows on Windows and because you're building Linux on Linux you're not trying to do the cross compilation so GitHub makes it really easy to say compile for Windows, compile for Linux but do so within the Linux VM and a Windows VM so that's why I would like to not have to solve the cross compilation problem because I think it's really, really hard not that it's impossible but I would like a shortcut so from your estimate how much do you think of the VPF when you looked at it is it generic enough for you is it like 60% or like I don't know I think like the attach points they are pretty much Linux specific and they could probably also stay Linux specific maybe and there are some attach points that are actually cross-plat so I would say XDP is an example of a cross-plat attach point it has a program type that's cross-plat XDP on multiple platforms I was just thinking if some of the library helpers that would load the VPF programs if they could be marked as platform generic maybe and other things not so I've done some of that right now I would say the code is not pretty because it was a shortest path not ready for upstreaming yet but if somebody wants to take a look it's really easy to take a look just going back to this slide it's really easy to take a look because it's just a fork in Github so it's really easy to click a button and say here's the comfortable diffs between the mirror and the fork because it's a fork of the mirror right and so Github can generate you can look at all the differences I can easily send a link to anybody who wants it it says okay well what would you want to do if all you want to do is see how difficult it is or see what the if-defs look like and see how ugly they are that part you can look at right now that's easy to do in 5 seconds if you wanted to say clean it up because I've given a proposal here that's based on what I don't like here's what I did that I don't like and I wanted to give a proposal here before I translate into something that I do like what do we actually like that would actually be the right thing so before doing that work that's why I wanted to give this presentation but saying okay so if you want to take what I did and try it because I can't just generate a Github per request and there's other processes and stuff that I'm less familiar with than everybody else in this room and so that's the hurdle here as I would love to work with somebody else who's an expert in doing that given that there's already a per request the ability to generate a per request in Github to translate that into stuff that goes in the BPF Next mailing list and BPF and so on I would love to work with somebody else who's an expert in that so I've gotten a couple pieces of helpful feedback questions here but I'm happy to keep chatting with people offline so so what would you like as a last question what would you see as an immediate next step that you could do like the compiler includes for example or yeah this is an increasing order yeah so like the if I were to take the changes right now which are sitting in a fork of the mirror right then it's really easy to factor out these parts the compiler specific parts and do it right because it's not like it's done this way right now but it's easy to convert to this and put that into a separate per request you know a separate branch or tag or whatever in Github and then say hey is there somebody that I can work with to translate this into the appropriate way to put it into BPF Next right if all you're going to do is merge it into the mirror if that one became authoritative that part's easy right it's the translation of the process from the Github per request process into the BPF Next process is the hurdle so it's not insurmountable I just need would prefer to work with somebody because this whole point is being cross platform to not just be one platform working on it to have people from multiple platforms that are collaborating so yeah the compiler specific stuff is the first easy one I would probably do them in this order right a compiler one then platform specific code which is a lot about you know dependencies on things like that and then feature specific code that's increasing order of difficulty right so I'd probably do them in that order Thanks Okay Alright Thanks