 All right, so I figured I will throw a provocative question to start the the conference Hope it will be interesting Thank you. First of all, thank you everyone for coming to this annual SFM MVPF conference It's always a joy to see new people coming new blood and always also very nice to see friends Who come and we see how this community keeps growing? So this is really awesome To be able to answer this Well somewhat absurd I guess question we first need to define what the anthem is and some is easy, right? So it's a uplifted uplifting song identified for the particular group, but What is VPF right? Realistically like there is only one correct answer here. So that's that's the key. So it says technology. It's nothing but technology but I've been reading some like negative comments about VPF on LWN sometimes on IRC channels and Like to me, this is weird like People cannot hate technology. So it means there is some misunderstanding. What BPF is so for people who hate it There must be something that's like missing and on the other side of the spectrum VPF is used as a buzzword I've heard Silicon Valley VC firms using BPF as a check mark the startups pitching the Slide deck to VCs when they if they have a BPF on the slides, they will have a higher chance of securing funds This is another thing absurd. So like there are two extremes here BPF is just a technology so we need to use wisdom of Humpty Dumpty to actually say to to define what BPF is In his words we can define it as nothing more nor less But as Alice would be saying well, there is always somebody else in there somebody else that will be thinking differently about it So my definition of BPF that BPF is universal assembly language As Humpty would say nothing else and nothing more. It's universal assembly language In other words BPF is a sequence commands that can be understood. So this is a scene from the movie Arrival and in this case Emmy Adams Explained in the aliens how to exit vi the analogy was BPF that's She used a language that aliens can't understand So and because because it's understandable. It's understandable. It's universal assembly language It's not only used by the user space to tell the kernel what to do in kernel Understands what it is then the kernel can say yes, I will accept this code I know what it's gonna do or it can say no like come back next time It's the same time it can be done the other way around the hardware can tell kernel what it intends to do So the the understandability of the assembly instruction is the key to understand here I also have seen people using BPF between two servers to user space applications it's again about and the verifiability and understandability of something and Some people will say like how does it compare it to like traditional instructions that like x86 in x86 CPU understands one instruction and does what it's supposed to do Whereas in BPF BPF understand the whole sequence of command the whole program understands That's the major difference between x86 spark risk whatever assembly instruction set and BPF assembly language There was a question about rust Partially, it's related to sandboxing. Some people say that BPF is a virtual machine BPF is a sandboxing mechanism It's not right. So BPF understands the program. It doesn't need to sandbox it Whereas any sandboxing like the AB assembly so the AB assembly keep brought up like all the time The AB assembly is a sandboxing technology sandbox doesn't know what will be running inside of it that's why like all sorts of God rails has to be done and In the instability comes the verifiability of this instruction set. So another I think I've heard people throw it BPF all the time is well I will be writing code in C because it will be faster in C. So it is it's It's not quite true like the experience and the benchmarking that XDP folks did clearly like demonstrated that like translating from C to BPF and then to assembly or directly from C to assembly you get exactly the same performance and back to Sandboxing because some boxing doesn't know what will run inside that has to use this foreign function interface To transition between native into sandbox code and that's where like all the performance overhead comes So that's the major difference when people saying like how is it different how like rust different versus BPF How some boxing is different versus BPF is being able to understand and having this native interface And finally we can answer the question that was on the slide. What should be done some of BPF? I didn't come up with it. Let me just read it. What AI said in the kernel There is a tool what it can do is pretty cool. It can trace it can filter it can load and unload It's time for you to get on board then I asked AI to Combine a B a P as a vegetable at the letter F and that's what it come up with I've tried couple times and every time it was this sad B. I guess because it's Got green from eating peas and the ice look like peas But that's that's what they I can do so I also asked yeah I way BPF is used and that was an answer. I didn't change a single letter Here it's like fully I generate it. I've read it a couple times. It's a bit repetitive, but pretty much looks correct. I I Also asked AI right to BPF program that parses a few for I didn't compile it But it looks correct in game here didn't change a single character all of the comments here written by AI it Seems to me it's pretty valid XP program So AI and search engines can answer all of these questions. What's the answer with BPF though? It's an absurd question. It can answer it where it can be where it's used write the program that does like foo but what they I cannot do is See through the BPF as its mission statement like to me it's really dear to my heart So the values that they post here are my values and the values I think of BPF community And that's what bring us together like in the first place It's innovation and enabling others to innovate it may sound I don't know artificial but This is effectively my checklist when I review the patches like one thing. It's a fix That's different, but when somebody proposes a patch set that adds a new feature. So my first Checkmark would be is it the new feature? Is it introducing something new to the BPF ecosystem and the kernel and if it's not It probably means that this particular feature could have been done as something else some previous feature could have been extended To do it. So it is it is it would be like a first-rate flag if people who presented something They didn't think through about extending something else. The second is whether it enable others to innovate It means that for this particular feature whether it's all in one particular need It could be like super innovative But if it's solving the needs of only one user or one set of users People who it means that the people who are doing it. They're not thinking about communities abroad They're not thinking about Enabling other companies are the people to innovate in the same space, right and the last part here is the challenge the status quo There are I think like also plenty of examples where people Repeat what they know they see like, okay, we've been using let's say net link all the time. So well, let's use net link again or there was Priorities in and in the way we attach things to to see like there is a prior there is let's just use the same stuff to use BPF It's not necessarily a red flag for me, but it's a sign that Status quo wasn't challenged. It means that there was not enough Thinking outside of the box that was done. So why I think it's relevant it helped us Maintainers and developers to get on the same page it reduces that hopefully reduces the friction once this values of innovation In enabling innovation to understood for example The real example for me here is the whole Helpers was the K-Fung discussion like helpers enabled others enabled us to like to grow the BPF ecosystem over the last like many years, but Year and a half ago K-Fung's for the drug use and the K-Fung compared to helpers just provide higher velocity of innovation Recently at cube con like a month ago somebody Edited K-Fung to call the BSM to call from BPF program through the K-Fung call a web assembly So they ported the whole web assembly into the kernel and through additional of K-Fung. They did it So that's an innovation that K-Fung's enabled understanding the values helps users and sponsors and by sponsors I mean I mean directors and VPs when they see what community is doing what other people are proposing to the kernel they see what's What problems they are trying to solve right when people when People see the problems they can like extrapolate where the community is going what the other companies why why it is matters for them Why it is matter matters for them? Another example of successful users. I would say is psyllium. So psyllium built a whole Ecosystem on top of BPF. It's a song like thriving open source projects that enables others to innovate on top of it another example would be all of the BCC tools and Now what is called now Libb Pf tools in the beginning? it was Brandon Greg who seeded this community was the first like 20 tools or so and People like keep saying once the BPF like goes into some pieces of the kernel There is no like everyone will be doing like proprietary stuff that didn't happen with psyllium that didn't happen with BCC Brandon did this first 20 tools now. I've checked yesterday. We have 130 BCC tools so after this initial people people didn't go out didn't start like doing their proprietary stuff That's for them is making the difference people do give back because of innovation and I Keep being asked. What's the biggest challenge for BPF? I asked the AI and I said that must be complexity and security and verification and so on It's I guess it's close enough to me. It's ease of use This is the graph we see all the time people who knew to be PR they start they have like a lot of excitement and then they quickly get burned by the verifier and I think we have to fix this slope we have to fix the score we knew about it for a while and We like the developers who've been around they own this productivity plateau, but the new people still go through the cycle So this is something we have to keep in mind. They have to consciously address moving forward And now the extremely interesting part that happened in the last year it actually I think was after was FMM Is a major shift in how we think about BPF in the past it was all the Standard stable hook stable set of helpers that call the well-defined maps and return cord is was also well defined So that was BPF and the kernel essentially. It was like standing next to the kernel Now I called BPF in the kernel There is nothing there is no boundary between so like BPF in the kernel pieces stock strategy dole interleaved The programs can now take the reference count on the kernel on object can stash it and all of it was a safety of the verifier we call this new tracing and Some of the subsystem already taking advantage of it the net filter the recent filter stuff all the struct ops the TCP congestion control scared XT is definitely in this category and Heat BPF like heat BPF added Started using BPF without changing the core at all. So it was completely done in the heat subsystem And of course, there is level no stable API so quiz to this room like which part of the puzzle is written in C Well, obviously the answer is like all of it at the end BPF is nothing but see it's Somewhat restricted because of the verifier smartness code code, but at the same time It's a safer see and it is extended see and my last slide is called to action so How do you see it's really the answer it's question like to you folks How do you see this vision applied to you don't need to answer now, but Can we do this together? Please help us spread the word well big big and community and That's and of course send patches Thank you Where is James? No, he doesn't care about trust. I guess you didn't want to hear the answer. Okay, let's move on Yeah, I mean feedback from you guys questions Just as an anecdote For the you know having the door open for feedback to make it better for k-pointers That was one area where feedback didn't fundamentally change the feature, but it really changed a lot like originally you had to Call a k-funk to get an act to get a ref count on a k-pointer And there were all sorts of kind of baked in assumptions like you know Is this rcu safe is it not and with sketty xt? We we said we really just want to be able to use rcu to protect a lot of objects And so a lot of changes went in to make To make Rcu kind of a core part of the verifier where you can say is this pointer actually rcu safe or not and that enabled sketty xt To be really useful for meta. I mean we're still trying to upstream it, but we were able to get a Scheduler that improves throughput by about one and a half percent For our main web workloads and I think and latency is also much better And I think that's because we didn't have to we didn't have to you know be calling k-funks in the hot bath and stuff So yeah, just you know, I just wanted to add an example of of how that's that's being applied Yeah, sketty xt was rewritten. I would say three times and like keep being changed but not to that huge degree and a lot of the innovation that happened in the pf over the last year was driven by sketty xt So it's huge. Thanks to june for pushing the boundary and actually like believing in the bpf values like innovating in a scheduler space and Challenging what possible he was telling us that well, why can I just like grab the kernel pointer and put it here? like well because we cannot guarantee the safety and It took us a while as they were just said to make the k-pointer useful at one point. We thought That kptr design is just great everyone will love it like to jay started to use in it and like well It's kind of not still not neat. How about you just like let it access directly without Atomic atomic exchanges because it's under rcu. So we went back to the drawing board and added all of this stuff It was explicit now RCU read lock-on-lock inside the inside programs Hey My question is regarding stability API like Remember it was discussed a few years ago when Brandon was giving a keynote You know safe and he was asked What if a kernel upgrade will break one of the very very useful the bcc programs? What will happen? Will the kernel developer need to revert the change? So I think there may have been development in this area, but can you say something about it? The last maintainer summit Linus was pretty clear about it There is it's not a concern to him in the first place It's not a concern to me and it's not a concern to Brandon Greg and if you follow it what bc is doing Kernel changes broke bcc scripts multiple times. Yeah, I experienced that and Well, did any of the kernel changes got reverted over the last 10 years because of bc script bc scripts existed for 10 years No, I'm not personally concerned about it, but there was there has been a precedent with I Think perth top So that that that has caused the revert in the past, you know, so The president I don't know. I'm just asking if there is anything new about it or is that the status quo? I Am not gonna repeat what Linus keep insane. It's like there are all the billionaire articles So as Linus is Linus said you remember what he said, right? Since it was me that he was yelling at I remember very vividly exactly what he said So basically he just says it hasn't yet hurt me So I don't think it's a problem, but hey, I may be wrong in five years He said that too. Yes in which he would refer something. So it's right now. The concern is basically Don't worry about it until it happens That's basically what basically Linus said Thanks very much You mentioned the new way to kind of add things as the K funks. There's no stable API, but there are going to be Maintainers that want to kind of expose something that is fairly stable, right? If you think about XTP one of the Kind of the original og use cases I would say at least for me is like, okay You have this program and you can this very reliable infrastructure you can build on you can rely on it For our critical infrastructure, like what's the idea going forward? Like how how would a maintainer do that in a world where k-funks are a thing? Is it just don't change the function signature and you're fine. Where is there more? There are more neurons to that see there is no Black and white right just like in the kernel. There is export symbol and there is Kmalik as export symbol that existed forever yet. Kmalik was changing and So it's always great, right? So some of the stuff will be as stable as it can get more stable than you API like Kmalik export symbol Did it ever change in the last 20 years now? And there are plenty of other export symbols that were added and removed and then change it many times Yeah, I mean, I think one of the one of the benefits in drawbacks I guess is the fact that we have something like core with BPF programs is You don't have to compile a BPF program to a specific kernel And that means that it's a feature where you can use this BPF program across lots of kernel versions But it also mean unlike modules, which obviously you have to compile for a specific kernel and it won't even load If you do a different kernel So there's some sort of inherent like, you know, it's a benefit But then people I think tend to look at that and they're like, oh, so if a kernel version breaks a BPF program It means it's a regression. I mean, who knows what Lienus is gonna say is as Stephen said, which is true But I would hope that if you look if you compare modules to BPF programs None of the things that we're talking about are a new API headers These are all essentially export symbol GPL Abstractions and so hopefully you wouldn't penalize the BPF program because it has more It makes it easier for for a kernel developers to use, right? I mean if you consider the guarantees the same but in practicality, you're probably less likely to have to worry about it I mean, yeah, that seems like a pure positive And if you if it becomes if it if it becomes, you know, like an interpretation of well You have to be stable all the time I would call that a misinterpretation and kind of an unfortunate outcome that really wasn't intended So another example would be the heat BPF Again maintainer summit So Benjamin really wanted to say to its future users of heat BPF framework that the API that K-Funk's expose it is stable and Linus like Don't like why even saying this? Why why do you want to make it stable when like why do you want this burden? I'm paraphrasing but roughly it's like don't make stuff don't promise stability when you don't know whether it's actually will be useful or not So that's what we're doing like K-Funk's all K-Funk's around KPTR. So like this Without they will be like great better than bread and butter. Well a year later. It turned out not that much Another question. You showed that graph or like that any ideas for how to Even the slope to bulldoze it. I don't know Well One thing is for sure help Don't Think of BPF as only like kernel and your favorite user space project community But it there are many pieces LVM and GCC and currently we do need your help with LVM in LVM way for years been struggling to Explain that BPF is a different assembly language They all think that well if compiler is Can do a valid transformation. It's okay to do this transformation for any assembly So this understand bill under this ability whatever verifiability of instruction set was missing For majority of the LVM folks they're saying well the compiler is allowed to do this So now when we still struggling in the past we did all sorts of hacks inside the LVM We undid some of them we do the Pattern matching on the back end and do the transformation that wasn't enough then we introduce fake in line assembler sorts of We see we pattern match now on a C code Then drop it in the fake in line assembler send it through the LVM pipeline so that LVM will not optimize it And this is like thousand upon thousand lines of code We would do this hacks only because all of you and people saying well, it's a valid optimization So we need your help to explain why it is necessary for us Do we have a Do we want to talk about that this week? Do we have a plan? I mean to put like just point out how bad this is right at one point on the tetra on our security tool I I forked LLVM and just cut out optimizations right because they were breaking the code Eventually that was like not maintainable because once it was more than just me working on it Everyone was like what like you can't fork clang right like it's You have to maintain clang. Yeah, so I believe Cloudflare did this at some point. They froze the LVM because I mean I forked it and was back porting everything off the ebbf branch into my fork for a year or two right eventually they lost a bunch of stuff because I'm The argument from LVM folks that this oh, it's a slippery slope. We don't know like how many optimization do you want to disable? They're actually very few like over the years We identified all of them that are a handful like really less than five and only because we don't disable transformation that like Disable those that make the code unreadable Where LVM changes something that cannot be inferred from the like the transformation the code that simply before and after You before like you can understand what it's supposed to do after you missing transformation because the information is lost So those the only transformation that want to disable and there are like For young honk and like though exactly but I think when you say people are burned by the verifier They're really being burned by clang half majority Because usually you get this a lot of a lot of errors on the verifier are coherent But then when you get these ones who are clang move stuffs around it just says like complexity is lost right? This is like and there's no way really that I see for the verifier to tell the user like at that point It just takes like intuition and understanding what clang's doing to really Fix that bug right like I don't know how to teach somebody like when the verifier tells you this complexity air Oh, you just need to move this line of code up to the rose and then change that from a static to a whatever right like the conversion to make those work is is sort of not There's no formula that I can give you versus like say like absolutely no bound like it's really like compiler Think people like the I've been working on compilers for 10 plus years before I was working on the kernel So I relate because like people who work on compilers They're all daily job is to find Katie new transformation that give me like extra 0.1 percent on some benchmark and that's That's great But the missing that all of this transformation compiler organization They heard and the last example was like we keep catching that new diffs in all the m that break stuff The last one was like people decided to enable Some transformation that in the past was only done at the short three level and they said well Why not I want to do this or two so and what this transformation did they've in some cases They added another argument to function So the function would take like five arguments with past five arguments and compiler said I'm smart I can just add another argument here like it's a six argument and the immediate BPM back ends it I cannot even compile it There is no way I can pass like six arguments and we are going with this LLVM people and they're like Why just pass it on the stack? We don't have a definition of the six argument today, what do you mean? We're just breaking everything like nope We're not gonna revert it. It's like it's your bug. Oh Yeah, this is not exactly the same as what you guys are talking about but somebody is also implementing Assertions so for the whole burn burn by the verifier slope A lot of that is also you just have to go through your programs and add like lots of boilerplate and bounce checking and stuff like that So hopefully that will allow programs to be simplified a lot at least in terms of what the human being sees So that's another thing. I mean, I think it's probably still in its early stages, but That should hopefully help a little bit Yeah, so you can so for example, let's say that like You call some k-funk that could return like a nullable pointer So you would have to check if it's null and then go to some branch where you like drop all of your References and then exit the program But if you had an assertion you can just assert that it's not null and the verifier and the jet will Will handle unwinding the program for you and like you don't have to have all that extra code Yeah, it does clean up and everything like that So if you've reasoned about your program correctly, it won't be hip But if you haven't then it'll the program will exit and everything will be taken care for you and other all the optimization was that Just for sake of example It was in the past where it was we had the code where function would return a Boolean And then after it turns we had some like check casted to integer like check and then the John zero one and Combined said well since it's a Boolean can only have like two values, but There is no Boolean and assembly So it was a 32-bit integer and the verifier said while you return so to the bit integer But this function can only like accept like zero and one So check return code. No, it's outside So those books are so hard to find too. So it'd be great to Improve that Participate participate in this conversation because it's really just like few people like me and honk and Andre They're target was all the M people and there's a huge community here Like they need to hear that it's not only like two people like when For LLVM folks when we were upstreaming the whole BPF back end the first question was with it's a toy project Is it the research something like what's this like who's gonna maintain it some of them like still think it's like yeah What's this stuff? And this discussion happened on the LLVM mailing list on the chain LLVM on all the MDs What I wanted to ask like from the feedback insight matter. Do you also see any other Burning topics. I mean obviously the verifier is the highest one that you make it that you have to make it more easy to use but you also see like for the broader community other aspects that we need to work on to Lower the barrier of adoption Yeah, and I don't want to I don't want to capitalize the mic so if other people have thoughts they can add but I Think it would be great if we were more consistent with features that we implement like for a lot of the times We'll add something and it's only supported by like Four different map types even though it really is generalizable to a lot more So it would be I think it would be helpful if either it was it was very easy to discover What features are are usable where or we? We were just like we had more of a culture where like okay Well, if you're gonna implement this for array maps and hash maps Implement it for local storage if it applies there Just to take the guesswork out of out of users, you know out of the user's plate Yeah, and another part that even in the hyperscalers the difference in the kernels and have to The services have to support all the kernels It's same as with any start up like most of the start of the sufferer from the issue that yeah The latest BPF and the latest verifier understand all of it, but my customers are still using the rail whatever Version X so it's some old kernel with some number of backports and who knows what this backwards are so Yeah skipping back to the LLVM discussions How do we make sure that people that follow BPF mailing list actually are aware of those LLVM discussions? Question do you have a suggestion? Not really like so LLVM uses a fabricator system like if we can somehow out to subscribe BPF mailing list maybe that would be basically like we need to make a decision whether we spam BPF mailing list with a LLVM discussions Related to BPF and then we can try to figure out like how to do this But basically I can see the point right like who knew about those discussions except for those like three people involved, right? Yeah, so in the past the fabricator had the email interface You can type your emails and it would automatically push it back as a comments on a diff and it was fragile So four years ago. They said like nope. They stopped this bot So now to comment on a diff you have to login into fabricator and that's the only way I guess maybe we should start it doing it like manually every time we start a new LLVM diff We just send like extra email with a link to the diff saying like well We started this feature or fix or whatever and then like whoever wants to follow go register and follow Yeah, yep, absolutely the latest one that Prog also could be controversial though shouldn't be is we add in BTF support to LLVM obj dump Should be no brainer just because well, we just want to learn them to obj dump like not only in mid-dwarf But the mid-btf But it is for some reason controversial So I think I've took way too much time already my half an hour expired long ago Like we started super early and actually your session would start now Initially so we now actually have half an hour But maybe we can continue with