 Okay, I just wanted to, I just put this down because I wanted to talk a bit about the problem we have. Okay, is that better? Good enough? There we go. All right, so I didn't make any slides and I just put this down because I think it's a discussion. So, we actually maintain our own Clang fork, okay? Not ideal. So, but the question is like, why do we do this, right? And I'm curious, like, I think this is a common problem and I have a feeling it is because it's both prosilium and for the stuff I work on. So, basically, we wrote a lot of our programs kind of early on in like Clang 12 or something or 13, I don't know, some version of Clang, right? And then as we tried to upgrade Clang to the latest versions, we would rebuild our programs and they wouldn't work anymore. Like the Clang would do things and the Verifier would explode. I think the most recent one that I had debugged, I think before I came here, as just a sort of a case point, right, is that on older versions of Clang, it was riding the socket cookie field of the socket as a 64, an eight-byte load. So, just wrote it over. And then I upgraded, I just like, let's see what happens when I upgrade. And I upgraded to sort of the, I don't know, nightly or latest Clang because I wanted to see what was going on and I was probably doing some of our back ports for Clang. And it basically decided to write that as one-byte loads. So eight one-byte loads that then got code moved around. So like split some other code in the middle and then did an eight, like one-byte loads eight times. And then the Verifier saw that as us breaking apart a pointer and decided that that was invalid because we had split up a pointer inside one of the functions. And it didn't, you know, it wants to copy pointers in big chunks. Anyways, that's like one very specific example. So what we actually do in practice is I have, I think some version of Clang that we had at one point and I basically backport all of the BPF patches on the BPF backend on top of that. And then I actually revert a few things from the optimization passes, some improvements from that side because they cause us trouble. So I guess two questions is, does anybody run the latest Clang other than the CI tests? And then do we want to have like a common Clang backport? I'm just throwing this out there for discussion. I will just observe from the Windows perspective we ran into the same problems with the different Verifier where Clang 10 and 11 are fine, but as soon as we move to 12 and up that it doesn't verify anymore. And so for the latest stuff we're stuck on Clang 11 which means you get maybe less optimizations but some of the optimizations are what can't be verified because it creates correlated branches that the prevailed Verifier doesn't do well with correlated branches. But same thing and there's like bug fixes, right? So I think Clang 12 added some fixes into the source code line number information BTF and so we can't pick up those fixes because we're kind of stuck. Or you do what I do and you maintain your own branch. We don't do that right now, right? But it does mean we're stuck dealing with all the same problems as you talked about. I need BTF and I need BTF for you tags and I just backport them and have our branch. It's not terrible actually. It works pretty well but. I regularly run Clang and with the BTF program latest one and yeah sometimes we do have bugs and most of these bugs are from the Clang optimizer. Yeah. And they make a smart optimization and actually Verifier cannot handle. And my typical reaction is to mitigate and firstly they try to Clang part and to see whether we can, in the LLVM we have several passes and basically intermixed with the Clang regular optimization. So we will try to see whether we can change the IR pattern and try to prevent these organizations. But sometimes it will prove difficult. And so you will see sometimes I also have patches in the current repository and I try to work around the issues. Yeah so consider as a regression and those things we kind of like use the upstream Clang. So these things is inevitable. We are not controlling that and they are not really wrong the BPF self-test every day and we are not covered with the self-test with the BPF with the Clang and you need to test that as well. So the problem I have right is is the self-test are small enough that we don't trigger a lot of the issues, right? And I have a really hard time. Unless I spend a lot of time to like look at the pipeline of LLVM, look at the RRC where the optimization pass went wrong, like walk that all the way back to the code. And then even if I do that a lot of the times it's like try to build a small sample that duplicates it sometimes just really, really hard, right? Like cause you're buried in a big code base trying to extract just the minimal set, right? It's quite tricky. And if it's not open source then you're even more in a world of trouble. Don't extract it. Like put the big, the whole big thing in there. It's thousands of lines of code, right? So that's actually fun. And so basically if you look at the self-test we have this pile of perf 600 and then recently Clang they changed the loop, complete the loop on rolling. They actually break the instruction set. We have an instruction set like a 16 bit and like assign the inter range for the offset and it exceeds that. So we have to mitigate the issues. But the point is the self-test do produce complicated code. And I think if you have such cases it would be good just to contribute even just for the loading part. Cloudflare has a big test. It's a couple, at least a thousand lines. So like answering your question like whether it makes sense to have an official Clang back port? No, like it's a dead end. It's a fork, right? You know how all the forks end up badly. So like the only way is to like keep up with Clang with Clang development nightly and like do more tests, add more tests and we'll keep fixing Clang. And if you didn't back port all of the Clang back end LVM back end changes that you know Hong did around summer of last year. There was a ton there like changes made to kind of pessimize some of the too smart optimizations that LVM was doing. So that was done. So after that things actually improved quite a bit because we were seeing this like yeah LVM would like add the constant there because it knows that the register is like with the constant delta. So it's like crazy stuff. We rely on all this like we back ported I'm back ported all of the BPF back end, right? Like there's nothing, these are not specific to BPF back end usually. Usually these are things deep in the core of the optimization layer. Like there was a, they changed some of the registry allocation stuff, right? At one point about two years ago, right? And this caused some issues. But I think in the latest version if the optimizations were perhaps more parameterizable to like specify instead of just saying you know minus O2 or if there was more granularity then there'd probably be a lot less fewer problems for either John or me. If I could ask for one thing it would be try this approach many times. So, and every time we got knocked by LVM folks so that's why we're doing this fancy wrapping into the function, into global variable helpers to PCMIs optimizations because upstream LVM refused to deselect optimizations based on the back end. So the problem that I'm running into may be different from everybody else's problem given that I got to solve it for two different platforms. I mean, I needed to work for both Linux and Windows which have different verifiers, right? And so if one verifier can verify things that one can't and optimizer is optimized for one verifier then that's when you start to run into problems or if you fall back to lowest common values. That's why I mean by configurable or whatever. I suspect the issues like any verifier will have issues because some of the optimization is just like not possible without doing full like compiler analysis. Another possibility would be for Clang if there was like CI CD for Clang that actually ran say two verifiers and any new optimizations that affected BPF bytecode. That would also be fine. So I would actually turn the question around and would ask like should we test our self-test with older Clang's? So like we know that we at least have like workarounds for some issues so like I don't know like should be because like for Libia for example we run like old kernels like latest self-test on old kernels we obviously just like have some subset of tests that run there. Should we do that with like major Clang releases? I definitely don't sign up to maintain this but I'm just proposing. Well, some of the tests we know will not work so like we can potentially but in the readme in self-test we list all of the building diffs that are required to even run the self-test. Sometimes they compile the self-test sometimes Clang will just crash to all the Clang's without bug fixes will just crash. But in the past we had one LVM build bot that we contributed back then for every backend there was a requirement that there should be public build bot and I think it died by now so we can like do it again and there we can let's say it will be running for every LVM commit. So instead of what we do effectively we do a night list of Clang we can put a spawn LWS instances needs to be really, really beefy and run all of the LVM stuff for every commit and then like upstream we'll see that all of your breaking VPF build bot. That'll be great. It's work. But you mentioned, right? I mean like other backends probably also have their own test suite and when something breaks they will revert to change upstream, right? So that too. Yeah, like windows build there are like I think four windows build bots for different, I don't know why but like they do different stuff. This would be ideal. So would this be acceptable from a LVM community like where you would use that as an argument to revert that optimization change or is that not a way forward? It depends like yeah, that would be a great area. At least they will notice and we will notice like first of all they will notice because often like when Jan Honk notices that the latest Clang broke something like he will bisect it to a commit that happened months ago because it's hard. Yeah, it depends. Mostly generalization is really hard to revert. Really hard. So you have to provide a workaround of VPF specific paths to change the behaviors. Yeah, it's going to be good. Well, yeah. So but like I mean like what is the way forward if the emerging optimization that is really hard to verify it and we are stuck kind of for we change to verify or like? Every time that happened, we managed of your own managed to find a way to trick LVM not to do it. So now we have like passes that run in the Clang phase then passes run at the MIR phase, et cetera. So basically realistically we can improve detection. I don't think we can prevent this, right? That's the conclusion. Without like a build bot, right? Like I mean if we went to put a build bot like VPF backend has to have enough political way to actually say like, no, this is not going in. And I don't think we have it. And like realistically, I'm not sure we will. Yeah, but it's still a huge struggle. Like I think that would be super helpful, right? Because the struggle now is like we are, you know, we notice it quite late. But we notice it with self-test. It's just like that self-test covers some patterns of the code and like you guys use different patterns. So like if you contribute your patterns like we will catch it sooner. Because we run it all the time. That's like, yes. There's no good argument against what you're saying by the way. I mean, other than I'm gonna tell you it's a bunch of work to find those patterns, right? But like what pipe, we have like pipe path. Like that's actually taken from a real application. Like strip down some like irrelevant parts and like it's pretty much full thing. So I don't know if you can do that, but try. Like in especially in the ceiling tests you don't need to run even run the traffic. You just take your 2000 lines program and just load them. That's it. That's good enough. That's what we do for pipe effects even. Like trying to like, no, we don't even try to run. Yeah, we just load. So the biggest amount of work, right? Is to get our stuff working on latest. There's nothing you can do to help with that, right? Like to do that. And then at that point then we can start doing regression testing, but like right now we're at a point where we can't even do regression testing because it just doesn't work. That's a resource problem. To-do list is getting longer, the more days I've been here.