 Let's continue, so the next topic we wanted to talk a little bit about was this... The way that the BTF is generated for the kernel. This is just because I would like to know your opinion about this. And actually, we have been talking with Jung Hong now and there are some things that we were not aware of and that's the sort of things that it's actually useful to come here to realize, right? Okay, so basically nowadays as we understand it, when you build the kernel you build the DORF, all those gigabytes and gigabytes and gigabytes of DORF. And then there is the bayhole component in the kernel that generates the BTF from this DORF. Now, that's as far as we know how it works, right? Now, this has the consequence of introducing a coupling between BTF and DORF in the sense that everything that you want to have in BTF at the end of the chain, it should be expressible in DORF because you have to convey that information through DORF from the C compiler, right? All the way there, all the way there to BTF. This has proven already problematic in some cases because DORF is very difficult to extend without breaking it, as we have seen with the type tags, for example. I think that is a deficiency of the DORF design and I'm going to propose, you know, this no-operation type qualifier for the future, but if you want to extend DORF, you have to go through the standardization committee. It would be now DORF 6 that who knows when it's going to be published and adopted by software. I mean, it's a long process and also DORF is very big and for the kernel is huge. It's very big, it's a lot and it takes time to generate it, you know, so we were wondering what if we, the duplicate and merge BTF in the linker at link time, right? Well, DORF doesn't get the duplicate as one of the problems it has, but you know this CTF format which is Causing Format to BTF which is used in different other contexts. In the new linker we already have support for merging it and duplicating it at link time and I think those formats are similar enough that probably it would not take a lot of work to get existing code to the duplicate and merge BTF as well, CTF. What do you think? I mean, I think this is worth it to have it in the linker anyway. It could be useful, but for this particular use case, I don't know because John Hong was mentioning that in this chain here, right, from the DORF, Payhole is getting information from other sources, right, not just the DORF. Yes, actually PAHO will be more than just transform DORF to BTF. It will check some Linux sections and try to, for example, per CPU variable and it will add global variables from the VM Linux sections and to the BTF as well and it will also try to check some functions if the argument is mismatched with the prototype and some currently will remove it. So a few steps actually in PAHO involved with VM Linux itself, not just DORF. Okay, so that means that something like this, oops, will not work? It's true, PAHO works. Yeah, but instead of getting DORF from the kernel build, if it will get BTF, it will get all the extra information from other sources and modify it to a final BTF. Do you think this will work? Yes, this should work. You'll get the BTF deduplicated through PAHO process. Possibly it will work. So it will work if we extend BTF as well with information about the choices, what compiler optimized and how. Because PAHO checks whether arguments match, whether this function, et cetera, whether like multiple static functions and they're all the same then it still limits it. If there are different static functions with different arguments then it skips it, et cetera. So you think this is doable, taking all those things in consideration? Yes. So I think this will be advantageous because we will get rid of DORF in the middle. And yeah, okay, good. So I think we will add support to the linker. So I think you can sell it to the overall kernel community and quantify this in compilation time speed up. I think Indu you were mentioning yesterday. So CTF, you know, it's deduplicated in the GNU linker and it has this capability that it can represent, it has a parent child. You can represent CTF, there is a concept called archive in which what you'll do is you'll have a parent child relationships. So if you have multiple compilation units and you identify unambiguous CTF, you promote it to the parent. So there is a way to, when you deduplicate there is a way to reduce the information. If BTF doesn't have this sort of representation then I think, I mean I'm not clear how deduplication will work very seamlessly. But payhold it deduplicates BTF I guess, right? It does, yeah, otherwise. And how does it handle the relate, you know, if you have different, because in CTF we have like a container and children, so like a parent dictionary that we call in children dictionaries for the different modules for example, right? Where the same type name could refer to different types. So currently when you deduplicate BTF and you realize that there is a common BTF type across compilation units, then you change the type IDs, right, of the types which are referring to that, which is fine. But then what about the unambiguous types? Sorry, the ambiguous types. By ambiguous types what do you mean exactly? So like for forward references they always ambiguous because they are just like the name of the type. But for structs it was the same name but different layout, then we keep them as is. For forward references I think we added like very relatively recently we added like extra pass, like can we see that there is only one forward reference? Actually if there is only like one potential candidate for the forward reference we resolve it I think as well. But generally speaking if you have, like we used to have this situation with like I think ring buffer, like we had two different ring buffers across like different C files, they were like completely different so we would keep both. Okay so I think we will push through this direction then and I'm glad that there are, you don't see like big problems with this. Okay, now the next question I know what the answer would be but I will just, yeah, but it is an interesting aspect and I think this is challenging, it's very challenging and I think that this is one of the reasons why this, actually this instruction set and this architecture is, is a special, in a way, in a good way maybe. The thing is we have abstract BPF in the kernel and this effectively, this is what mandates and describes the format and coded format of BPF instructions. No bats, no house, right? So this is part of the UAPI and this cannot be ever changed, right, or broken, the backwards compatibility as far as I can understand. I'm not a kernel hacker but this is what everyone tells me okay, so considering that and also the fact that in the opcodes space in this format it's very small, it's just 8 bits, right? What has been happening is that we are basically abusing sort of, you know, reusing or using like multi byte fields in the existing extract or infra byte fields like the destination and source registers as opcodes and this has a complication, I mean, it's a little bit inconvenient because if you have more than one byte in the field then the NDNS plays a role there but also if it's smaller than one byte, like the four bit fields, NDNS also plays a role too because the bit fields are actually switched, right, depending on big Indian or little Indian, by GCC and clang which is basically an implementation dependent behavior but whatever, it's common enough, right? So it's also sort of wasteful because if you use like for example this signed division size module here the opcode is 3 but then it's using the offset 16 field as an opcode too, right? Then the offset 16 is a 16 bits field in the overall instruction and if you use that as an opcode that means that you cannot use, you cannot use split that field, you know, in other sub fields too for other purposes, right? So I mean wasteful in that sense and complicated, well, complicated, it complicates the instruction set because depending on the NDNS well, you know, you have to adapt to that. You have NDNS dependent instruction opcodes which is very unusual, you know, when it comes to instruction sets. Now, you are all very smart people, you know the kernel upside down, right? So do you think that it could be a way, some way, somehow, right, to actually evolve this abstract in a way that it would be sort of compatible enough so we could have like proper instruction classes? So for example, let's consider the following. This is science fiction what I'm going to show you, right? Because of this UAPI thing, but still, right? So this will be an ALU instruction that it has the normal opcodes for general instructions but then it has an additional level of opcodes, you know, and then some unused bits following. This is the typical classical way in instruction sets to evolve them. You don't have one single format for instructions because you don't want to dedicate that many bits for opcodes for instruction, for instruction formats that do not need that many opcodes. So that's how it is done, right, hierarchically. So my question is, do you think there could be some way to achieve that and without breaking the kernel? Somehow. I don't know, I guess in the kernel UAPI there are other abstracts with similar situations. You can see the way we did already, right? You just have a union code tool unused with this of, right? Yeah, maybe to use a union there. Yeah, it's the same thing, right? Yeah, that could be, it will be the same thing, yeah. You don't have to have a separated the struct type, but would that work? Sorry, have all the first fields, have all the codes already been used up, but then like 0, 255 are all used up in PPF? Well, yeah, I don't know. Because if you could have like 255 be a special one then you could have it to indicate that it's a special instruction that would have a different struct shape. So it would be backwards compatible that way. The other thing is that, yeah, we have extensible structs but that's a different discussion. But yeah, so does anyone know if all the op codes have been used up or not? Okay, well, I can maybe do a very concrete question. No, okay. It could be, okay, so we could add new, okay. Well, the thing is that we have the general op codes here, right? Which are divided in three, right? I'm always confused here, yeah, the class, the, okay. That was the question, I'll stay there. Are all of those being used already in the current UAPI? Are they all used? No, there is one free and they wanted to use it for SDFAN but they didn't look like that. Yeah, I think how many we have free? One, right? Or two. Yeah, this is an array. Yeah, so that means you can't just, sorry. But because it's an array, you can't just have like, for one code, we have a different sized one. It's the problem, right? Yeah, different sizes are absolutely in a way, right? Yeah. And we kind of have an indirection as well, right? Sorry. We have different sizes already. We have 8-byte and 16-byte. So we have infinite space, really. Nothing is... There is infinite space, okay. We have 8-byte instructions, 16-byte instructions. We can add 32-byte instructions, whatever we want. There's plenty of room. We can have like what you're proposing but it won't be called like structBPF in the sense. Yeah, right. There's structBPF in the sense too, but sure. So a while ago in the kernel when we added a bunch of new system calls we were faced with the problem of how to make the structs extensible in a backwards compatible way and we called them extensible argument structs but basically the requirement is 64-bit aligned and you need a size parameter. In the system calls you can pass them separately but you could also have the size parameter within the struct itself and then you can have a version struct by size. Every time you add a new field to extend the struct you bump the size and then you require that if you pass an old struct that the additional fields are zero-filled, right? I'm always getting this wrong. But the additional fields are zero-filled and the kernel will verify this. So you can have it backwards and forwards compatible, extensible. There is a description of this algorithm description of how this works for open at two and for clone three in demand pages. That's assuming that this is somehow even feasible and that you would require a new structBPF instruction two. But there are other schemes to make. Okay. Well, in the mean... We are following the new V4 instructions and we are using the off-16 field as an encode but it was just to actually take a look at this. That was all. Okay, thank you for the feedback. Um... Right. Okay, I will do the last three but yeah, so... Yeah, this will be very fast. Okay, then... There was an ABA change implemented in Clang like a year ago or more which was that before all the structs were being passed by reference but now small structs are passed by value and we have to actually adapt to this in GCC and... This is our understanding of this, you know, but we have a couple of questions that maybe we can work with Young Home later, right? Like what about back-destructs and underlying fields and recursive structs, right? Because it was not clear to us exactly what to do with those. And for the stack when yes and we require 16 bytes as a boundary and... Yeah. For the... Yeah, I think that we potentially just follow C standard that we didn't do anything special. Well, the X86 ABI for back-destructs and... For strutters and aligned fields it passes it always by reference. Okay. So I think we could follow which is quite prudent because I mean it's... I think it's a wise decision but well, yeah, it gets complicated. Yeah, it's possible BPF also pass by reference for this pack struct on... We could, but we will need to agree on that. Yeah, yeah. And also I did not see any of the tests that were added in this change about the structs that contain other structs. But are you applying this recursively or they are passing them by reference straight away? For the recursive we define the struct I think so but if the struct is separately defined it's not. Okay, well, so we have to... Double check. To check and explicitly... Okay, cool. Okay. Thank you. Okay, first, rest. Yeah, well... Well, that I mean... I don't know a lot about the Rust type system in this sense but I don't think it is reducible to the C type system. I don't think so. I think the Rust types they have all sort of things which are out of the scope of the C type system so that makes me wonder if BPF is going to be able... I mean it will have to be able, right? Unless the same way that there is BPFC I guess that it's going to have to be BPF Rust as well, right? And basically I've been wondering lately now we have Rust C which is a little bit based as far as I know now they can use the BPF backend and this is used by AJA another Rust BPF infrastructure and Rust Red BPF I think it's called which is another Rust thing that this one, yes, AJA works with object files that you compile with Clang or whatever but Red BPF as far as I have seen it uses the Clang compiler underneath along with LLVM to compile Rust code into BPF assembler, objects, right? Bad codes. So then I was wondering what BPF is that thing generating? I don't know. So I think we need to coordinate on this somehow before it gets out of hands because the LLVM backend the BPF supporting in LLVM I guess it is independent of the language that is used in the frontend, right? Yeah, if I understand it correctly the Rust C frontend just generate IR and then pass everything to the LLVM Right, yes, yes. So we need to take a look about the different types of the BTF they are generating for the common Rust data types and most likely to define a subset or something because otherwise I think I have no idea. I didn't try. Well, I would suggest maybe that if any data structure that is not expressible in current BTF probably is too complex, right? Something like that. I don't know if I can pass them. I was there but I had my own deaf room and I could not Yeah. But the recording is there so it might make sense to reach out because they've been looking into this topic. Yes. Right, that's the last of the, yeah, right, yeah. So coordination between tools and because now it's not just clang and working and let's do this fast. I mean, this doesn't need a lot of time. So we have the BTF main list. So we get in CC when some for example, he's always very nice and he puts us in CC, you know, me and or Faust or when he changes something that is particular interest of DCC or compiled BTF. But, you know, I mean, this list is so difficult to follow and particularly with people who are, we are related to BTF, right? So it was wondering well first about is there people from the Rust compiler in the BTF main list? I have not seen anything then there. But, and second I know I hate more main list because you know and I hate overhead and but it could be that for compiled BTF, you know, for tools in a specific things maybe we will need a separated list maybe where we have the Rust people, Clang people, you know, you know, Hong Kong and maybe I don't know. I have maybe a side comment or question. Have we thought about doing like BTF types for K-pointers like metadata with a pointer so that the BTF knows the length? So my Rust is rusty. So like the, like Golang where you have like slices, right? So you have a pointer, it's not a pointer and see we don't have any notion of a slice per se. But a K-pointer is a slice in some sense of that notion. Sorry, what was that? Yeah, the K-PTR, yeah. Yeah, okay. And then that's what I'm, but should they have a BTF type? No? But how would you program that from C? How would you program a dynamic pointer from C? Yeah, okay, I'll look. Yeah, okay. Okay, so this was about, you know, if whether maybe it would be a good idea to do a separated list for compiled BTF particular issues and having the GCC, the clang people and also the Rust compiler people in it. Well, that's just, thank you for summarizing it all. Definitely we need something as you pointed out what we have is not enough, it doesn't work obviously. So, to answer your question whether the plan to standardize ABI, yes, PSABI is in scope for ATF standardization. We'll get to it once we're done with ASA, whenever it is. But we cannot wait for this to formalize because as you pointed out Rust, IA people and RedBPF people are already doing their own. As far as I know, IA folks they contributed back some of the LVM backend changes but not all. They still have like a fork specifically for Rust because they do some fancy linking that somehow LVM linker cannot do. New mainly list, I guess. Should it be in the in Vigir? Like, I guess. Vigir now, so if you folks haven't noticed we just moved the last way to BTF Vigir to links in the LF so it should be stable now unlike in the past where we had like mass unsubscribed events twice. So if we can definitely create a new mini list there. So if someone creates this list we definitely want to be there. It would be nice, yeah. Okay. Then we have this BPF Wiki in the GCC Wiki that gets including binutils and it's possible to subscribe for notifications so if we had a compiled BPF list we could subscribe the list to this Wiki for example, right? So... Just a comment. The Vigir list is a member of the IETF list right now. One is a wider distribution right now so that you can't actually send some to the IETF list and people that subscribe to Vigir won't get, right? And so I'm just looking forward, right? As Alexi mentioned, PSAPI will eventually be in the IETF and then if you're going to create a new list then we may want a similar structure there whether it's a separate IETF list or a member of the future one when that actually becomes there. In other words, in the future we could take BPF at IETF.org and also subscribe whatever the new list is, right? At such point as that becomes relevant. I'd say in the short term the relevant people that might be on another list would be the equivalent of a design team in IETF language, right? Which means it's not part of the standard. It's just people trying to come up with what the proposal would be and then it comes into the IETF later and that seems to be the right way that the IETF would be the process there. So I think that any of the different solutions you're talking about would be fine but I just want to say how that's how it kind of relates to the future. Okay. I think we talk about this maybe because we tend to eat all the time as you can see today with those topics. So maybe having once per month some dedicated to compile the issues in the BPF for these hours. Okay. So thank you and then Faust will say a little bit about the type tags, I think, because... How much time do you need? I could cut this entirely. It's mostly stuff that we need to coordinate between GCC and Clang for the BPF back ends. You can enumerate them. Yeah. I can go through two minutes real quick. Okay. Two minutes, fair enough. And then we will have your... Okay. Yeah, sounds good. We have buffer at the end of the day so it's not that critical that we strictly stick to the schedule. Yeah. So this is just related to the type tags that I've been working on in GCC and that we... Clang has support for. But the idea... Since I've been working on it, there's a couple of problems that we've found and we discussed these in one of the BPF office hours or two of them and took all the time that the dwarf format for the type tags originally was only meant to support pointer types and we said, well, we should be able to tag any type, not just pointer types but to change the dwarf format to place the type tags as children dies and dwarf of exactly the thing that is modified with the type def. But since then, there have been a few more patches to Clang to make the changes that we agreed on and some more things that we've discussed with regards to how type tags interact with cons volatile or strict qualifiers and, importantly, how type tags interact with type defs or something that we found is that currently in the BTF loader if a type has both cons volatile restrict modifiers and type tags associated with it there's an ordering in force that any type tags come before the CVR qualifiers in the chain and that's fine except that the helper function that is used and this is credit to Edward Zingerman for looking at this. This helper function treats type defs also as modifiers and the problem becomes that it introduces an expectation that type tags should jump over type defs to the beginning of the chain which means that you end up losing the actual original ordering so if you have some type def like this type def int tag1bar and then two uses of it in the dwarf representation by attaching the tag to the int type that it applies to you can sort of use that to reconstruct you know exactly what sort of declarations it's coming from but in BTF if you follow the ordering that the tags should go before modifiers and treat type def as a modifier then you jump the tags to the beginning and you can't reconstruct necessarily the declaration so the issue here is that sort of the BTF type is modifier function that is used for enforcing this ordering sometimes type defs are filtered out at the call site but not everywhere and so it's sort of a question of should we really just say type defs are not modifiers and that's something I think that we looked at that would probably be best fixed in the kernel I think the original expectation is type def is always immediately after the pointer so it's not for all rather modifiers so it works correctly in the current kernel it just doesn't work with new format so in that particular case it just need to be fixed kernel just need to be fixed yeah okay I can that's not that important I think just to close things out one of the things that we talked about during the BPF office hours and like since type tags are implemented as an attribute the compiler looks at the attribute and says okay what element of the declaration does this attribute apply to and part of the original motivation was you have sparse attributes that are used in the kernel for like pointers to say this is a pointer to kernel memory or user or whatever when I was looking at this we found issues that sometimes in some declarations sparse would actually associate the attribute differently than when GCC parsed the same declaration so you couldn't use the regular underscore underscore attribute syntax as a direct drop in for the type tags for replacing sparse attributes the C2X standard attribute syntax I don't know if people are familiar with it but basically it sort of borrows the C++ double square bracket looking attribute syntax and that is actually more precisely defined in terms of basically if you put an attribute it always applies to the type that's unambiguously which means they don't sort of slide around as the new attribute sometimes do it seems like that's the best path forward for the type tags we sort of agreed on it in the office hours but you have the issue that since it's C2X standard attribute syntax it's you know if you explicitly specify I'm compiling the kernel with standard equals C99 or whatever then you can't use that attribute syntax so I guess the proposal is that we have maybe we need to add to the compiler some sort of flag that enables the attribute syntax and then when you're compiling the kernel with the type tags you can use that no I'm saying we can use exactly the C2X standard attribute syntax but if you're explicitly compiling to an older standard then the compiler says I'm not going to follow that syntax because it doesn't exist yeah but what's the problem with C2X is well pass dash STD equals C2X to the kernel will it compile or not so you're saying it'll just won't compile because of other C2X things in there and you will just that's sort of the question have you tried compiling the kernel with C2X is there stuff that will break I mean can we just compile it with C2X for the sake of type tags there is a building always a test whether there's a BTF type tag in GCC just say okay I support this unless it's like a C2X C23 for example and then it's available otherwise not available so I think that's okay thanks that's it