 Hi, I'm Iirka. This presentation will be basically overview about the ongoing work on U-PROPE multi-link, which is basically a way to attach multiple to attach BPF program to multiple U-PROPEs. I'm going to do it also now, but there are some difficulties and the U-PROPE multi-link is trying to to address that. So first, what's U-PROPE? U-PROPE is basically probe that you can install on the application or the library. Basically, it's defined as the offset of the space where you will actually enter the probe and anytime when the application is executed and going through that place the probe will be hit and sequentially the BPF program will be executed. U-PROPE is also used when you want to actually attach BPF program to the U-PROPE so like the user statically defined transport if there are in the application. If you are actually attaching BPF program to it the the way it's done is that we actually use U-PROPE to attach that. As I said, U-PROPE is defined by the path offset and it's also the U-PROPE which will limit the probe execution just for that process ID. There's one extra field which makes sense for U-SDT but is part of U-PROPE configuration is this RF counter offset. It's basically another like Boolean value which got said when the U-SDT probe is attached and it will be set like to true and it's counter. For me it's just another field that I need to take care about so it's properly attached and it's ongoing work. At the moment how it actually looks like when you want to create U-PROPE is that you need a perf event to do that so basically you are attaching BPF program so you need to load the BPF program you have the program FD then you need to open the perf event where you basically need to use the K-PROPE PMU sorry U-PROPE PMU where you specify the path the offset and inside the kernel the perf event installation will actually do execute the U-PROPE register which is like the API how to create the U-PROPE the path translates to inode and inode and the offset are like the keys are like the interface how to create the U-PROPE then you need to perf event with the U-PROPE and then you need to connect it with the program that you just loaded so there's IOCTL command for that set BPF on the perf event and it will actually attach the BPF program to the perf event then you enable the perf event and from that time you actually monitor multiple probes like this there are two problems so it takes time it turned out that actually creating 1,000 perf events together with attaching the BPF program on it is costly when you try to attach for example 1,000 probes using the BPF trace to the application on my system it would take like 20 seconds so that's one problem that it doesn't scale well another problem is that by creating the perf event you spend the file descriptor and if you are creating multiple probes for each U-PROPE you need extra file descriptor this is actually this is output of procfd directory of the BPF trace and it's actually wasting the file descriptor also for the BPF program which I don't think doesn't need to be like extra program for each U-PROPE we could probably use the same file descriptor but still 1,000 U-PROPEs you will waste 1,000 file descriptors because of the perf event so that's the reason for the creating the U-PROPE multi-link basically it attaches the BPF program to multiple U-PROPEs without using the perf event basically you load the program and then you create the link with the program you specify the U-PROPEs you want to create and the kernel insight is basically just running a sequentially U-PROPE register for each U-PROPE that you define and fortunately that's actually very fast so you don't need to change there like you can register I tried like 50,000 U-PROPEs and it's within 1 second so that's actually why it's all possible so how does the interface look like we are creating new link so we needed to add new fields to the BPF ATTR union under the link rate and there's new U-PROPE U-PROPE multi-structure and basically it has all the fields that you need to when you are defining the U-PROPE but each field not each but the paths offset and ref counter offset and the cookies they are actually arrays and when you take like one index it defines one U-PROPE so if you go to that index to all the all the arrays it will define one particle U-PROPE can't just tell you how many U-PROPEs you defined and flags at the moment there is just one flag that will tell if that's K-PROPE I mean U-PROPE or U-PROPE so this is like the kernel interface there's the BPF interface as well so for BPF link create function basically we mimic the kernel interface so there's a new struct in the link rate ops that you can basically put the same data and then go to the BPF ATTR more user friendly function is BPF program attach U-PROPE multi ops there you can actually specify binary together with the pattern of the function and the function will go through that binary and look up the symbols that match the pattern and attach the program as a U-PROPE to that to all those symbols or you can you can basically use the same way as for the link rate and define all the all the arrays for the U-PROPEs that you want to specify together with this interface there's also U-PROPE in the RFC but it's planned to be support for the section definitions so currently there's like U-PROPE multi and you specify under the slash you say what path and the function pattern and you can define program like that and it will be automatically attached as U-PROPE multi program and we will have the U-PROPE as well and the last change in the LibBPF is that basically when we have the function that attach U-SDT U-SDT and inside that function we actually we will detect if we can use a link interface and use that instead of currently like using just U-PROPE so effectively when you will use that function and there will be the kernel will have support for the new link it will attach the U-SDT probes with this U-PROPE multi link there are two important helpers they are not added of course but there's these two helpers have like flavors for different program types and also for programs that are attached with U-PROPE multi link so get func IP will get you the user space value of the IP register at the probe and the attach cookie will get attach cookie will actually get you the cookie that you can specify when you when you create U-PROPE so you can have your own data specific data for a specific probe available I mentioned the PID filter that's something that the current RFC doesn't support but as per discussion it seems that to be able to replace U-PROPE current U-PROPE interface with this new interface we need it and it looks like it's actually possible and not that bad so this will be added in next version and yeah I have some users that actually I'm using to test the feature so there's support for the tetragone and BPF trace for that you can check it out your questions sorry that's more of a generic question I don't know exactly how the multi-link interface works but like what happens if one of the things that you're trying to install a link fails like do you fail everything or do you fail only the thing that failed yeah I think we fail everything we don't have any like so you have to roll back afterwards yes there's a specific error path like rolling back everything that we created so far would it make sense to like try everything and return I don't know like the successes or the failures yeah we don't do this type of things right then do yeah I mean it's cheap enough to like to fail if you have like thousands of functions and like one or two of them like might fail I mean it's only you like to break it maybe and like do the binary search of like which one are not attachable but like you shouldn't instruct Colonel to do that one comment like if you can go a few slides back about like UAPI after talking a bunch was container guys I think like nowadays maybe in addition to accepting paths we should accept file descriptors they're like everything file descriptor that's the most reliable way right so like I didn't define the file to which to attach like we should probably have like a flag to say like this is the array of file descriptors versus array of like pointers to strings that was one comment and I know we discussed that like on the mailing list about like multiple paths instead of just saying like everything stays within the single I know single file I'm still not convinced honestly it seems like a complication in UAPI in implementation and in practice you always work with one file so look at like BPF trace at like UAPI at like Libbpf section definition there is one path to the file right so I think like we can simplify it both for users and for the implementation like furthermore like when you have the BPF program like the high level API BPF program attach you probe attach multi you probe right if you specify if you can specify multiple L files and then like you have a pattern per each file that just makes Libbpf like do so many like elf parsings right like it would be much simpler to optimize it for single file right like I open L file once maybe I sort like the patterns or like for function names or whatever and they just do one pass otherwise like doing it multiple times it's like super expensive I think like overall like I don't see a lot of benefit to having multiple file paths per one like multi you probe link like what are the realistic use cases where you need this it's true that it for BPF trace it's always like it will be separated by one path for Tetragon we can actually configure like multiple them but I doubt there will be like many of them so it can be yeah so like you can have a few multi links doesn't really matter I would just make it like in most common case you'll be allocating an array of exactly the same pointers hmm true and then like you by the way like so completely different question right you're saying the you probe register is super fast why is it fast and why the K probe was slow like remember like we had to do like the multi IP attachment K probe and all stuff why that doesn't apply to you probe so it's doing different thing right I mean we are so what you probe register is doing it it actually installs the software breakpoint on that offset of that I know and that seems to be very cheap well if I remember correctly for K probes right like the slow part was like we were synchronizing CPUs and stuff like this so like if you do it one by one that's like where we spend most of the time just waiting for CPUs to get on the same page like that that doesn't happen in you probe case so for K probe we had to actually add the F probes to have another layer basically what I'm getting is like do we still need to like improve you probe further and like do something like F probe for you probe or like there is just no point because it will be exactly the same no well it doesn't look like I mean from all the testing this I am doing this actually for like 50,000 you probes it's very fast it's always like within one second it's I didn't even like measure it because it didn't make any sense that's great it's I also from all the profiles the main main thing that was slow was like Perf event installation because it's doing like really some unique things like using the IPIs to install to the practical CPUs and stuff like that and it's probably costly you mentioned that you tried you have a tester to install the thousands of you probes it's a quick very quick but my question is that when you doing that the tester is that every objects or fire already been loaded into memory or is just somewhere and no process using them because that may make different I don't know how it works but I imagine maybe happen at somewhere when the library have been loaded or finally been loaded into memory we'll do that in quotation so the benchmark I was doing was on the file that wasn't mapped at all it was like on the file system it had to be read like yeah I don't have too many too much data on like all the profiling I just did the cases that were easy to do and that look like I mean covered the whole thing but I definitely we need to also check like the boundaries of the performance if there are like some conditions where the you probe register might get actually slow but so far it looks like it's fast it's just a small comment were you discussing before having a single path for like a single binary for all the so one use case that maybe doesn't really match this pattern and we don't really have it but it's a potential thing to think about is like looking for a single symbol in all binaries so basically it's like P thread mutex log in all of the binaries of the system it's something that sounds reasonable to do I guess well that would be like always tied to some library right or I mean there is a more usual case it's use DT like very often you don't care which binary use DT is defined in so you just like find all the use DT's blah blah blah column blah blah blah but even in that case you have to like discover all the binaries and process them one by one and that's the expensive part not like creating 10 links I think so I would still keep it simple like the BPF proc attach use DT even if you need to do it across multiple binaries like it's very easy like to to do this because like you still have to like identify every single binary where you need to attach to so that's true alright thank you very much thanks