 Hi, thank you. So my name is Yiri Osha. I work for Redhead I work on perf and one of the latest thing we actually did is That we moved some of the perf functionality to the lip path. So This is more of commercial presentation for you guys to start using the lip path. So lip path What is it? It's basically Another kernel tree library. So if you are familiar with Libbpf lip trace event and other libraries in the kernel tree lip path is right there with them What does it do? It's the interface For the perf events subsystem At the moment We took just the counting and sampling interface which I will be Discussing later on But basically the The idea is to take any mature and stable code from the perf tool Which is basically most of it and move it To the lip path We will do all the things that library supposed to do so we will have versioned symbols and we will have very nice documentation that I will Show you shortly so as I said currently we are in the in the Linux kernel tree So you need to actually follow some steps to we are So far we are not in the package So we need to follow some step to install the library which I will just show you so assuming you are in the kernel tree kernel gtree you just go to the tools lip path and This is where the lip path is Residing and to actually install it. It's very simple. You type make install That's dear where you were to Install that the magic will happen and you will see everything Everything under the desk there directory, so We have the standard things that you would expect from library we have some header files and We have we have some Libraries so once you have this installed you can start using it What can we do at the moment the interface is quite easy and at the moment we support Counting and sampling which basically mean we allow you to create perf event to attach it to anything We can attach it to and get the value from the event We can do that for the counting so we can get just the number we can do also sampling that means We will open ring buffer between kernel and user space and you can read the samples and Do whatever you want with them so I will just Go like quickly through both of the interfaces. So you've got the idea. How simple that is If you actually go to the installed tree, you can see we have very nice documentation even the web HTML file so You can actually go to the you can put it to the browser and You can read Everything that is to know about accounting. I will just highlight some points How do counting worse so I will just go quickly about the interface the Liper as any other library We have the initialization function Which will do some basic setup and allows you to Provide the callback that will actually get any Debug message from the library itself. So anytime the library complains about something or do some debug It will go through this callback So once you do this you define the event the event Is basically defined in the same way when you want to create event Directly through the kernel. So we are using this perf event ATTR function Which You populate with the data of the event you want to create for this example I'm creating two events software CPU clock and software task clock and They are both created as disabled Next step is to decide what are you going to measure? For this example the example is measuring the process itself like it's measuring itself Again, we have object for that. So perf threat map you create the object like this and set up with PID That you are interested in zero is for the current process next steps is Basically common for any time that we access events in the perf We are using this way of accessing events meaning we have perf evalist object You create it and then you populate it with the events and later on you are doing You are working only with the perf event list so Create the evalist create the event object from the ATTRs that you defined put it to the event list Now you set the map basically saying to event list What you want to monitor and that's the perf threat map object that we created earlier. We will monitor ourselves This function will open the events. So at that point We basically have all the file descriptors for all the events in the event list and As I said the events are created Disabled you need to specifically enable and disable events and do something that you want to measure in between So this is the counting interface meaning that you create the events you Let them count something and you want the number the number Here is how you get the number. We have some iteration functions, which you can get every event through the event list and this perf excel read will actually Read the counter to you and this structure counts Will get you the value and the times for the value And you can print it you can do Whatever you want to do with it at the end, of course, we provide some cleanup functions Just to clean up So this is the counting interface really easy create event Measure something get the values you are done The other interface that we currently have is sampling. It's a little more A little more difficult, but not that much again if you go to the installed tree and check the HTML file You will find very nice documentation I Encourage you to read everything. It's really nicely explained So we are doing the sampling. There are some common steps. It's just different at the end So again, you need to initialize the library, of course When you define the events for this Example is basically doing the sampling for the hardware CPU cycle. So Basically, this is the event when you use per free court without any Any parameters it will it will do basically this it will define the hardware cycle and start sampling on this You initialize it with some black magic, which is explained in the in the dog If you are interested Again, you need to decide what you want to monitor for this Example we will sampling we will do the sampling system wide. So we have special object for that per CPU map Which will this way it will initialize to all the CPUs in the system and Now again the thing with the event So we have the event list with which we populate the event list with the events and then play only with the event list So this is how you create it. This is how we create the event from the attribute that we Defined above you edit to the event list. You set the map You tell the event list what to monitor You open and Here is where it's different this function per every map Basically creates the ring buffer. So you you have some events in the in the event list And when you call this function, it will create the ring buffer for every event and for every CPU or whatever object You initialize the event list and it will create the ring buffer for the sampling so whenever the kernel generates the sample it goes to the ring buffer and That's how you do the sampling. So this is just to create the The maps the ring buffers The event is again created disabled. So you need to enable it and disable this example sleeps for three seconds The example is doing the sampling system wide. So when it's actually sleeping for three seconds the events are The sampling is working on every CPU and every CPU is getting the data to the ring buffers so normally you would like to After you enable the the event list with the events you want to start read as as fast as possible Because the ring buffer the kernel will not wait for the ring buffer and the way it works is that The user space needs to read the data to make the space for the kernel to produce the data So you need to read as fast as possible But for the sake of this example I am just getting the three seconds of the data to the ring buffer and this is how you read the ring buffer again we have some iteration functions and For this will iterate through every ring buffer and this is the way how you read the data from the ring buffer So there's some in it function some final function and in between You will get pointer to every event in the ring buffer and When you have the pointer you can process the event Which is at the moment somehow? Black magic for those that you are not familiar with actually how the event is stored This is another interface that we are planning But at the moment this is how you parse the output and this is what you get out of the output the CPU PID TID IP and period of course. This is all configurable but you need to know how the things are stored in the buffer and This is the way how to parse it from the buffer it will we will provide the interface for that later on and Again clean up Clean up clean up at the end So this is How you do the sampling? The motivation for this interface to actually expose this interface We've been asked for some time to Make some of the functions and external I put some of the functions to the to the library and The most of the time people ask for is To be able to read the trace points like to have the way to put Some function calls to their own library to get the data from the trace points and To use it for debugging basically because you can define your own trace points and and see what's happening So I put together example of which it does exactly that so it's basically the sampling And it's connected with lip trace event which takes the data from the trace point and display it to the user and I will show it to you So when you download the example you will basically You will get make file and the C file which you can see it's not not that big it's just like 250 lines the make file only wants you to to put together the kernel tree and The director is where you want to install the lip earth and lip trace event so when you put it With your setup you can say make lips and it will make both lip trace event and lip earth for you When you say make it will make the TP binary for you and You can basically use it like TP scat scat switch which means you provide The subsystem name for the trace point every trace points goes with the subsystem name and the event name itself so you just put the subsystem name and the name of the event and You will get the values from the from the trace point if I put Something less verbals you will yeah you will basically It hooks to the to the trace points get the data and print the data to the user If you actually check the check the file it is actually what 200 and something lines 264 so it basically follows the example it creates the trace point The lip trace event needs some initialization so you will you will actually tell the lip trace event that you want which trace point you want to Decode and later on it's just lip earth. So the lip earth event in it and The standard thing that you saw in the example and it ends up with Reading the ring buffer live and printing printing out the data the display function for every Trains point is the black magic again a Little more explained, but basically you need to know how the kernel stores the data to the sample so you can actually parse it out and And played with the data the lip trace event is actually used to get you to nice line output from the from the trade point. So this is like really simple 260 line example to Have in the in the binary like your own monitoring of the trace point Quickly just going through the future interface. So this is just the beginning to have something to work with We will add the next item on the to do list is to add the API for parsing the per data files and because we've been asked from from the guys from other libraries like Babel trace To have some access to the per data when they won't probably to merge the data with their own data So this is what comes next Another interface we've been asked is to provide something that parse the events names and provides The perf event ATTR structure defined with the data So to the perf tool when you're using the perf tool You can use really nice perf event names like almost human readable names and Perf tool actually parses that and put it to the structure that you can use with the kernel So this is something really useful that would be in the library and everybody would be allowed to do that Then of course the sampling parsing and some hardware events counting That's another items on the to the list on the to the list is also packaging There are two options that we are considering now It's either to go the way the Libb pf went and had some external mirror on the github and The source would be still kernel kernel tree But once in a while we would we would store the data to the github and make package out of the github source This way we would be maintaining our own release numbers. The other way is to put another library to the kernels tools At least in federal I'm not sure how the other distribution maintain this but in this way we would actually Be dependent on releases of the kernel. So we need to we need to see if actually One way or the other is is better And that's it So if you have any questions so You haven't initialized size field instruct perf event ATTR Is it initialized by the library I presume? Yeah, if it's filled by zero, it's it's the current one Is it filled? No, it's only it should be filled by a strike prefer event utter size and My question was well, what happens if the applications and library ideas of well, what stock event utter size is different Yeah, I think it will not work Well That's the issue because well, it actually have grown in five five Well, you need to The the ATTR structure is what you provide to the library. Yeah, but you need to make sure it's always the size field and Look how it is always for the syscall So probably it's done in the library. Yeah, there's some initialization in the library. I'm not sure about the size Well, I'm also not sure if currently we provide the zero It will actually go ahead and try whatever's in the kernel and see if that matches the the attribute so That should work as well Thank you. I have a question like for example, if I want to measure some bottlenecks in application Like if I select LTDNG, which is much easier to set up and run and analyze How could it be compared to perf is perf more precise in such situations or it's Couldn't be compared. That's all Yeah, I guess can't can be compared but it's case by case Yeah, the LTDNG is another word right so They do something we don't we do something that they don't do so it's case to case what what are your needs? Any more questions how stable is the API of this library stable no, so The interface that we just ported to the library is one of the oldest thing in the perf and The shape it has now has for maybe several years now. So we are quite confident. We will not The perf itself is using this interface the perf itself is using this library. So There's some guarantee that it will not change. However the The library itself have the version symbols. So Is it? Yes, we have support version symbols. So once we release the library. We actually We will care about the API stability through the version symbols. So Thanks Does the library only use the public? Linux API or I see it's in the same git repository does it mean in use the same API as any other application or if there's some Well, it's built for Linux it's written for Linux if that's your question or maybe I misunderstood Like Linux has a API stability it will not break things so So Leap perf use only this API to access Inux or is there someone no no the library Using the the syscalls that are provided by the kernel. So it's It's simple as that. It's just the user space user space glue That you can easily easily access and make those examples that I show Thanks. We have time for one or two questions So how do you think the experiment of maintaining user space code in the kernel is working? You kind of touched on this but like I guess a recent one is like live you ring is actually maintained outside the kernel tree, right? But you know, obviously has a large kernel component I'm just curious like it's a very interesting dynamic and now that you're I guess proposing to add more I'm just curious how you think it's going or Uh, so maintaining the user space inside the kernel tree that your question how It's actually not that bad because the kernel people don't care too much so And the maintenance is quite permissive so Actually, he didn't even know it got merged recently so So it's quite uh, yeah, it doesn't have There's a group of people which will give you always feedback the group of people which is interested in that Uh, but like the core kernel community Like for example, Linus himself I saw some email. He doesn't care that much what goes inside as long as it's not total crap So not a bad. Okay. Thanks. I'll be sending patches to add system D Okay, we're almost out of time. So if you have some more questions for your car I suggest that you meet him in the hall and thank you