 Hello everyone. My name is Iri Oshai. I work for Red Hat and I work on Perf upstream. This presentation is about the way the Perf is handling build IDs of applications that it's profiling. And it's basically about the new feature, new way of handling build IDs in Perf. This feature is mainly focusing on people who needs to record the data on one server and report, investigate the profile on another machine. So we believe that this change, this feature actually will make this workflow easier, so easier report and moving the data to another machine. Basically, we can cut the Perf archive execution, which I will get to in another slide, but it's quite important achievement that we like. And by a consequence, it actually makes the profiling the record even faster. So that's a really nice side feature. So before actually going to explain what's the new way of handling build IDs, let's see what's the current workflow when you want to profile on one server and investigate the profile on another. So currently, you run the Perf record. Of course, you do the profiling as you want. At some point, you end up with the Perf data and now you need to run the Perf archive. If you want to investigate this Perf data on another machine, then on this record server, you need to run the Perf archive on top of the data that you just recorded. The archive will go through the data and identify all the binaries that were involved in the profile and it will pack all the binaries to one table. And you need to take this table and the data and copy that on the report server, the server that you can do the investigation on. When you are on the server, you need to unpack the table and populate the debug cache with its data. And right after that, you can basically run the Perf report and everything. I mean, the report will have enough information. It will have the binaries that are linked to the Perf data. So it will correctly translate the samples and you will see the function calls, not just the functional addresses. So this is the current workflow. Sometimes it's really horrible to get not just the Perf data, but also the Perf archive from the customer because Perf archive depends on how the data are big, but it can be really big. Also with the applications that you are profiling are big. The archive will be really big and get it from the customer through possibly multiple VPNs is not always great experience. So this is something we were aiming for to make it better. And I will now go through the new workflow that can be used. That feature, this feature is actually enabling. So again, you have two servers, one you record the data and other that you investigate the data on. So on the record server, you run the Perf record as you are used to, but you need to specify one new option build ID dash M up. I will explain the name will become obvious and in few slides. But now just for a quick presentation. So this is the only option you need to specify other than that you can put any option that that that you are using. It's not like tradeoff within options. You can use always all the options available. And you end up with the per data, of course. And now you don't need to do the Perf archive. The tradeoff here is that the reporting server of course always needs the binary. We just provide them from the different place and the different place is debug info demon. So you need running instance of debug info demon somewhere for my tests. I was using the debug info demon on the on the server that I was doing the profiling. But actually it will be. It can be available somewhere in the network and the server just needs to have access to the binaries to the or even just rpms the debug info demon can go to the rpms and provide the binaries and the debug info. So that's the new way you record with the extra new option and you provide the debug info demon. No Perf archive step. So when you are on the report server, if you run the Perf report on top of the data, of course, you will not see the data being translated. What you need to do is the new functionality in the build ID cache, which you will add the perf data dash a perf data and you provide the address of the debug info server. And build ID cache will actually go through the data, identify all the needed binaries and debug info. And talk to the debug info demon to get the binaries to the server and it will download, populate the debug cache. And the report will suddenly start to work, of course, it will, it will have all the necessary binaries and debugging for information to display. Nice, nice profile. So that's basically, that's basically how the new workflow looks like. So we you don't need you no longer transfer all the table with the binaries, but we change it for actually Perf report and build ID cache being able to download the data afterwards when when they are actually, actually needed. Before going little bit little bit more deeper on how this feature is implementing. It's actually probably nice to explain what build ID is. So build ID is kind of unique ID for the application is generated by the linker. So when you compile the application, the last step the linker will make a hash of the, of some sections of the binary and produce the, produce the build ID. So there's a, there's an option in the new linker it's build ID. And you can specify if you want MD5 or SHA, SHA one, it's stored in the node section of the elf files. So any application that can display node sections like in here, I'm using the retail dash and you will see probably more node sections. So the one that has the type new build ID will will get you the build ID of the application. Perf is actually using build ID to identify every binary that it comes to, because as I said, it's the unique, unique ID for the application. And if you upgrade the application, the application with the same name will have of course different build ID. So the profile, when it's reporting the samples, it always needs to know which application I should go to and find the data for the profile and it's, it's based on the build ID. So that's why we care about build ID. So how it all works a little bit more deeper on how the, how the profiling profiling works. So you run perf record. And the profiling will start you configure any events that you want. At some point you will say, okay, that's enough. You stop. You stop the perf record and things will start to happen. First perf will disable everything that was enabled. So all the, all the events and all the extra stuff that we allow to be configured will be disabled. And right after that we flash the data to the file. So the data is spread over multiple, multiple buffers. So at some point we take all the data and make sure that the file contents all the needed information. So in idle word, that would be where per record would actually stop. But what's currently happening that per record is opening the per data yet again. The profile that was just stored to the file we open again. And we process it, we process it almost like per report would do. And the reason is that we are actually looking for the binaries behind the profile for the binaries that were used in the profiling that the samples are actually, that are actually behind the samples, and we need to store them to the, to the debug cache, which is actually time consuming. Of course depends how big is your profiles, how many samples you stored. But this is the extra processing that the per record is doing when it's actually finishing. So you might have maybe notice if you profile really a lot of data and you press control C. Sometimes it takes a lot of time for per record actually to finish. And most likely it's spending time in, in this, in this processing it goes through the data. It looks for the binaries behind the, behind the samples and it stores them. It reads the build IDs and stores them into the debug cache. I have simple diagram showing what's actually happening as I said, we go through every sample. And for every sample, we have the IP address and the PID, like the process identifier. So we find the process and we find the memory portion that the sample happened. So based on the IP address and we find the memory map and the memory map has many things inside. One of those things is also the full path of the application or of the file, which is providing the data for the memory. And we take that path and we read its build ID and we store it to the, to the debug cache. So this is really what this extra processing look like. And here comes the place where the new feature actually comes to the game. And the new feature is all about storing the build ID directly in the memory map event. So the, and the record option at the beginning, the build ID and M up is basically telling the kernel. Then you store the memory map store their build ID right away. So if I show it in the diagram, it looks like this. So we have again the sample. We resolved the sample. We resolved the memory for the sample. But in addition to the path of the application, we also have the build ID for the application. So there's no need for the extra, extra loop anymore. So that's why this feature is actually making the perfect record faster. We can cut totally this extra processing. But of course is a trade-off. We still need this binary, this debug info that the loop was actually getting. But we need them at the time when we are reporting. So we added the support to download those binaries to the Perf report and Perf build ID cache. So when going through the data, both applications can actually download and store the needed binaries to the debug cache. So where can we get those debug info binaries as I already said? It's the debug info demon. So Perf knows how to talk to debug info demon. Of course, it's really simple interface, client server, basic interface. You Perf specify if it wants the binary or debug info based on what is in the sample. And it specifies the build ID. The debug info on the server side will check its database if the build ID is there together with whatever info is needed. It will send the information back. And the Perf will store it to the debug cache. And it's there for Perf and reporting. Debug info demon is part of Elfutile. So if you do the search on like the DNF search under the Fedora, just the debug info D, you will find the package. It's been there since maybe, I don't know, maybe one year. So it's definitely in the latest Fedoras. So it's available. So let's go once again through the new workflow. So on the server that you do the profiling on the record server, the only thing you need to do is to do to run to add the option dash dash build ID memory map. Which per record will use to inform the kernel store the build IDs into the memory maps. And this option is also to inform the Perf record to cut the extra processing at the end. So that's the only thing that's happening here in the Perf record. Then of course you need the debug info somewhere. Maybe I should note that it's really highly configurable. I'm using it like this during my testing the debug info slash which will actually provide you your whole system into the debug info demon. But it's highly configurable. You can specify even like a bunch of RPMs and it will go through the RPMs and provide the binaries and debug info from the RPMs. So that's it on the record side. On the report side. What do you need to do? You need to download the binaries. We added the dash dash debug info demon option, which actually specify the address together with the port of the debug info demon. Both the record option and this debug info demon option can be also specify in the config file. So if you put it to the dot Perf config, you don't need to always put it to your command lines and it will be always there. The status of the feature it's quite young feature. It was merged just a few weeks ago. Both kernel changes and tooling changes are merged. It will be in five dot 11 if I'm not mistaken. And of course there are more things, more things to do. We have some to do list probably the biggest that we were asked by guys from Google is the ability to detect if the build ID parsing in the kernel failed. So the thing is that now when the kernel actually reads the build ID, there are certain circumstances that this parsing can actually fail. It's like you need to take care extra lot for this parsing to fail. Mostly it's always there because it's in the first page of the application, but these guys are using huge pages and they like to swap them out if they are not used. So there are occasions where the where the page is not there and there's nothing kernel can do about it. So it will just keep but as I said in the normal workflow, if you don't do anything special, the build ID is always there. However, we will actually fix that. We will detect that the build ID parsing fails and we will tell the third record the user space part to actually read that build ID instead of the kernel. So that's one of the fix that we are still working on. And that's basically it. So if you have any questions, I hope this will improve some user experience with the perf hopefully. Okay, there is a question in the chat by Lukas. I'll read it out. I'm still not sure how that would work with a customer. Do I understand correctly that the engineer who wants to create the report would have to access to the build ID demon? Isn't it easier to just create an archive and pass it directly? As a customer, I would be hesitant around some data available from outside to the engineer during the report that is supposed to send some packages from my system to the engineer. Yeah, well, of course, it's situation by situation. The old method is always there for situations like this. But yeah, very mean for places where this is actually possible, where you can, where you have the available debugging for demon somewhere. So yeah, it's definitely not solution for everybody, but it's, it's there if it's, if it's possible. Okay, there's another one question from Takayuki Nagata. Are there any expected use case for the feature? So well, the use case is basically as we described. As I said, it's new feature is like really fresh. So apart from us, nobody, nobody is using it at the moment. So we will see. I mean, the use case is as was described in the, in the presentation. And we will see if people, people will like it.