 Hello everybody. My name is Yordanka Rajov. I work for the VMware open-source technology center and in this talk I'm going to show you how to write plugins for KernelShark. Before we start, I want to say a few words about this talk. Our original idea was to make a tutorial, but because of the current situation with the conference being online, we decided to switch to a regular talk. But so this is something I'm calling quasi-tutorial. You see a lot of quotes. You're not supposed to try running this quote during the presentation online, but the examples will be available together with my slides and you can download it and try it and we'll be very, very happy to help you if you have questions offline. With that being said, let's move to the actual talk. So we assume that you are a KernelShark user already or at least that you know something about KernelShark and you are interested in using it. For this talk, we have one main goal, which is to teach you how to customize the KernelShark GUI by using the KernelShark libraries and we have something which is like a bonus goal for us, which is to actually show you that you can use the library without using the GUI. So let's see if this will be successful. Before we start, let's see some very basic definitions in one extremely simple example. How do we access the data with the KernelShark libraries? So in this tutorial, I will assume that you have a Ftrace data recorded with Trace-CMD. In principle, KernelShark can open arbitrary data format, but this is outside of the scope of this talk. So for the moment, we assume that you have a trace file recorded with Trace-CMD and you want to open it. So in your file, you may have a million of tracing records, very different event types, hundreds of different event types. Each event type has specific data being recorded in it, different data fields. But not all of this data is actually used directly by the visualization. In case-shark entry, that's the basic data unit that we use in KernelShark. It's a structure which holds only the absolute minimum of information from each trace record that is needed in order to visualize. Because we show kind of time series of trace records, of course, we need the time of the record. We also need the CPU core, the idea of the CPU core on which the record happened. We need the process idea of the task this record belongs to. We need the kind of identifier of the event type. And basically, that's more or less everything that we need. Of course, there is more data being recorded and we need to have a way to access this data. So this can happen on demand when you need it via the offset in the file of the actual original record which corresponds to this entry. And now let's see how we can use this in a very simple example. So how can you open a data file with library? You first initialize the context of the session, then you open the file. Because I want everything to fit on a slide, here the file is hard-coded. So I assume that your trace file is called trace.dat in the same directory. But you have to make this more sophisticated. So you check if the file was successfully opened. Then you load the data into this array of k-shark entries. If this operation is successful, you can loop over the entries. And for example, plot things like the CPU core process ID, the time of the record. The important thing here is that after you load the data, all k-shark entries are sorted in time. Which means that when you do a loop like this, you will see the time of the records progressing. That's important. We will see why later. And so let's say that you are interested, for example, in sketch switch events and you want to check what is the next priority when you do a context switch. So you can check if the event ID of the k-shark entry is the same as the event ID of the sketch switch. And if this is the case, you can use the offset of the, which is inside the entry to retrieve the value of the next priority for this particular event. Be aware that this is a relatively expensive operation. So if you want to do this for a small fraction of the events, that's okay. But if you try to do this for every single entry this will slow down your program substantially. And you will see that there are actually clever ways to do this later in the presentation. Of course at the end, you have to free all the resources that have been allocated, but this is trivial. And so far so good. We are able to print the data to the screen, but other tools can do the same. And this is not interesting at all. We want to see how we can visualize data, not to print it to the terminal. Before trying to visualize something, we have to understand the way the visualization model used by Kernel-Shark works. Back to our case when we have a huge file containing a lot of records. And let's say that you have this file opening Kernel-Shark. You've been zooming in, zooming out, scrolling here and there. And at some point you visualize only as a fraction of the tracing data that has been recorded. And let's say that you are visualizing everything that is inside a time window, which here is called t. So you are somewhere in the file and you visualize part of it. And it's within this time interval. The way we visualize this is to actually subdivide this big time interval t into smaller, uniformly sized intervals, which we call bins, or we can call it bucket if you want. It's pretty much like a histogram. And when you want to visualize something, you only check what is going on at the beginning of the bin and at the end of the bin. Everything that is in between gets ignored. Remember that when we loaded the data, it was sorted in time. So in order to determine the state of the model, the only thing that you really need is to know the index of the first event in a given bin and the index of the first event in the next bin. So everything that is in between will belong to the bin we are looking for. Okay, this was the explanation of the model. The diagrams, they look cool, but let's see some code. So back to the simple example from the very beginning. We open the file. We load the data into this array of kernel shark entries. Up to here, everything is the same as in the example before. So now we initialize the model here. The model is called histo. So we initialize the model. We set the binning because we want to keep everything as simple as possible for the moment. We'll get a very simple model which has only five bins. And the range of the model will start from the time of the very first record and will continue up to the time of the very last record. So this basically means that we will visualize everything. And now the state of the model is determined. So now we can fill the model with data. This is happening here in this fill function. And at that point, the model is ready to be used. So how can we use the model? We said that we want to see what is going on at the beginning of the bin and at the end of the bin. So we'll first ask the model, okay fine for me the first entry which has this value of the CPU ID. So we were looking for first data from CPU Core 2. And we want this to be in bin number 3 for example. And so we call this callback function match CPU. So this will return for you the very first kshr entry from CPU 2 in bin 3. And from the entry you can retrieve actually the task name. So this is the name of the task that correspond to the process ID which is recorded inside the entry. You can do the very same thing with the last entry in this bin. And here again you can plot okay this is bin 3. It starts with this task that has this process ID. And the same bin ends with another task probably which has a different process ID. Okay so far so good but this is still printing something to the terminal which is not what we want. And I guess it sounds too abstract to understand what is really going on. Here is actually seeing kernel shark visualizing something. So of course here the number of bins is not equal to five. What we do is the most natural one. So we just check what is the total number of horizontal pixels available for drawing. And then we set the number of bins to be equal to the number of pixels available. So basically one bin is one pixel. And here you see a really huge data file open in kernel shark. So because this file is so so big each bin actually contains hundreds of records. And so if you click somewhere on this graph the marker will select the very first entry in the bin or pixel that has been clicked. In this case you see marker A has something being selected. Just imagine that you switch to marker B and somehow you managed to click to the very next bin that is exactly next to the one which you clicked before. So in this case you're not going to select the next event. You're going to select the first event in the next bin which will actually be thousands of events distant from the one which you selected with marker A. So this is one extreme case of the visualization model when you have thousands of events in each bin. But let's see the other extreme which is when you do a very very deep zoom. In this case the time interval which is being visualized is much much smaller. That's why you see much less events. And so here with each vertical tick that you see in the graph corresponds to one bin. And the place where you see no ticks these are bins which are empty. They contain no data. So most of the bins in this case are empty. And you have some bins containing just one entry. So this time if you click on a vertical tick you select the single entry in this bin. And then if you click to the next tick you select the next entry. You see the text of the next record showing up in the table. So these are the two extreme states of the model and of course it can be in any state in between those. That depends from you. If you zoom in you go deeper. If you zoom out you see a broader view. Okay. So this I hope with this you understand the way the model works. But this is the built-in model. So that's the the visualization that comes out of the box. That's not what we want to actually learn from this talk. So let's see how we can customize Kernel Shark. And let's start with the equivalent of the Hello World program which is the Hello World plugin. So this is the customization that you want to inject inside the GUI. So you want the GUI to print Hello World. That's your callback function which you attach to the the code of the GUI. So how can you make this thing a Kernel Shark plugin? Basically what you have to do is just to build this code as a shared library. And inside this shared library you have to provide those two methods the initializer of the plugin and the initializer of the plugin. And inside the body of the initializer you have to register your callback function and you have to unregister it in the day initializer. It's as simple as this. So if you build this and you open this plugin in Kernel Shark what will happen is that you if you start Kernel Shark from terminal you see Hello World every time when you you change something in the graph for that when you zoom in here zoom out or any time when you change something you will see this thing being printed and actually you see it printed two times because in this particular example you have two graphs being shown by the GUI currently. So you see this thing printed once per each graph. Again this is completely useless but it's just a starting point to explain you how you can write a plugin. Okay so let's do something which is still completely useless but it's at least a little bit more interesting. Let's instead of printing Hello World to the terminal let's draw Hello World inside the visualization area of the GUI. So how can we do this? That's the modification that you have to do again because I want everything to fit to a single slide. Some things are hard-coded here for example the font file is hard-coded but together with my slides I will plot the code of the example. So there you can see how this can be done in a more appropriate way. So what do we do here? We just load the font here in this get font function and inside the body of your callback function we call this routine print text. We provide the font. The color is no which means it will be plotted or drawn in black. Here we give the coordinates and that's the text that we want to see. And if we build such a plugin and if we load the plugin with Kernel Shark we will see this. You see the Hello World in the middle of your graphs. The important thing here so this text is absolutely static. No matter what you do you can zoom in zoom out it always stays there unchanged and that's because this plugin knows absolutely nothing about the data that is being visualized together with the Hello World text. So again this is completely useless. This can't help you solve your tracing problem but it's a good step to learn. Now let's try to implement another plugin which this time will know something about the data which is currently being visualized. So this time you have to implement a different initializer. So in the initializer you will register something which is called event handler. So the previous time we registered the draw handler this time we will register an event handler and that's the user action which we want to inject into the original plotting algorithms used by Kernel Shark. So this action will be executed when you load the data. So you register a callback for a specific event. In this case it is for this event ID which corresponds again to the sketch switch event. That's how you retrieve the ID of the sketch switch event. So you register this pluggable action for this event and inside your pluggable callback you first retrieve the value of the next priority. So this time this is a different operation. Here you don't use offset to jump up and down in the file because you already have the record in the memory by the time when this callback is executed. So this is much faster actually. So you directly retrieve the particular field in the tracing record you are interested in next priority and you can fill this information into something which you call data container. So for each if you have a plugin like this for each sketch event you have an entry in the data container you initialize data container here in the initializer of the plugin. I have no place on the slide to show the initializer but basically here you have to free the memory of the container and you have to unregister the callback and that's it. So this plugin is as simple as this and at the end with this plugin you will be able to know all the next priority values of all sketch switch events inside your trace file. There is nothing magical about this plugin and I want to show that you can do exactly the same with the very basic example from the beginning of my presentation. Let's see how this can be done in the example. Again everything here is the same up to this point where we open the file. Here we initialize the data container then we have to find the stream object which corresponds to the trace dot dot file. With this stream object we can retrieve the event idea the sketch switch event and now we will register the event handler with the callback. So the callback function is the same I'm not showing it here because I don't have a place on the slide and below everything once you load the data you can do the following. So instead of looping over all the data you can only loop over the entries inside the data container so be aware that this data container is not going to be sorted that's because the data doesn't show sorted in the file but you can sort it this is optional it depends it's up to you if you want the data to be sorted you can sort the container and now you can print the same thing CPU ID process ID time from the container and you can get also the next priority and as I said printing is not very useful but you can use the very same mechanisms in your plugin. Okay so far so good we know how to draw from a plugin we know how to access data from a plugin let's make a plugin that can do both. This time for this plugin I will use C++ for the drawing callback just because the GUI itself is written in C++ so written making the callback developed in C++ makes the integration easier in principle this will be also variable in C if you don't like C++ it will be possible to implement a drawing callback like this in C this is not fully functional yet but it will be available so let's try to understand what is going on inside to this call drawing callback function so first of all we convert the C arguments of the function into C++ objects then because so we want to plot to see to visualize how the priority changes when we do context switch on a CPU so how the CPU switches between different priorities that's why we want this thing to be plotted only on top of CPU graphs if this is not a CPU graph then your callback will just do nothing we'll return then we are trying to check if we have a data container for this particular data stream and if we don't again we return this part here needs a little bit of extra explanation if you remember the discussion we have in the case when you open a huge data file and you have thousands of events in each bin it may happen that you have multiple sketch switch events in a single bin or you can have a sketch switching almost every bin in your histogram then the different bins will try to plot different things on top of each other which will result in a kind of ugly situation there are actually things you can do to solve this you can make your plotting logic a little bit more sophisticated but again this is hard to fit on a single slide so here I would just say okay don't do anything if this is not a deep zoom if you have too many events being visualized at the same time just ignore and here is the actual routine which is used to plot your elements on top of the graph you can write your own loop which does this it's not going to be something very complicated actually it's very simple but instead you can use also some of the routines which are coming together with the library in this case we have very simple needs so we just want to plot one data field from one particular event so we will use the most basic routine that is available which is event field plot min so why it's min because as I mentioned you can have multiple sketch switches in the same bin of the model and in this case if you want to see only the high priority tasks you need the next priority to be a smaller number so if you have multiple sketch switches in a single bin you want to visualize only the one that has higher priority or smaller value of next priority as a number okay so let's see what we do in this routine we first provide the the arguments so here you have from here from this arc cpp object you can retrieve all the coordinates of the graph that is under the plotting that you want to do then you provide your data that's the data container you provide this check callback function which in this case is just a simple c++1 the function which does one very simple check so you know that this will be plotted on top of a cpu graph and so you want to only care about the entries which have the same cpu id so you don't care about sketch switches and other cpus so you only want to visualize the sketch switches of the cpu which is currently plotted under the the region which the plugin now tried to draw something on top of course you can make a more complicated logic here for example let's say you really want to see only very high priority tasks so you can say okay I want next priority to be below a certain threshold for example you can add this logic into this callback function and this will do the job for you and there is another callback which you have to raise so this is the make shape callback we will see the implementation of this thing just in a minute and you provide the the color of the shape that you want to plot and its size so here is the implementation of the actual drawing again because this is the most simple possible example you are actually plotting only one data field of one particular tracing event and you don't care about things like correlating different events between different graphs so you still do the very basic thing so in this case you in this function you receive only one graph only one bin number inside this graph only one data field and you receive the color and the size you can retrieve the coordinates of the the bin on top of which you want to plot something over the graph and with this coordinates you create a text box object you have to provide the font you convert your integer value into string you provide you give the color you ignore the size because the text box the text box doesn't care about the size and you just say okay these are the coordinates of the text please draw the text here okay now we can do demo so here you see the plugin here let's first clean everything okay we'll rebuild the plugin you just need to provide the the path to the libk shark includes okay so we have the plugin now let's open kernel shark with this plugin so we will load the plugin directly from the command line you can do this from the menu of the GUI but this way it's faster so we do okay and we'll open one trace file I hope you can see this on your screens I hope it's not too small okay so you see nothing because as already explained you the plugin will only work if you do a deep zoom so let's select something and zoom in so you can see the the priority being plotted on top of every sketch switch event unfortunately there is nothing really interesting here let's try to find something which has higher priority and here it is so here you see tasks with different priority switching switching out for example if you click here you see that the post audio task is switching in and it has priority 109 and you see this thing actually being visualized on your graph okay so just to demonstrate for you that this is there's nothing magical happening here and this is the very same code that you saw in the presentation so we do so what what do we have here so that's the the get font function which is used to load the font well what is different here is on the in the example given in the presentation I assume that you already that you have only one file being open we can open multiple files so then you you need to have a one data container for each data stream so instead of having a single data container we have an array of data containers now here is the processing callback it's it's the same so you just add to the data container which corresponds to your stream here is here you can see how how is the what is the proper way to actually find the the font file not to have it hard-coded you initialize the container here you register the two callback functions here you unregister you free the memory of the container and that's it okay now let's open the the callback function that is used for plotting again it's exactly the same as the one I showed you in the presentation here is the make shape function and here is how you registered how you you run the the plotting routine okay so going back to the presentation what else can be done with plugins one you think which is possible with plugins with kernel shark 2 is that plugins can add clickable shapes here you see a visualization of the plugin which was already available in kernel shark one the visualization of the latency between sketch switch between sketch waking and sketch switch event you you probably know this so this was visualized with this small green box plotted before the actual start of the task now if you click on the the latency box this will automatically select for you the sketch waking and sketch switch event how can you implement something like this it's very simple you just have to make your custom plotting object which in this code in this case is called the latency box it inherits the rectangle object and in your customized plotting object you just have to implement the interface which is made of two functions double click and distance in the double click you just implement what you want to do here it's it's very simple as you see you just say okay I want to select with marker a the first entry which is the sketch waking event and with marker b the second entry and the next thing you have to implement is this distance functions distance function so the GUI uses this thing to decide if your click is close enough to the object or it's too far and it's not going to execute the double click in this particular case this function returns zero if you click inside the latency box and infinity if you click outside which means that you really have to click inside the box in order to select something the plugins can now register additional menus for example you here you can see the menu of a more generic version of the plugin that we just implemented this generic function can plot any field of any event so here you see from the menu we are actually selecting the same thing we are selecting the sketch switch event we are selecting the next priority field I hope you can see this and we are selecting that we are interested in seeing the minimum values so once you click apply yeah this is probably too small for you to see it but so on top of each sketch switch event you see a small dot or a small line being plotted and for example you see this line is kind of bigger which means that you have a high priority task here in every place in all places where you see just a dot means a low priority task okay how can you implement this it's again very simple you just have to provide another in plugin initializer in the implementation of your plugin and in the case which I show you you saw on the previous slide it's as simple as this you inside this menu plugin initializer you just create your the widget of your new dialogue of course you need to provide implementation of the widget and then you say okay I want to register this menu the name of the menu will be plot event field and I want to register this into the tools menu so you basically say okay uh my the name of my menu is tools slash uh plot event field and you you say to the GUI the chrono char GUI add plugin menu with this name and that's it it's as simple as this if you don't in this particular case it uses a qt defined widget but if you want to have something different you can implement whatever you want it it will just start in a new window so it doesn't need to be to read to be a qt object plugins can be used also to load data that has arbitrary format for us this is very interesting but unfortunately it goes out of the scope of this talk but if you are interested in opening your own data with a specific format that you have in kernel shark please contact us and we will show you how you can write a plugin which will make this possible so we'll be able to open your data together with the ftrace data for example or with uh I mean you can mix whatever data formats you want and open all this together merged inside the GUI with the help of a plugin uh we started the upstreaming process of the kernel shark to to kernel dot org uh we are right now in the middle of the code review or maybe in the first half of the of it let's say uh this process will take some time because steven rosted our team lead is very busy with other tasks but I guess in couple of months the we will release the kernel shark 2.0 at the kernel dot org uh at the same time a beta version is available uh here on this github repo but yeah please keep in mind that because we are reviewing the code right now uh this branch here will uh be rebased probably several times at least so if you want to use this right now you can download it from github uh but the final version will be uh will be available on kernel dot org uh we hope this tutorial was useful if you have ideas for plugins that can help you solve the particular problems that you have please give it a try and as I said we'll be very happy to help you in case you have difficulties implementing your plugin you can contact me my email is on the title slide or we can contact steven rosted it's uh the same thank you very much if you have questions please