 Hello, is that okay? Okay. Hello, everyone. Thank you for joining this session. This session shows that the kernel good time tracing feature in the recent Linux kernel. Let me start. I'm Masami Hiramatsu. I'm working for RINARO and RINARO members as tech lead for social next landing team. Also, I'm a maintainer of the K-Probes and the rated tracing features and the tools. For example, Power Probe and F-Trace and the dynamic events. So, okay, here is today's agenda. The agenda, yeah, according to the agenda, we'll start from our background and the extra boot configuration and the boot time tracing. So, why we want to the kernel boot time tracing? The reason is to debug and analyze the boot time errors and the performance issues. For example, measuring performance statistics to shorten their boot time or analyze our driver-in-insured essential field errors or debugging boot time process were continuous tracing from boot time to user space. So, what we can do now? Currently, we already have some kernel command line options for tracing. For example, setting up options, output to printk and enable events and tracers, filter inks, and K-Probes events, et cetera. So, you can see these options in our kernel parameters.txt file. So, here is the example of the kernel command line parameters, like in the graph config file. So, you can pass the many parameters like this in one line. So, but there are some limitations. One is the size limitation. The kernel command line size is small. The minimum size is 256 bytes. But usually, we expand this to the whole queue byte. But who want to read or write the four queue byte length single line options? And also, it supports only partial features. F-Press has many features, like par event filters and actions, instances, and histograms, but those are too complicated to write into the single line. So, we need more flexible options. So, one is the solution. The easiest one is expanding the kernel command line, but it's not easy to write down the complex tracing options on single line. Then, how about expanding the device tree? It's well documented and structured data. I actually tried this, but this was not because it's only for the hardware description. So, finally, I introduce a new boot time structured data. This is extra boot configuration. And, okay, so one is the extra boot configuration. The extra boot configuration is a new kernel command line extension. We call it boot config for short. Boot config is a plain ASCII text of three structured key variable list like this. And, sorry, this supports up to 32 queue bytes or 512 nodes, sorry. So, this supports up to 32 queue bytes or 512 nodes. The nodes mean that the number of the key words and values. So, here's the extra boot configuration syntax. It's a simple key value set. The key consists of the single or multiple words and the key may have a values or array of values. The values are treated as a string so you can use double or single quotes. We also have some assignments equal defined the value of the key and common equal overwrite the previous definition. And also plus equal depends a value as an array element. And here is the important features or feature of the extra boot configuration. We can use our structured key. This means that the same key words can be merged with race. So, for example, key.word1 equal value one and key.word2 equal value two can be written as a key brace and word1 equal value one, word2 equal value two and close brace. So, that's our, and also this tree structure can be nested. So, this allows us to write complex nested key values in a simple form. So, how is the extra boot configuration passed to the kernel at boot time? The extra boot configuration file will be loaded with the int ROD image. The boot command, yeah, there is a boot command and this command appends our configuration file to the int ROD image as this. So, you just need to boot the kernel with this int ROD image and pass our boot config kernel command line. Then the boot config is enabled on the kernel. So, here is, yeah, how to use the boot config command. This commands operate that boot config for int ROD or int RAM-FS image. The boot config having a option applies their given our config file to int ROD image. And the boot config minus D option will remove the config from int ROD. And the L option will show that what config is currently applied. And in addition, if you just pass our configuration file to the boot config command, it passes the file and format it in the key value list. Also, boot config supports the ProcFS interface. So, when you boot up the kernel with boot config and it's enabled, you can see what boot config is applied via Proc boot config, like a Proc command line, like this. And this is our key value list because it's an easier to be handled by a share script. And, okay. So, and also the extra boot configuration allows you to expand the kernel command line. The configuration which start with the kernel or user as a part of the kernel command line. So, for example, if you apply this boot config, these options will be shown in the kernel command line like the below list. Okay, the extra boot configuration also provides kernel APIs. So, these are functions start with XBC for the kernel functions. This means that because after boot up the kernel, these APIs are gone to reduce the memory footprint. There is a main data structure which names are XBC node, which is a tree node that represents keyword and value. So, there are many APIs, but yeah, here is the main APIs I picked just for like XBC find value will find a value from keyword or XBC find node will find a node which will be keyword node or value node from keywords. And also we have some iterator macros. So, here is an example of the boot config kernel API usage like you can find the value node by using that XBC find value. And also you can check that the result and keyword has, there is a keyword but the non-value key or if you have a get a V node returned, this keyword has a values. But yeah, you need to check that these are array so that you can use the array iterator for that. Okay, so next is our boot time tracing. So by introducing the boot config, so we can implement the boot time tracing on it. Here is the boot time tracing options in a boot config. Boot time tracing options start from ftrace or kernel. The kernel keyword is for Grover ftrace settings. And also this support that the par event and par instance settings like this. We also support the K-Probes and the synthetic events. You can see that there are details in this boot time configuration, boot time trace, RST text file. So here's some more boot time tracing parameters we will check. Like a Grover parameter, for example, that's a Grover parameters, kernel.tprint.ql is a flag or which output that are trace event data to the console. The par instance parameters, for par instance parameters, we can use that are ftrace.instance and instance name and buffer size. For example, buffer size will set up the buffer size for this instance. You can skip that instance name, then this will change the set up the parameters in the global ftrace instance, the default instance. And also we support the par event parameters. So that's our ftrace instance and event group name, that event name and enable, for example, enable parameter will enable their event. In this instance, for example, and also we have our instance, sorry, event group name, event.actions, which can allow us to set up that are trigger actions. And if you set a cluster K probes as a group name, you can control that or define that a new K probes events with a probes parameter. And also we can, if you set the synthetic as a group name, this will define a synthetic events with a specified field. So here is one example of the root time tracing. This one will set up the global ftrace instance with these parameters, options, set up the options, but for size. And also other K probes events. K probes, for example, we face the sheet events in a probes with this probe definition, like a kernel lead function, argument one and argument two. And we also set up the filters, which mean that the common PID, if they're below 200, it will enable the event as a trace data event. And also we have our kernel global parameters, we can set. Here's another example for synthetic events. The ftrace events, dot event and the synthetic event, this one will define a new synthetic events for init call and set up the summer histogram actions on it. So this actually will make our history of the init call callbacks execution time. So each, how long it will take for each callbacks. Okay, another interesting example is here. This shows that how to make a partial function call graph in a specific function. On the function, so that's our global maker, user function graph tracer, but tracing by default tracing is off to disabled. But on the specific function entries, entrance will put their K probes event. And at that point, we enable the trace. And also the function exit, we put the return probe on it and the trace, disabled trace again. So that's our, in this between these events, the function graph will be enabled. So we can get our partial function call graph. By the way, with these congregations are still a bit complicated. Yeah, because that's our F trace is very big. So you may need some help to write it down or test it. So here we have our two, to help us shelf script for good time tracing. These are under the tools, boot config scripts. The one needs our F trace to be conf.sh. This converts a F trace current settings to a boot config file. And this is good for making the prototype boot config by setting F trace interactivity. Another one needs our be conf to F trace. This converts given boot config file to shelf command to set up the F trace. Of course, if you use the apply option, which will try to apply the generated command to, so that you can check that it can pass the error check or something like that. And this is good for checking your boot config before reboot. So how we can start using the boot time tracing? So at first you have to enable that the config boot time config, sorry, of config boot config and the boot time tracing was enabled and build the kernel and install it. And I also built the tools, boot config, boot config. And I set up that this kernel tracing so that the F trace setting up as you like. And I run this tool F trace to be conf.sh to generate the boot config file. And yeah, maybe you will edit it and apply the config file to the init.d image as follows. And boot kernel with boot config on the kernel command line. Then you will get user the boot time tracing. So finally, I'd like to explain that when to start the boot time tracing. Until Linux 5.9, the boot time tracing starts from the end of FS init call. So only device init call or later can be traced. Yeah, but this one is a bit too narrow. So yeah, to expand increase that traceable code, I decided to move the kernel K probes and also the boot time tracing initialization routines in a more earlier stage. So finally, from the 5.10 kernel, we can start the boot time tracing from the end of core init call. So that are almost all drivers and more device frameworks and file systems can be traced now. Okay, so we have some time, so let me show that the demonstration. Okay, can you see my screen? Okay, we have some boot config file. So that are the examples. Applying boot config, boot config file, down to this boot time tracing to init form FS, sorry, then. Yeah, this will use the QMU to boot up that are, we do it up. Okay, then you can see that the VFS read events now were output to the print K buffer. And also, okay, tracing cat size Q byte. Okay, we can see that the buffer size is set to one megabyte. Then also we can use that the boot time to be gone. This one is the number two, one, so that are the making a histogram. Or let me try boot config. A, boot time to be gone to init all the intram FS, and run it again. Okay, so we will check that it was past correctly, boot config. Yeah, this one is set past correctly. So, DD, this is tracing. And the histogram is in, should be in there. Synthetic events, so that are synthetic events it call histogram. Yeah, so you can see that there are the latencies yeah, it's shown in this histogram, yeah. So you can, as you can see that there are, you can check, change the options just with their, with using the boot config command. And the lat need, for example, this one is the sad one, so that are making a partial function call graph. This is kind of the back tracing. And we'll see the trace buffer. Okay, it just started. The function graph call graph it's made, but it just started from a PCI proc init. And this one is the city one. And, okay, the, in the, at last. Yeah, actually this one, stopped by your K-let probe. So that are, you may see that the trampoline handler or the K-let probes call graph, but this one actually finish at the PCI proc init. The end of the PCI proc init. Okay, that's the demonstration. Okay, go back to the slide. So, here's today's summary. So today I explained that there are the extra boot computation is introduced, let's say, introduced in their recent kernel. And this one, yeah, I showed that there are complex and the many boot options are supported. And also it does not need any upgrade, update the boot loaders because it's loaded with the init.ld. And also it, let's say, provide kernel APIs, yeah, and some commands. And also I explained that the boot time tracing supported on our boot configuration. So it, and explained that there are five options, five instance options, and the histograms, et cetera. And also it shows that there are helper, shell script, upstream kernel support that are in it called tracing. Okay, here is the current status. Yeah, we almost done this work. So you can use that after this to, yeah, boot config and also a boot time tracing on the recent upstream kernel. Maybe your start boot time earlier tracing, early time boot time tracing will be supported in our 5.10 next release. So here's a future work. Yeah, I will enhance the histogram syntax and also some boot time, boot point support on the boot loaders, if possible. So that's all for today's session. If you have any questions, please feel free to ask me on the chat or Q and A. Okay, I, yeah, I got several questions or two questions. Yeah, one is in example number three, does it capture all the nested function called in the PCNIT function? Yes, so that's as I showed in the demonstration. This one actually that's captured all the nested function from the PCNIT function. So that's you can make the function call graph on this part. And oh, yeah, and also is that boot config API usage for other kernel called like drivers? Yes, yeah, as far as the drivers using the init functions, yeah, it could be possible but note that the boot config APIs will be your freed after boot. So that those are drivers has to use, mostly has to embedded in the kernel at this moment. Maybe we can remove that these limitations if there are some drivers want to use that the boot config APIs directly. At this moment, actually that's my use case, the boot time tracing is only for boot time so that I decided to make it just for init, let's say init functions. Thank you, Doryu. And is it possible to demo? Yeah, I just demonstrated there, yeah. And okay, let me show that, oh, great. Let me show it again in our boot time configuration. Sorry, boot time tracing number three. So that this one needs the boot time tracing number three is here. So that it will make a partial call graph tracing and just apply it and boot up. And you can see that it will take a bit longer time because the function call graph tracing is still running in background. So anyway, this kernel tracing, sorry. Data tracing, yeah, trace. We show that there are only the partial functions call graph. Yeah, actually that are the mixed with the other CPUs so that in that case, we start the CPU zero. So you can try pass CPU, CPU zero trace. Then you can get the trace data only on the CPU zero. Yeah, it shows that there is a nested function call graph in the end again. Okay, so that's all from my side. So thank you very much for your time to join this session. If you have any further questions, please email me or ask me on the Slack. And yeah, enjoy the rest of your conference. Thank you very much.