 Thank you for taking your time to join this session. Today, I will talk to you about the extra boot configuration and boot time tracing. So let me introduce myself at first. I'm Masami Hiramatsu. I work for Renaro as a tech lead for Lena members' landing team. In the Linux kernel of the document, I'm responsible for the K-Probes boot config and its related tracing features and the tools. Okay, so let's start today's talk. So first of all, I want to make a notice. Actually, this session is not explaining F-Trace itself. So if you are not familiar with F-Trace, please check Steven's session. So why do we need the kernel boot time tracing? Because we sometimes need to be debugging and analyzing boot time errors and performance issues. For example, the major performance statistics and analyzing the driver initialization failure or debugging the boot time process and continuously tracing from boot time. So what we can do for those requests? So we actually have kernel command line options for tracing. There are the options for tracing and F-Trace. So let's see what we have now. So here are the command line options for the boot time event tracing. So TP-PrintK, for example, will show that the trace event data on the console. This is important if the kernel boot process has a bug and cannot reach login because you cannot dump the trace buffer in that case. So also, you can enable each event via trace event option. And you can add K-Probes of events also. It's like a breakpoint from the debugger. So you can add new events dynamically. You also can use the function and function call graph and other various traces like IRQs off, wake up, or preempt off or something like that by using the F-Trace option. Note that you can specify only one tracer at the time. And optionally, you can filter the trace function by its name. F-Trace filter and F-Trace no trace for the function tracer. And F-Trace graph filter and F-Trace no trace for function call graph tracer. You can specify which function you want to trace or not trace by its name. There are also some other options. For example, trace options will set the options for F-Trace. So you can pass some more options to that. For example, the stack trace option if you pass, that will add a stack trace for each event. And there are some other options too. Yeah. To check that those are options, you can see that the documentation admin guide, the kernel parameters, text. So here is an example of the good time tracing via the kernel command line. In that graph configuration, you can write the options like below. Yeah. It's nice to, it's not nice to read, but it works. So with these options, if you boot up the kernel, you will see that this, let's say, example output, let's say, this kind of output. This one needs a part of the boot log and of the kernel. And yeah, it shows that there are some more init call, start and init call, finish events, page force, are logged on the console. However, there are some problems over the kernel command lines. We have a size and coding limitations. And actually, that are the most of their architecture supports four kilobyte command line or more. But that must be a single line option. So four kilobyte single line, that will be hard to read and also hide. There are some boot rather rocker can allow us to write it in several lines, but it's still hard to read or write. And also, we cannot use the power event filters and actions instances and histograms with the legacy kernel command line. So what solution we have here? Oh, the first solution is we can use that are in it long way face. For example, embed the summer script chair script inside that are in it long way face, but it too late for the kernel boot time pressing dinner expanding the kernel command line. No, it's not so easy to write down their complex tracing options. So finally, I decided to introduce a new boot time data, which is the extra boot configuration. So what is the extra boot configuration? It's a new kernel command line extension. I call it as a boot config for short. This boot config is a plain ascii text of tree structured key value risk. So that are something like sis control dot com file, but more structured. So you can see that are the example. The boot config will actually that are loaded with the unit of the image when boot. And there is an incarnate APIs for flexible option parsing. So here is the extra boot configuration syntax. It's comfy. It actually consists of the simple key value set. You can set that values or sorry, you can set that value or array of values as a comma separated risk. And you can merge our same keywords with brace like this. So that if you have a key that word one equal value one and key that word two equal value two. This can be written as a key brace words one and a word one equal value one and what the two equal value two and close the brace. And here they are here are the value assignment operators. We have three assignment operators. The equal defined that value of the key and con equal override the previous assignment. Yeah, if you like that there are the equal was either assignment to assignment continuously on the same keys, you will be a was a error. So that's a you need to use that are colon equal. And there is a was a their plus equal assignment. It actually adds their value to their existing key as an array element. So that's a if you have a key equal full and key plus equal bar. This is the same as a key equal full comma power. And here is the latest update of the root config syntax. We can mix the keys and the values on the same key. So our parent key can hover a value and sub keys. Note that you can not match their values and the sub keys by brace, because it cannot identify a value or a sub key. Yeah, because of their value will be here. Yeah, in their, the was a their in this case we their value can be about sub key, not a value. Well, this is an important syntax. The comment. This is a comment. So you can add a comment in the root config. So if you put the comment, you can remember why that is said. Yeah. So that's all of the syntax of the root config. So here is. Yeah. So how the root config expand that the kernel command line. If the key. So that are in the boot config file. If the key start from the kernel keyword. Those are passed to the kernel command line. For example, kernel dot root equal, you ID equal, blah, blah, blah, it will be will set that there's a root option for the kernel command line. And also, if you set that there are the keys start with the unit. So pass to the kernel one line, but after double dash. So this means that these are the options are passed to their unit process, like a system day. So how we can now pass their boot config to the kernel. As I said before, I said before, boot config file will be loaded with unit Audi. So these are tools boot config boot config. Yeah, actually that are in a kernel source code. The, it's a cut as a common to handle. Was a hundred eight. So that's our. For example, hyphen a option will append your boot config file to the in the Audi image. The hyphen D option will delete it. So this figures how the root config file is appended to the unit Audi. The appended to the, yeah, I'll say, or the root file is appended to the unit Audi with a footer. The kernel will check that the footer to decode it. So if you are not that you need to set that are boot config kernel command line option to enable it. Or if you omit that this boot config option on the kernel command line. It is ignored. What, even if there is a boot config in the tail of the Intel D. Actually, that's all about the boot config. Would you have any questions with their contents so far? Oh, okay. Allison. Did you have questions? There does boot config works with flooded image three. So that are if you mean that there was a device tree. So are. Yes. So that are. If if you have a unit Audi image, it works. Yeah, even with or without their device tree. So. Okay. What's the kernel version, the root config, sorry, they're available. Is the next question actually that are, I think, kernel of 5.9 from the 5.9 or 5.0. I think the 5.9 will support that are the initial boot config. Yeah. Okay. Then we'll move on to the next slide. So there are how we can expand that the boot time tracing with this config. The boot time tracing has two types of the configuration in the boot config. So one need, yeah, actually that are. One needs are the kernel command line and one needs are good time tracing in F trace. And the boot time tracing in F trace will be here. We have our dedicated top level key. It's our F trace in the boot config. And another keys start from the kernel, which is there will use that are existing the kernel command line option for tracing. So. To support that are the power event, power event and power instance settings, we will use that the F trace keys. And also the K-pros synthetic events actions histograms available, available with the boot config. So let me show you that are how it works. So here is the example, which I showed you before. If you use that boot config, these options will be written like this. So compared with the command line version, you can easily understand what is set and why. So, so for example, you can see that the find that the buffer size is set to the one gigabyte, one megabyte. And also there is a two K-pros events defined and also in it go and exceptions are traced or something like that. Okay, so let's see more examples. So here is an example of the power instance settings. You can see that there are three different different tracers set at the same time in their different instances. The function call graph is in the default one default instance. And IRQs off tracer is set in the IRQs off instance. And wake up tracer is set in the wake up instance. And each tracer has a different buffer size. So here is the result. So you can see that the instance are automatically created. Yeah, for example, the IRQs off and wake up instances are created. And the each instance set that different different kind tracer. Yeah, you can see that the default kind tracer set to the function graph. And the IRQs off has IRQs off and wake up instance as a wake up. And also the buffer size is set in a different way size. And if you cut the trace file in each instance, you can see that the result of the tracer. For example, IRQs off instance, if you cut it, you can see that the tracer shows that the maximum latency of the interrupt off period. Like this, this shows that three million, yeah, about the almost four millisecond someone stopped the interrupt off. Anyway, here is another example. This tracer traces all function call graph in a specific function. Yeah, as I said that the function call graph tracer is the powerful tracer. We can see that the precise information to filter the result we had a function name filter. But that is sometimes doesn't work because we don't know what function is called inside that function. So combined with the power event actions, we can limit the function call graph between one event and another. This example uses the K-prox event to put that a pair of the event on PCIe proc init function start and exit. And trace on function at start action actually to start and trace off action at exit. And also we stop the tracing by default. You can see that the first line said that are the tracing on call zero. This means that the disabled tracing by default. So this mean I'm in total we start to call graph tracer at the start of the PCIe proc init function and stop it at the end of the PCIe proc init. There is a question in the question box. Do you like to answer that now? Okay. So how much overhead is there with a very minimum trace with this trace? Yeah, it's a good question. Actually, this depends on how much trace function you are was enabled. For example, the function call graph tracer will get more overhead. And if you enable that several different tracers, it will make more overhead. Yeah. Something like without actually that are in my QMU case, I saw that are if we I just enable that some events, it will just, I think it's just a 10% or something like that. But if you enable that function call graph tracer, you think that the more than five times slower than normal case. So it's totally depends on what tracer you enabled. Okay, and another question from Leo. Was there configurations for ftrace and kprose to start off from boot config. These two enabled after the kind of booting? Yes. That's a good question. Yes. The setting will be continued to the other was after the kind of route, because we cannot decide when we will say clear that we can clear that the settings. Currently, it will continue to set the same configuration after booting. So you need to clear that was something like that. So I recommend you to set up the summer event which was a trace of action. Then all the tracers will stop at that point. And also another question is after the kind of booting, how it handle the multiple instance. Oh, yeah, it actually that you can say you can usually use that there are multiple instance after kind of boot. So if you after the kind of boot, you can get the trace data from the each instance. And if you don't need any more, you can delete that instance. Yeah. Okay, there are another one needs are from Mikhail. Do we have a similar functionality for K probe K exit case. Sorry, I don't know about that the K exit cases. Maybe we can. I think that are if they're cake, the other was a Unitality can handle that the Unitality image, I think it's okay. Yeah. But not sure I didn't test it yet. So I will try to test it. Thank you for the question. Okay. So the next one. Here it's there. Okay. The result of the partial function code graph. Here is the result of the function so that you can see that the trace start from their PCI proc in it and stop at the exit. And here is the more complex example of the histogram. Yeah, actually that are event histogram can make a histogram of their event parameters, usually, but if you use it with a synthetic event, you can expand it to their histogram of their like it up to time between several events. Here is an example of the search histogram. It major that the time of the each unit to call functions. And make histogram on their unit call latency event, their new synthetic event. So you can see that are there in it calls that actions will set that are TS0 with their timestamp. And in a finish event there, you can see that are there. It will make a latency, the lot variable by using that are the elapsed time from their TS0 to the current common timestamp. And if they're the master to the start event, it will call that it call latency new event with the function name and latency. And in to call latency make a histogram of their given function name and other the latencies. So here is the here is the result of the histogram. You can see that are the function name and ERAPS time under those are sorted by the ERAPS time. So you can find that are what kind, what in it call function takes how long time to do that. Yeah, in this case that are the late six set algorithm will be take a long time. This is because that are this in equal will run a say benchmark function. Okay, let me explain the options which you can use in the boot config for the boot time tracing. So here is the other options which start from the kernel. These are the some options borrowed from the kernel monline. So I call it Grover options. So you can use that the TP print K and other options. Yeah, one question coming from Sarah. Is this initial initial function tracing same as a unit call debug? Yes. Yeah, good question. Yeah, actually that are this one is almost same as a unit code debug. But you don't need to enable that are the unit code debug with this feature and also we can filter it on the online actually with a good time tracing. So good time tracing is including that are such kind of features. Thank you. Okay, and here is the instance options. Those are started from the after race. So that are those options are setting that the instance rated parameters. If you specify any instance name, the instance is created automatically. Yeah, so that are if you set that they're tracing on crowd on their instance who will create the full instance and tracing off. Yeah, so that are you can set up that are some options tracing on press clock and buffer size. And there are more instance options. Like Alex snapshot or city masks, event tracers after a stress filters after a notice is or something like that. Most options are similar to the options in the current one line, but extended to support multiple instances. And here are the power event options. Since their event is disabled by default, you need to specify the enable without any values. So also you can configure the filters and actions. As you saw in the examples. And also, there are K probes events and synthetic events. So, please focus on the group name. So if you put the group name, sorry, if you put the Cape Robes as a group name, if their event will be the Cape Robes events. And if you put the synthetic as a group name. This one will be your synthetic events. So that's our Cape Robes is kind of their dynamic event and kind of breakpoint. And the synthetic event is a kind of a stuff event, which is used for storing the histograms, usually. Okay. Anyway, if you look at the three of their boot time trace options, most of their F trace started options are actually that are corresponding to their trace surface interface. Of course, some sub kits changes their work changed for the simplifying the name. For example, if trace.instance.full.options, this will set the value to the trace surface thrash instances thrash who thrash trace options. And some other options, which which is that a trace surface does not expose user kernel command line option with the kernel keys. So for example, you can set that the tp print k or dump on oops. Those are not exposed by your trace surface. So I will use that reuse their actually that the kernel command line option. So that's our syntax for their boot time tracing. Would you have any questions about boot time tracing. I have one question. Yep. If you were to recommend. This is a super set of in it called debug also right because it includes in it called debug. So when would we, when would you recommend using boot time tracing versus in it called debug. Do you have recommendation. Obviously, if you see early boot problems, you want to use boot time, but I just wanted to see what kind of debugging or tracing would be appropriate in situations, which one would be better. Yeah, that's a good question. I think that are if you have some suspicious functions for the, yeah, for the debugging target. You can use that are there was a the second example technique. So you can start with their function call graph, the partial function call graph on their function and understand what their other was a was a function execution pass is done in a in that function. And yeah, you can put that summer cable of the events on that and get that summer or say some local variables, values, how it changed or something like that. But if you just focus on our unit call was a her debug. Of course, you can use that are the unit call debug option. Yeah. But I actually that I if you was a one filter out and focus on our summer. The result. You can in that case you can use that are this good time tracing. Because it's it can harbor the filters. So filter out there are some latencies or something like that. Is that good for you, sure. Yes, thank you. Yeah. Okay. Okay, the next question is a song a song and which it's a we don't use any to our MFS. Is there any other way to use that the boot country. Actually, that's a good question. Yeah. Currently, we don't have any way other way to use that the boot config without intram MFS. So, sorry, you, sorry, you need to use that intram MFS for using that boot boot config. I'm, I'm planning to, to add some more. That's a was a feature to add a boot config. Was a what appended the boot config to the kernel image. But it's not was a finished. So, if you are interested in please help me to do that. Thank you. Okay. Was a or these are the additional question was a informations. Oh, questions is coming and the cows question. Why are you or try type was a type to any term a face. Yeah, actually that there is no was a no good space for their. Any drum. Other than any term a face. Actually, at first. I tried to use that load other boot config without me to MFS. But in that case, I need to say other was a change their boot loader. Like a changing that grab or something like that. And also the kernel need to be changed to say identify that they're the boot config area. So it depends on their. Except was a outside of the kernel. So at this point. I'm using that they're the unit on my face, because the unit on my face handling is, you know, all the car. All the code is in the kernel. So I'm decided to use that. Okay, that's a question from their. From the adrenaline. This new kernel boot config type with these tracing options is ideal for monitoring micro services at startup, or maybe more focused on their monitoring systems security at boot. Oh, yeah. Yeah, I'm, as I said that are in the first order of this session. I think that are the boot time tracing is. Yeah, if you are saying, you know, there are many ways to use that. Sorry, there are many reasons to use that. So I don't think I can say focus on that. There, those are the usage is. Yeah, if you, you know, want to use that for the system security. Yeah, you can use that. And of course, expand the boot config to support that. Like, making a secure secure security. Was a top level case, and you can add some more options. For that. Is that the question. Okay, the next one is the room. In the case of boot failure. Will there a crash file, or the kernel be stored in the kernel dump on variable. Oh, kind of sudden. You mean that are there the boot on kernel dump on variables. No, I'm not so sure about that. Sorry. I think that are if you are usually use the K dump or something like that. You can get the summer or say boot. I'm tracing. Was a trace buffer in the dump file. Yeah, I, I saw some script to get the F trace. Trace buffer retrieving from the crushed. Was a kernel image. So maybe you can use that. Okay. So this one needs additional information for their good time tracing. So for, for the purpose event. I think, do you think that are you need some more assistance to to set it up. If so, yes, we have a path probe, which is a part of the kernel path to it now support that are good, good config format. So you can define that the new K probes event by your source level information at below. So this example, actually that are put their probes on their 55 fifth line of their C groups in it function with SSID local variable. So as you can see that are that's that's boot config and that's that define will will show that are the definition of their capers events in a book boot config format. So now you can. Yeah, easily to get that they're they're capable of definition. Okay, our next slide is there. So, and also, if you feel that it's not easy to write down their boot config. There are two script to help you to make it. Yeah, one needs our efforts to be gone. This is a combat current after settings into boot config file. So that if you set up that the address, the trace service, then our runner after is to be gone script. You can get the boot from your file. And also, there is a beacon to after this SH is vice versa. It's, it read their given boot config file and make a share script, make a share script to set up that after this. So that's a apply option, which tries to apply that those, those commands to, to after this or check. And here is another information. I'd like to touch that the start timing of the boot time tracing. Yeah, actually, that I since they're a boot time tracing depends on the basic traces and also summer trace events was capable of the event. So we need to wait for those initialization. So there are the boot time tracing is actually started with their started at the last of their core unit call. Anyway, most of the corner initialization has been done after the quiet call the quiet call usually used for the initializing that the core functionality, very quiet functionality. So that's our, for example, the subsystems fire systems, or device. The servers will be initialized after that. So those are the traceable. And here is the initial, say initialization order. Sorry, internal initialization order of the, the boot time tracing. Also, note that the boot time tracing is initialized in this order. So not what you write. So, for example, even if you are enable the was an event. At first, if you are you are good to come to the right that the event enable at first and settings buffer size. It will be your was the event was, sorry, the buffer size is set at first. And afterwards there is an event is enabled. Always. So here is the summary of the usage of their boot time tracing. In the following steps, you can learn the boot time tracing on your kernel. So, at first, build and install your kernel with the config boot config and config boot time tracing. And build the tools boot config boot config. Actually, that you can learn make tools boot config will make a boot command. So, set up the F trace as you like. And generate the boot config file using that this F trace to become share script. Yeah, here. And applying the config file to the unit of the image as well as boot config high from a and in it out the end reboot machine with the boot config option on the kernel command line. Then you can get the other good time tracing is working on your kernel. Okay, that's all. All of their my talk today. Would you have any additional questions. You can use your voice or question and Q&A box. Okay. Okay. That's great. Thank you so much. Just want to do one last call for questions. Yeah. If anyone has a question from a Sami, please go ahead and type it in the Q&A box or the chat. Or feel free to unmute yourself and and use your voice. I'm sure do you want to talk through the different resources available. Of course. Okay. Okay. Yeah, thank you very much for joining this session. Yes. Thank you. I'm sorry. Oh, there we go. Perfect. Yeah. She will we'll just give a quick run. Oh, we have a question. There you go. Okay. I think that the last question will. Yeah. If any term is error happened, then we will. Not able to run this. It's right. You mean that's right. If they're in it from a face other summer. Was it their format errors or something like that. In that even in that case, actually, the boot config can work. Because that's our, we will get the boot config before was a passing or was a. Decoding that the image. So that's our, even if they're the image is broken. The boot. Unless that's a boot config isn't broken. Yeah. The boot config and the boot time dressing is work that can work. Okay. And please, how do I get there more resources to help me because I'm. Junior in the world. Yeah. Okay. I actually that are in this slide. I showed that the summer. There was a documentations. What was the year actually that the documentations are in there. The kind of source code. Sorry. Kind of documentations. There is some more. Documentations about the boot config. In there. The documentations under the. Admin guide. And also the boot time for the boot time tracing. You can find that the documentation. Was it find it under the documentations. Slash tracing slash that the boot time tracing. The RST. So you can find that those are. The resources under the. The kind of source code. Sure. Thank you very much. Yeah, sure. Yeah. What they are shared. They're eating their chat. Yes. I put the link to. In the chat for you to see everybody. Yeah. Okay. Thank you very much. Is there one more question there? Okay. One more for you. Yeah. At the bottom there. Is device three or ACPI very. Alternatives to the interim. Is that true? Consider. Yeah, actually. At first I tried to. Change the device three. To support that the boot time tracing. But it was great or rejected. Because that the device three is only for the hardware. And not for the software. So we cannot use that. And the ACPI ACPI is actually. Was a made by the hardware vendor. So I cannot use that. So, so that's the, that is why I introduced. There are extra boot configuration file. Okay. Thank you. Thank you. I'm sure. Also. Yes. Actually the reason you would want to not tie to ACPI. It would be that it would be. Sorry. I cannot hear you. It doesn't depend on the firmware. If we were to. Not to tie it to device three or ACPR. The. I was wondering if that's another reason. To not have a tie with the format. Yes. That's right. So that's our current three. Some protons to use that the device three. From the file. But the recent three more modern. The device three from the family. So we cannot change it. Yeah. Correct. There seemed to be another question for you. Oh, yeah. Okay. We have time. So don't worry. Okay. Okay. The question is. Those are tracing option. Straight to go. Or permanently in. There's some new kind of version. Very. I hope so. Actually, I hope so. I will. Expanding. There was either. The options. Because that's a trace. Introducing that there are new features. So that's a. Actually that depends on the trace. API's. So unless there are trace. Fs. I hope so. I hope so. I hope so. I hope so. I hope so. I hope so. Okay. But one thing is for you guys. So unless there are trace. Fs. Does not change. This. These are parameters will not change. In a new corner version. Thank you. Okay. Okay. Sure. Would you or try to. explain that there this material. Yeah, go right ahead Shua, thank you. Thank you Masami for doing this. Thanks everybody for attending joining us and asking great questions. We hope it will be helpful in your journey to learning more about effective and productive participation in the open source projects. This is why we do this. We offer this as a self-study for not only new developers but also people wanting to know a different area in the Cardinal or about open source. We have various, we have webinars ranging from open source to software engineering and Cardinal specific. And we also leave you with the other resources, the next one Foundation Mentorship Program and outreach internships and training opportunities on the slide. These are all links, you can go and learn more about them. And we also have Linux Foundation events to attend a range of, we have educational content that you can learn from attending these events. And thanks again for joining us today. Thank you so much everyone. Thank you Masami, that was wonderful and hope everyone has a great day. Thank you. Yeah, thank you very much for joining us and I hope you have a great day and great good time tracing. Thank you. Thank you Masami. Thank you.