 today's talk is what's new in kernel shark 2 and the reason why I submitted it because I like the rhyme Oh, is this thing not on there? She would help if I turn the thing on There we go. So what is kernel shark? Whether or not you guys know what it is I'm expecting you probably do know what it is if you're coming to this talk if you want to know what's happening with number two, but I will go on it's a front-end GUI in our utility for trace command I'm not going to talk about what trace command is because that's a front-end utility command line interface for ftrace Ftrace is a kernel tracer inside the Linux kernel The um, it basically what it does is it shows plots of cpu Events where each task Is a different color so you can see where the tasks are You and also has plots of tasks where the color changes depending on what cpu that task happens to be running on It below the graph it shows a list of events where you can map the events to the little ticks on the graph So I want to go back a little history of kernel shark It was created in december of 2009 So it's over a decade old. I mean that was that makes it Almost 12 years old now. It's a couple more months It was originally written in gtk version 2 um It was mostly used to debug the scheduler because with this graph you could debug the scheduler and there's a nice little article that I wrote on lwn Using kernel shark to analyze the real-time scheduler But the problem with this was it was maintained as a idle task. Now. What does that mean? It means that I was never told to create kernel shark. I was never In my career to say okay Maintain kernel shark whatever blah blah blah. So whenever I had nothing else to work on I worked on kernel shark So that was how it was developed And this is what it looked like back then. This is the gtk version of Kernel shark you could see the lines Little ticks the graphs is each cpu each color is a task down below it is the task and each color is a Different cpu when there's nothing when there's nothing there It means the system is idle either that the task is not running or on the cpu. It's an idle You could do little markers on it and you could there was a marker a and a marker b and when you connected to you See the delta between it But the problem was it really needed some loving It like I said, I I developed it without really focusing on it I worked on it in my spare time so which means that when I added a feature it was rushed I really didn't do much to it And I kind of programmed myself into a corner I made some really poor design decisions that made it really difficult to update it And when I was starting the more I did I was adding more hacks on top of hacks on top of hacks And it was really ugly It was also extremely slow when you zoomed in it could take a long time because it used, you know on algorithms to determine the drawings And had a really strange and archaic Interface that was totally non-conventional So it was Really not as intuitive to use although I heard a lot of people still I still see people using the gtk version which I'm kind of laughing at So it was useful to a lot of people So then in 2017 I joined vmware Well, how is that relevant? Well, vmware was trying to make an impact an open source community. They my job at vmware was Basically to help vmware to become an open source friendly company And they said hey, steve you have anything that you could work on that could be Helped out Same a little while ago. I got someone else. Yordan Koderdorf joined I know I mispronounced his last name he gets mad at me for doing that But he joined vmware at the same year later on So we said hey Let's give a developer a developer the task To work on kernelshark Yordan said to me. I don't know gtk never used it But he knew cute or qt And because gtk 2.0 was deprecated and it really needed to be rewritten in in gtk 3 I said well gtk is notorious for not being backwards compatible And I'm getting sick and tired of it and I didn't want to write it gtk 3 and then maybe years when gtk 4 comes I'll have to rewrite everything again. So since it had to be rewritten. I told him. Hey, you know cute go ahead have at it And in 2019 Yordan came out with the finished completed version of kernelshark 1.0 And this is the cute version looks a little prettier if you ask me, but basically the same And that was because the 1.0 version We said once cute gets to the level of handling all the features of gtk version Then we'll release 1.0. That was the milestone once it's equal to the gtk version We'll call it kernelshark 1.0 because before that kernelshark was 0.9 So Yordan did more than what I tasked him to I just told him go ahead and Do the gtk features But he Really looked at the design and he created an abstraction layer between before the kernelshark was just an interface to trace command now it just Is an interface for what it displays and trace command which creates a trace.dat file Is a lower layer that feeds into our upper layer and that feeds into the graphs He also included ways to save sessions So if you're in the middle of doing something and you say or if you're like at a point where this is like I want to I want to get back to this spot because you zooming in zooming out you might lose It was hard to keep track of you could go and say save this session And he also used more conventional interfaces for zooming in zooming out moving the graph around So it was more intuitive to people who understood GUI interfaces And most importantly he threw away the oh Big the big o and logarithms and made it into a log n algorithm So the old way this is gtk version When it drew these lines It would actually search it will go right down the data path and at every single event It would draw a line Well, actually it would say he'll look at the pixels It'll do a division by how many events by how many pixels are there and when it got to a new pixel It says okay draw a line then search for the next The time that would be at the next pixel So we searched the events in a linear order until I found that one and then drew a line And then searched the next one and found and then drew a line and so on and so forth so This was approximately 873 pixels for this timeline so It looked at you could say average of 27 events because you know the 23,891 divided by 873 Is 27 so it looked at 27 events for each before it got to the next line now what Jordan said was this is a waste of time because Well, why don't we just search for it? So if we have 23,000 events And he knows what the time of what the next pixel is because it might not have any event in the next pixel So he would draw the one line and then he did a binary search of the whole data path to find the the event that belongs at that next Pixel and he draws the line there and then he did a binary. So each one of these lines did a binary search So using the same 23,000 events with 873 pixels It looked at approximately 15 events not 27 15 events smaller per each pixel And I know this because 2 to the 15 would be the binary search is greater than the actually I have a typo there says 27. This must be 23,000 so a total of Third so I multiplied 873 by 2 to the 15 or not to the 15 but 15 events 873 times 15 is 13,000 events instead of looking at the entire events That's yeah, it looks like a little bit of increase. Well, let's look at 8 million events So in the linear algebra linear algorithm, you know each pixel we looked at nine over 9,000 events to draw that next pixel When I did this I In like I changed the size to change the size or to recalculate drawing it takes about three seconds to draw this The olog in version of this Looked at about 23 events not 9,023 events less than that so It was much It was actually much quicker to do that and once he did that like I said here total of 20,000 events to draw this instead of 8 million And because it was so quick He was able to do You could scroll in and scroll out and if I get trying to do a demo here I will show you that you could scroll in and scroll out and he actually could zoom in and zoom out smoothly So the abstraction layer of this is he had this buffer that he called buckets and For each bucket is the event Where basically represents a pixel And he would load the trace data into this abstraction layer only recording the information needed to draw the the graph So each of these little lines are pointing to information in this abstraction layer Which means by the way that this abstraction layers means that it could Look at anything. It doesn't have to be traced that that file. We actually have code to look at perf Data and we also have code that looks at the you know, ESXi VMware's proprietary data and all that The list information since it shows everything and shows all the information in the data This actually goes directly to the trace file. You need to have an interface that says okay to draw the list I need to go directly to the trace.dat file What's really nice about sessions though is you could save and restore where you left off You can restore the last session You can save multiple sessions and then retrieve them later So here if I like this I just go say if I open up open this up. That's right if I open up Colonel shark and go to file sessions restore last set or restore request import session. That's it. I can't see I just take my glasses off so I can see Gives you a window you could pick it's a json file And then boom it takes you right where you left off You can do this from the command line as well You could say okay colonel shark dash s pass in the file and open up colonel shark right where you want to start it And say if you're using colonel shark and you close it and you're like oh wait I want to go back to where I left off you could do colonel shark dash l which is you know last And it'll bring you back right where you left off so Colonel shark 1.0 also introduced the idea of plugins This is something I was actually saying that need gtk version needed for a long time But I never got to it. It was hard-coded everything that the gtk version was Set by me writing code and that's it you you know What you see is what you get The v1 added plugins which The two plugins I mentioned here is one that plots some Special functionality is added by the plugins and one of the things that we add to mist events. I'll talk about this so When you looked at the task of gtk version So these are all the tasks. I have a migrate test which the migrate function here I think I used in my article for lwn Which runs a bunch of tasks and it forces migration and you can see how things migrate around depending on priorities And I use this because this is one of the things that colonel shark really shows well And the top level the one task because different colors Members shows the migration different colors or different cpus because these are all tasks to the left They're called all the migrate threads that are running and each color is when it got pushed off to another cpu That's you could see there and I could select one time But this is what it looks like and what it gets when a task gets preempted If it schedules out where that green line is in the beginning right here This guy scheduled out because it's blank here Supposedly and then over here it was preempted it switched to a cpu switched to another cpu And here this red line means that it wanted to run but it was kicked off the cpu by something else And the thing else ran probably one of these guys was that's the same color right there And yeah, then this guy took over there and then it ran here So you could see how it was kicked off and it wanted to run but something else came in The green little line means that this guy woke up at this moment and this is the green may woke up It's on the run queue and when it turns solid it means it's on the it's on the actual cpu This is The qt version of the exact same data which is kind of interesting because I noticed that Here it shows that it actually scheduled out and it was a bug in the gtk version It should have been read there. It actually wanted to run and it wasn't Where the gt this version it actually wanted to run this whole entire time. It was just being forced out So the gtk version had a bug, but this is a schedule with a plugin That means that this isn't the code that does this is not actually in the kernel shark proper is actually Plugin that when you download kernel shark source code it comes with the source code and when you install it It's automatically installed so it's there when you see it You won't notice it's a plugin because it comes with the features But it was added as a plugin to extend kernel shark And you could disable the plugin and which I did and then it disappeared those this is what it looks like without the plugin notice something here When I was running this I thought this might have been a bug in the plugin Hey when I'm like what the hell why did that thing disappear? So I had to investigate because that why you saw that it was something here and then something disappeared And what happens is I clicked on that line that disappeared if you look down here You'll notice that The task running is some pool gsd smart tcc. Um, see This is not the task that was being this is not the migrate test so This guy wouldn't be traced normally But the plugin says trace it because if you look down here This line It actually is the sked switch event and it sked switched and it woke up the migrate task so That normally this wouldn't be traced because when you trace say Or if you want to show the event you're looking at all the events that happen under that event But a sked switch the one that actually records that is the previous event Not the event that got switched to so The guy that was running here Did the event and the sked plugin caught that saying oh, we want to print we want to include this in our Work, so you actually got to see it get scheduled back in Missed events is nice because you see all these blanks here You might say oh, this must be not running Well, I ran the function tracer which Basically is a fire hose into the ring buffer and it's really hard for the reader to keep up with the writer And it drops tons of events And if you're looking at visually you kind of want to know about those Events so we created a missed event plugin that creates these little flags whenever there's Telling you that there's enough there's these events You don't know it may look blank, but or the fact that there's events missing and if you click on it It tells you and it also tells you there's 2,090 900 events that missed so What about kernel shark v2 heck that's the name of this talk You're finally over gate to hear enough of kernel shark v1. That's not what this talk is about, but I'm hoping that was useful We need to do more features that the api for the plugins weren't that great So we actually had to rewrite the code for the plugins not we you are done So it's not quite the plugins for 2.0 are not quite compatible with 1.0 I'm so much really big into backward compatibility, but since I didn't see many people writing plugins I was like, I don't think we're gonna hurt anyone if we do this update So risk in versus reward the risk is we might break someone's plugins But I haven't heard of anyone writing plugins for the v1 kernel shark So let's re let's break the api so that we could have a better interface and add it to v2 These plugins are even more flexible and can do a hell of a lot more It could be associated to different streams from different files And this is what makes kernel shark special One of the new plugins and then I'm going to just talk about the plugins because really the whole rewrite of kernel shark 2 Was implementing these plugin facility. That's basically all kernel shark 2 was great created new plugins And also add a few minor features But the plugins is basically we'll see what it is and and we included some nice plugins and we're hoping that people might Write more plugins So there's the event field plugin Which shows the maximum minimum As a marker so if you want to see the maximum value of a field in event This will allow you to see it or you can do the opposite. I want to see the minimum value as an event And it creates a little graph above The graph With little lines that you could look to so I go into Kernel shark under tools you'll see plot event field and if you download Kernel shark from kernel.org and install it it's got the default is kernel shark 2 And you do your make install and install they will include the plugin So you may not you notice that plugins are added it should add them automatically for you This pops up and what I did was okay I'm using the data stream as the file that I'm reading which is this trace guest net It's the guest network thing and I want to say I want to look at this event called k mem cache alec K mem cache alec is one of the allocation mechanisms within the linux kernel that allocates memory And I want to say let me plot the fields of bytes alec That's how much memory that this event actually allocated And this is what I get and you'll notice that's above the Little line you'll see this line that has a little graph that shows you these green things here are the are the maximum values And everything here is a little tick of how much was allocated So I kind of zoomed in one and I took a couple zooms I only have one thing so you you select your mouse you move it to the right and it will zoom and then when you let go It zooms in nicely And I zoomed in a few more to find this guy and right there You can't really see the bar on that because the marker on top of it is over the bar But there's another green one here and you'll see this guy allocated. It was here couch alec and I lose I can't believe I cut off the how much it allocated. I was supposed to talk about that and then you're to check my screenshot, but if we have time for a demo like a demo The latency plugin is another thing that's similar to the event plugin, but it takes two events And you can select a field between these two events that they map that would map each to each other So if this field has a value that matches The field of the event two it's going to select these two guys And it will show you the difference in time between those two events So I go here and I select plot latency And it gives me this graph this dialogue which all this is developed by the plugin the plugin This is the plugin defined the dialogue it defined all this and hopefully in the future We'll have a tutorial on how to write plugins And I looked at sked waking which is when the event is or when a task is woken up The sked waking event is triggered with the PID of the the PID field down here Is the process ID of the task that's being woken up Then I go over here and I say select sked switch event Which is the event that's triggered when one process schedules out and another process is scheduled in and next PID is the process ID of the task that gets scheduled in So basically I want to see the time between when a task was woken up and when it ran on the CPU And I get this now. I didn't zoom in this is kind of funny because These lines down here. This is the my I think I ran the the migratest and they all run at real-time priorities And you've noticed this line here and the gap here and a gap here. Okay real quick anyone know What those gaps are for from Yes Yes on the real no, it's no it's on the it's on normal kernel But they're real-time tasks the real-time task is the hint Okay, I'll tell I'll say it so what this is is the RT throttle kicking in This is using the RT throttle is a utility a safeguard in the Linux kernel That have a if real-time tasks run for more than the time listed in the kernel, which is by default uh I think it's 950 milliseconds if it runs for 950 milliseconds in one second So 950 out of a thousand is going to throttle those tasks and they're not going to run for Five milliseconds and then they'll be able to run again This allows non real-time tasks to get onto the cpu because the real-time tasks will never allow A non real-time tasks ever score. So this if you actually put the measurements here That's five milliseconds and you get to visualize that that's why it's happening It's triggering on all cpu's throttle throttle throttle And right here if you look above this this this the guys where this what what they got preempted But there was a one here this green one see red means it was just preempted It wanted to be on it wanted to be it wasn't preempted. It was just throttled The red means it wanted to run But couldn't so that's not part of my latency, but down here This guy got woken up at the beginning Of the throttling And then it was scheduled in here and that's why you see this huge spike And here happened again huge spike. So this guy and this one didn't there's no green bar So it didn't get woken up. Here's one was a green thing where it got woken up huge spike So you could see that in the graphic you could find where in the Graph that something Visualize that's a maximum and then you can zoom in and look at the trace events to figure out what actually happened Now tracing between hosts and guests So this actually requires trace command 3.0, which is has not been released yet But the development code is in the git tree. So if you go to the git tree, there's no tag for this But you could download the trace command git 295 has the functionality for it, but doesn't have all the things you need So if you want to play with this you need the 3.0 So you run an agent on the guest and then On the host you could actually tell the guests to Well, this actually works the between the agent gets you actually tell the guests to execute something But what's not in 3.0 or what? 3.0 or is required in 3.0 is the time synchronization the time synchronization algorithms to synchronize between hosts and guests is in 3.0 Where just the communication level is in 295 So on the guests I say okay trace command agent and then on the host I do trace command record Well, so I'm going to record to a file. I'm recording the events the kvm all kvm events All scheduling events all networking events and interrupt event or irq events and irq vectors So interrupts are happening all on the host and then dash capital a means agent This is the vsoc id for the guest which is uh channel 3 port 8 23 8 23 because that's my birthday dash e I'm telling to record all events on the guest so the events The guest is going to record all events But on the host I'm going to only record these events So like I said, that's the I highlighted this because I didn't know if my mouse would be able to work Then I moved the the trace data because uh There is code if it no there's some depending on what utility you are to run the guests you can actually Put in the guest name and it can usually find with the guest name. So if I had my guests called, you know Goliath I could put dash a goliath and it should find it if it's if it can but sometimes if it can't you have to actually put in the The vsoc id And if you put the vsoc an id it will give you Dash 3 if you put in a name it actually the trace file for the guest will be the the name of the guest So I just moved it to guests actually this I changed it I did ran this a couple times and the second time I ran a guest dash net So actually the screenshots are probably going to be for dash net But you get the idea then I ran kernel shark with trace data and I'm using the command you could do this from the From the menus driven so you could say file append Which will do this but I did the command line what I did was kernel shark the main host data first so trace host I'm appending the trace guests as This is telling kernel shark that this guy kind of is dependent on this one And then this is what comes up The this here is The eight cpus of the host I had to scroll down so you're missing cpu one because it's up here And because I wanted to show cpu the two cpus of the guest Not much useful information there, but it's nice to see it together and they're time synchronized So these events are actually synchronized with this it actually uses the kvm hardware clock to Say look at the offsets and figure out what the actual event mappings are so it doesn't do it while the recording happens It just says this time is it maps to this time Then I go in I say plot kvm plots And I go here. I say it shows you Okay host guest and if you have multiple guests, you'll be a scroll down for multiple guests And then I select the two cpu or the cpu one and two And it gives me this I did some cleanup. This is I Removed the cpus. I removed all the cpus from the other hosting guests I didn't want them and I selected the wget that I used For on the host and the nginx Web server that I was running the trace On I grabbed I said trace. These are all the guests and then you'll notice this combo here This is the guest cpu On this is the thread on the host that acts as the agent on the guest So basically what happens is, you know when a guest's virtual cpu is running It really has to be represented by a thread or something on the host So the host thread that's running For the guests is it by it determines finds out what that is and here it is So this is the host thread or host task and this is the host task with cpu zero and one below Now if you zoom in you get to see some really nice information And I even did some clicks here So this to show you here This is on the host it woke up here Scheduled in and it's these little lines that go up and down and up and down This is when there's a kvm enter goes here and then kvm exit and this Shows you that whenever you see this funky thing means that the thread The guest virtual cpu Is scheduled, but it's in host context So that's why you see all the events that happen on the host scheduled back to the guest the guest events happen Scheduled back down to the host events And if I wanted to click here I could click when did it interrupt come in it's like this guy was when I woke up And by the way when I click this it automatically does the second marker There's a marker a b and tells you the delta so when I clicked on this guy it showed you how long it took to Wake up So this event happened and you see that the host It's interesting that the here there was no events going on It's scheduled to the guest But there's nothing going on and then scheduled back to the host But then that's probably because it said oh we have nothing to do scheduled the guest But then an interrupt happened on the host and then it's scheduled back to the host Because a lot of these things are interrupts that are happy within the host So in summary Colonel shark 2 is much more flexible now We're looking for more plugins to add to it um Allow others to extend the interface And hoping as always patch is welcomed So thank you and if I don't know if I have time or not how much is that over or do I have time? Am I early good? Screw it. Let's so I don't know what oh it's right here. This is empty So let's see if it crashes open trace file Uh traces What's a good one? Oh my my great one So there it is much quicker You can zoom in. Whoops control zoom in zoom out um Yeah, this will be this let me move the And this is where I said if I click here Then I go up to marker b Look here It's five milliseconds. So that's how you can see see um Let me zoom in here just to take a look at it Uh And I'll quit Then I'll do colonel shark dash a no l which means bring me back to the last place where I just left off So that's really a nice feature Let me see here I get append trace files sessions sessions Import session Let's see. I think I had one latency. Do I have anything with the G? Two lines. Oh shoot. Maybe I'll just pick that one heck of it. I have no idea what is Oh went to this screen here But that oh, this is where I want. Yeah, this is where this cpu is. So whoops. No, that was a bug That happened once minus. Oh, where would I have happened? Yes. Oh, I somehow lost the marker. That's why But let me zoom in again zoom in again zoom in again Just double click here and let's see how it when I double clicked it knew that this is a wake-up event So I'll automatically select the sketch switch event. So it showed you right there and you get to see The time which is what 26? Um Micro seconds to wake up not too bad. And I go like remember I said what the heck is going on here I don't really know. So I'm just clicking here and down here. It's like kvm a pick Emulator. Oh, it's maybe oh, it's a page fault. So it's actually emulating an instruction that the guest did So it jumped to the guests. There was no events that was being traced, but it did something that The guest wasn't allowed to do And guest context. So it faulted went to kvm Went to yeah, and then kvm emulated it for the guest and then went back That's why you can see and you can even see what I don't know what this instruction is, but it shows you the instruction that it emulated so questions Yes Oh the different colors. Oh the different colors. Oh, like I said, it's um, actually a good thing. Let me zoom back out so the different colors I said What means is it depends on if it's a task or a What's the word if it's this is a task because it's a task with the process ID So each color at the bar the color here. It means it's a cpu This red is actually if I go onto the cpu where it's at which I don't have plotted Let me go plot cpus Uh, the guests. What was it cpu zero? Shoot. Let me go find out which cpu it was Um, this guy says up on top. Yep. If you look up on top, I can't move the mouse Um way up here. You'll see cpu zero That's so cpu zero there. So if I plot cpu zero On the post You'll notice it's red and that's because so right here this red represents this task So that's why the color ticks are the color of the task So and say if you find a case where Now like maybe right here if you're saying geez these two colors don't look too good If you could go up into tools there is a color scheme Oops that you could actually change the colors and it will change the scheme it uses a hash table So you could play around if you don't like it if like right now actually probably made it worse So you could actually change the colors of what everything is So every process ID just goes throws to a hash table and then it just could do you know red the rgb red green blue Cuts it and says that's the color So nothing too sophisticated, but works really nice Any other questions with that? Oh wait. Nope useful ones Yes, we have to make the tutorial that's definitely on There's so many things we want to do And we realize like oh this will be great to write this tutorial. Oh, it's great to do this Oh, it's great to do that. Then you realize wait a minute. I need four of me, you know I think we all kind of understand that so yes If someone else wants to drive in and write a tutorial And try it so we're actually working for people to actually look at because sometimes when you're Too much into the code it's hard to explain it So sometimes having someone work with you that and then you that just ask you questions How do I do this? How do I do this? How do I do this? You could probably create a better tutorial that way than us just hey I know how to do this and I'll write it and then I'll say things that to me is Obvious, but other people are like what the heck are you talking about? So that's we're all Susceptible to them But anyway though like I said anything you could think of I like the events Maybe play with the events differently. Maybe there's something else you want to do One thing I might want to do is have more input like maybe add code to do something specific for when a trigger happens Filtering we need to filter. I just maybe want to filter tasks right now that I could filter it by just Saying only on this task stop the What's the word I want to say only on this task Do I want to see the latency or something? But right now I could actually just I could go up and just say filter and just say I don't want to filter for like here's all the cpus I could Say what tasks to show? You know If I just do this this should be interesting apply They all go away So I could actually just have certain tasks and I could plot certain tasks and I can say I just want to see the information on This task so I can do the plot that's how I have to do the filtering but maybe have the events thing do a filtering or Have something that's maybe easier or I don't know. I'm like basically When we need a need these are the things that I had a need for So I asked yordan. Would you please write plugins for this? And he did So I even need to learn how to write plugins because I I'm not really good at it myself So maybe I'm the person that needs to write the tutorial and say hey yordan How do you do this because most of this work is done by yordan pretty much all of it So I don't want to take credit for this I've kind of helped lead him and I helped review the code and everything that but He's the one doing the code. He's doing a fantastic job And he's in bulgaria and couldn't come here to present this So if you have two questions, why are you presenting it and not him there? And I also I had to have an excuse to come here. Yes Well, ideally if it's not well suited to the bug By the way before we go on I just want to say that if the event it doesn't exist that it's clicked onto Event that's being filtered out. You notice the line turned dashed Yordan actually thought of that so um It's really great on any scheduling utility the scheduling folks that does that do all this dead The sked deadline and all that they use kernel shark I was a lpc for the scheduler and kernel shark popped up a few they just had Just had a cut in the screenshot of the bar and I'm like that's kernel shark, you know So it's used a lot in scheduling folks because you can really visualize how things are but I'm trying to get it done For locking problems and a lot more other things. Well, this is a good for what's not good for I sometimes don't I don't always use it unless I need a visual but if I just want To trace something and debug something I just go right into trace command and I usually can find it But then when there's latency issues, I'm like, okay, this latency thing's new So I haven't really gotten to use it yet because it's rather new. So now I plan on using it more So that's why the more features it gets The more it'll be used Yes, yes, yes Well, there's a record feature which won't work because I'm not really I think it will Now I have to be careful. This is not this is like a debbie in kernel and And there it goes. So it runs actually in different utility It's not this actually isn't kernel shark. It's actually a kernel shark plug-in type of not plug-in But a helper it's because we don't want to think and now I could go in here by the way and say I don't know. Here's the V sox, but let's say what's a good thing to let me just Kick off sked I could click and let me turn off. Let me just do sked switch and sked wake up Which what's wrong one? Wake up and let's see here plug in. Do I want function graph tracer or something? No, I'm not going to do anything Um sleep one is the command I could change it to ls You know, I haven't done this in Sometimes so I actually I'll think this is probably the first time I've done it with kernel shark too So if it crashes, I'll blame you but Why so you can see it shows you the command that does it does trace command record Maybe Then capture Oh, here it is. This is shows you what it's going to do then this and there it is it recorded it Now let's see if it works There What was that and there's the record that I just did On this laptop and I was a little worried because when I last time I did a trace on my laptop I locked up my laptop So I was like don't walk up That's why I kind of limited to those two events because I didn't think they would be a problem But I turned on all events and then the system So some driver or something use it and this is a 510 kernel from debbie in so it may have a bug So I did have time to debug that So and I'm going to zoom back out if that's all that is zoomed out so zoom all the way in there we go So and there's also filters advanced filter. That's really bad TEP that tep is just trace event I forgot what the piece trace event trace point. Oh But this is I could go up here and filter on it shows you here's the events sketch switch And insert and this is like a type of thing that has that this is my poor attempt at Doing what wire shark does and I didn't do it. Well So this I don't usually show this because I'm not Proud of that one, but patch is welcome event. So hey if you want to add more you want to fix it I suck at I suck at ui so Any other questions anything on slack GPU activity so basically events from the GPU only it only shows events that are being traced now the gpu if it's running okay, if you have Trace if you able to extract events from the gpu And you write a plugin by the way the remember I showed you that there's a abstraction layer We've actually made it in this code that the trace command is really a plugin So the f trace work is just a plugin that feeds into kernel shark So it's not even hard code anymore. That's a plugin that's done in and now we have a perf plugin So if you want to write a plugin for events if you could extract events from the gpu and put it into a plugin You could get all the visualization from that plugin Because I said that's an abstraction layer Anything else on yes Well, if you had the trace output by the way I haven't tested it recently but crash, you know the crash utility when the system locks up Well, I mean if the system's still running you get a deadlock and the system's still running and you traced all the events up to it Yes, you could trace f trace could easily trace Debugged deadlocks because that's one of the reasons why f trace was created in the first place It was part of it came out of the real-time patch and we caused deadlocks all the time and f trace was used to debug that So although a curl shark curl shark might help if you want to like the visualization of it But the trace command and f trace and if your system's still running great if it crashes it locks up and If it locks up and you could still trigger a watchdog that would go into k exec And k dump and you get a core file you there is a you to a plugin Not my plugin, but a crash plugin that comes with crash called trace So you load the trace plugin and it will it will extract the f trace buffers from the Core dump level and create a trace dot dat file that could be read by trace command or kernel shark And I've used that to debug Customers Problems where they would trigger a bug, but we couldn't get their stuff. So they just tell them enable all these Events and then they would give us a it would lock up crash. They get a core dump. They send it to me I extracted the data and then I brought up, you know kernel shark and everything I was able to debug The customer's problem From that output Any other questions? Okay, thank you Great to have a live audience And I still miss that sound You