 Hello. Thank you for staying for like one of the last talks of the day. I'm Steve Rusted. And for those that you may know me and actually seen some of my talks, you're probably waiting for what I will do right now, which is my little gimmick. If you know what this is, it's actually, it's called a camera. People don't use these anymore. It's like from the 20th century. And what I like to do is always give us do a selfie with a real camera because this I've been doing this since 1984. So smile everyone. See if it worked. Good. Good. So like shows a nice empty audience. What have been here earlier by those running plumbers? Anyway, I guess you don't want to hear anymore about that, but let me talk about what I'm here for. Tracing VMs on the host and guests and seeing how things interact. So I like to talk about why we use VMs. I think pretty much everyone here should know the rundown. It's you could run several services. I like to run my I have a mail server, a web server, SSH server, all on VMs, you know, it's easy to migrate from, you know, hardware. If you want to upgrade your hardware, you easily migrate it. It's like an extra layer of security. I just say security and quotes because there's ways to, you know, basically break in front to the host from the guest. There is ways there, but it's an extra layer. Well, any time there's any time you make it harder for the attacker, it's good thing. They but they do add a little bit of overhead. You have two operating systems to deal with the hypervisor and the guest. The sometimes the guests will need to do things through the host. So it takes a little bit of interaction to get to the devices. Sometimes the host even has to emulate some commands. But luckily, if you want to look at what this overhead is, KVM provides several trace events within the kernel. Over 100 trace events are there for KVM telling you all sorts of good things. If you don't know what KVM is, it's the kernel virtual machine that basically runs the guest in a Linux environment. The most useful that I find of the KVM trace events is KVM exit and KVM entry. It tells you when the guest goes from the guest into the host context and back to the guest context. And that's basically kind of like what I'm going to be focusing on today, but it's more of an example of what's possible, what's out there, not exactly what you have, like all that you have to do. So like I said, it tells me when the guest falls into the hypervisor and back and also shows you why it did it. There's also trace events in that timeframe that shows you like what the guest or the host is actually doing for the guest. Now, tools to determine the overhead. Well, I'm the author of Trace CMD or Trace Command, like I like to call it. It's a command line interface to the Linux kernel tracing facility, aka Ftrace, although Ftrace is just a collaboration of a bunch of things. It's not, Ftrace is really just a function tracer, but people call things Ftrace when it's really just a tracing facility within the Linux kernel. Basically anything that's in the TraceFS file system. What Trace Command could do, you could start tracing, you could stop tracing and then show the events so you don't have to know the TraceFS file system. It will do everything for you. It will mount it. It will even mount it if it's not mounted for you, which means you obviously have to be root user or pseudo trace command to do these operations. You can also record to a file for post processing and that's basically what this talk will be about. Sysbench. Some of the things I wanted to do to analyze this, I needed an example. I decided to pick Sysbench because this is a pretty much a common benchmark to use. It's out there. It's easy to download. It's available for most distributions. It's in Fedora. It's in Debian. I'm sure it's in Ubuntu, although I don't use Ubuntu. I'm sure it's Susie. It also has a MySQL benchmark, which a lot of times you use database operations and VMs, so I figured something that's appropriate to look at is database transactions. Sysbench has a lot of benchmark, has a MySQL plugin, and I decided to use that because it seems appropriate. How do you run it? This is the command that I use. Most of the ugliness here is actually the Sysbench code. The first thing I did was I did trace command record, which just says start the tracing and start recording it into a file. Dash E means enable these events. KVM is the KVM event group of events. Remember I told you KVM has 100 trace events? By just doing dash E KVM, I just enabled all 100 KVM events. I could actually just specify individual events if I want, but that's a lot of dash. I don't want 100 dash E's there. Then the trick was, okay, I'm going SSH into the guest as root because I want to run suspense as root. Obviously, you may not want to do it. I have keys and stuff like that. It's just a test box anyway. And this is for analysis. But to make things a little bit simpler, this was actually, I gave another talk on this, and I wanted to look at a single CPU. So I decided to use task set. If you don't know what task set is, it's a way to change the affinity of the process. So I said task set dash A, which means if it creates a bunch of threads, it will make sure that those threads have the affinity set as well. If you leave off the dash A, it will only set the first task of it forks and stuff like that. They just go all over the place, which confused me at first until I forgot, oh, I didn't do dash A. And then dash C means the CPU you want to find to. And I said CPU 1. Don't use CPU 0 because a lot of times CPU 0 is used for housekeeping from the kernel. And you'll get a lot of, unless you want to see what the housekeeping interruptions might be, but I usually use CPU 1 because I more or less want to focus on a single thing. And a lot of times, yes, things will migrate. It won't be the only thing running, but I don't have as much interference from the kernel housekeeping threads. Finally, I actually execute sysbench, which is the rest of that nasty command line. And I just know there are basic things that you could figure out on your own. If you just look at this, I'm not going to, this talk is not about how to run sysbench. It's just the fact that I'm using it. But I created four threads. I said run for 10 seconds. And there it ran and gave me a bunch of times. So now I want to go and take a look into this. So I do, and I do trace command report. This is the output. So I just did trace command record. And by default, it records to a file called trace.dat. You could change where it records to if you, in the trace record, trace command record, you could put in, you know, dash O to make it go someplace else. But I don't, I seldom do that. So, and if you use, if you do put it to a different file, since trace command report will only read the trace.dat file by default, if it's not there, it'll give you an error. But if you want to use another one, you have to, you have to specify the name. If you just specify the name, we'll read it. But sometimes if you want to specify more than one, you got actually add an option, say dash I file name. We'll see. But anyway, going back to this, great. You could see all the KVM events and that's going on here. And you know exactly what's happening, right? So the point is what I, the problem with tracing and getting raw data and stuff like this is the fact that unless you really know what you're doing, it's meaningless. It's one of those cases where too much data is probably worse than not enough data. It's this, you know, getting rid of the noise. I got all this data. I'm sure it, there's a problem happening and I'm sure this data shows me what that problem is. But I have no idea what that or what, where in that data I could find it. So looking for the needle in the haystack. So introducing live trace command. Now, this is something I've been working on quite a bit lately. So we had, there's three libraries you'll need. They're all, if you go to trace command, trace-cmd.org, which is being, thank you very much for hosting it. And colonel shark.org. It's, it's basically on the same machine. Anyway, the trace command.org has links to the, the colonel.org get reposed that have, you'll see three libraries, live trace event, which is a library on how to parse those, parse the events from, so you'll get the raw data from the trace.dat file. You'll look into the trace of best directory to get the, the ways and the information on how to parse that binary stuff and parcel it nicely for you. So you don't need to figure it out yourself. There's also trace of us that actually has all the accesses to trace that best file that you don't have to know which files is it will mount the file system if you need it. And finally trace command, which is the third one, which by the way, when you add trace command, if you do package vague trace command, it will include live trace of S and trace event because it's all, it's in that order trace of, live trace of S depends on live trace event, live trace command depends on live trace of S. So when you run this, this, this file specifically is how do you parse that trace.dat file? I don't want everyone to use trace command. I want you to build your own tools. I'd rather have your own recording. I'd rather it'll be great that you could record your stuff. It's slowly building up this library. Ideally, I wanted to allow you to start a recording to say, Hey, kick off tracing the kernel and create a data file for you and then be able to parse it and analyze it without going through trace command itself. You just have this library and as an API interface, by the way, this talk is not about this library. I'm not going to show you how to use it. I gave a tutorial at scale on how to use this, but that's not this talk. And plus one, please wait before you try to do anything in this talk. Please wait a bit because I did, I wrote this talk on my layover, 10 hour layover in JFK on my way to Dublin and I realized I need to add some more features. So I added more features. So some of the stuff I'm going to talk about that my tools do only works on my laptop. So I plan on pushing these features up. So I'm going to have links to code, but don't download it quite yet. So like I said, this tracing me allows you to analyze the trace that death file. And this talk, I said, isn't about it. It's only about that the file actually exists. First tool I'm going to talk about is a tool that I wrote. It's a very small tool. Actually, you could download it. Now I was going to add more comments. I will add more comments. So if you download it now, download it again in a couple of weeks because I plan on adding more to it. That's rosted.org slash code slash kvm-exit.c. If you want to download it right now, it's available. There's a little tool like maybe I forgot if I uploaded it not. So we might have to wait a little bit to download it. So what this does is basically simply examines kvm exit and kvm entry events. So when I run my trace.death file here, I'm going to use my mouse to actually do a good point, but I want people who are watching remotely, if they are watching remotely, and hopefully they are actually recording this. So I execute trace exit dash capital C1. And I'm trying to remember what the dash capital C1 was because I wrote this on Saturday and I can't remember what I did it, but there was a reason why. Well, I know what it is. It says only show me CPU one. I don't care about trace. So I executed that. Remember that command I wrote? SSH up dash capital C1 was just a only show me virtual CPU one. Don't show me any of the other CPUs. The reason why I did that because remember I did I pinned everything to CPU one. I have had a way to I only care about CPU one because I pinned everything to CPU one. That's why I did that for this talk. So I go here. It shows you virtual CPU one and then it tells you the host process ID. So you have to understand what kvm works is on the host. It actually has a thread that that will represent the virtual CPU. So every virtual CPU has a dedicated host thread. And this is the process ID for it tells you how many times it exited the total amount of time. You know, it's like exit three seconds tells you and breaks it down why it exited. Here you see an NMI exit, you know, three times exit due to NMI total time of 19 microseconds. So if you notice the first few by the way timings is all in microseconds because that usually makes more sense. The very last one, the min time is in nanoseconds. Reason wise, when I first wrote this tool, my min time was always zero. And that was kind of useless information. So I'm like, okay, I had to get it to a all time stands are in nanoseconds. And I usually do a divide by 1000 just to give you the idea for the microseconds. And also external Europe and penny Europe. And then it goes on a few times, like 255 times that CPU when I ran the suspense, it did a CPU ID call. When you execute a CPU ID system call in the guest, it actually actually goes to the host and asks you information tells you why did it. The whole instruction here means that it went idle. The CPU went idle. I'm like, okay, for like what 2.5 seconds. So I really don't care about that. Yes, it went idle. That's not something I want to analyze because, you know, I don't care. It's doing what it wants to do. That's not where I want to focus my attention. And of course, read, write, pause instruction. Here's a little thing. It also was interesting. I got this extended page table violation and extended page table misconfiguration. So I'm like, okay, this looks strange. And there's quite a lot of them. And it happened for like, what? The first one was 24 milliseconds. Other one was almost 10 milliseconds, something I look into. So how do we do this adding another little thing trace command agent, what you do is on the guest, you write trace command agent. And what it does, it will open up a V socket. And what a V socket is, is basically like a networking socket, but only for a virtual environment. Whereas every single virtual guest will get an ID assigned for it. And the host will get one as well. And this way it will let you start tracing on a guest. It helps with the synchronization between the guests and the hosts of the time stamps. So you can actually interleave how and events happen together. On the trace command record side, you do dash A. And this is how you will connect to the agent. Dash A means connect to the agent. So it will tell the record on the host. So you do trace command record dash on the host dash A. And then you tell the agent to do some recording. And this is where it negotiates the time stamp synchronization. And this is kind of like how it works. So you do trace command record dash A guest, or first thing you do is the first one is over here. You'll see dash E host events. That means you could start tracing on the host, followed by dash A guest one, which will be here, which is running a trace command agent. It will connect there. Then it will, you pass in the guest, oops, went back. Guest agent. And then over here, this is guest two, guest three. And this is what it looks like when you run on the guest. So you just go in trace command agent, it'll say listening on at board colon 823. By the way, 823 is my birthday, August 23rd. That's the default. So send me a birthday present. Anyway, the socket, this is the context ID, which like I said, it gets an ID signed for it, for the virtual socket. So the four, this guest had four as its virtual socket. The other one is the port that's, that is like a TCP UDP, stuff like that, but in the virtual environment. So now this time I do trace command record exact same thing. This time I'm including the scheduling events. I could just care. I only really care about sketch switch, but I said, I'll include all events. And then I do dash A. And over here, you'll see the same number for the SID and the port, followed by the name guests. If you leave out the, you know, the game name guests is optional, but it will create a separate trace dot death file for the guests. And if you leave this off, you'll get a trace that you'll get trace for 823. That that's not very useful. But if you put in a name option, it'll actually say trace guest dot death. So you actually label what output everything else is the same, except for on the guest side. I said, enable scheduling events as well. Then SSH up does everything. So the way this works is the trace command agent runs, it opens up the socket. Oh, this is interesting. I downloaded that didn't work. So I've got the PDF conversion from Google didn't work very well here. And then you run the command, which then will connect to the socket. It'll switch over to the file descriptors. And then that's, it does this, the metadata and then it will open up two more sockets per CPU. So as the guest is recording, it just sends the data straight to using splice to send the data off to the host. Splice is a way of passing data from one file descriptor to apply for whatever without ever going through the guest. So it's much, there's no copy. It's a zero copy transfer. So this time I run KVM exit, but this time not just passing the trace dot death file, I'm going to add the guest file. And what happens, the live trace command has really easy way of iterating through all the events. You can set up a callback for every single event to call a function. And then you can just say iterate through all these files. And each of those callbacks will get called as it hits the events. You can also pass in multiple files. And it knows that you were the first one is the host and the other ones, the guests that did this negotiation. So it will actually do the, it'll read the metadata that tells how to coordinate the two. So the timestamps are synchronized. And then you'll be able to see both sides. So this time it gives you a little bit more information because this I could also, let's see here. Oh, I'm going to talk, there's a preempted, but let's see, I got 20 minutes. I want to go further though. And yeah, this is where things got prior, it actually monitored the guests. And I passed here and it shows when things got preempted. This is okay. What this means, the preempted here is when the external interrupt happened, it actually monitors the host side of things. And it was interesting because on the external interrupt, the host was preempted. That's right. I put the sked, I put the sked command, remember back here? Here I added sked on the host side. So my KVM exit actually monitored the host side and included when the, when there was a KVM exit and the host was preempted, which means it was in the running state, but another task came in and ran instead. So I actually, I could see that, hey, it got preempted. And it tells you how long for 10 milliseconds. And a pending interrupt, it got, you know, 11 milliseconds, or actually microseconds. And it also tells you it got migrated while the halt instruction. So it actually even monitors migration. The migration, remember I said, everything's pinned to virtual CPU one, but virtual CPU one was not pinned to anything. This is actually telling you that this virtual CPU, the thread representing virtual CPU one was actually bouncing around the host. So this is more information, not just give you idea of what is available. So if I wanted to, I could actually, I, this was a trace command report that that where I did the dash I, and I filtered off of KVM events. And then I passed in the guest file and I'm filtering off of common PI. Okay. This was a trick because I wanted, if you put a filter in, it kind of filters everything on, and there was a kind of bug in here. So what I did was I wanted to see only the KVM events from the host. I didn't want to see the scheduling events of the host. I only wanted to see the KVM events of the host. So when I ran the dash, I have to fix this, but trace.dat, I had to add a filter for the guest. So I, oh, I wanted the, that's right. I didn't want any events in the guest. So what I said was filter something that it will never find. So I only wanted to see the KVM host, but I wanted the guest. The reason why I wanted the guest because you get some extra information if you have a guest there. So the filter, the last filter that I did here was basically, I don't want any data point. I only, I don't, I only want KVM events from the host. I don't want any data from the guest, but I did want this communication because it reads the metadata, does the synchronization. And on KVM exit, when I do this, it tells you the, on exit, actually, here, like, or here's the KVM exit, the MSR, right? It gives you the, the instruction pointer, the, this is the instruction pointer of where it exited in the guest. But when I do this, the library will do this for you. It will actually say, Hey, here's, this is the, this is the, the function, the kernel function in the guest that matches this IP. Now disclaimer, this was the code I added over my layover is only on my laptop. So if you want this functionality, wait a few weeks, I had to push it up, but this is what I added. It's a little bit of a crazy thing to get this to work, but it does work. And I could clean it up. Like I said, you're talking about new features. Yes, this talk is more like a plumber's talk, because I'm talking about things that aren't quite done yet. So yes, and staying down here, I, the, the, the violation was always, the funny part is always in this information. And because this is just trace command does it. I can also, well, here's all the native stuff. And I also added it. So it automatically shows up in the trace, the KVM exit. So like I said, you probably download this, it may not compile, because it doesn't have all the features that it requires from lib trace, lib trace command. But here it not only does it break up the IP, the extended page table violations by just exit. It actually breaks it up to where the violation happened. So you'll see here that the first one was in the iNet lookup establishment. And it happened at 2113 times at that function. So that function is doing something to cause these exits. Same thing for the clear, clear page ERMS inside the guest. So if you want to analyze why the guest is falling into the hose, this actually will give you more information into that. So all these things, you could see why things happen. I'll write for the misconfiguration at the end there. Kind of makes sense. It's doing something that's faulting and probably faults. Maybe that's the design. Next I want to talk about is a kernel shark. So by the way, I have enough room here. For those that are in here, I guess, I have somewhere in here, a bunch of kernel shark stickers. So if you want, I'll give you a sticker on it. If you stay up to the end. So can't quite show it here. Some of the talk that I gave here is in this tutorial, which is at rosted.org slash host guest tutorial. FYI, it's a rough draft, but it's been a rough draft since kernel recipes. And I haven't really had time to update it. Ideally, this will be on kernel shark.org. But I don't want to put up on kernel shark.org until it's a little bit cleaner, a little bit better. Anyone out there is that writes HTML and stuff like that or knows how to make nice web pages. Hey, if you want to volunteer some things, do this because my HTML skills are horrible. And everyone has their niche. I'm great at kernel writing. I'm good at maybe writing libraries, but when I make things pretty, I am pretty poor at it. So this is why it's there because I'm too embarrassed to put it on the official site. So the way you run this, so if I did that host and the guest thing, you'd start kernel shark with, you know, the host data file, I just put trace that host to exaggerate that that's the host file. But when I ran it, it was really just trace dot dat because by default, that's the host file. Dash a means append followed by trace guest dot dat. And like I said, it denotes that you're appending it. You're telling kernel shark that this is a guest file for the host. And that this is what it looks like when you open it up. So when you first open up, this is what you see. You see each of the CPUs for the host followed by the CPUs for the guest. Each color here is a different process that runs. So the color changes as processes change. And what I would do is I'm first thing I want to get rid of the plots. So I would hit plots, hit CPUs here pops this up. I would click the turn both hit the all claim them all off, apply it. So there's no plots there. Reason why is because I'm not interested in those plots. I'm interested in these plots, the KVM combo plots. So I click on the KVM combo plots. And then this pops up, I select all. So I want to see all of them. And then it shows me this, this is kind of nice because what it does is it attaches the guest CP virtual CPUs to that host thread. Remember that host thread I was telling you about? That's here. This is the guest CPU on top host thread down the bottom. Same here. This is for virtual CPU or virtual CPU zero, one, two and three. And it shows the the events and if the colors ever change, it looks like everything got pinned or maybe not. Maybe I zoom in more. So when you're looking at the CPU plots, which the top layer is the CPU lot for the virtual CPU, it changes color depending on what task is running on the CPU. The bottom one is a task plot, not a CPU plot follows the task. But when the task bounces between CPUs, it will change colors as it does that. So the first thing I do is I left click, slide a little bit over just a little bit and right click or sorry, left click, drag, let go and boom, it zooms me into that location. And this is what I get. So I'm going to do the same thing. I want to look at this little carrier and we click, slide a little bit, zoom in again. And this is where you'll see the top level is the guest CPU. The bottom level is the the host task. And here it shows me KBM exits, KBM entry. So if I click on this, so I clicked on one, clicked on the other, you'll see down here, there's a KBM exit, the first one, the second one is a KBM entry. So this, every time there's an exit, it goes from what you'll see the line that comes down. It does some events and come up. Now I just noticed here, you'll see, is this a bug between, you'll see there's a little there. So events can only happen at the guest or the host. It can't happen at the same time. So you might see some cases where, wait a minute, there's a, there's a event here. This event should not exist. So it must be a bug in the system or maybe something wasn't coordinated. No, what I found out too is the fact that the KBM entry, you could have, it doesn't go directly into the guest. There's actually even an interrupt could go in. The interrupts aren't even disabled when it does KBM entry, which means that if something happens, so you get the event from the KBM entry and then an event could happen again, and that will show up there. It's because it actually, you had an event happen before it ever made it to the guest. So even, so the Kerl shark is just looking at KBM exit, KBM entry, but then you might see some events after there. That's because those events actually happen on the host before it actually made the transition, but we just, you just have to know that. That's why. But anyway, I zoom in here and if you looked in these, these events here, I actually look at, okay, we had an MSR right. Then I go, go look, here's all CPU one that was, or this is CPU one for on the side, the host for the zero. I just want to make sure I'm on the correct CPU. So it says, you can see the little direction, all the KBM operations that it was doing for the host or for the guest that the host was doing for the guest, and then before it entered out. And right here you probably, yeah, that's, yeah. So next thing I do is I'm going to search for the EPT violation. So, you know, I typed in here, I clicked info, contains EPT violation, hit run, there was thousands of them. So I had a wait a while. So actually I was able to take a screenshot while I was waiting for it to process. And I finally got in and if I here's one with the EPT violation, I'm like, okay, let's zoom in. So you could zoom in and take a look and say, hey, I zoomed in here, there's the EPT violation. I want to examine it some more and figure out why it's doing it. So it happened here ran here by the way up here gives you the when you click marker a will give you the first click marker B will give you the second click. And then when you zoom out or when you have those two event clicks, you can look here and you say, okay, that was four. This is in microseconds, right? That's Miller. Well, that's, yeah, these are Miller, Microsoft. So four microseconds and now seconds. So it gives you the full time stamp. So it was four microseconds difference looks like it was out. And I think that's all I had to talk about questions. That was not bad 72 slides in only 30 minutes. Yes. Okay, so the question is, when did I check it outside that boundary because Mark here said he's changed that recently, but did you change on x86? How recently? Well, I'm I did this on this on the Debian testing. So it's, it's like probably a 518 519 kernel. So I mean, heck, okay, might be an oral one. So let's just hear just what the heck. Since there's no question. I mean, I could just play around and you'd play with us. So if you want to do something, come on, let's try it. Let's say, do I have my learning if I yes, it's running. So let's do I want to switch. Yeah, I always do it this way. Okay, now maybe I do it this way. Yes. So make sure you guys see this bigger, bigger. Ah, the agent's running. Look at that. So by the way, this is a 518, which is probably the 518 or this is 519. Has your patches in it? It should have it. Well, let's see here, this is. So if I were to do trace command, record dash e sked, what if I have migrate at all? I always like that. I always like this one, my real time thing. So if I'm going to do trace command, record dash e kvm dash dash a for a 23 name, guest, dash e sked, SSH root at primary root doesn't matter, but just team. Let's see. I think it's see migrates. Oh, I got to do clicks. I need to put in, oops, not that guy. Oh, I probably don't need anything. Let's just try that. See what happens. So I'm going to do trace my record. Oh, no. Really? Let me see here. Kill all. Just I don't know if I only know which one is. So let's do trace command agent. Yes, three. I was using another one for my, that's probably why. And I'm going SSH to the guest, negotiates the protocol with this no such file or directory. I didn't quite, really shoot. There we go. Here's a suspension. Let's run suspension. So I already had it before. I'm not doing a task. I'm just going to run it. So now it should connect to the host or the guest. See here negotiates. Now it's running suspense. I should have it for 10 seconds. Is it reset information, records the data? No events were lost. If your event was lost, it would tell you so many events were lost. So all the events are there. It's trace.dat file. You'll see, I guess I already had this directory because IRQ, I must have done something before. But anyway, trace command report, dash I trace.dat, dash I trace guest.dat less. So here's all the things I do, Colonel. Well, we do my KVM exit. Where did I put my KVM exit? Just do it this way. Oh, I want to do, what was it, suspense, maybe trace.dat, trace guest.dat. So it's analyzing, it's running through analyzing all the files. And here I get the host. Let's see, where's my mouse? There it is. Got the host PID. Here's all the events down. It doesn't show the, oh wait, I want to show the functions too. So it was dash F I had to do. And there it is. So some of them don't, like you notice this says external interrupt and there's no, it doesn't show you, oh, because it ignores anything of interrupts. Because interrupts are just random. If you want to see them, I think you have to put dash, I think I have it so you do dash A. So if I do dash A here, then I see the interrupts where the interrupts even happened. Only if it's in, only if it happened in kernel space. If it happened in interrupts, now I can see the interrupts. Because usually I found out that since it breaks it up, when you put the functions in there, it will break it up per function there. So I would go here. Anything from the remote at all? Okay. And that's what you have for here, the breaks up here. And just if you want to see, for those that didn't look at the code, but the code is okay, ignore my template. But at the end, let me go to the bottom. But the main one is trace command. This is the iterate, the trace command iterate events multi where you pass an array of all the handles for every file that you open. So you get a handle when you do a trace command open, it gives you the handle. And then I just pass it. And over here, you'll see that I add follow events, which is a function that says when you hit the sked, sked switch function, call this guy, the KKVMX, call this guy. So this is all I did. And then I just created some data to store. So it's the analysis. Yes, question. Sure. Go to Mike. Yeah, so usually what I do when I want to trace is that I use trace CMD start and stop. Is there a way to start a tracing session on the host and stop it on the guest? Let's say when we use cyclic test with the break test command, the break trace option. So you want to stop it from the guest? Yeah, because I'm running on the host interesting event on the guest to happen. Right now we don't have a way. Well, I'm sure if you want, if you could send a patch to figure out a way, I'm happy to accept it. So sure, I that's just like add a bug report to it. And if you could write the code to do something, figure out, I mean, right now with a trace command agent, it might want to use a trace command agent to do it and have a way of going through that or signal the agent to say, okay, stop tracing. Okay, that might be a better way. Any other questions? Yes. Okay, we have like one minute left. So for the wait, does it read me the automatic? Are you talking about the guest side? Does it automatically correct? Well, yes, that's because it picks because that's actually it's whatever was given. That's not trace commands control. That's when you create the guests. The sock those those CDs are given by the like QEMU or whatever will give them to you. Well, the random number of number was negative one. It looks like a random number because it was unsigned negative one. So that was like that four billion thing because it was an int. So it's basically unsigned number. And yes, we got to do an error to it. But that was like when you had that issue, it was that it was done. But I guess my time is up. So unless there's anything else, thank you very much.