 Hello everyone and welcome to my talk which is a technical deep dive into the QEMO emulated NVME device. My name is Klaus and I'm a software engineer with Samsung Electronics and I'm also a co-maintenor of the QEMO emulated NVME device. So we will begin with a low level introduction to PCI Express attached NVME controllers. And we will then proceed with an overview of the NVME controller device architecture in QEMO. And we will look at the core event loop and ION processing. We will then continue to look at a implementation of a recently ratified NVME technical proposal called the Shunt Name Spaces Command Set. And we will then look at an implementation of a custom vendor specific command and see how that can be extended into the QEMO device. We will also have a couple of live demonstrations just to show on screen how this stuff actually works. So NPM Express or NVME is a scalable interface for PCI Express based SSDs and fabric connected devices. It is characterized by having a single memory mapped IO register right in the command submission pad thus making it highly efficient. It is also from the ground designed for massively parallel IO and it does so through host control and lock free IO queues as we shall see in a second. It also consists of a number of streamlined and very efficient command sets, the admin command set that we shall go through as well as the standard NVM command set from the space base specification. And then we shall look at the new sewn command set. So queues are very central to NVME. And a queue is basically a circular buffer with a fixed slot size and defined by two pointers. Queues are used in pairs. So they always come in a pair with a submission queue and a completion queue. The important thing here is that there is always only one entity writing to a queue. So the submission queue is used by the host for submitting commands, which means that there's only the host of writing to this submission queue. And the controller will then pick up commands from the submission queue. Similarly, the controller will post its completion of these commands to the completion queues and the host will be reading from that queue. And as I mentioned previously, these are always used in pairs. So there is one completion queue that is associated with the submission queue and vice versa. Now, the two pointers, the tail and head pointers are used in such a way that the tail always point to the next empty location of slot in the queue that can be used for new commands. And it is advanced whenever new entries are added to the queue. That is, in other words, there is a completion queue tail pointer and a submission queue tail pointer. There's also a completion queue head pointer and a completion queue tail pointer. And these are used whenever something are consumed from the queue, then the head is advanced. The queues are typically allocated in host memory, but they can be allocated on the device in some cases. So the admin command set is used exclusively with the admin queue pair. And it's important to understand that there is only a single admin queue pair and the admin queue pair is a static queue pair that is always available from the device. Whenever the device is bootstrapped and started by the host, the admin queue pair will be there. This is because the admin queue pair is among other things used for the creation and deletion of the other kinds of pair, the IO queue pairs that are associated with the admin command set, but the various IO commands. The admin queue pair is also used to post the identifier command, which is used by the host to get various information about the controller or some of the namespaces that are attached to it. The getlock page command is used to get various dynamic information from the controller. These could be arrow locks or information about the firmware currently loaded. It could be information about the temperature of the device, stuff like that. There's also a number of controller and namespace runtime parameters, and these can be changed and queried with the get and set features commands. So the NVM command set is the IO command set that is specified in the base specification, and it is an example of an IO command set. So the IO command, the basic IO commands for this command set is flush, write and read, and they are the own mandatory commands. There is also a number of optional and specialized commands such as the write zeros, which is used for the controller to efficiently zero out long range of the blocks on the device without actually requiring the host to transfer a huge buffer just filled with zeros. So these commands that are submitted to the submission queue comes in the forms of what they call submission queue entry, and it is 64 bytes. This command, or these 64 bytes includes an 8-bit upcode uniquely identifying the command. It also includes a command identifier in 16 bits, which is something that is assigned by the host. The host will also, in the case of IO commands, use or set up a namespace identifier, which actually points to the namespace that the IO operation is operating on. Typically only the data pointer is used, and the data pointer is typically two 64-bit addresses pointing into host memory. Typically in the form of a list, that means that the PRP list is a very simple data structure where we point to a 4K page in the operating system host memory. That includes another list of 64-bit addresses pointing to various locations in memory where the data to be read or where the data should be written is defined. There's also the possibility of using a 16-byte STL descriptor or scatter gather list descriptor, and the scatter gather list is an extremely flexible and very fine-grained way of describing memory that is scattered all over the host memory. The final six d-words or 32-bit d-words are used in commands in various ways depending on the command. So whenever these commands have been submitted to the device, they go into the submission queue. And for the host to actually notify the controller that something has been submitted, it will ring a doorbell, as we say. And a doorbell is just a write-only memory mapped IR register, and the host will write the submission queue tail doorbell. What this means is that it will write the new value of the submission queue tail pointer. That is the new value after it has added a number of commands to the queue. What is important here to know is that the host can, if it wants to, submit any number of commands, as many commands as there are room for, and then update the submission queue tail pointer once. So we don't have to need to have a doorbell write or a memory mapped IR register write for each command. Instead, we can batch it up, which is more efficient. Similarly, when the controller has read the commands and processed and executed the commands, it will post a completion queue entry. This is a smaller 16 bytes and doesn't contain that much information, but they do contain the command identifier which the host can use to match up this completion queue entry with the command that it actually posted. The submission queue head pointer field is a way for the controller to inform the host about new value of the submission queue head pointer. That is a way to inform the host about how many commands that the controller has actually executed and that these commands or these slots in the queue are now ready for reuse. The face tag is also special. This is used for the host to be able to actually pull the completion queue for new entries. So, similarly to how the controller needed to be aware of new commands, the controller also needs to tell the host about new completion. It does so by typically generating an interrupt, but the host may also rely on just calling and reading from the completion queue in memory and just reading until this face tag is actually changed. When the face tag is changed from 1 to 0 or 0 to 1, then we know that it is a new entry and we can read the completion queue entry. And we will continue reading completion queue entries until the face tag is inverted and then we know that this is an old entry. This means that the host or the controller does not need to inform us how many new commands or completions have been posted just that there are new completions posted. And finally, when the host has read the completion entries, it has to notify the controller that there is now room in the completion queue and that the completion queue entries have been consumed. And it does so by updating the completion queue head pointer and writing it to the completion queue head dongle. So to actually configure the device, the host does this through a number of read-only-memory-met IOR registers. These read-only-met registers tells the host about controller capabilities, such as the number of queues supported, there are various different registers. Some of these registers are writable and these are used for the controller to actually be configured. One of the things that the host has to configure before the controller can actually be enabled is the location of the admin completion and submission queue. So the registers which are located in a well-defined memory location called the PCI base address register are, for instance, the capabilities as I discussed, which is a read-only register. But most importantly, these controller registers are the admin submission queue base address and the admin completion queue base address. This is something that the host needs to bootstrap in the device before enabling the device. Enabling the device is also done by writing a certain value to the controller configuration by the register. And we can also see here that the DAW builds for this various submission and completion queues are located in well-known addresses relative to the start of the bar zero. So QEMO is an open-source generic emulator and virtualizer. What this means is that it is not just a x86 PCI-based virtual machine that is booting up and presenting to the user. QEMO can emulate and describe a bunch of various architectures such as x86, MIPS, Spark, and as well as a bunch of various storage controllers such as a completely standard IDE, SCSI, the para-virtualized Word.io, but also the NVMe storage controller, which is the specific storage controller that we are interested in. Now, a basic PCI-Express attached NVMe such-state device, the architecture of that looks something like this. You have a PCI-Express controller, you have some memory on the device. You of course have all of your NAND flash packages that actually hold your data, as well as typically a dedicated controller to handle this. And then you have the NVMe Express controller, which holds the NVMe Express logic. Now, the PCI-Express controller and the memory available for the controller is something that is already implemented by other parts of QEMO that we just utilize. And the whole part of actually managing all the intricacies and various characteristics of media such as NAND is completely defined away. We do not worry about that. And we are just relying on the block layer provided by QEMO. So the logic that the QEMO NVMe device is actually implementing is the pure NVMe Express specification logic. So in other words, this is a pure NVMe controller with a little bit of PCI-Express specification logic. There is no flash translation layer or no NAND management and because we simply assume the presence of a flash translation layer. What this means is that we are directly using the logically addressable and linear addressable QEMO block layer instead of trying to emulate NAND flash. The point of the controller is to implement the NVMe specification, not to simulate NAND flash. So the basic controller architecture is that the device actually consists of two different QEMO devices. The controller device is the core device that implements core controller logic such as setting up the PCI-Express endpoints, processing the IO commands, stuff like that. The other parts or the other devices are the NVMe NS devices or the namespace devices. The namespace devices are holding the reference to the actual underlying QEMO block driver. This could be a file, this could be a network file, could be pure memory, but typically it's just a raw file on the underlying file system. The namespace device has various attributes such as the size of each logical block, is it 4K or is it 512 bytes? What is the command set that this namespace uses? Is it a standard NVM command set or is it one of the sown commands that we're going to look at in a second? So these devices take care of namespace initialization and also certain parts of persistent management such as managing various state that is required for some of the commands such as the sown commands. So the NVMe device is basically an extension to an abstract QEMO PCI device model. So the NVMe device begins when booting up by initializing a parent PCI device. It does so by setting up PCI IDs, vendor and device IDs. It initializes base address registers. It sets up the misses signal to interrupt and other PCI Express based stuff. It also sets up a number of internal controller states such as these read-only and writeable memory net IO registers that we saw earlier, as well as initialize the namespaces that are attached to the controller. When all of this is enabled, it will basically wait for the host to configure the controller and enable before it begins processing IO. Now QEMO is essentially a single threaded application, which means that while the guest code that actually runs on the virtual CPUs runs in dedicated threads, the device in relation to it by default runs on the main QEMO thread. This means that if a device in QEMO does something that shouldn't do like simply blocking at certain points, it will block the entire QEMO interpreter. So device code must yield appropriately and not do any too time consuming jobs. This means that device code is typically designed as being run in response to various callbacks when the IO is completed. It works with expired timers and something called button haves, which is basically timers that expire immediately and used for the sole purpose of deferring some work for other work to be done and then come back to this when we have time to do it. So let's look at the basic asynchronous command processing QEMO and how this works. Now everything, as we saw earlier, everything happened, everything begins with a memory mapped IO write. So at this point we assume that the host has written some commands into the IO submission queue. And when the IO write to the memory mapped register comes in, we just begin by looking at whether it is below the 4K mark. That means that it is a register that we are trying to write to and we will have some routine preview in the device that takes care of that. Otherwise, we know that this is actually a write to one of the door bills. So we will assume that it's a write to one of the door bills and by looking at the final bit, the first bit, the least significant bit, we know whether this is an update to a completion queue head pointer, or if it's an update to a submission queue tail pointer, that means that we have new commands to process. So assuming that it's an update to the submission queue tail pointer, we will schedule the submission queue processing. And what we mean by scheduling here is that we won't begin processing this immediately. We will schedule a threat, we will schedule a callback to run at some point in the future, basically as soon as possible. But then we will yield after doing this memory management, after handling the memory mapped IO write. At some point we are allowed to run again, and we will then go through the queue of submission queue entries. So we will read them from host memory using direct memory access, then we will increment the internal submission queue head pointer, and then we will execute the command. So the first thing we do here is that we look at the submission queue identifier. If it's zero, we know that this is an admin command. If it's non zero, then it is an IO command. The admin command typically does not block because they do not need to perform any really time consuming jobs like actually doing IO. So typically, the admin command will just be executed immediately inside our current contacts, and this will result in a completion queue entry being enqueued or set up for being posted. Now, again, we do not post immediately instead we schedule the completion queue entry to be posted at the various convenience. The point here again is to allow a bunch of commands to be executed at the same time and only post and write to memory in one go instead of writing, instead of doing this in a bunch of small updates. If it is an IO command, typically the device will end up yielding, that is it will issue some kind of synchronous IO, and then it will just wait. At some point, the IO command or the actual read or the write that the IO command was supposed to do will complete, and we will end up at the same point as the admin command that is scheduling our completion queue entry posting. At some point, the scheduling is run, and we will go through all of the completion queue entries that are queued up, we will write them to host memory, and then we will raise the interrupt. The point here, as we can see, is that we have now batched up the interrupt, so instead of just doing a single, an interrupt for each of the completion queue, we do a single completion, a single interrupt for all of the completions that we have processed in this batch. So let's actually look at how this works in practice. So this demonstration, I want to show just what actually happens when the device first boots up. Now, up here I'm using a small tool that I hope will be open sourced by the time of this Q&A associated with this talk, but what this tool basically does is just wrapping QEMO and generating a QEMO invocation, which includes a bunch of stuff. And it does so from a simple configuration file, written in batch, instead of writing and remembering all the stuff in QEMO, you write this small configuration file. Now booting up the device, we get a bunch of stuff in the PCI, in the log trace. And what we see here is that there are two namespaces, which are registered on the controller device. We also see that the host operating system, or the host at least, is going in and actually reading some of the capabilities we were in, but those were low numbered registers on the device. So reading some capabilities, figuring out stuff about the controller. And then one of the important things that we talked about is that it's going in here and it's actually writing the address of the admin submission queue, both the submission queue as well as the completion queue. When those addresses are bootstrapped, we're actually ready to enable the device and that is what the host is doing right here. It is writing to the controller configuration register, and that last bit here is what is actually enabling the device. And we see that exactly from the trace, we see that the start is a success, the controller enable bit has been succeeded. At this point, the controller is ready to actually receive commands, and we see here that they're actually receiving a identify command. So if we log into the host and try to do some IO, then what we're going to do here is issue a very basic IO. We're going to issue an IO to the first LBA on the main space. We're going to be issuing a four LBAs. We're going to write that 16 bytes, 16 kilobytes, because each LBA is four kilobytes. By doing so, we see here that the host will ring the ball bill to notify the host that we have new commands in the queue. The command will be parsed as an IO command, because it's an IO queue, specifically a write command. The write command specifically is now parsed to go against the name today's named one for LBAs 16 kilobytes against the first LBA. The data associated with this is located in a PRP. So we see here that we have 16 bytes, kilobytes of stuff encoded into PRPs. And we know that the first PRP is the first page, the first 4K of this data. The other PRP, the PRP2, is instead a pointer to a page that contains more PRP addresses. So what we see the device now doing is actually mapping the first address in the first PRP, then following this pointer, reading the data here and having each of the addresses located in that memory location. We see that's what I'm in here. Having all four means that we have now mapped the data in all four memory locations, and we actually issued the asynchronous IO. And at some point that IO completes. And we include the win queue, the completion with a successful return status, and we raise and interrupt. And finally, when the host has actually read this, the host will write back to the completion queue doorbell and say, okay, thank you. I've actually read the completion and it is now free for you as a controller to reuse. Okay, so let's now look at the sown namespace command set and the NVMe sown namespaces. So a sown namespace is basically just a normal namespace that is divided into contiguous non-overlapping logical block address ranges that we call sowns. Now, while we still have the entire LBA space from zero to n-1 in being the capacity of the namespace, each of the sowns has a certain capacity and a certain size. This means that in the same address range, we now get a number of sowns from sown 1, 2, 3, and so forth to the maximum number of sowns that can fit into this namespace. The sowns are defined with a number of properties, including its state. These are an empty state, which means that the sown has not had anything written to it yet. It is a number of open states, which means that it is basically partially written. It's also something called closed, which is actually also a partially written state. And then there is the full state, which means that you can write anything to the sown anymore. And then there are a couple of error states, the read-only state and the offline state, which means that something has happened to the sown. Another important parameter is the sown write pointer. And the write pointer is pointed to the location at which the next write must begin. And whenever a write completes, the sown write pointer is advanced and pointed to the next free location in the sown. And any attempt to start a write at something not pointing and not at the write pointer will result in an error. There's also, because of this write pointer, there is a one sown, one write rule that basically says that you should not submit multiple commands against one sown at the same time. You have to submit one write against the sown at the sown write pointer, wait for that to complete. When you know it's completed, you can write to the next position with the updates sown write pointer. So there is a way to solve this. It's called sown appending, but it is, and we shall look at that in a second. So one of the things that comes with the territory of using sown namespaces is a certain amount of explicit sown resource management. That means that the host is now responsible for a number of the tasks that the sown state device usually used to do for you. What this means is that the host gains a certain amount of control over how resources are used on the device. So, and this is the reason for all of these states. So basically a full state means that the control, a full and an open state basically means that the sown is not really using any resources. But as soon as you start writing something to the device, and it moves into an open or closed state, it will take up a certain amount of resources on the device. The device must keep the current value of the right pointer somewhere in memory with open sowns. There's typically a write buffer involved. So there's a bunch of memory that needs to be managed here. Now the open state is actually two different states. So it's an explicit the open state and implicitly open state. The implicitly open state is something that the controller is a little bit more involved in. That means that the controller is allowed to move the sown out of the implicitly open state and into the closed state. But this is not something that is allowed if the host has explicitly opened a sown and moved it into the explicitly open state. It is at that point the host's responsibility to make sure that there are enough resources on the device for all sowns to be in the states that the host wanted to. If any of these resources are violated, this will also give an error when you're trying to write to a certain sown and resources are not available. So as you can maybe see there are a bunch of complexities involved. So there is this explicit sown management that you certainly have to do that you didn't used to do. There's a new command used to actually get information about all the sowns in the namespace. You have to manually reset these sowns whenever you've written something to them if you don't want to use anymore or you want to override the sown. You have to manage the number of resources. You do this through the sown management send command. If you want to do a higher queue depth write to a single sown, you have to use something called sown appending, which solves the one sown one write rule. But at the cost that you don't actually know where your data is being written to, instead of writing to a certain point in the sown you are just saying I want this data to go to this sown. When the write has completed the controller will inform you in the completion queue entry of where the data actually ended up. So as you can see there are a number of complexities. And this is where a device model such as the NVMe controller in QEmov that implements this becomes extremely useful. This means that using this device you can start developing applications to use these extremely efficient drives. And I will leave it to the marketing people to actually sell these drives, but let me just say that there are a number of efficiencies by using sown devices. They are way more efficient, but of course it comes at the cost of this slightly higher complexity. So using this device you can actually start developing applications for it. And to serve a sown namespace device in QEmov, what you do is that you on the NVMe namespace device you add that it should use the IO command set with the identifier 2, which is the identifier allocated in NVMe to the sown namespace. And you provide the capacity of each sown. What you get here is that you get full mandatory and most optional support for everything in the ratified technical proposal. That is you get the sown management send and receive commands. You get non sequential that is hierarchy depth one sown command. You get tracking management and enforcement of of so resources, even though they don't mean that much to the actual device, the QEmo device. But you can at least emulate it and you can see how a real device would would would would behave. And you get a lot of trace events that can be used for debugging and protocol inspection. So fundamentally this requires the implementation of two core features. It requires the implementation of the sown resource management state machine, which is basically just a big switch statement. This is what needs to handle the to validate the sown transition between states, and it needs to track various so resource usage. Another big part is handling these sown right pointer updates. That is advancing the right pointer whenever a right completes. It is making sure that the right pointer is not violated when a new right comes in stuff like that. So this is a relatively modest chain set and where we are actually using while while it is about 1500 lines of code inserted into the code. The vast majority of this goes to basically the state machine. We are commending a couple of the commands. Which is just accounting commands and the 250 lines of specification and internal data structures. So, all in all, there is just about 350 lines of code that's actually hooking into the actual IO path in the device. And this is mostly basically handling the right order. You can get this headset from the links I have here on the slides. So let's briefly look at how these right pointer updates is actually managed in the device. So a very obvious and maybe naive way of handling the right pointer updates would just be to update it by the number of blocks written when the right is complete. Now the problem here is that if we were to verify the rights, then we would suddenly allow multiple rights to go and be written at the same location in the zone, which is not allowed by the specification. The point here is that this is not allowed by actual NAND. So, so so this is why I would not be allowed here. Now, we also have the problem that we would not know what the next next value of the right pointer would be so we cannot really efficiently support. Now, the obvious solution here is to use a staging right pointer. What we mean here is that we use a right pointer. So now we have two right points we have a right pointer that is the authoritative right pointer that is tied to the actual zone. And then we have an internal in memory staging right pointer that we advance immediately when the right comes in. Having that right point means that now we know that we have something in flight, and that the right point of all these succeed, then the right pointer is at this location or it's going to be at this location at some point. This means that as right comes in, it is trivial to determine if we have a right point of violation. And it's also trivial to support the sonar pen because we can now just use the staging right pointer as the location of the next right. So we will update the authoritative zone right pointer, the actual zone right pointer whenever the AI goes complete. And while this actually allows a higher normal regular rights to be issued in order to the device and actually being completed and processed without any big issues, this is an implementation detail of QEMO because the commands in the condition queue are handled in order or sequentially. But this is a implementation detail of QEMO. So QEMO allows this to be catched and actually replied to the user as an error that hope you're doing something that you should not do. And this is another example where QEMO device can help in developing this because this is the subtle bug that you could end up doing, unknowingly in your application, but a real device would not actually tell you that you're doing something wrong. It will just suddenly start spilling out errors and saying what are you doing? So let's look at a flowchart like the one we did for the basic IO command processing and look at how soned IO processing hooks in. So basically on the IO path, what we do is that we say if this is soned namespace, then please check the zone. The zone at least has to be not offline. So if it is a read, it's very easy because there are no differences. We can just use the completely conventional read path that we already have implemented in QEMO. There's no right pointers we have to take care of. The only thing that's important here is that the device or the zone is not in the offline state. If it is a write, we will transition it autonomously to the implicitly opened zone state, which will mean that we will go over and call the resource management state machine. Now, the zone resource management state machine might require to close another zone, or maybe even transition another zone to the full state to actually relinquish some resources to be used by this transition to the implicitly opened state. And that might cause a change to the Soned Changed Soned List log page, which will further issue what's called a asynchronous event notification, something we shall look at a little bit more detail in the final part of this presentation. So if everything goes well, we will be advancing the staging write pointer and we will issue the asynchronous IO, and then we will yield and wait for the IO to complete. At some point, the IO completes and we advance the actual write pointer. If this causes the zone to move into a full state that is writing the last LBA in the zone, we'll again go to the zone resource management state machine and transition the zone to full. And finally, we just enqueue the completion queue entry as we do in the normal IO path. So for the final part of this presentation, let's now look at how we can add a custom NVMe command. In this case, what we're going to try to implement is a asynchronous event notification trigger command. Now, NVMe includes a mechanism for the controller to out of band notify the host software of certain events that happen on the controller. And the way this works is not unlike what you would call long polling if you did some kind of web development. Basically, the host submits a request, the asynchronous event request command. This is an admin command, and then it just waits. The command does not have a timeout and it will not reply or complete immediately. Instead, when at some point an event occurs, the controller will complete the command and the command completion queue entry will include information about the event. As we can see below here, you see that it includes in the D word zero of the completion queue entry. We get the type of the event, we get some particular information about the event, and we get a log page identifier. The point here is that all events are tied to a log page. So whenever an event happens, the way for the host to sort of acknowledge that it actually saw the event and chosen how to respond to the event is by reading that log page for additional information about the event. And then the controller sort of knows that the host has acknowledged and is at least now informed about this event. That could be critical, it could be transient, it could be a lot of different stuff. So the events are grouped into types. There are the error types, which is general error that is not associated with the specific command. Then there are smart and health information sort of like a the temperature is too big too high on this device, please turn up the cooling various notices that the oh something changed here you might want to know about that, but not really critical. Then there are some command specific events that can be defined by the various I command sets as we saw earlier actually this is one of the commands that the zone commands that has this particular event that is that oh some other command or whatever in the the zone namespace suddenly cost the state of this zone to change. And that's something that the host would like to know about. Instead of just constantly looking at all the zones, the control is now able to actually tell the device or tell the host that this zone, this particular zone change the state. Now, there's also some vendor specific stuff of course that that anyone can choose to implement the way to see fit. So the basic information is included in the CQE as we saw. And, but if you want more information about error, you have to go to this log page that is in the identifier. So the, let's build a identity asynchronous event notification trigger command. Now the point is that this the, the asynchronous event notification mechanism is very difficult to test. The CQE includes a test case for this and what it does basically is going in and forcing the device to, to basically raise an event because of a temperature going to high because you can control the threshold. So instead here now is trying in QEMO to produce a command that will include from the host include the type, the information and the log page that should be included in the event generated. So upon receiving a command like this, the QEMO device should include a asynchronous event notification if there are any outstanding asynchronous event requests. So here we have the actual code required to implement this new command. So what we see here is that we have enabled or added a new, a new upcode for this vendor specific command. It is in the range of the value of reserved for vendor specific commands, and we have chosen the first one available. The actual command is pretty simple and it will just extract the keyword that we have chosen to hold the information about the event. It will extract the event type, the info, the log page, and it will verify that the log page is valid, and then it will include an event with this information. So if we go down here, we can also see that because this is an admin command, we are hooking into the NVMe admin command function, and we're just going to match on the upcode and issue that command. So let's see back here in our QEMO how this actually works. So what we have here is a QEMO with support for this command. So the first thing we're going to do is be using the NVMe tool with the admin pass-through command and passing any asynchronous event request command. It doesn't have any other parameters than the actual upcode. We see in the trace that we're not waiting here. So the next thing we're going to do now is generate an event false to control to generate an event we're using this new command. We're going to use the admin pass-through command again, and we're going to put with our new upcode that we have defined. And then we're going to put the information about in dword 10 that we are generating a smart type event. It's about temperature. It doesn't really matter. And we want you to read the smart information log page to clear the actual event. Now generating this command, we now see that the actual AER completed immediately. And we see in the trace here that the trigger came in. This calls the event to be queued. The AERs that are currently queued up will be processed and the completion queue entry for that AER is posted. Now, interestingly, if we now hook up another AER and we try to generate a new event, we see that the event is actually in queue. It is generated, but it is currently matched. What this means is that because the host has not yet read the log page, we're not going to be generating any more events of this type until the log page has been written. So this event is basically ignored. So we can read the smart log to clear this event. Doing that, we can now again generate the event and the admin AER should complete and so it does. So that actually concludes my talk and thank you very much for attending if you got to this point. Please reach out if you have any questions or feedbacks, but I hope that this talk maybe demystified the NVV device for you and maybe encouraged you to actually try and work with this yourself. We would be very happy to accept contributions as always. The patches and the stuff that I've been using in this presentation are available at the get repository in the bottom of the slide. And if you have any kind of questions that you cannot handle here at the Q&A right after this talk, please email me at the address also on this slide. Thank you. Good evening.