 Hello everyone and welcome to my talk on transport-level testing of NVMe devices using VFIO. My name is Klaps. I'm a sub-netware engineer with Samsung electronics and on my database basis I work on open source. I'm a co-maintainer of the QEMO emulated NVMe device and I also work in various other areas of the open source NVMe ecosystem. We'll be talking about VFIO today and some of the core concepts that we're going to have to go through to understand what we're going to work with is we're going to have to go through some low-level stuff in NVMe and PCI Express. Then we're talking a little bit about the IOMMU and how it works and the VFIO kernel framework. We'll use this to try and write a driver in less than 30 minutes which will be able to do low-level commands submission to NVMe devices and we'll also see how we can interact with the controller memory buffer which is a PCI Express-based feature. So NVMe one slide. NVMe is non-volatile memory express and it's a storage interface that's designed to exploit the low latency and inherent parallelism of NAND flash memory. Now some core terminology here is the controller which is basically just a PCI Express function that acts as the interface to the host and then there's the command queues and the command queues are the core feature that makes NVMe tick. The host will write the submission queue entries that describe a command to a dedicated submission queue and the controller will write completion queue entries to another dedicated queue called completion queues. So while this might be easy we won't be able to do with one slide we're going to have to need way more than that and because by the end of this talk we'll actually be having a rudimentary NVMe driver. So we will focus on core concepts which are the queues, the submission and completion queue entries, PCI doorbells and core controller configuration and we will only be talking about a single NVMe command, the identify command which can tell the host about various stuff about the controller which means that we will skip all other commands like IO and you can read these up in the spec or look these up in the spec afterwards if you're interested. We also won't be talking about specific namespace types, the standard NVM namespace type and shown namespace type if you're interested in this I suggested you go see my talk from last year as an open source summit which have a talk specifically on the shown namespace implementation in QEMO. So let's first look at the actual submission queue entry. It's a 64 byte payload and it describes everything that the controller needs to actually execute a command. There's a lot of fields in here but we are only specifically interested in three of these for what we'll be doing in this talk. So the first one is the command identifier which is a host chosen unique ID for a command and then we have the opcode which together with the actual submission queue that we submit stuff to uniquely identifies the command that the controller should execute and then there's the data pointer which describes the payload that's associated with the command. Now the payload or the data pointer is a 16 bytes so but it actually consists of two 64 bit memory addresses what we call physical region pages that always point to something that are of the size of the host operating systems page size. So the queues that are central to NVMe are basically circular buffers that are defined by two pointers. There's the tail pointer which is incremented when we add something to the queue and then there's the head pointer which is incremented when we are removing entries from the queue or reading entries from the queue. And as you can see in the example figure here we have a queue where we have three entries in it which and the reader has not read any of them so the head pointer points to the first location and for the producer or the writer to know where to write the next entries we have the tail point to the point to the next empty slot in the queue. So the queues are in that way locked free because the reader will never read past the tail pointer and the writer will never write past the head pointer. So the host and the controller can act both as queue readers and writers depending on the actual queue where for instance in the case of a submission queue the host is the writer and the controller is the reader and for completion queues the at the other way around. So the queues are normally allocated in main system memory and with this means that from the host point of view writing a command or reading a completion entry is as simple as a memory copy. So this brings us to some questions. So first thing is how does the controller or the device actually know where to find these queues and how do we get the device to fetch and execute these commands when we have written something to it and how do we know if the command was executed and when it was actually executed. So to configure the device until different stuff the PCI device can expose a set of configuration registers we call this the PCI configuration space and it's mapped into into the host memory address space. This means that the host can go into this address space and configure the PCI device and read various stuff about the device in this configuration area and the device can also expose memory areas to be mapped into the host address space we call these the base address registers and they contain an address which the host configures and this tells the device that at this location we will be reading these configuration registers that the device expose. These are different from the PCI registers so this is custom registers that are unique to the device type say NVMe. So in the case of NVMe the device will expose an area of memory called the memory base address register and this memory location contains a basic controller configuration and is used for the basic controller configuration and communication between the host and the device. So it looks something like this when we look at it in tabular form. So there are a bunch of registers here there's the controller capabilities register which is a read-only register where the device can tell the host about various capabilities and features that it support and features that it does not support then there's the version register that tells it about what version of the specification are we implementing and then there's the control configuration registers where the host can set up the device in a certain way with certain parameters that are needed for the device to operate. We also have these two special registers called the admin submission queue base address and admin completion queue base address. These are special registers that tell the device where to fetch admin submissions, admin commands and where to post completion entries for these commands. So now we know how to basically bootstrap the device by configuring this controller configuration register. Now how do we get the device to actually fetch and execute the command? For this we use something called a PCI doorbell. A PCI doorbell is just a common name for write only memory map register. We can read from them but the behavior of that is undefined and so from the point of view of the host we will only be writing these registers. As you can see here they are also located in the M bar or the first bar on the device and it starts at the address of the memory bar plus 4k so the address of 1,000 looping hex. The tail doorbells are four byte wide and they come in two forms. They come in a tail doorbell and a head doorbell and we shall see how these are used. So when we write this we call it ringing doorbell and the way we do this is that when the host has written one or more entries to the submission queue we sort of can kick the device and tell it now you need to start executing stuff. We do this by writing the new value of the tail pointer to this register. So we can see here that it's basically just a write to a memory location at a certain offset with a certain value. So the value tells the device that I have inserted entries up to this point in the queue and you may now fetch them from your previously head position. In this case this tells the device that it can execute and fetch and execute entries 0, 1 and 2. So what happens when the queue is full? In this case that's when the host has produced enough submission queue entries that the tail pointer goes to 7 in this case and there's no more room in the queue. When that happens the controllers somehow have to notify the host that it has executed these commands and that the host may reuse the slots in the queue. Note also that is the responsibility of the host not to override the non-fetched entries in the queue. But to understand this we have to look at how NVMe piggybacks this information in completion queue entries. So specifically how do we know if a command was executed? To do that we use the completion queue entries and these are posted or written by the controller whenever it has finished executing a command. The completion queue entry is 16 bytes so it's smaller but it contains just enough information for the host to know about the status of the command. Was it a success? Did the controller encounter some kind of error? Maybe it didn't understand the command, the opcode that we requested or maybe we encountered a memory error or something like that. It will always add the complete command identifier which is used by the host to know what command this relates to. It also includes the identification of the completion of the submission queue that we originally submitted this identifier to and it also includes a new value of the head pointer. That is, this is the new value of the internally maintained head pointer to the controller which tells the host that I have now executed these commands so you may reuse the slots. To see there's an action when the controller picks it back something and gives a new head pointer location of four then the host will know that it can now reuse entries zero, one, two, and three for new commands. How do we know that when these commands are executed or have been executed? Since there's no doorbell register that can be ringed on the host to inform that the completion queue contains entries, we have two options. The host can choose to simply keep reading the memory location of the completion queue and wait for something called a phase change. This is a polling technique. We can also rely on interrupts that are maybe generated by controller or a combination of these two things. Let's first look at what polling means and what it means to look for a phase change. There's a special bit in the completion queue entry called the phase bit. The phase bit is inverted by the controller whenever it writes or overrides an existing entry in the completion queue. When we begin operation with a completely empty queue the phase bit is zero and the host may keep reading this memory location, the tail, and keep reading it until the phase bit changes. So when the host writes something it inverts this phase bit, goes to one. The host will immediately notice this and then it knows that this is a new entry and then it will advance the tail position and it will check the next entry. If the entry or the phase bit has changed again then we know that this is not a new entry and we stop the polling or we wait for another interrupt. So while polling allows very low latency it usually comes at the cost of dedicating an entire call to keep doing this constantly. So another way to do this is to rely on interrupts and it is the standard way of working with NVMe devices. So in an interrupt based system we rely on the controller to generate an interrupt which basically says that something's up because the interrupt doesn't carry any information other than something's up you need to take action. Now depending on the configuration and depending on the capabilities of the device the interrupt may indicate that there is new stuff in a specific completion queue or it may just indicate that there's new stuff in some completion queue. So we still use the phase bit because when we get this interrupt we don't know how many completion queue entries the queue holds. So we still keep reading the completion queue until we see the phase change in the phase bit once again. So finally we need some way for the controller or the host to actually indicate to the controller that it has read this completion queue entry and that the controller may reuse this slot in the completion queue because the controller is under the same restrictions that it cannot override the entries until they are acknowledged by the host. So we do this with another doorbell we call this the head doorbell and it's also registered in the bar and basically what happens is that we tell the device the new location of the head pointer, the maintained head pointer but notice that the entry is not zero or cleared in the memory it remains in place which means that the phase bit value remains in place. So at the next point when we go around through the circular queue and we come to this position then the controller will again invert the phase bit and we notice a new entry. So that's basically it for the low-level NVMe stuff that we're going to need. Now in part two we'll look at the VFIO or IOMU and how to actually use this and try to do these steps manually on a device. Okay so what we've been going through right now is basically what the kernel driver or the user space framework does for you. So you don't need to worry about these details to do amazing stuff with an NVMe device. You can use the block layer abstractions of your operating system coupled with a high performance synchronous programming model like raw IOU ring or a slightly higher framework like XNVMe or you can use user space frameworks such as SPDK for even more control. But neither XNVMe in its pass-through mode or SPDK allows manipulation at the actual transport level. Now all the abstractions of these frameworks are at the command level which means that if you look at the API the SPDK lowest level commands is about writing a command an admin command or an IO command and it's about simply waiting and processing the completions. This is similar in XNVMe where you can pass through a command to the device and you can wait and you can look at the completion queue. So this means that there's no manipulation of the queues directly and one of the things is that both XNVMe and SPDK relies on a concept where we always have a one-to-one correspondence between submission queues and completion queue. NVMe actually supports that multiple submission queues can result in completion queue entries going to one completion queue so you can have a many to one relationship between submission queues and completion queues. This is not supported by any of these frameworks. You also have no specific control of the interrupt vector configuration so you will always I believe that the SPDK will try to attach SPDK actually doesn't use interrupt vectors it uses polling on all the queues but you cannot set up a set of completion queues to use one interrupt vector or another set to use one vector each and stuff like that and you basically also have no low-level controller configuration abilities at all these frameworks. So this makes them a little less suitable for doing really low-level inspection of your devices and testing of your devices and the code paths in the device. So if we wanted to actually do some of the stuff that we learned in part one how would we go about doing this? So we could go to the standard NVMe kernel driver in Linux and we could modify it to do some custom stuff we could add a custom iOctil to do a specific test and stuff like that or we could simply disable all security features of the kernel and just go completely crazy as root and do this through the iOuPCI generic driver or we could actually utilize the virtual functions iO framework to do this kind of custom stuff but in a very safe way. So how low can we actually go? So the problem is that interacting with devices at the level of the registers had traditionally been a job that is for the kernel to do. It's a very privileged activity reserved for kernel and kernel headers but the virtual function iO framework actually changes this at least sort of. So Vfio is a driver framework in itself and the VfioPCI is a driver that the PCI devices will attach to a boot bound to instead of the normal driver like NVMe. Now the point here is that we retain the kernel retains authority of the device it's still in charge of the device it still protects the device from user space but the driver the VfioPCI driver provides full access to this device from user space which means that we can actually read and write from the PCI configuration space and we can read and write to the bars and it gives us very fine grained interrupt control to configure it as for pin based legacy pin based interrupts or MSI and MSIX configurations. Now it all relies on IOU based DMA translations which allows the driver to limit what the user can actually program the device to do to to make it safe and it also confines this DMA to the DMA of the device to the address space of the user space process. This is essential since traditionally the IOU devices have been working with physical addresses such that you could you could basically instruct the device to perform a DMA into another physical address which may belong to another process or something like that but the IOU provides a translation between the IOU devices and the CPU it basically does what we call it adds something called IOU virtual addresses which are sort of like virtual addresses a virtual address space but for the IOU devices that match to physical devices that can then go into the system memory. So it's basically just what the memory management unit does for virtual for regular virtual memory when the CPU is operating. So and as we can see here in the in the diagram you have a bunch of IOU devices behind this IOU and they will work with IOU virtual addresses. Now we can actually have multiple IOU devices behind the IOU and in this case the IOU devices can technically depending on how the motherboard or how it's actually wired up these devices may actually be able to interact with each other. So we can still have one device that could read or write to the other device through DMA. So the VFIO framework and the IOU framework in the kernel has the concept of an IOU new group which is basically the lowest granularity in which we can ensure safety. So for a perfectly isolated IOU device it needs to be in a singular group with no other IOU devices otherwise we need to accept that the security or the isolation granularity is at the level of the group marked here. So to to work with VFIO there's a lot of boilerplate involved. So VFIO worked with something a concept called a container and when the first thing that you have to do when you start working with VFIO is to create this container and then you can verify the VFIO capabilities like what's the API version supported by the kernel is there actually an IOU that we can use and then we have to determine the group of the device the IOU or VFIO IOU group and we need to verify that this group is viable and what viable means is that each of the devices in the group has to be either not bound to any driver or at least everyone or all of them bound to the VFIO PCI driver. So when the group is determined to be viable we can add it to the container and the cool thing here is that since we are since we we can ensure that different groups interaction between different groups will always go through the MMM IOU we can add more groups to the same container and work with all of them independently. When all this is done we can enable the IOU and we can retrieve a list of the ranges that the IOU virtual addresses can have and that we can use and then finally we can actually open the device we get a device handle which is a file descriptor and we can start configuring it that means we can set up the memory regions the PCI configuration space any bars that we're interested in and we can configure the interrupts of the device. So the API or the user space API from the kernel is very minimal but also extremely extensible for future features and capabilities. It looks something like this when you work with it all which means that it's a bunch of IOPTL calls for instance this we have an example here where we just query the group for the status of the group and we can check if it's viable and then there's an example here of how to set up interrupts as you can see you set up a data structure of a certain size you set up a bunch of fields you but what you do here is that you set up the exact way that the interrupt should be configured and then you issue another IOPTL to actually configure this for the device. Now as you can see all of this is agnostic to PCI it's an abstraction on the device which uses this common framework. So there is a pretty good documentation on the kernel website for the actual VFIO framework but there's also a lot of good code in the QEMO source code. The QEMO source code has extensive support for device pass through this means unbinding a device on the host and basically binding it or passing it through to the machine and what happens here is that basically the virtual machine or the hypervisor becomes a user space driver that manages the device and passes it through to the guest operating system. This is done by the VFIO subsystem in QEMO. QEMO also has an NVMe driver for using raw NVMe devices as block storage and this is VFIO based so there's a lot of good stuff in there. The VFIO helpers we call a library but the object in QEMO contains a bunch of various very nice helpers to actually work with this. So in general in QEMO there is a bunch of great code to learn from and to look through to actually understand better how this works very deep. But one thing that we don't have is that there's no lib VFIO. There's no user space, there's no library available that actually tries to unify all of these best practices that are especially in QEMO code and all these utility functions. There's no library that does this. So I've been working on a library called, so far just called VFIO in my local repository and it's a set of utility libraries written in C to work with the VFIO user library, user API. The library contains functions to do the basic VFIO device initialization of the spoiler plate code and it includes extremely simple naive and fixed IO VA allocator. It just helps you allocate IO virtual addresses but you cannot free them. You can just allocate a new one and then the allocator just makes sure that you don't reuse these virtual addresses. So in some sense, while this can give you unused IO virtual addresses you still have to manage them yourself but it's a plugin-based so you can add another allocator of your own writing if you want a more advanced allocator. Then there's a bunch of utility functions, functions for doing portable IO, indianness, correct IO. That's support for or utility functions for setting up the interrupt configuration and there's utility functions for doing DMA mapping of memory. So all of the boiler plate stuff that I talked about previously can basically be done in one single API call here. It's called the VFIO-PCI init and it will initialize a VFIO-PCI state from a PCI ID. And then there's another call to configure the IRQ basically setting up a Unix kernel event file descriptor and match it to a specific interrupt vector on the device. Now this is all intended to be open sourced, I'm still working on it but my intention is to open source it, no less than Q3 this year. So besides just basic VFIO functionality, it also includes a bunch of indianese specific functionality because that's what I'm working on. It provides a bunch of utility functions for this like enabling or initializing the controller, resetting the controller, enabling the controller, and creating IOQ pairs to do actual IO. So it also has very low level command submission. So the call API is at the level of posting a command to the submission queue. It's at the level of kicking the device, riding the doorbell, ringing the doorbell with the kick call. There's a peak call which allows you to peak at the top entry in the completion queue and then there's acknowledgement function that allows you to acknowledge all the entries in the completion queue. It also has some mid-level convenience functions such as the post-kit weight acknowledge combined command that basically allows you to synchronously submit a command and wait for its completion while giving you a reference to the completion queue entry. So when we go about how to try and emulate all this stuff, what we can do is try to use QEMO which is a super nice platform to experiment with this kind of stuff because we need an IOMMU. Most platforms, most workstations already have this but it's nice to be able to work in a virtual environment. So setting this up requires you to configure QEMO with the Q35 machine type and you need to enable the kernel IRCOOT chip to be split or completely user-based dream and then you need to add the Intel IOMU device, the virtual IOMU and then as usual you add your NVMe controller and you add a namespace to it and then you set up your regular boot drives, configure the CPU type, amount of memory and so on network and so on. So when you have all this up and running, the first thing you do with the library is initialize the device at the PCI node. You do this with the VFIO PCI unit call and then at that point we can now map the controller registers. So in this case we're interested in mapping the M-bar or the first bar, bar zero that we talked about previously. We're mapping the controller registers and the doorbells separately as you can see here. We're mapping the first bar zero. We're mapping the first 4k of bar zero which is the controller registers and then we're mapping the doorbells which are the next 4k bytes on the bar. When we have this mapped we can basically reset the device which means that we write a certain value to the controller configuration register and the device resets and is ready for use. Then we can empty some memory. We use regular virtual memory using the MIMMAP which guarantees that you get a nice page aligned set of memory. Then we assign some IO virtual address. This is something we choose because we control the mapping. So in this case we just choose the null address. It is a valid address and this actually found a bug in the QEMO emulated device because it didn't actually accept the address of zero for say the admin queue or stuff like that. But the zero address is a valid address, hot address, so no problem there. Then we call the VFIO DMA MAP function which maps the virtual address to the IO virtual address and from there on we can use the IO virtual address in commands to the device. So what we need to do as we learned was to set up or bootstrap the device with the admin queues. So we assign two IO virtual addresses at the zero and address one thousand here. We memory map space for the admin queues and then we map them. The next thing we do is that we tell the device about the size of these queues. We do that writing to the admin queue attribute register and then we inform the controller about the actual address of the queues through the admin submission queue address register and the admin completion queue address register. Then we set up an event file descriptor and we assign that to interrupt vector zero which always corresponds to the admin queue. When all of that is done we can enable the controller which writes the configuration register, the cc register and from here on out the controller is ready to use. So let's try to execute an identified command and get some information about the device. Again we allocate some memory of a certain size, in this case the identified data structure given returned by the device is 4k in size and we allocate that. Again we choose an arbitrary virtual address, we can choose that ourselves so we can use the allocator from the library and then we map that address. Then we set up our command. We need to have a data structure available that actually maps to the submission queue entry that we saw in the beginning of the talk and we set it up. We set the opcode which identifies the command to be executed, we choose our controller or a command id and then we set up the data pointer. In this case we just use the first address in the data pointer and set that to be assigned by a virtual address and then specifically for the identify command there's a field called the controller namespace selection which chooses the sub command. In this case we are asking the device for information about general information about the controller. So to write the command we simply use a mem copy as we discussed. We write it to the virtual address, the address space that we see as the user space process and we write it at the current tail of the submission queue address and at the size of the command. Then we update the tail pointer and we take care of wrap around because it's a circular queue and then the final thing we do is that we write the doll bill and we use the utility function again here to do a memory map IO write of 32 bits to the doll bill address and writing the new tail value. So when this happens the controller will pick up the the command and it will execute it and it will generate an interrupt when it's done. So to do that we simply read from the event file descriptor which is a blocking file IO so we block here until it's actually until we get the interrupt and as soon as we have the interrupt then we can read the completion. What this means is that we start reading the completion queue, the admin completion queue at the current head position and then we wait for this face bit or we accept allow we expect a face bit change here. When we read that we constantly increase the head pointer value that we maintain internally and when the face bit changes again we know that we're done. So the final thing we do is that we inform the controller that we have acknowledged and we have read the completion queue entry and we write that back to the completion queue head doll bill. So let's look at this with an actual demo of how this works. So here I've booted up a virtual machine like I described previously and over here we have the trace log output of QE mode that allows us to see what it's actually going on on the emulator unit device. Over here I have my VFIO and the first thing that we'll do is that we'll check that we actually have devices and as we can see here we have a NVMe controller device, a QE mode emulated NVMe device which currently contains two namespaces and we're not really care about that right now. So if we look at the LSPCI we'll see that we have the controller here and the first thing that we do is that we unbind the device from the NVMe driver and we bind it to the VFIO PCI driver. So as we can see here VFIO loads up and what actually happens is that the controller is shut down and it's in a sort of a clean state. We don't know this for sure but in this case that's the kernel nicely shuts down the device as we unbind it. So the next thing we can do now is we can try to run this IdentifyExample and just to see the actual code here we can see that what the Identify is supposed to do is to initialize the PCI device, the writeouts of stuff from the registers in this case, the writeout version of the device. We will map some memory, use the allocator to reserve an IOVA, we'll map that virtual address then we'll set up the Identify command and then we'll use this convenience function to post, kick, wait and acknowledge the command. Finally when the command has been executed we're going to see what the device actually responded to us and print out something from it. In this case we're going to print out the vendor ID of the device. So running this we see that we get a bunch of debug information but over here in the trace is actually what's most interesting. So what we see here is that we see the library configuring the device, specifically it's configuring the admin completion queue and admin submission queue addresses and finally it's enabling the controller and there's some other stuff here that just is part of the initialization but finally here we see that the host or the application we have here is writing the submission queue doorbell and the controller picks this up, notes that it's an Identify command, it executes the Identify command, it maps the address that we have assigned in this case address 2000 and it writes stuff back into this and then it includes a completion and raising an interrupt on vector 0 and when the application here has read that completion queue entry again writes the completion queue doorbell with the new value of the head pointer. So we see here that we could read the register and get the actual specification implemented by the controller and we also read out the vendor ID of the device. Okay now so let's do some bit more tricky stuff. So higher end NVMe devices support something called the controller memory buffer. It's basically a region of general purpose read write memory that is located on the device and it is exposed through a bar, it might be an exclusive bar that means it's at offset zero or it can be an offset into a memory location exposed through a bar. There are two address spaces for this area there's the PCI express address range which is basically like physical addresses and then there's all IOBA addresses and then there's controller virtual controller address range. Now the the area because it's on a bar can be mapped like any other bar in this case we know that it's a 16 kilobyte controller memory buffer it's located on bar two at offset zero so we map the bar like we would map the M bar and we get a pointer to this virtual memory. So we can write it through the PCI express address range by simply writing directly to the pointer with a MIM copy or typecasting and write it then greatly into it. We can also instruct the controller to carry to execute a command and then post and then write the result of that command into the controller memory buffer instead of writing it into host memory. We do this by setting up like a virtual virtual address for the controller this is what we call a controller base address and this address is only visible by the controller so this is not an address that's backed by physical memory or something like that it's only available inside the controller and the controller uses this address to know if it's supposed to be doing a DNA operation or writing into its own internal memory. So we simply choose something in this case we could choose the maximum IODA address so we know for sure that we're not using an address that might be used for DMA so in this case we can use the max IODA address we can add one and we can align it to page size. Then what we do is write a certain register on the device called the cmd memory space configuration register and we write this address into that register and we enable it by setting the two lower bits to one in this case. Then we again set up the identify command but in this in this case the the the memory pointer that we ask the device to write the result into is this controller base address which doesn't have any backing in an actual memory on the host and then we use our convenience function to a post kick wait and acknowledge the command. Now when we have written this into the controller memory at controller memory buffer we can read out the data again we could either ask the controller to to read it from the to read it from the memory but we can also like just use our virtual address reference to read directly on the device and ask the kernel to perform the memory map IO for us. So in this case we just take the address of the controller memory buffer we typecast it to a identify controller data structure and then we can simply access the fields as we would any other data structure. So let's see this connection. So back in our demo setup now we're going to be using this example program called cmb and as I showed in the in the previous slides the the the only particular different thing we're going to be doing here is that we're going to be mapping this address we use it we look into a bunch of other registers on the device to to get information about the size of the controller memory buffer that's what we do here we read some specific registers that defines the size and and and it gives us a and we also can get the physical address of the actual device and or the actual the the bar and which we won't be using but but but it's nice to to to know what it is here in this example and then we we map the cmb we simply assign a controller base address and we write out what the result of that is we inform it the controller base address to the controller then we execute the command and then we write out the version field of of the result of that identified command so running the example and we'll see here that the controller setup here or what we're doing here is that we're setting up the this is all this stuff where we're actually setting up the the cmb and we see here that it says it's located on a physical address this is the physical address that the operating system has assigned to the bar and but we won't be using that because we're not interested in using the physical address so instead we assign the controller base address and we just choose something in this case we choose something that that's this address plus one and the the last i o b a range available and then we carry out the command and then we can write out the version field which tells us that it's yes this is specification version 1.4 okay so rounding up on my talk there's some key takeaways here i wanted to to highlight i think the the key thing here is that you can actually write a driver like you don't have to be a kernel hacker or kernel expert to to write a pc i driver you with a relatively brief overview of the device specification you can do this driver stuff from user space and as as we've seen even with going through the theory you actually wrote a very simple driver that can execute commands compliant with the specification in less than 45 minutes but i also think that that one of this what this shows is that we need some kind of lipo i i think it would be nice that we had something that we could all use that the community would maintain so i'll be outsourcing this as a start and then we'll see where we go from there i'm very happy to collaborate on this if anyone else is interested on interested in trying to get something like this up and running so i'm gonna say thank you thank you for for attending my talk i i hope you enjoyed it and if you have any questions i'm available for q&a both on email but also after the talk thanks again goodbye