 texts for software-enabled Flash, a newly announced technology from Kyokusha. Today I will be covering what software-enabled Flash is, why we created it, the concept and technologies it contains, and I will also show some demonstrations of the software-enabled Flash technology running on our prototype FPGA. I will finish by adding a little section on coding examples and then directing you to where you can obtain more information. So what is software-enabled Flash? Well, software-enabled Flash is a media-based host-managed hardware approach. We'll talk more about this in a few slides. We're trying to redefine the way the host interacts with Flash to allow applications to maximize performance and provide functionality that would be difficult if not impossible to achieve with existing interfaces. Giving the host control over media enables the host to define the behavior of the media and the performance characteristics for Flash. This can enable new storage functionality that maximizes performance and efficiencies or stated simply the ability to extract the maximum value from your Flash resources. Coupled with this media-based hardware approach is the software-enabling API. This API is designed to make managing Flash media easy while exposing all of the capabilities of Flash. So software-enabled Flash technology isn't just software, but it is software-enabled. The highlights of this API are, first, it is an open-source Flash native API. Next, it's designed to make it easy to create storage solutions with powerful capabilities. It abstracts low-level details that are vendor and Flash generation specific so that your code can take advantage of the latest Flash technologies without modification. Making an open project, any Flash vendor can build software-enabled Flash devices that are optimized for their own media. And finally, even though just the API and documentation are published today, Keelkshow will be releasing an open-source software development kit with reference source code, utilities, and examples. So before we go into more details, I'd just like to reinforce the message that software-enabled Flash was created to enable the development of custom storage systems that provide innovative solutions to storage problems. Software-enabled Flash is free from legacy disk interfaces and paradigms. It offers total control over the Flash media and can extract the full performance and efficiencies of Flash by providing solutions that are Flash native. Software-enabled Flash is extremely easy to adapt to new storage applications, to foster innovation in storage, and also to enable rapid deployment of new storage solutions. And I'll show some examples of this too. So in summary, Software-enabled Flash is shedding the legacies of hard disks, allowing applications to have much more control, to have predictable performance within the Flash device, and enabling flexibility and scalability for software developers. As previously introduced, Software-enabled Flash consists of hardware and software components working together. Over the past several years, I've had the fantastic opportunity to meet with the storage developers of most of the world's hyperscalers. Taken with the input of other engineers at Kyokusha, it has allowed us to distill a list of basic requirements and features for hyperscale customers. I should mention that although the hyperscalers face similar problems, their individual priorities and approaches vary significantly. On the requirements side of the slide, Flash abstraction is not just about code reuse. There can be major economic and performance advantages to transitioning quickly to newer generations of Flash at scale. Scheduling is an item that's going to be covered in depth later. Access to all the Flash media. In hyperscale environments, they refer to RAID within a device as a RAID tax. Their systems are designed with redundancy at the system level. And so they are trying to access all the capacity that's native to the device since they're going to be ensuring resiliency at higher levels in their environment. Also, for these hyperscale environments, CPU offload is important because they view the CPU as a sellable resource, particularly in hosting environments. And finally, device DRAM configurations need to be flexible. Device DRAM has to handle worst-case scenarios, but often that leads to stranded resources in their environment. On the functionality side, data placement is important to try and minimize right amplification for Flash media. Isolation is both for security and noisy neighbor problems. Latency control is a topic that we're going to cover in depth later in the presentation too. But for management, well, that's tied back to the flexible DRAM configurations, and we'll show more around that later. And finally, adaptability. A hyperscale environment is a very dynamic environment and changes quickly behind the scenes. The preferences for standard configurations that can be provisioned and deployed in real time to meet changing workloads. In response to those requirements and required features, the software-enabled Flash API covers these major areas and a few more that aren't listed. Group broadly, the areas of the API are concerned with the hardware structure, buffer management, programming of the media, and error management. Not listed separately, but touching many of these areas is latency control. In order to maximize the performance of Flash storage, it is necessary to be aware of the geometry of the Flash resources. The software-enabled Flash API allows storage applications to query this geometry in a standardized manner. Some of the characteristics of those are listed here. The API also allows control over how many Flash blocks may be open simultaneously and can control the management of the associated write buffers. When discussing the programming of Flash, it's important to note that the optimal programming algorithms vary from vendor to vendor and often between generations of Flash from single vendor. The software-enabled Flash handles all the details of programming and lets the Flash vendor optimize for their own particular media. The API was created with consideration for other Flash vendors, as well as all the foreseeable needs of Kyokushia's own future Flash generations. Finally, with respect to error management, software-enabled Flash allows vendors to optimize error reduction techniques for their specific Flash media characteristics. And it also controls the time budget for error recovery attempts. This is a high-level block diagram of our software-enabled Flash controller. It is one possible design and other vendors are free to implement different architectures as long as they comply at the API level. For example, although we currently use the toggle interface at the bottom of this block diagram to connect to our Flash chips, other vendors may implement on-fee interconnects or a different style of interconnect. What that our design uses standardized interfaces wherever possible, including the PCIe interface at the top of the block diagram. The controller itself is focused on management of the Flash media, programming, lifetime management, and defect management. The controller has advanced scheduling capabilities on a per-dye basis and hardware acceleration for garbage collection operations if needed, as well as ware-leveling tasks and other administrative tasks. Another important call out here is the use of on-device DRAM is optional in the software-enabled Flash technology. Host memory resources may be used instead. So as we just mentioned, the use of DRAM on our software-enabled Flash devices is optional. Why is that important? Well, the answer lies in the fact that hyperscale customers often require thousands of simultaneously open Flash blocks on each Flash device. The actual numbers vary from customer to customer, but we've heard requirements between 4,000 to up to 16,000 open blocks on each individual device. Since each open block requires a write buffer, this can create a demand for a lot of memory within the device, potentially tens of gigabytes of DRAM dedicated to use as a write buffer. It's often the case that the actual number of open Flash blocks can vary over time as a function of the system load. So sizing the device DRAM configuration for a worst-case scenario creates what we call stranded DRAM resources within the device. They must be present to handle peak load, but in normal operation, that's DRAM that is essentially idled and not used for other purposes. Software-enabled Flash supports device DRAM configurations. It also supports host-only DRAM configurations and, most interestingly, hyper-configurations that allow the DRAM to be sized within the device for normal usage and for the device to drop on host DRAM resources during periods exceeding the limits of the drive's internal resources. We should note that in systems that use host DRAM resources, there are system requirements to protect against data loss in the event of unexpected power loss. Many hyperscale environments already have this in place with non-volatile memory resources, system-level mirroring, and system-level erasure coding. So in summary, we support both DRAM within the device, DRAM only in the host, or hybrid solutions that contain DRAM both in the device and in the host. There are benefits to each approach. With the unified write buffers that can draw upon host-side resources, we can dynamically adapt to changing requirements and workloads, and we can optimize DRAM at the system level without stranding any DRAM resources. The conventional architectures are more appropriate for systems that cannot meet the resiliency requirements at the system level and rely upon the drive for data loss protection in the event of sudden power loss. Now I'd like to introduce the software components of software-enabled flash. Keopsia has created and will soon release a software development kit. This will provide open-source reference block drivers, open-source reference flash translation layers, and open-source device management utilities. Bundled with the software development kit will be an API library, once again in open-source, and also will have a device driver too. This block diagram shows how the pieces of offer development kit interface with each other. Two notes on the software layering. First, as you can see, it's possible for a user application that is software-enabled flash native to interact directly with the device, bypassing file systems and traditional flash translation layers and block drivers. We have a couple of proof-of-concept applications running today on our own FPGA prototypes. These include a software-enabled flash engine for FIO, as well as a version of ROXDB and a version of Firecracker that are software-enabled flash native. Although these applications are currently just proof-of-concepts, in the future we plan to include open-source native applications as part of the SDK. The second note is that the block-labeled SDS stack is a software-defined storage stack. Hyperscalers are already running their own software-defined storage stacks in their environments. These software-defined storage stacks can be modified to interface with the software-enabled flash library and are not dependent on the use of the SDK. In such cases, the SDK serves as the detailed example of the best practices for using software-enabled flash. The APN library is once again open-source, runs on Linux, it has both C and C++ bindings, it supports use in kernel space as well as user space, and provides easy access to all the features of software-enabled flash. And now for a system-level view of one possible deployment of software-enabled flash. Note that the items outlined with the red dashed lines are items that would likely be customized for different environments. Let's start at the top and work down. So at the very top of this diagram, you see unmodified applications running within a virtualized guest. Those applications typically use the POSIX API to talk to a file system, and the file systems use block device IO to talk to devices. In this particular case, since these are virtualized guest environments, they're actually talking to a hypervisor within the kernel. In this case, we're calling out that we have built a software-enabled flash QEMU block device driver. This interfaces with our reference flash translation layer and the software-enabled flash library to bring block-level features and functionality to the virtualized guests. The software-enabled flash reference FTL communicates to the software-enabled flash library using the software-enabled flash API. The library, in turn, uses IOU ring to talk to the software-enabled flash driver in kernel space. This allows us to bypass system call overhead for greater performance. And the final thing to note is that on the far left side of the diagram, outlined in red, there are soft native applications, software-enabled flash native applications that can interface directly with our library, utilizing the IOU ring mechanism to allow applications to interact with software-enabled flash devices without system call overhead for high performance. Finally, the software-enabled flash QEMU block driver, as I mentioned, it's part of this SDK, is useful because it allows unmodified applications to take advantage of many of the features of software-enabled flash, such as isolation, latency control, and die-time allocation, even without modification. So I encourage people to look into the possible uses of the software-enabled flash QEMU block device driver. All right, at this point, we're going to introduce some concepts and features of the software-enabled flash technology that are going to be necessary to understand later examples. This diagram is a representation of a possible software-enabled flash device. At the top of the diagram, we have the actual control logic. And then this particular device has 32 die that are arranged on eight channels with four banks per channel. The first concept that I would like to introduce is that of a virtual device. A virtual device is a set of one or more flash die that provides hardware isolation. So when you define a virtual device, you get to define how many channels the device is spread across, as well as how many banks. Individual flash die are mapped one to one to a virtual device. A flash die is never shared between virtual devices. The next concept is that of a quality of service domain or a QoS domain. This is a mechanism that allows us to impose capacity quotas and scheduling policy as well as to provide software-based isolation. Note that unlike the virtual devices, you can have QoS domains that share a single virtual device. So in this example, QoS domain 0 and QoS domain 2 are sharing virtual device 0. Likewise, QoS domain 4 and QoS domain 5 are also sharing a virtual device. The final concept is that of a placement ID. And this is a mechanism that we'll talk more about later that allows us to group data at the super block level within a QoS domain. This slide describes how the concepts introduced on the previous slide provide control over data isolation and data placement. Super blocks within a virtual device start out in a free pool. As QoS domains allocate storage, a block is drawn from the free pool and assigned to a QoS domain. The device can choose any free block from the pool, so contract blockware and block health and assign the optimal block to maximize device endurance. Super blocks are never shared between domains. There is no mixing of data at the block level. Once a super block is released, it's returned to the free pool. So over the lifetime of the device, ownership of a super block may change several times if there are multiple QoS domains within a virtual device. So the virtual device provides physical isolation at the dye level and can be used to shape both the capacity and IO throughput for an application. Within the virtual device, blocks are combined to form super blocks that are striped across each dye in the virtual device. This controls the amount of parallelism and the amount of potential throughput for the virtual device. To summarize data isolation, the two main mechanisms are listed here with their benefits and restrictions. So we have dye level isolation in the form of virtual devices and block level isolation in the form of QoS domains. Dye level isolation is the most effective isolation. It provides full hardware isolation between applications. However, typical devices only have a small number of flash dye per device. This can be in the range of 16 to at the high end, low hundreds. So there's a small number of potential virtual devices, and it's limited by the actual number of physical dye on the device. However, block level isolation using QoS domains is far more scalable. It can scale into the thousands of tenants. And while the isolation is not total hardware isolation, we can guarantee predictable performance and extend lifetime and keep data separated at the block level between different applications for either security or performance reasons. Closely related to data isolation is data placement. Here we're introducing the concept of a nameless write mechanism to control data placement. Why is a new write mechanism desirable? Why did we do this? Well, although the system benefits can be realized with control over data placement, if physical addressing is allowed for writes, the host becomes responsible for device wear. Flash memory is a consumable. It wears out with use. And so poor choices for physical data placement can wear out flash devices quickly. So even though the host would like to be able to control data placement to minimize write application and group data from applications or tenants that have potentially similar lifespans, the host really doesn't want to be responsible for the lifetime management of the media itself. How can the host have control over data placement without this responsibility of ensuring the device helps? Well, the answer is nameless write. When a new superblock is required for a placement ID or if a new superblock is manually allocated, the device chooses the optimal superblock to use. This is the framework for nameless write mechanism. Now let's show how the actual nameless write works. Nameless writes allow the device to choose where to physically write the data, but allows the host to bound the possible choices for where the device will put it. As mentioned earlier, QoS domains are mapped to device nodes in the system. And so the nameless write operations must supply a QoS domain handle as well as either a placement ID for auto allocation or a superblock flash address returned by a previous manual superblock allocate command. The QoS domain maps to a virtual device, which in turn specifies which die can be used for the write. So by virtue of the QoS domain, you've specified a subset of die where this write can be performed. The placement ID or flash address specifies the superblock, which is owned by the QoS domain, that should be used for the write. If a placement ID is specified, a nameless write command can span superblocks, and additional superblocks will be allocated to the domain as needed. In manual allocation mode, nameless writes cannot span superblocks. As illustrated, the device is free to write the data to any empty space within the bounds specified by the host. And when the write is complete, the actual physical address is returned. This enables direct physical read path with no address with translation required. Direct physical read optimizes performance and minimizes latency. The nameless write operation automatically handles all media defects so that you always are given the actual physical address at which the data resides. Similar to the nameless write operation, the operable flash has a nameless copy operation that can be used to move data within the device without host processing. This is useful for implementing garbage collection, ware leveling, and other administrative tasks. The nameless copy function takes as input a source superblock, a destination superblock, and copy instructions. These copy instructions are powerful primitives, supporting valid data bitmaps, lists of logical block addresses, or even filters to select copy data based on logical block addresses. A nameless copy command can operate on entire superblocks with a single command. This animation illustrates the difference in impact for the host to implement a garbage collect using standard read and write commands versus the nameless copy. As you can see, for the manual copy operation, the host had to issue 20 commands, 16 of which were reads, and fours were right to harvest the data for garbage collection. The nameless copy operation issued a single command to perform the exact same work. Now I'm going to show a video that was captured on our FPGA prototype system. So on the left-hand side, manual copy is being used for garbage collection. On the right-hand side, nameless copy is being used. The two workloads are identical. The blue rising graph are the actual garbage collect operations. Over the course of this workload, there's a dramatic difference in the impact to the system. As you can see, from the center graph, CPU utilization was much lower for nameless copy. Why was it lower? Well, to perform the exact same workload, nameless copy only needed 24 commands to do garbage collection, versus the manual copy needed over 800,000 commands. Likewise, nameless copy only transferred 120 kilobytes of data. These were the copy instructions, versus over three gigabytes of data that flowed through the host for a manual garbage collect operation. Pretty dramatic results. Another key feature of SAPRAEVAN enabled flash is the advanced queuing and scheduling features. We'll spend a bit of time here with several slides. Scheduling and queuing controls how die-time is spent. This means how much time is spent for read operations, versus write operations, versus copy operations, as well as the prioritization of multiple workloads. Consider a multi-tenant environment. There may be business reasons to enforce fairness, or give certain tenants priority. And these business needs may change over time. These tenants can share a device, and weighted fair queuing can support the performance goals. The host is allowed to prioritize and manage die-time through the software-enabled flash API, and the device enforces scheduling policy. This is the basic architecture of our software-enabled flash scheduler. First, I'm going to go down the feature list. Each virtual device has eight FIFO command queues. These are eight input queues to the scheduler. And since a virtual device can be as small as a single die, it can get down to a one-to-one ratio of a scheduler, eight-way scheduler per die. The device scheduler handles all suspend-resume operations for programming and race commands, giving read operations priority. The host can specify which of these eight queues, input queues, are used for each type of flash access command. So for example, you could configure all read operations to be submitted to queue zero, all write-off or program operations going to queue one. The assignment of flash operations to scheduler input queues is done on a per QoS domain basis. So one QoS domain can be submitting its reads to queue zero. A second QoS domain could be submitting its reads to queue two. Anything's possible there. Every queue can specify die time waiting for both read operations, program operations, and a race commands. It's a credit-based system that allows us to allocate die time based on very complex and flexible needs. And finally, the host can override the defaults for both queue assignment and operation wait for each individual command. Now let's talk a little bit about the functionality. We mentioned that each one of the queues can have separate waits for each type of operation, an erase, a program, or a read. If you set all the waits to zero, this works as an eight-level priority scheduler with input Q zero being the highest priority and input Q seven being the lowest priority. When all the waits are the same, it works as a round robin scheduler. And when all the waits are unique, it works as a die time weighted fair queuing scheduler. And now another demo of weighted fair queuing running on our FPGA prototype. So some things to note. We started off with QoS Domain 1 in red, having a wait of 198 and QoS Domain 2 in blue at a wait of 202. We did that so the lines wouldn't overlap each other. We're now going to alter the wait for read operations, giving a much lower wait for QoS Domain 2. This graph is a graph of read response latency. And so as we lowered the wait, the latency was reduced in real time. Now we've gone ahead and shifted the wait over to favor QoS Domain 1. And you can see the latency response curves for the two QoS domains have now inverted. QoS Domain 1 has priority over QoS Domain 2. I promised we'd talk about latency control and it's a really complex issue. These two graphs have the same average latency, but the top graph has a better minimum and a much worse maximum latency. So which is the better response curve? Well, the answer is it's a trick question. It really depends on the application. Some applications prefer consistent latency and others need the best possible performance and can tolerate long tail latencies. With this in mind and with the theme of host control, software enabled flash allows control over the latency response curve. We demonstrated a bit of that in the last slide. We do this using many features that we've already introduced. So we can isolate workloads to make sure that they do not interfere or impact the latency of other workloads through the concepts of virtual devices or QoS Domain scheduling. We can map commands to different die queues per QoS Domain. Once again, prioritize and shake latency response. We can wait the commands per die queue and per QoS Domain to prioritize the different commands. For each individual command, we can provide an override. So in the previous demonstration, that's what we were doing. We were altering, in real time, the weights assigned to read operations by using this override mechanism. The system also provides automatic program and erase, suspend and resume functionality. Software enabled flash also gives host control for garbage collection with the offloaded nameless copy accelerator. You can prioritize and assign die times nameless copy operations separate from user read and user program operations. And you can adjust the weighting to account for right amplification to give consistent latency response curves. We also provide host control over where leveling and patrol operations. Once again, to allow the best possible response curve to the needs of the individual applications. This is the final demonstration of our FPGA prototype. This is just a single still frame taken from a live demo that we gave a while back. In this demonstration, we define three separate virtual devices and they were running three different workloads with three different storage protocols. In this example, we were running the ZNS protocol, a standard block protocol and a hyperscale FTL protocol in the different virtual devices. The graph is a heat map of die activity. The blue bars represent read operations and the red bars represent write operations on a per die basis. None of the workloads are impacting each other's and the same device is supporting the three different protocols simultaneously. In this first domain, which is circled, the workload was a mix of reads and writes, a fairly balanced workload. The second domain was doing a redominated workload and the third domain was doing a write dominated workload. We really don't think that running different storage protocols simultaneously on a single device is a common use case, but the value is that a single device can be deployed at scale and then provision and configured in real time to match dynamic needs. As new storage protocols evolve, software enabled flash can adapt quickly. And now we're going to have some actual examples that are taken from the software developer kit. SCF CLI is used to configure a software enabled flash device. It is a command line tool, it's open source and included in the SDK and any of its functionality can be incorporated into an application if needed. There's extensive built-in help and this is the top level of the health output. In addition to allowing configuration of the device, it also supports all of the API primitives. You can read and write data with this command line tool if you want, but probably the neatest feature of this is the built-in Python shell for interactive programming of the device. So in the target section of the help, you'll see that the different functions supported by this CLI tool are grouped by functionality. So we have functions that deal with virtual devices that could be creating a virtual device, listing virtual devices, deleting virtual devices or changing virtual devices. We have functions that apply to super blocks, the shell, which is the interactive Python shell, unit level controls, the block layer controls, root pointers, many things that we aren't going to be able to cover in this introductory webinar. But getting back to that Python shell, it's extremely useful tool for examining the device for diagnostic purposes while you're debugging your software development. You can even supply Python scripts to the SEFCLI program and have it execute those scripts for you. This is really helpful, once again, for common debugging tasks. We use this all the time in-house. So now that I've introduced the SEFCLI program, let's go over an example of its use. Note that in all of the coming examples, there are many possible settings that aren't illustrated in the pursuit of clarity and simplicity. When you don't supply an argument or possible setting, there are default values specified for each parameter. So let's go over how you would create a virtual device with the SEFCLI tool. As you can see, you can specify the starting channel and the number of channels. This defines the channel level parallelism of the virtual device. Likewise, the next arguments start bank and number of banks determine the number of die per channel that you've specified that will be assigned. The combination of channels, number of channels and number of banks defines the total number of die in the virtual device, which in turn defines the maximum parallelism of the device and the size of a superblock for the device. When we create this virtual device, we assign it a virtual device ID in this example. It's being set to virtual ID zero. And we can also set a parameter to limit the number of QoS domains that can be created within this virtual device. In this example, we've set that to one. Once you've created a virtual device, you can also use the SEFCLI command to list the virtual devices and that's illustrated at the bottom of the slide example. Next, we're gonna create a quality of service domain or QoS domain. With a QoS domain, although it's not shown here, you can specify the Q assignments for each of the flash operations. So here we have told SEFCLI, we want to create a QoS domain. We are creating this domain minus S zero is specifying which software enabled flash device we wanna create this domain on. We have to say which virtual device on that software enabled flash device, we wanna create this QoS domain in. So we're going to create this QoS domain in virtual device zero, the one we just created in the previous example. We're gonna give this QoS domain an ID of two. We're going to specify the capacity of this QoS domain. This is always expressed in terms of atomic data units or ADUs, and we're gonna specify the size of the atomic data unit. In this case, we are specifying a 4K atomic data unit size. The last two parameters are fairly interesting parameters, number of root pointers. The number of root pointers is a argument that allows you to specify how much metadata, how many metadata root pointers you want to store within this device. So if you were implementing a key value store with software enabled flash, you may have a set of key value mappings that you need to store or persist. Or if you were implementing a standard SSD with software enabled flash technology, you would have a lookup table for the LBAs. That metadata you would want to be able to store and retrieve. The problem is, is where do you store your metadata? Well, our answer is you store the metadata within the QoS domain itself because QoS domain can be running potentially a different protocol and have a different type of metadata, unique metadata in fact, and that poses the problem. If you store the metadata within the device itself, how can you retrieve it? Well, you need an anchor point and that's what a root pointer is. It is a piece of metadata that we guarantee that you can retrieve on system initialization that can point to all of your other metadata that you need to initialize your storage application. And then the number of placement IDs, as mentioned earlier in the Nameless Write discussion, one form of Nameless Write is to specify that this data needs to be grouped with other data and all data with the same placement ID will be put in a set of super blocks that are actually allocated for that placement ID and we will not mix data with different placement IDs within the same super block. In this case, in this QoS domain, we're specifying that we can have four different placement IDs active within this QoS domain simultaneously. And so that is what it takes to define a QoS domain and once you've defined the QoS domain, you can list out all the QoS domains that are present on the device with the list QoS command illustrated below. This next example is a Nameless Write and this is the minimal program necessary to form a write of data into a previously existing QoS domain. So if you would use the SEFCLI command to define a virtual device and then later to define a QoS domain within that virtual device, you could write this program, simple C program. The first call is to get handled on a CEP unit. So this says I would like to interact with this particular software enabled flash device. They are enumerated by the system automatically. And of course, you would need appropriate permissions to be able to open the device. But once you've opened the device, you get a handle to it. Once you have a handle to the device, you can then call SEF open QoS domain. You supply the handle to the device and the ID of the QoS domain you would like to open. You can also optionally supply a notification function to get asynchronous notifications for that QoS domain as well as a context, which is a piece of data that we will preserve through the operations that we don't use internally at all. It's user context, very helpful for maintaining asynchronous state. Finally, key was the encryption key for the QoS domain. And the last argument is a address of a QoS handle that will be returned by the open command. So first you get a handle to the software enabled flash unit, next you open a QoS domain and get a handle to the QoS domain. And with those two pieces of information, you can issue as many writes as you would like. And so here is a call to SEF write without physical address one. It takes the QoS handle that you just opened. In this case, we're doing auto allocate. When you're doing auto allocate mode, you need to supply a placement ID of what placement ID you wanna group this data with. You can supply a user address, which is a piece of metadata that will be recorded along with the user data. You specify the number of atomic data units you're trying to write, an IO vector mapping out the write buffers for the write. In this case, the one is a count of how many entries are in the IO vector. And we will return the permanent address. Once again, you've set the bounds for where we can perform the write. The device actually chooses the actual physical address and returns that to you in the permanent address field. And then we also return the distance to the end of the super block in case you're doing manual allocation mode. It's helpful to know when you're approaching the end of a super block and need to consider allocating another super block. And the final parameter to name was write is an override structure, which allows you to override either Q placement or Q waiting. I mentioned when you opened a QoS domain that you could supply a notification function to handle asynchronous events. There are several asynchronous events that a software-enabled flash device can generate. Once again, in this example, we're showing that you open a QoS domain and you supply a handle to the asynchronous event notification function. And then beneath that, we have the actual asynchronous notification function. This is just a small stub of a very large switch statement showing some of the different types of asynchronous events. There is an address update event, which is used when data is moved. Data can be moved by the device under a number of circumstances, but one of the most common would be if in fact there was a right error when performing the right to the device, we could have decided, we could have told you that we intend to place the data at one location and end up that we have to write it to an alternate location due to a media defect. And the next example there is a block state change notification. So you can tell when a block has been filled and closed. There are also some capacity-related events. When we defined the QoS domain, we gave that domain a capacity quota. It is possible to do thin provisioning and in those cases you can hit events for running out of space on the device if you've overcommitted data and need to take action. This next example is an example taken from our reference flash translation layer and it shows how we update the lookup tables, the translation between a logical block address and a physical flash address for a block mode implementation or traditional SSD implementation or block mode driver. And the important thing to note here is that you see once again a context which is useful for managing multiple independent flash translation layers. This tells you which flash translation layer you're operating on, the user logical block address, the actual physical flash address. And what you can see in this example is that we are using atomic exchange operations and we are updating in this example, both the valid bitmap showing that this is valid data within the lookup table and then we are performing the atomic exchange on the actual lookup table itself so that we're entirely lockless upon our table insertion. The next example is an address update. This could happen once again, if in fact we had to move data internally as a result of some internal event in the device. Lots of comments here, but the key takeaways are that the address table update is also lockless. We supply the expected flash address as well as the new flash address. And this allows us to handle race conditions between data overwrites and data movement without doing any locks whatsoever in the FTL. So once again, the reference FTL is a very high quality, high performing FTL. This example is an example of a direct access read. Once again, the housekeeping of opening up a unit, opening up a QoS domain, and then reading the physical address by specifying the QoS handle and the flash address you wanna read. And I think it should be fairly obvious. Once you've opened up the QoS domain, you can issue as many read and write commands as you want, you only have to open it once. But those operations are listed for completeness here. And there's also a little bit of a note here that the error recovery mode for a QoS domain is set at the time of creation of the QoS domain. And this determines if the software able flash unit will perform automatic error recovery on the QoS domain. And so you can actually set a read deadline that allows us to limit recovery attempts. And this is very useful in environments where you have mirroring at the system level. It's oftentimes quicker to retrieve data from a mirror upon a read error than it is to actually do the heroic recovery attempts on read-read trial. I'm sorry to jump in here, Rory, but we do just have a little bit of a break. But we do just have four minutes left in the presentation. So yeah, feel free to continue on here, thanks. Thanks, I knew it was gonna be close. So this is actually the last of the API commands. And this is just an illustration that all of the previous command examples I showed were synchronous command examples. They all have asynchronous counterparts that take IO control blocks and operate asynchronously. So let's get on to the summary and how to learn more. So hope you've seen that software-enabled flash technologies is a pretty interesting new approach to handling flash media, giving control to the host. So it is purpose-built hardware. It is an open-source flash native API. It's leveraging the industry standards wherever possible. And it can be used as a building block for multiple drive protocols. So we've actually demonstrated traditional block SSDs built on top of software-enabled flash technology, ZNS drives built on top of it, custom hyperscaler FTLs implemented on top of it. And as well, we've gone flash native with several different applications. And we're trying to combine full host control with ease of use, it's a very delicate balance to strike. So for more information, the first thing would be to direct you to the microsite. That's www.software-enabled-flash.com. There you can find the white paper, as well as pointers to the actual repository for the header files and API documentation that have already been published. And with that, I'm done with the presentation. Thank you. Awesome. Thank you so much, Rory, for joining us. And thank you to all of our attendees. Hope you enjoyed the presentation and we look forward to having you join us next time. Bye, everyone.