 Hello. OK, sorry. Hey, guys. I am Abhijit, presently working as a software engineer in the customer session team at Uber. So I'm going to talk about the new Async I interface in Linux, which was introduced in kernel version 5.1. Welcome, everyone. I hope that this talk is an enlightening session. And we get to learn through things about the interface that was introduced in the recent version of the kernel in the next slide. I'm trying to go to the next slide, although I'm not being able to. Yeah. So first, we'll talk about existing Io interfaces in Linux for doing file Io. Then we'll discuss about Io Uring, which is the main focus of this talk. We'll also talk about a libc library called libUring. We'll also discuss about system calls specifically that have been added in this interface, the examples, various options they provide. We'll discuss about the performance of the interface along with the observability and debugging aspects of Io Uring interface. Lastly, we'll converse about the internals and its adoption across the industry. So I'm having a bit of an issue in going to the next slide. Yeah. So yeah. So let's talk about the basics. So we have a basic system called read. That is, if you want to read any kind of file, be it a socket, be it a file on the disk, be it a pipe, we can issue this read system call, and we can get that file. So what happens here is that it is for regular files, that is files residing on the disk. If the requested pages of the files or the blocks are in page-gash, then it is fetched and returned. If not, it fetches the data from the disk and then blocks before returning to the user. So the user call to the disk is blocked until the data returns. If the file, of course, this is with exception that if the file is not opened with O non-block flags, let's say we have opened the file with O non-block flag, then depending on the file, this call might block or not. So for regular files, the O non-block flag does not work. So even with O non-block flag, it will block. But for sockets, spikes, and character devices, if we open the file via O non-block mode, then the read system call will give an error until that data is ready for reading. Next slide, please. So here we have an example of how the flow works for making a read call in a file in, let's say, an EXT45 system. So let's say we issue a read call. We go by this syscall interface. We issue the series of calls in the kernel. And what happens, ultimately, is that the user is kept waiting, or the application is kept waiting until the data is fetched from the disk, and then it is returned, trying to issue the animations here. So we see that the whole call chain is being called here. We issue the sysread call read. The kernel goes through all the series of these calls and then returns back data to the user while it is kept waiting for, let's say, a simple file that is being read from the disk. Now let's discuss if there are free pages available in the kernel or not. So if there are no free pages available in the kernel, this call might block. If there are free pages, then it will succeed without blocking. This is true even for O non-block flags. Files open with O non-block flag. So let's say if you have a socket or a character device or any kind of pipe, the write will block until the data is ready for writing. But if we use the O non-block flag, it won't block. And if you throw an error until the file descriptor is ready for writing. Next slide, please. So let's talk about what are the present interfaces existing in Linux or in general. So the POSIX async interface is something that we should discuss. This async interface is actually implemented in libc, glibc. And how it implements the async nature of read and write calls is that it maintains thread pulls. Thread pulls for offloading any kind of blocking calls. So let's say if you want to do a read operation over a buffer file residing on disk or let's say, residing on a block device, then we can use this library and it will basically offload the task to a background thread. And that's how it can become a non-blocking interface. So but we have issues with this interface. Its implementation is non-uniform across different architectures and its performance also can be improved. Next slide. So we have to talk about a flag called O direct. With this flag, if we use this flag when we are issuing system calls, let's say read or write, we can make this call non-blocking for files residing on disk. That is regular files. So the problems that we're having is mostly with regular files because as you have seen that the normal system call of read and write will be, you know, can be asynchronous if we use sockets or pipes or character devices. But if you are using the regular files or files residing on disk, which have to pass through the page cache, then we are having problems that the call is blocking. So if you use this flag, which means that we don't want to use the page cache at all. So if we use this flag and if we also use lib AIO interface, which we'll discuss later, then this call can be made non-blocking. So this flag helps us in issuing reads or writes via the lib AIO interface, which are non-blocking. So this also is one of the flags that we should be aware about apart from the existing interfaces. The next slide, please. So the current ways, how do we achieve, so we obviously have a lot of code, a lot of applications out there who are using, who are doing reads and write databases. Everything that we mostly use in Linux goes through these system calls. So how do we do it currently for asynchronous nature of this, to achieve asynchronous nature? So we use IO multiplex interfaces. So we have the select interface, ePoll or poll interface. What we can, how we leverage them is that. So these interfaces provide us system calls like ePoll, wait or like poll or like select. So what the system calls do basically is check if the file descriptor that we're interested in is ready for reading or writing. So they basically check that, okay, if the file descriptor is ready, then yes, your read can proceed without blocking. Or if your file descriptor is ready for writing, then your write can proceed before blocking. So with the help of these system calls, we can have asynchronous nature. We can check the readiness and then the readiness of the file descriptor and then issue our reads and writes call. So does it solve the issue of reading for buffered files? No, it doesn't because this files, let's say the files which reside on this are always considered readable and writable. So poll or select or any of these interfaces won't work. So if we operate using let's say any kind of interfaces like select ePoll or poll, these interfaces won't help us to achieve a synchrony because these files are always considered readable or writable because Linux in general considers disk as something in which result is guaranteed. So although there is latency but the result is guaranteed. So these type of files are always considered readable or writable because they're always ready for reading and writing. The readiness checks don't work. And that's why these system calls will say that your file descriptor is ready for reading or writing and hence your call can proceed. So but we still have to block because these files are residing on this and they have to be fetched from the disk but these are always considered ready because they are on disk. Other than that, if let's say if it's a file descriptor on a socket, then until the data comes on the socket or until the socket buffer is available for writing, the readiness checks won't be firing, let's say unless the event occurs, unless the readiness events occur. They don't, so the readiness events don't work nicely for awkward files. And that's why even using these interfaces, we can't achieve true asynchronous for regular files. Next slide please. So LibAIO interface. So again, this is an interface that is, that is again a native Linux interface. Basically what it consists of is essentially three system calls. So the first system call is IO setup. So what it does, it basically sets, it initializes the metadata, it initializes the unique instance for this IO operation. Then you submit the IO operations using IO submit system call and we can issue multiple read and write requests using the system call. And then finally we have the final one which is IO get events to synchronously wait for these set of requests. So all the operations that we, IO operations that we issued using IO submit system call, we will have to wait for all of them via the IO get events system call. And the IO get events system call is a blocking system call but the IO submit system call is actually a non-blocking call. So does it solve the problem of regular buffered file IO? Yes, it does solve the problem of regular unbuffered files but regular buffered files still blocked. So this is the interface which we are talking about before that if we open the file descriptors with O non-block flag and if we issue the IO request using this interface, it won't block because it will bypass the page cache and it will issue the calls asynchronously. But if we do not open the flag with that interface with, sorry with that, if we do not open the files with that flag then this interface will also block. So we're talking about IO submit calls, blocking for file descriptors upon which we want to issue, read or write operations. So still it's, I think it's a neat way of achieving batching of multiple IO requests. And as far as I learned about it, it is slated to be the default O6 IO backend in the future. Next slide, please. So now we come to the IO viewing interface. So this is talking about it quickly. This interface was developed by Jen, it's Xpo from Facebook. It was added in Lex 5.1. We have three system calls in this interface, IO ring register, IO ring enter, and sorry, IO ring, the last one is setup. I think so there's not correct data here. So IO ring setup is the third system call. Basically we have two ring buffers or circular arrays which are shared by both kernel and user. So these circular buffers or ring buffers are essentially used by user space to communicate with the kernel. So this is very unique as per what I have seen because normally when we issue a system call, we initialize the pointer to the data, we initialize the metadata and issue a system call. But here we are using ring buffers to actually communicate with the kernel. It's quite interesting and it is quite unique to this interface. So using this, using these circular buffers, we basically submit our iU request and we also look at completion of those iU requests. Next slide please. Yeah, so I believe if we click, we can see how these ring buffers are operating. It's an animation, so we have to click. Okay, so I have got a few questions here. Let me ask the questions if I can. So Praveen's question is that what about AFXDP which is a similar system call interface. So as far as my knowledge, AFXDP flag is being used is being used to do XDP networking via the VPF interface. So if we open sockets, if you have to open sockets, then we use that flag to open a raw socket and then we do the XDP networking. So I'm not sure if the VPF XDP interface, sorry, flag uses the similar interface. I do not think so. But yeah, I think so. I have a next question from Phong. What is your question? Can you stake please? Okay, so I think I'm not getting any answers from Phong. We can come back to it later. But let's see about this slide. Let's continue with the slide. So we have, we here see two ring buffers here. One is the submission queue and another is the completion queue. And we see seven submission queue entries here. So the user has initialized seven submission queue entries and the user has populated the submission queue with these entries. Now, after populating the submission queue, so we have some animation here. So basically with the queue, you consistently update the tail of the queue. So the user space app updates the submission queue and keeps on adding SQE to the circular ring buffer. After this has been done, then the user issues a IOU ring enter system call. So this is the main system call that indicates the kernel that, hey, we have a few entries in the submission queue ring and please read those entries and please complete the operations that are corresponding to those entries. So after the kernel knows about it, the kernel comes into picture and starts reading all the SQEs. So one by N, it will read the head of this ring buffer. So it keeps on reading the head of the circular array that is the submission queue and then it will issue the IO operation for each of the submission queue entry. And then finally, when the IO operation is complete, it will go to the completion queue and then start populating the completion queue ring buffer. So the completion queue ring buffer will then have all the completion queue events and then the user can start reading via, so the user can in the backgrounds keep reading the completion queue. So it's asynchronous in that the user can independently fill up the submission queue, issue a call, sorry, okay, yeah. I got a message from Swam, he just wanted to say hi to me, sorry, Swam, I was expecting a question, sorry, yeah. So I was telling that the user app can update, keep adding SQEs entries to the submission queue ring and then the kernel reads the entries from the submission queue ring and then the kernel when after the completion of each and every IO operation updates the completion queue ring, which the user reads. So these both submission and completion are decoupled, which results in asynchronous. So the user space application can independently keep filling the submission queue and asynchronously also reading the completion queue for the completion of the events, next slide please. So discussing about it in general, so this interface provides true asynchronous for all operation types. So if you're talking about storage networking, if you're talking about storage or networking, then any kind of operations are handled here. We also need to talk about the SQPole mode, which is very interesting because we do not have to enter and we do not have to make any system call if we use this plan. So we'll discuss about it later, but we'll just discuss briefly about it now that we can completely bypass the kernel and issue all our IO operations and which helps us in not making any context switches at all. So this can really transform the way that we interact with the kernel for making IO operations. Next slide please. So here I just presented example of the SQPole mode in which you can run IO-euring interface and in this mode, there are animations here, in this mode, the user space app just fills the submission queue entries in the submission queue ring buffer. They basically just update the tail and then the kernel creates a background thread and the background thread pulls for any entries that come in the submission queue and then reads all the entries and then basically performs the IO operations and then fills the completion queue with the completion events. The kernel is eagerly creating a background thread and then processing all the requests which are without having the user space application to make a system call. So that is the difference between this mode and the normal mode. So in this mode, there's no system call as we are seeing. It can operate without that. Next slide please. So discussing again about, so apart from read and write operations, it has support for send, receive, open, close, connect, fstack, if allocate. So all this system calls that we normally make that we somehow interact, somehow do an IO operation here. All these operations are supported. So let's say if you want to issue a connect operation and if you want to connect to a socket, we can do that. We can do a read, we can do a send, we can do a receive. So we do not, we normally we do not use the system calls that have been provided to us by this interface directly. We can though, but it's a helper library has been made available to us to use or to leverage the system calls. The helper library is liburing and this library provides us several functions that we can use to avoid the complexity of the ring buffer management and the ring buffer management of the submission and completion queue. So again, one thing that we should take note of is the both the submission and completion queue are shared by user space and the kernel. So those, the ring buffers that were used for communication between the user space and kernel are being shared by both of them. So we do not make any copies, we do not make any copies of data while we're transferring the request from user space to kernel. So there is significant amount of savings in copying data. Also, because the IO operations itself are unpredictable, so we cannot guarantee the order of submission and completion requests. So if let's say we issue some submission requests in order, the completion events for those requests might not be in order because they're all asynchronous. So depending on which one of them complete first, we'll have the completion queue events for each of those IO operations. Yeah, performance wise, lastly, yeah, performance wise, we have seen a good performance, we have seen much better performance from IO-euring interface than compared to previous interface phases. We'll discuss, we'll see, look at the performance numbers. Although there are quite some resources there which point that IO-euring is performing really well in at least in buffered reads, in normal use cases where we are doing reads and writes call, also in networking cases, let's say where we are trying to run a server. So, yeah, performance wise, it's also good. So let's look at design with similarity. Circular buffers, as we know, are an elegant design pattern, elegant architecture. Why? Because they have back pressure built in. So you do not, it's not like a linked list where you can keep adding entries. There is a fixed size to it. So your memory is bounded, your memory is reusable and your back pressure is built in. So let's say if your ring size is full, then you won't accept any entry from the user space application. So reverse is true for the kernel as well when kernel is filling the completion queue ring buffer. So when the completion queue ring buffer is full, the kernel will have to wait. So the back pressure is built in. Also, ring buffers are used ubiquitously. For example, in networking, we use ring buffers where the socket entries, socket buffers, then we create them in Linux after getting them through the networking driver. The memory is bounded. So it's a ring buffer which keeps on being reused for the sockets which are being allocated feed continuously as the networking layer processes those packets. Microsoft Windows uses IOCP as the traditional IO stack which is the IO completion ports which is similar in design to IO ring as it also uses ring buffers which is a generic asynchronous IO interface for Windows. So Windows has something similar which also provides asynchronous interface for making IO operations. Next slide, please. So the first set of calls that was wrongly mentioned in one of the previous slides is IO ring setup. So this is basically to create an IO instance, a unique IO instance, IO ring instance that we can, we'll be using in the system calls after this. So basically we provide the number of entries we want to make the number of entries that the number of operations we want to, IO operations we want to make and some parameters. So in these parameters, most importantly are the flags. Next slide, please. So most of the entries here, if you populate the flags most of the entries here will be populated by the kernel itself. So flags is something that we discussed SQL mode. Here we can configure the IO instance to start from SQL mode and we can also configure it from IO pole mode. So in the IO pole mode, the application has to pull the kernel for any kind of completion events. So normally the application just needs to read the completion queue events, but events from the completion queue. But if we use this flag then the application will have to pull the kernel for any kind of completions. We initialize struct of IO ring patterns and then using some checks, we can initialize the flags that if you want to use IO pole mode or if you want to use SQL mode. And if you do not use any of these modes it's also completely fine. We then issue the system call where we want to initialize IO ring instance for 100 entries and some parameters that we initialized. Next slide, please. Yeah, the submission. So let's talk about the SQL struct here. So this, so when we talk about submission queue ring buffers it is filled with instances of these structs. So each entry of the submission queue ring buffer is an instance of this struct. So if we look at the fields that are used here the opcode is what specifies which operation we want to perform. Do we want to perform a read or a write or a send or a receive, connect, close, accept, whatever. So flags is something that we'll discuss later on. IO priority or IO prio is basically to indicate if we can prioritize this request. File descriptor or FD is something that which, what is the FD that you want to do the operation upon. Other fields are also kind of metadata related to the submission entry. The user data field is something that we that the user provides. So let's say if we issue a read operation then we want some buffer upon which the kernel can write the response of that read request too. So the user data is actually that. So this is what is passed to the completion queue entry as well and the user can read this data completely. So we also see a few flags here that are for each operation. So you have some flags for F sync, you have some flags for accept, cancel. So these are per operation flags. So that is that, so let's move to the next slide. We will discuss how we initialize this previous idea. So this is a bit complex as from the code that we can see, not, I mean not a lot, but yeah, somewhat. So we basically, because it's a ring buffer, we know that there are lots of, there are head and tail. We know that we have some flags and we also keep some metadata around the dropped or submitted around the offset. So basically what we're doing is we are creating a shared memory space between the user and the kernel. And then we are initializing the metadata around head and tail entries. And finally we are after having all this metadata we are complete with our submission queue entry struct, next slide please. So now this is the completion queue entry. This is basically for the completion queue ring buffer we have each entry of that ring buffer is an instance of this struct. So in this struct we have three fields and it's pretty simple as compared to the submission queue entry struct where we have user data, which is again the same thing that I pointed before that is the address at which the kernel can populate data of populate the result data. So after the completion of any IO event, whatever the result was of that IO event the kernel can populate the result of that IO event in this field. Result, RES or result is used to indicate if the flag, sorry, it was to indicate if the IO operation was successful or not. So that is indicator of if this operation was successful. Flags is something that is used to signal that which buffer was used for submission. So we'll talk about again this flag later on. Let's move ahead. So here we see, here we try to initialize a completion queue entry. So here again it's very similar to submission queue entry where we do an M map and we just update a few metadata fields mostly head, tail and some metadata around the completion queue. Overall, what we're trying to do here is that we are trying to study the submission queue entry struct and initialize the metadata around the struct and the completion queue struct. Next slide please. The kernel that we have a few submission queue entries we want entering entry call. So this is the system call that is used by the application to issue and wait for an IO request. So here I have used the word wait for an IO request but it's not necessary. So normally if you indicate via the flags that has been used in the parameter. So this IO ring entry call has five parameters, the file descriptor first of them, the two submit second of them, min complete flags and sync. So in flags you have two flags that we want to talk about, IO ring enter get events and IO ring enter sqwaker. So if you use the first flag, it means that we want to wait for min complete number of events to be in the completion queue till this call is complete. So this call will block until there are min complete IO completions. So if let's say if we pass min complete as three and if we set this flag as two in the bit set, then it will wait for three IO operations to complete. And then only after that it will return. So if we do not use this flag, it's not necessary that it won't block and this will complete. So IO ring enter sqwaker is a flag that we use to signal the kernel to wake up and start polling again. So normally when we use IO ring in sqwale mode, the kernel creates a thread which continuously pulls the submission queue ring buffer for any metadata. So any kind of entry in the submission queue ring buffer will be pulled by the kernel thread. But let's say if the submission queue ring buffer is quiet and there are no entries there. So then the kernel thread will start sleeping and this flag is used by the user space application to indicate that the kernel can start the background thread again. Next slide please. So IO ring register is basically to pre-register set of buffers or files which the user space application can later on leverage for writing or reading data. So let's say if we have a file descriptor, the kernel will want to know the file reference of it. The kernel will want to know what is the buffer, what is the buffer that he wants to access to write data of the result too. So for those meta data or for those if the kernel wants to do that for each and every IO operation, the kernel will have to get the file reference. The kernel will issue get or output calls. The kernel will also try to copy the user data that is there in the submission queue entry and then copy it back to the completion queue entry. Now what these system call does is that it provides an option to the user application to pre-register those buffers. If they pre-register those buffers, then the kernel will not have to copy data while reading an IO while the, let's say if it's processing an IO operation. So it makes things cheaper. So when we have read all of this, we need to summarize ourselves with the complexity that are involved in this whole thing. That we didn't discuss about one important thing that when we are reading and writing to the ring buffer via the submission queue entry and the completion queue entries, we want to have some kind of synchronization between the kernel and the user space. And that synchronization is brought by memory barriers. So all of this initialization plus synchronization is a bit complex. And then the liburing library helps to alleviate all that complexity. Next slide, please. So if we use the liburing library that is provided by the glibs, that's provided by glibc. We just, we say here we see how we use this library. We first issue a IO ring queue I need to call to get a unique IO ring instance. Then we get a submission queue entry. Then we set the submission queue entry with some data that we want to provide. And then finally we submit this IO ring operation. So this is the submission part of it. Next slide, please. So then we take the ring instance that was given to us by the IO ring queue I need to call. And then with the completion queue entry that we have initialized it, we basically wait for that completion queue entry to be available. And then finally we get data from it once the IO ring wait CQE call returns. So using these set of calls, we have completely avoided the submission queue entry initialization, the ring buffer initialization, the synchronization between the kernel and user space, all that is completely avoided. So it becomes quite simple to use this IO interface. But yeah, if we wanted to, I think so for both control, we can still use that interface. People prefer more control, people prefer simplicity. I think so, people prefer it either ways. But yeah, this library helps it much, helps makes it much easier to use the interface. Next slide, please. So here we have an example of reading a file by a LibUring interface. So first we open the file using an open system call. Then we initialize the structs that we require. So you ring interface for IO ring struct, then SQE IO rings SQE struct, then IO data, IO ring SQE struct. So the IO data struct is something that we have initialized ourselves. This is not something that we have to do normally. So this we have initialized ourselves so that the kernel can understand it. So what we do is that we first issue an IO, using a QINit call, then we finally, then we get an SQE entry. Then we prep that SQE entry with the read request and then we set the SQE with the data, then we submit that request. And then finally we'll wait for the completion Q entry and then finally we'll get data once the IO ring wait SQE call has returned. So we have made here five to six calls and we have completely ended the complexity that was associated with the previous, that was issued with the previous set of code. I'll discuss about these flags a bit later. Let's move to the next slide here. Yeah, so when we saw the submission Q entry struct we saw a few flags there. Let's discuss about those flags now. So this flag, the first flag, IO SQE IO ring flag, what it does is that it creates an ordering between the SQE's. So all the SQE's which we populate in the submission Q entry, it forces all of them to be ordered, not asynchronously each of them. So let's say one, two, three, four, five, six, seven, 10. Then we'll process first, first, second, later, third, fourth, fifth and not each of them asynchronously. So the second flag that is the IO SQE IO ring flag it forces that each entry of the submission Q entry submission Q ring buffer has to be processed first before the new ones have to be taken. So the new ones will be taken later and then after the existing ones have completed. So obviously this is quite a dream and this is actually a performance killer. So the SQE IO SQE Async flag, using this flag, we can the IO ring, the kernel will straight away offload this operation, the IO operation to a kernel thread right from the start. So right from the start, you have a background thread which takes care of all this and you don't, the kernel does not rely upon page callback mechanisms or page callback mechanism or interrupt driven behavior to signal IO completion. Next slide, please. So we have discussed these flags, IO pole and IO SQE pole flags. So using this, the second flag is quite interesting as I discussed that the user space application does not need to make any system call here. It can directly populate the submission Q ring buffer and the kernel will create a background thread and will read entries from the submission Q ring buffer and populate the results of them to the completion Q ring buffer. In the IO, using the IO pole flag, the application will have to pull for any completions using the IO ring entry system call. So normally this is done with applications to have low latency. So we have other flags which I'll skip but yeah, I think we can read about it but yeah, I think I'll skip here. So I have a few minutes left. So let's go to performance, the size which we discussed about performance. Yeah. So let's discuss about performance aspects of it. On AWS, in CEC2, I bench, I had, I made, I connected to performance benchmark of using FIO. So it was an M5B large instance with x86, 64. There was two, they were two, a VCQs, a Geo physical memory. It was in the storage was 75 GB of non-rotal memory. The kernel version was 5.4, 5.4.0. It was, it was, the operating system was going to do 20.04. We have IO ops and bandwidth benchmarks and the IOQ that use was 16. Although higher IOQ lens were also indicating same performance. So let's look at the numbers. Yeah. So the green one is for, is for blocks which are of four kilobytes, red from red, being red from the disk and the red ones are for the blocks which are of eight kilobytes being red from the disk. These are all, so one thing I forgot to mention was these are all random read operations being done for a file on the disk. So before any, before any benchmark result, we had to clean the, I had to flush the caches. So I know dentry, page caches, everything was flushed and normally we were doing a random read operation upon the file in disk. That file size was 4 GB as we saw in the previous slide. I have a few questions now that have come to me. I'll try to answer them later. So please be with me. Sorry. So in the previous slide, we saw that for direct IO, sorry, can we go back to the previous slide? Yeah. So the previous one again. Yeah. So we see in these benchmarks that for direct IO, for direct IO, if you want to bypass the page cache, the IO ring instances is comparable or better than performance amongst all the engines, but LibIO comparably fares well. So this is true for both block sizes, eight kilobytes and four kilobytes. And we have, we have compared POSIX, AIO and SYNC. SYNC is normally reads and writes system call, AP issue reads and writes system call. Then SYNC, we use SYNC for that in FIO. So this is for direct access to files, which means that we want to bypass the page cache. The second, let's go to the next slide. In the buffer, so this is basically, we are measuring IOs when we want to use this page cache. And we see that LibIO is clearly slower here because LibIO will prove, will actually not, will prove uses because LibIO is not asynchronous for buffered files. It is asynchronous only for files which are opened with four direct flag. So here we see IO ring clearly winning in IOPS than LibIO, POSIX or the SYNC interface. Next slide please. Bandwidth wise again, SYNC numbers are pretty similar that for direct IO, the numbers for IO ring and LibIO look similar, but POSIX and SYNC are still slower. So I have, it's saying that I have a couple of minutes, I'll just rush to the slides. Next slide please. So for buffered IO, we see that IO ring bandwidth is clearly better than all the interfaces. Next slide please. So observations here that IO ring is performing well. The instance that I used was bound by CPU, but not by the disk bandwidth because the NVMe would, NVMe not volatile memory storage could not be saturated, but the CPU utilization was exceeded 100%. So we needed more CPUs to test the two performance, but what results, what we can observe here is that even with limited CPU, IO ring pairs much better than other interfaces for bandwidth and IOPS. Next slide please. So debugging and observable. So we have a few static trace points that have been added in the interface. These static trace points, using these static trace points, what we can do is we can use a BCC or BPF trace or we can use modern debugging, next debugging tools to actually measure what is the latency or what is, if let's say if an IO operation is submitted but not complicated, if an IO operation is indeed submitted, we can measure all these things using these trace points that have been listed here. Next slide please. So here's an example of using BPF trace for measuring latency, where we are putting trace points using the submit SQT trace point and the completion SQT trace point. Next slide please. I wanted to talk about internal mechanics, but I think we are short of time. I'll, let's try to go to the adoption slide. Yeah, I think so this, I think so let's discuss about this. So yeah, ROXDB is using IO rings in its backend. There have been significant improvements in the multivit queries using IO ring as the backend. LibEV, which is a library for applications to leverage the event loop, might be using IO ring as the backend. So yeah, I think this is what I could have discussed in this time frame. Sorry for rushing through the slides. Big thank you to these two folks who reviewed all these slides and made several comments about the content of these slides and helped me improve on these slides. Gens also is the author of the interface. I'll try to answer questions now that have come from the people. So the first, so William has a question, William Lavender has a question. Will you be putting your slides on the ELC website? Yes. Answering question from Steve. What interval does the IO ring K thread use in SQL mode? Is the thread triggered by dirtying the off the submission Q page? Yes. The second part is true, which is let's say if the submission, if the submission Q ring buffer is, if there is an entry in the submission Q ring buffer, the kernel thread will start processing that entry. I'm not, I do not have knowledge of the interval, so I'll have to read more about that. Could send file use this IO ring? So I'm getting message that I'll have to say goodbye to all of you folks. Thank you for having me. Thank you for listening to this talk. Yeah, I'll answer all the questions in the, I synchronously. And please feel free to ask me any questions. I'll share this, I'll share the set of slides within the ELC website as well. Thank you, thank you guys.