 So, can everybody hear me? Great. My name is Julius Voreva. I am software engineer at Red Hat, and I'm going to talk about a year-ring and how we integrated it in Qoom. So, first, we're going to discuss a year-ring API and how cool is it, and what's new inside, and then something about Kimu and what's inside. So, this is IO path in virtual machine. We can see that a lot of stuff is going on. So, IO path from user space to kernel, and you need to choose some Azure kernels IO there, and then it connected to Kimu with a block driver, and then we can finally pass through our IO requests to kernel. So, what's now existing in Kimu? We can use Linux IO, but it's not really good because it's not true as in. We can use thread pools, we can use NVMe pass-through, which works well, great, but it's not migratable, and SPDK is hard to manage, and so, what we can do? We know that we have this path from Kimu user space to hardware, and we can now improve it because we have new IO-ring API instead of existing Linux IO. What does it IO-ring? It is a new truly asynchronous interface, which was created for block devices, but now it can do everything. It can just a good interface to pass messages to the kernel in an asynchronous way. So, you can pass network also and do other syscalls. You can change syscalls. For example, if you have asynchronous open and asynchronous read, you can make that read will go after open, and this all way will be asynchronous. So, that's cool. It's part of Linux 5.0.1. So, it's in use. So, if we use it to Kimu, we need to make sure that we're on the newest kernel. What's inside IO-ring? If you remember in Linux IO, we have events and we submitted into a single queue, and now we have two separate queue for submission and completion, and their memory, they're shared between user space and kernel. So, you can use it directly. You don't need to enter kernel that, and you can use asynchronous flush, which is not available in Linux IO. And because of that, we needed to switch back in Kimu to synchronous flush, and this doesn't work good. So, for my submission, it's when it's from user space to hardware, and completion is backwards. So, basically, IO-ring is three system calls. It's IO-ring setup when you can manage, you can choose different regimes, and you can pull from submission site or completion site. You form your submission queue, and form your completion queue, and do enter syscall. What's cool is that you submit requests simultaneously with fetching, completion queues. So, you don't need to syscall for that. And a great feature is file descriptor register. So, you don't need to do a get and a put on each submission and completion. So, you just register a set of files and then refer to index and array. And you can register first. This is not a great improvement, but since we have a workload, such improvement helps. How fast is it? Well, it is quite good. It is better than Linux IO, but not entirely great. So, you can see information which I test. I use FIO and NVMe SSM. So, if IO-ring is so cool, then let's integrate it in QEMU. And that's how Stefan Heidner came up with our three-tier idea. And our mentor, our master implemented it. And basic function is actually merge upstream, and it will be in QEMU 5.0, with some problem existed. How we integrated it? We use the same as Linux IO actually implemented. We have QEMU event loop. It's based on IO context. We just add IO context in the pool and use file descriptor to pull completion check. And so, this is how we can submit requests with inter and check completion on IRQ. We also have a use LibUring. And I can recommend much to use it because it's really helpful, it's easy to use, but now we are tied to LibUring versions. Launching is very simple. You just remove your... You put instead of your native or thread, IO-ring, and this is how it works. It works with IO Direct and Cache 2.0, so we can do the same, yes. And you can see that even without extra features, it can do good. It can take a look at base. What we can do next? We can implement of the registration. So when we have our IO in QEMU, if we know that it is entered from different file descriptor, we just register it. And that's how we can save some time. Yes, it is improved a little. It doesn't really help much, but one thing of it, it is really helpful when it's come to submission polling. What is submission polling? You can run a kernel thread to wait for submissions. You need to wake up it with a special syscall, but if you have a heavy workload, you don't... Kernel thread just don't sleep. You don't need to wake up it. So it's really helpful. And of course, when we submit... When we pull submissions from kernel threads, we always check FDs and that's how FDs registration helps. Yes, and now we submit requests without syscall and get completions on IRQ. You see that we don't have any syscalls at all if you have a heavy workload. What can we do next? We can pull completions from to site. We can do user space completions. This is a feature in Kimu itself instead of using E-Poll. We can do... We can do manual polling sometimes and it could be good, but it doesn't. It just doesn't work. The feature in IRQ itself is completion polling. We just submit syscall, IonEnter and it's busy waiting, just polling completions and we're just waiting for it. And you can see that it uses extra CPU, but this is the fastest way. So we have a thread that pulls submissions. We have... We have polling completions and we don't have any extra context change. Yeah, it's not implemented. This is what we need to do. What we need to do is merge submission polling and file registration. It is implemented, but it needs to be merged. We need to implement file buffer registration and IO poll. It is difficult with IO poll. It takes some time. And we can probably switch to IO ring as default IO if it's supported by the kernel. As we know, it needs decently risen kernel. And the final idea that was suggested is switching main pool to IO ring. So you know that we use Linux IO context in key main pool we can switch it to IO ring context. But this is a lot of work. I'm not sure if we can gain some performance of it, but we can try and take a look. Yeah, I guess that's it. That's a quick presentation. Any questions? Yeah. The way I understand IO ring basically is a more than a specific way of implementing BIOS. Different protocol, different things. Let's go with the restrictions, but it basically is BIOS. I was wondering, could we literally just expose IO ring to the guest with a couple of restrictions so that it can directly issue calls? I'm not sure if it's possible. Yeah, so the question was, it's not really a question. We have BIOS and IO ring is some sort of BIOS, but from the kernel side, can we take this to the Kimu directly? To the guest. To the guest, sorry, to the guest directly. So, well, it's not supported in Linux. What we developed between code. Sorry. Well, probably we can. Yeah, well, I don't know if you have something to add. Yeah, so today it's not possible to do that, but there is some important. Yeah, so the question was about V host, and it would be possible to expose the rings that Julia described to the guest, and there is someone who's working on it, so we'll see those un-patches and we'll see what happens. Any more questions? Okay, thank you.