 Okay, good morning. Let's get started. My name is Jun Nakajima from Intel. We've been working on KBM enhancement for OPNFV. And so we're very good at achieving a low latency high performance. And we think the kind of techniques or technology we developed for OPNFV should be applicable to the general cloud and then that would help to build a real-time cloud. And we also share some issues and then that we experience and I think that would be a good feedback or input to the architecture decision of OPNFV stack. Okay, so agenda today is, first of all, I do real-time kind of one-on-one. And then talking about use cases. Then after that, talk about the software stack that you need to work on to build real-time cloud. Then then followed by the more specific setup and the configuration for open stack. So real-time, it doesn't really mean to fast, but it's about a deterministic execution. And in this case, the delay variation is the important. And there are two types of delay variations. One is timing variation. For example, the delay from expected events. For example, the time alarm or time meant, like a timer, right? If you see that kind of different delays that we call the jitter, and then you really want to minimize to achieve real-time, right? That's the timing variation. And the other one is the processing variation. For example, if some processing takes 10 seconds, and sometimes the same processing takes 15 seconds, then you see the latency variation. That's not also real-time. So you really want to make sure you see the deterministic execution. That's basically about the deterministic real-time. And sometimes real-time system often imply high performance. The reason is to achieve real-time, one of the effective way is to reserve system resources, I mean, like a CPU memory IO. And if you do that, typically you get the good performance and high performance. And that's the implication about the real-time, the kind of consequence of a real-time, okay? And in terms of the meaning of a real-time, different meanings of real-time, for example, how do real-time versus soft real-time? How do real-time means it's something you cannot, the deadline, you cannot really miss. If you miss the deadline, you're gonna have serious consequences, right? For example, nuclear plant, for example, right? If you miss that deadline, that would be serious. I don't know what happens, but, or medical system, right? On the other hand, you may have some kind of soft real-time, soft deadline, it's kind of guideline. And also the order of time, okay? Are you talking about millisecond or microsecond or even nanosecond, that's maybe insane, but that kind of order really matters, right? This is the first time I use this one, so it's a little bit tricky. So in terms of use cases, like I said, we've been working on our technical NFV areas, so where it's low latency, high performance is very important. Other areas like financial trading system and the IoT devices, this is kind of my guess over bet, but in the background server for IoT devices, because the IoT devices are not human, right? So you may need to meet more, like a very sensitive process, real-time requirements for them. And then a highly interactive system. My son always complains about Jitter when he plays multiplayer games, right? And for VR, virtual reality system, you really need to have a very high interactive response, or the AR system. This should be popular in the future, right? And high-performance system, again, HPC is not really high-performance system, but, sorry, real-time system, but one of the techniques to achieve a real-time, you can use the same technique to build a high-performance system because you deserve system resources. And also, you don't want to oversubscribe, accelerate, or even virtualize FPGA, okay? So the good news is, we already have kind of a real-time support in OpenStack. For example, the bird driver for pinning guest BCP to the host PCP, physical processor you reserve, or you can reserve a memory for guest RAM, okay? You can do that today. And then also, we have a good NUMA support to use system resources more efficiently. And like I said, we've been working on OP NFV, and then as a by-product, we have a real-time KBM, okay? And real-time KBM can support both virtual machine, real-time virtual machine, and also bare metal, like a container. So you can run the containers, bare metal containers in real-time. So this is not really bad news, but it's kind of a fact idea you need to point out. One is, basically, this is a cost associated with building a real-time cloud. In general, you need to have more hardware resources because you need to reserve system resources. And this is actually what Ironic is doing, right? Because you have a bare metal, and there you don't do any over-subscription, okay? You just have one virtual machine on a bare metal system, okay? No oversubscription, you don't run two instances of virtual machine on Ironic, okay? The other thing is that lower throughput caused by the scheduling, okay? To meet real-time, you have to kind of reserve system resources, and then also in terms of scheduling, you need to have a real-time thing. So sometimes you see lower throughput. Yesterday, I actually heard at like a self-optimization session where they can get very high IOP, but they see the high latency. So it's possible. So when you achieve real-time, sometimes you see lower throughput, okay? Then the mitigation is basically make part of the cloud real-time, okay? Not entire cloud. So I'll talk about that later. So now I'll talk about what kind of a software stack you need to work on to build a real-time cloud. Before that, I'll talk about what kind of a software stack and how they can cause the latency. So if you look at the software, system software, like a hypervisor or OS or application, if the OS or the hypervisor uses a sharing over subscription, there's no way to get real-time basically. Well, there are some academic guys working on very complex real-time scheduling, but that's very complex. There's still kind of research level. So today, the most effective way and the simplest way is reserve system resources. I mean the CPU, memory, and IO accelerators. Then also the code. Code in the operating system, schedule or interrupt handling, okay? If the code is not written with real-time in mind, then also you see jitter as a latency. For example, like a non-deterministic loops. If you don't know how many times the code will iterate, then you cannot tell how long it takes, or it uses locks, right? It's hard to tell how long it takes. The lock will be released. And there are also hardware issues. Sometimes the modern system depends on the cache, right? If the hardware misses cache, then it causes latency. So now I'm talking about the software stack and how to achieve real-time in the cloud. So as you see, on the open stack side, first of all, you define flavor for real-time, okay? And then Nova configuration, okay? With that, you can reserve system resources. Then from our experiences, you can achieve millisecond of real-time, okay? So this box, yeah, this one, right? Misecond of real-time by just using Nova and the flavor thing, okay? Now to get even more real-time, which means microsecond kind of a real-time, okay? Then you start using real-time hypervisor, okay? I'll talk about more on the real-time hypervisor, okay? Now somewhere in the middle of a microsecond, if you're talking about 10 microsecond, 20 microsecond, kind of order, then you need to enable some hardware-based resource control, okay? So real-time hypervisor, so real-time hypervisor supports basically real-time guest and then the way it handles is provides, you know, support, preemption support for real-time guest or processes, okay? And also it has a real-time scheduling and the interrupt handling, okay? The example is, like I said, the Linux with preempt RT configuration, some people call the real-time Linux N plus KBM, okay? So like I said at the beginning, we are working on this real-time KBM. So we can use this one for real-time hypervisor. So now I'll talk about more specific, you know, setup for the open stack. So this is a summary of the setup, it's a bit busy, but basically start with the host aggregates, okay? Based on your real-time requirement. I'll talk about the details more later. Then one thing is the setup of boot parameters. This is something we learn, you know, this is a bit painful and I really want to share this thing with you for architecture decision. We actually need to change some boot parameters to reserve memory and then the CPU. And then also for real-time, you know, if you want to achieve a micro-second of a latency, you need to have a real-time hypervisor, like the real-time KBM. Then the setup of novel configuration, then flavor, okay? So this is kind of recap from the previous fours, if so, right? Each box shows host aggregates, okay, and the dots are basically the host, okay? Then this one is the host with real-time hypervisor, okay? So as long as you use the same host, the same Linux configuration, if you change the boot parameter to reserve CPU or memory, okay, you can continue, you can use just a novel to define L1. L1 is basically the millisecond of a real-time, okay? And essentially this is equivalent to what the ironic is doing. So I believe if you are using ironic, probably you might want to try this one because then you can use, you know, the virtualization, you can run that, the instance on real-virtualization, and that way you can take advantage of, you know, more efficient management, okay, of a virtualization, okay? Then if you go to two, you actually need to change the hypervisor in that case, you need to, oh, oops, what the hell? Need to use, you need to reboot, so. And sometimes if you, you know, because you change the boot parameter, when you go back L0 and L1, you know, you also may need to reboot. So this is kind of a bit painful point, but I don't think you guys change this kind of configuration often, right? It's mostly static, and if you do, probably you just extend rather than change the kind of partition, but if you design the kind of, you know, these hosts aggregate efficiently, that's good. This is more details, well, I just skip this one, okay? This is a boot parameter. I'll share this for us anyway, so probably, you know, you don't need to take a picture, I promise, okay? So this is from the next kernel parameter, maybe some number you know, but you don't know that's fine, okay? By doing this, you can basically reserve CPUs for real-time guest, okay? Also, you can reserve a memory for real-time guest. That's something you can do with this one, okay? So this is how you set up a host aggregate. So you, for example, you create a host aggregate for L1 and L0, and there is a keyword, for example, RT equal true for L1, L0 is non-real-time, is RT equal false, okay? Then essentially, you add those holes to each aggregate, okay? Then in a novel configuration, you reserve the system resources, right? And then you use the VCU pin set to point, you know, oh, I think something's wrong, sorry. Maybe I put some button. So remember that, just a moment, this one, okay? We reserve CPUs using ISO CPUs, right? Then those CPUs need to match the numbers here, okay? This means, so the VCU set means you use the processors you reserve for the host, okay? This way, the usual novel guest doesn't use the CPUs you reserved, okay? So it's kind of exclusive. So if you reserve at the ISO CPU, then for real-time guests, you set those here. This way, you can use the reserved processor for this one, okay? And also specify no CPU over subscription, okay? And then no over memory subscription, no memory over comment, okay? The other thing is IO, like Nick, right? If you're using an ironic, of course, that Nick is attached to the physical machine, right? So you can do the same thing with using a novel open stack today. So you can reserve, for example, some certain Nick or in a virtualization called SRRB, right? The SRRB has so-called virtual function. They look like a device, and you can reserve for some of the virtual functions for a real-time guest, okay? So you can do this today. The PC, so I just took, this is the PCI device you reserve, okay? And then this is a keyboard for path through, okay? And set up a flavor, okay, for real-time. Those are already there. So you can take a look there. And then, for example, that the policy dedicated, right? And isolate the memory here. Then also this one, remember that RT equal to this one. And then also if you specify Nick, you can do this, okay? So this is kind of a, I mean, all set up for the open stack. So with that, I want to conclude my presentation. So like I said, open stack is ready for building real-time cloud. And as I mentioned, you need to work on a flavor, noble configuration, and if necessary, if you want to achieve like L2, where you want to achieve micro or second kind of real-time, then you need to have a real-RT5 advisor, okay? And design that the host aggregate to reflect different real-time requirements, right? And then we used actually Ironic to modify the boot parameters actually. And today, Ironic can allow you to add the boot, just add boot parameters. So I think we need a tool to deploy and then manage multi-hypervisor configuration. And that's, I think, input to the architecture decision, okay? So is that, I would like to take questions. Can you go to this? Oh, yeah. Thank you. It's not done. I have a question. I have a question on how you're dealing with hyperthreading in real-time systems. How do you deal with, what do you mean, should you enable or disable? Collaborate on any special biosettings or similar that you need to do? Hyperthreading itself doesn't cause additional latency, but once you enable that, you have basically the number of CPU will be doubled. So that has implication to the scheduler. So you may expose more scheduling issues with the scheduler. So I think from our experiences, if you start with disabling hyper, I mean, HD, W, I think better, but then I think what I can say is try, enable HD how it changes. But like I said, with HD, you get more throughput, same thing, throughput, but that may cause some latency, that's true. It may expose a software issue. For example, the number of the bCPU processor will be doubled. So if OS software has a loop for the processor, then it needs to cause more latency. Does that answer your question? Yeah, so it sounds like right now it's trial and error and see how it depends on your application workload. I think what I can say is you have a more kind of risk if you enable HD. Okay, thank you. I assume we were talking about not only networking, but CPU and other. You're creating a host aggregate. So why not just do for networking, do SRIO-V and create a host aggregate for that? So what is the real-time hypervisor bringing? If I still have to create... So this is basically the networking, right? That's like I said, you reserve network SRIO-V. Oh, so you are doing SRIO-V? Yeah, yeah, yeah. Okay, and I guess the other question would be why wouldn't you run RT on all hosts? Is it just the cost then? Why am I not running just a real-time hypervisor on all computers? Yeah, this is what I covered at the beginning. Okay, sorry. You missed the first one? Yeah, I didn't. Okay. I don't know of time, but... Well, I get it. So it's basically, it's the cost, right? And what about live migration? So since we're creating, we lose all the advantages of live migration. So this is what I said. Basically, with real-time, you tend to, you need more hardware resources because you avoid over-subscription, right? Or you don't share resources, right? And also some scheduler, I think somebody also mentioned like if you enable HT, it's basically throughput versus real-time. Sometimes if you achieve high throughput, you sacrifice real-time, right? So that's the mitigation I, you know, the proposed basically suggested, right? You just make part of your cloud real-time. Okay, so this real-time hypervisor is not a standard term. It's just a term you're using, or is it something that is standardized, level one, level two, level one? Oh, okay, that one is not standard. It's my term, okay, okay. And then, okay, just to make the explanation simple. Yeah, yeah. Any other question? Okay, I think then I'm done. Thank you.