 Okay, let's get started. People are still kind of gathering. But okay, my name is Jun Nakajima from Intel. We've been working on the real-time and then especially in the context of OPNFB. So we are really real-time guys because we're working on, for example, the real-time KBM. So today I wanna share some of BKM, some of the methodologies, how we're gonna analyze and debug real-time applications in cloud, okay? So this is actually kind of sequel to the presentation we gave at the last summit in Austin, which was basically the building a real-time cloud using OpenStack. Once you build a real-time cloud and it's possible, or even very likely, you're gonna see a problem, right? So to solve that kind of problem, I think we need to have a structured way of handling those problems. So I'll share some simple methodologies to solve such real-time problems. And then we propose some monitoring of the architecture for real-time OpenStack. So this is what I presented at the last Austin summit. To refresh your memory, first of all, the OpenStack is actually ready to build a real-time cloud. We have a software stack, for example, Nova Flavor and then Hypervisor. We even have a real-time Hypervisor, like a real-time KBM. And of course, you can use containers if you use a real-time Linux, for example. The point here is to make a real-time cloud. You need to reserve system resources, for example, the CPU or memory. And also, depending on the real-time requirements, what I mean is, for example, you're talking about 100 millisecond or 10 millisecond or even less, okay? Depending on the real-time requirements, you may need to, for example, use more hardware-centric solution as well. But the point here is again, you need to reserve system resources based on your real-time requirements. That's kind of a given. Oh, sorry. This is very sensitive. This is very sensitive. Okay. Then, also, other thing is, why it's working, I know. Okay, I'll stop using this one, probably. I need to use this one. Okay. The other technique is host aggregates. I think you are familiar with host aggregates, but the purpose of a host aggregate is to provide a consistent host environment in a cloud. I'll talk about more on this one, why. But if you don't have a consistent host environment, every time you run, that people will see different environment, right, different machines. So it's very hard to reproduce the same thing, okay? And if you're interested in the last presentation, there's a link here, so. So analyzing debugging in cloud is actually more difficult, okay? The reason is the host machine can change every time you run, right? Because you don't have a control. And other thing is the neighbor. Neighbors can change at any time, right? And that's not under your control. It's either, right? So because of that, it's hard to reproduce the same runtime environment, okay? So that's the reason we are suggesting you use the host aggregate so that you can have a consistent, or you as a user or you as an admin to provide the same consistent runtime environment every time, okay? That's the key. The other challenge is in the cloud is it's hard to get the performance data and performance data from the host machine because if you're running a VM or a process, right? The scope is very limited. You need to get the real performance data from the host machine. And today the OpenStack API doesn't provide that kind of information, especially for real time, okay? Today's data is not good enough. It means not precise enough to resolve the real time issues. So I'll have some proposal to enhance this. Okay, so I'll talk about kind of methodology. This is just a structured way of again handling the problems. So the first foremost, the prevention is the very important, right? Before jumping into debugging, you really want to prevent and check the runtime environment is something you are expecting. Okay, it's good enough or not. I'll talk about more on how you're gonna check, I mean how in the following slide. But again, the checking those, for example the context switch or how many interrupts that you're getting or how frequent the VM exit. Or sometimes you may want to look at the cache misses as well. If you're looking at the very tight real time, like a micro second kind of thing. So once you make sure your runtime is good enough, okay, it's something you are expecting, then you run, right? And while you run, probably you want to monitor. But when you monitor, you don't wanna bother that real time environment. So the monitoring cost must be minimal. Okay. Then when you find some issues, problem, then you start using more, you know, expensive tool to get detailed data. So that should be kind of on demand, okay. In this case, it's okay to disturb real time thing because you already have a problem, okay. So having a expensive, you know, tool, expensive, not the price, okay. And the system or workload will be okay. Then basically you repeat this kind of process until you don't see the problems. Okay. So this is the cyclic test. Cyclic test is popular among real time, you know, application developers. And this is very simple. This is just a benchmark to measure latency. Oh, I didn't, I didn't touch. Just don't know why, I don't know. I don't know how to debug this one better. So, so what it does is very simple. So it's set up a timer, let's say T1, and at the time exploration, you measure the time, which is T2, then you calculate the delta T2 minus T1 and just repeat, okay. That way you can measure basically the latency from the time exploration, okay. What it can suggest is, for example, you have some issues, by the way, you want to run this in the VM, okay, or container, okay. Then you can tell if you have a scheduling latency into latency by looking at the output from the cyclic test. This is a typical output, okay, and you probably want to look at this one. This one tells that the maximum latency, okay. So 28, the unit is a microsecond, is good. Of course we have a reasonably good environment when we run this, but if you run in a VM and if you don't see anything, then you're gonna see much higher value, okay. Even sometimes a millisecond or 100 millisecond level, okay. So make sure, run this in a VM, okay. Oh, so, okay. So that's thing done with this one and then make sure you don't see some outliers, okay. That's the first thing, okay. Then other tool is a PARF, okay. This is a portable Linux and distros usually provide some package for PARF and you can install this one. And this can be used to analyze various events, including hardware and software events, okay. Again, sorry, I don't know what's happening, but. And you can also, I'm not doing any magic here, but anyway, okay, and I'm fine with this one, okay. And then, so in case of, this is how you use a PARF, okay. And if that guest or process is well-isolated and this is example when you run just the, you know, the PARF for five seconds, okay. And count the number of context switch, okay. If it's well-isolated, then that should be zero, but in case of a non-isolated case, you see 103 times during the past five seconds, okay. So that way, you know, if you, this is not good, right, because you try to, you probably want to isolate the guest, you reserve the, for example, system like a CPU to the guest and don't do any oversubscription. But if you see this kind of number, something's wrong. So other tool is, other tool is F-Trace. And this is also good to get more detailed data debugging analyzed latency, okay. You can also, you know, get the context switch information, BMX it, or hardware interrupts. And overhead is, it depends on what kind of event are you going to measure. And this is a typical output from a F-Trace. So you can see the process and the CPU number and then this is a timestamp in a microsecond. This part is microsecond and this is second, okay. And this is an event. And more information like a PID, other things, okay. So this provides very detailed information because it has a timestamp in microsecond and also provides that CPU, okay. The example, okay, it's probably, it's hard for you to see this one, but this is very simple, okay. I go through this one. We are looking at just the BMX data, okay. And this one tells BMX it happened at this time, okay, on this processor. Then go back to the same BM, okay. Look at the time difference. You see only one microsecond difference, right. So means whenever BMX it happens, the CPU go back to the guest within microsecond, okay. And also look at BMX it, this one and this one, okay. So the difference is about one second, okay. So I can tell two things from here. One is BMX it happens every one second, okay. And whenever BMX it happens, the CPU go back to the guest immediately. Less than one microsecond, for example. In this case you see the no difference, right. So means the time must be less than one microsecond, okay. In this case it's one microsecond. So again you can tell two things from here, okay. So it's not difficult to read this kind of thing. And this is a good environment, okay. So in case of a non-isolated guest, if you have some over subscription, let's say, you didn't assign the CPU to the guest or process, then what kind of things you're gonna see. So this one is QMU, which means the guest, basically the QMU process, okay. Then somehow on the same CPU, okay, same CPU, something happened. So you see, for example, in this case, a compiler. And compiler ran something and also kernel stuff happens, then go back to the QMU again, okay. So if you look at the timestamp, okay, here to here, okay. You see the difference, the time side is same, but if you look at the microsecond portion, okay. You see about 1.5 microsecond, no, so millisecond, okay. Difference, so means for during 1.5 millisecond, this guest, this process was not run, okay. So depending again, depending on the real-time requirement, this tells that something's wrong if you set up the environment to meet that kind of real-time requirement, right. So now I propose some enhancement to the monitoring architecture of our open stack, okay. So those are the requirements of our proposal. And then one thing is, one key is what we need to debug, you know, when we debug real-time applications in the cloud, we really need to get the data from the host machines, okay. And to that end, we need to extend our open stack services. At the same time, as you saw, for example, some F trace data can expose other things. For example, you saw some compiler stuff. That has nothing to do with your guest, right. It's basically capturing some other process on the host so that can expose some other activities, other tenant, for example, activities on the system. So when we expose that kind of a, provide that kind of performance data, we need to filter so that we can hide that, you know, unrelated or some security issues, okay. And again, to monitor this one, we cannot bother that the system, right, because that can break real-time stuff. So the overhead of monitoring must be very low, okay. The other thing is we want to provide the standard or popular real-time debugging tool, like Power or F trace in open stack. And then other thing is the on-demand I mentioned because like F trace, depending on the events, it's expensive or it's a heavy workload from current point of view to provide some type of F trace data. So that kind of debugging or analysis data must be provided on-demand, okay. So this is a high-level overview of a proposal, okay. So green stuff is actually we already have in open stack in Newton release, for example, okay. For example, the Nova and Reboard has a power of support for limited events, okay. So for example, the context-switching information is not available today, but it can provide some limited, I think, cashmere stuff, something like that today in Reboard and Nova API, okay. But what we need is probably we need more, for example, scheduling or KBM information, let me switch this one. This is from data. So this one, KBM, and then the scheduler from Perf, and then on-demand monitoring, okay. So this is an overview of our, you know, proposal for monitoring architecture. Oh, one thing is, this CMT, MBM is discrete, the cash monitoring technology memory monitoring, memory bandwidth monitoring, which is required to get more, you know, low-level data, okay. But it's already in open stack today. Okay. So this one basically just summarizing what I just pointed out in the previous picture. So we already have a summer level of monitoring capability in open stack, in Newton releases, Newton release, and then, but for real-time debugging and analysis, we need to extend. For example, we need to add the F trace and then also Perf, for example, around the scheduling, and BMX it was switching. And the other thing is on-demand, so whenever, you know, developers find some issues, we need to provide capability for developers to get more detailed information available from F trace, for example, okay. And so this is kind of my summary page. Like I said, the open stack is ready to build the real-time cloud. You know, real-time cloud is useful to build, for example, high-performance computing or backend for IoT, right, so, or media processing. And then you can, for example, use Ironic or use a virtual machine, but you need to resolve system resources, okay, so. But when you run real-time applications on a real-time cloud, it's possible to see issues. Then today, we don't have the, you know, infrastructure to provide the debugging analysis information. The key here is, you know, key data, what we need is available on the host machine that the application is running. So in order to get, I don't know what's happening. We need to provide more data from the host machine and those data are coming from, for example, the Perth or FJS data, okay. And now we propose this, the monitoring architecture, and then, yeah, and then one, I think a point I want to mention is the on-demand again. You don't want to disturb real-time activity while doing the monitoring, but when you find the issues, then we really need to activate the heavy weight monitoring to get more, you know, the precise and then the detailed data, okay. So anyway, if you have any thoughts and comments, please send the comments to us. Email here is, and then I have references here and then I'll post this one later. With that, I think I'd like to take questions. Oh, okay. Maybe that was almost the first slide, so I need to go back to this one. This one, I think. So the question was the comment on the SMI, probably are you, many people, yeah, fine. Many people may not be familiar with SMI. So SMI is, you know, typically invisible to software, but this was invented to handle some platform issues, or for example, some walk around in a platform, or for example, some events happens, you know, for example, the machine gets hot, then, you know, something needs to be done, and then cause, and once this happens, that can have a harmful effect to real-time, because some SMI takes more than 100 microseconds, and then it's, who is implementing SMI? It's basically the OEMs, right, that the company is providing the platforms. And there are some ways to disable SMI, and then we are kind of working on that, but the flip side of disabling, you know, the SMI can harm the platform also. So anyway, SMI is basically, you know, in a way to protect the platform, but some cases, some SMIs are not really necessary, for example, around USB, for example. But for real-time, these can have some unexpected behaviors, and it's not really so consistent. So if you run, for example, a cyclic test, and if you see some unexpected, for example, latency, very high latency, and if you're doing the right thing, then probably you might want to suspect SMI. Does that answer your question? That's right, right. Other questions? Okay, I think I'm done here. Thank you very much.