 Good morning. I'm sure everyone's bright and shiny. Last night, Thomas Gleickschner took me out or where it went out, and I think I came back around 2 a.m. or whatever, and I think he did that, so he'd think if he got me drunk enough that it would make me speak slower. Good luck with that. Anyway, my name is Steve Rostead. I work for Red Hat. I'm on the real-time team there. I've been working with Thomas and Inglemonar from 2004 with the real-time patch. I'm right now the currently the maintainer of the stable releases that you can see. It's from 3-2-RT, 3-4-RT, which I found out on Monday or Tuesday or whatever the RT summit was. I mentioned that 3-4 is no longer going to be supported, but then later that day, Lisa Fon sent 3-4 patches out. So I went, oh, what's going on here? It says it's going to be end of life in September. So I sent him an email and he said he's going to do two more releases, approximately two. That could mean 10. So I probably will continue supporting 3-4 for a while. I also support, let's see, 3-10, 3-12. I think 3-14, I guess is done finally. 3-18, I'm still supporting 4-1 and 4-4. 4-6 is end of life now. Development has moved to 4-8. And when 4-9 comes out, if that turns out to be the long-term stable, they're going to develop on that. And that's going to be what I'll support later. So I'm not going to support anything from 4-4 up until probably 4-9 if that becomes the long-term stable. Anyway, that's not what I'm talking about here. I'm talking about SCAD deadline. So does this work? Help if I turn it on? There. So first of all, what is SCAD deadline? I'm sure a lot of people here know what it is and maybe a lot of people here don't know what it is and I think maybe that's why you came here to see what SCAD deadline is. One thing is, since you guys always take pictures of me, I have to take pictures of you. There. So there's other scheduling classes out there. When you first boot up your computer and you start your tasks and run your tasks, the default is SCAD other. That goes under the completely fair scheduler and it basically tries to do the CPU time slices for each task to be as fair as it can. So you know, you keep a bunch of tasks. A lot of times the ones that are like most interactive are usually the ones that hopefully gets the best or quickest time frame. So you have a smooth desktop experience. But then there's other scheduling classes out there that's SCAD FIFO and SCAD RR. SCAD FIFO first in, first out is just that. It's the first task to come in at certain priority. We'll run until it gives up the CPU or another higher priority process comes in that will preempt it and it will continue. There's a bandwidth, RT bandwidth schedule that's been out there for, I forgot which one it came in, but it's been out there for a while. If you look into slash proc slash sys slash kernel slash, you'll see SCAD underscore RT underscore runtime and SCAD underscore RT underscore period. That's actually kind of like a SCAD deadline type of schedule. It's a bandwidth scheduler that keeps real-time tasks running at most like 95% of the CPU. So for 950 milliseconds, real-time tasks could go into an infinite loop. But after it hits that, it will be throttled and you'll get a message in your console saying it's been throttled. And the reason why we do that is because of a real-time task where to go into an infinite loop because of a bug, it would lock up that CPU and bad things usually happen. So a lot of bug reports said, hey, I got some task that's where my system locked up and we find out that some bug in user space was using a real-time task and went on forever, so we added code to throttle those tasks so your system could still be responsive with only 5% of the CPU. But I'm going to talk about the concept bandwidth scheduler and the deadline a little bit later, but let me continue. Your processes in SCAD other are modified by a nice variable. Everyone here knows Unix and you have that nice command. And the nicer you are, the basically more CPU you give up. So the higher nice value, actually, the lower priority you are. And that's, you could use that if you have a batch job, you want to give it a really high nice because you don't really care about the speed, but if you have something that's interactive, you want that to have a higher priority. SCAD Fifel, I mentioned it was the first thing, first out. And finally, there's SCAD RR, which is SCAD Round Robin. And I don't know why we even have SCAD RR. It's a POSIX scheduler, so it's out there because of POSIX reasons, but I don't know anyone that actually uses it. I can't even give you a reason to use SCAD RR. I kind of find it's a weird scheduler because when you run at a priority, if it's another task of the same priority, schedules eventually will run as well. But you really have no control over when that happens. So it's just basically the CPU is going to give two tasks, three tasks, four tasks, time slices of the CPU, and there's really the, it's up to the kernel on how it does that, which makes it really non-deterministic, which is the whole point that real time is supposed to be, is supposed to be deterministic. And that's why I never understood the concept of SCAD RR. I avoid it. I tell people why use it because it's confusing. And not only that, SCAD RR has a little issue with scheduling on multiprocessors. If you have two CPUs and you schedule three SCAD RR tasks, two of them will be, if two of them get loaded on one CPU, they'll run 50, each one will get 50% of the CPU. The other one will get 100% of the CPU, and there's nothing there to migrate them. So that's one of the issues with SCAD RR. And I really, no one's complained about it. It's been like this for years, so I don't think we're ever going to bother fixing that. So let me come up with a little scenario. The start interest used to with like, you know, bandwidth schedulers and all that. You have two programs. And running at a nuclear power plant. And the power plant are, you know, they're cutting costs. They're really cheap. And they give one machine with one CPU on it to run the control system, the safety system of the nuclear power plant. And on that same processor, they have a washing machine attached to it. And that processor is going to run the washing machine as well to save costs. So the nuclear power plant needs half a second every second in order to make sure all the safety things are going on and everything's fine. So 500 milliseconds of every thousand milliseconds. The washing machine needs 50 milliseconds every 200 milliseconds on that same machine. Who gets the higher priority? Well, okay, who thinks the power plant does? Okay, you know this is a trick question. You just know it's not the power plant because that's the obvious answer. Right. So we have our power plant and we have our washing machine. The power plant, like I said, takes 500 milliseconds and every second. And the washing machine, like 20 milliseconds every, what, I forgot what it said. What did I say? 200 milliseconds. Or 50 milliseconds every 200 milliseconds. So you got all these chopping going around. That's what it would look like if they had the CPU by themselves. But if you gave the priority to the washing machine, so the nuke is higher priority than the washing machine, where you'll see in the red the washing machine can't run. So it's definitely going to miss its deadline because it has a 200 millisecond deadline. The washing machine is going to run for the nuclear power plant is going to run for 500 milliseconds. So it's guaranteed the crash as soon as you turn it on. And what happens here is your nuclear power plant is moving fine, but all the employees are trying to wash their clothes because they're glowing green from working in there. And the machine keeps breaking down. And now there's a huge line, everyone's trying to wash their clothes because they're afraid of their own health. No one's watching the nuclear power plant anymore and the system blows up. That's why the entire system matters. So now let's, if we swap the priorities and we have the washing machine higher priority than the nuclear power plant. The nuclear power plant can be split up. If you notice back there, if you look at it, it shrunk and it's a little bit, I actually made this, I actually measured it, so the size actually moved. The nuclear power plant can easily make its deadline and the washing machine can easily make its deadline. And what this is called is rate monotonic scheduling, RMS. And that helps you prevent any stall. So you have computational time versus a period. So a computational time is like how much execution time you need for the process. It can be implemented with schedule. The smallest period gets the highest priority as we just mentioned. So if you figure out the little mathematical formulas, it's the sum of all the computational times. Like I said, the 200 milliseconds over 1,000 plus the washing machine that had 20 over 200 or whatever, that's the formula. The utilization of the CPU is the sum of all your tasks with the computational time over the period. Okay. So let's throw a dishwasher into the mix. Yeah, they're really cutting costs now. So we're doing the dishwasher onto the same machine because we say, hey, we have a lot more utilization left on this CPU. Let's utilize it. And now we have the nuclear power plant. I kept them in the same 500 out of 1,000. The washing machine actually went up to 100 milliseconds out of 800 milliseconds. And you have your dishwasher at 300 milliseconds out of every 900 milliseconds. And because the washing machine has the shortest deadline or shortest period, you give it the highest priority. The dishwasher will get the next priority. And the nuclear power plant is sitting there way back at the end. So we start off. And the first tick you'll see is the washing machine that needs 100 milliseconds. So this is 100 milliseconds each number. So the first is 100 milliseconds. The washing machine goes off. And it finishes its execution. So it throttles. It says, okay, I'm done. It gives up the CPU and goes to sleep. And giving the next task up, it's going to be the dishwasher that has 300 milliseconds out of every 900 milliseconds. So it could still make its deadline. It's only at 400 milliseconds right now. And then finally, the slowest thing, the nuclear power plant comes. And it's chugging along. It needs 500 milliseconds every 1,000 milliseconds. And it hits at the 800 millisecond point, the washing machine wakes up. And it runs preempting the nuclear power plant. And then we're at the 900 millisecond period where the dishwasher's turn comes in. It starts running. It chugs along. And finally, the nuclear power plant runs. Look, that's that 1,200 milliseconds. So it failed. It crashed. But how could this happen? If you look at this, we had only 95% of the utilization. So going back to the slide, we find out that we do our calculations. You know, we come up, oh, see here? It was, yeah. The calculations come up to be, there's a lot of papers out there. I have links that have some of the stuff I looked up and read. I've read this stuff back in 2001, 2002. So this has been out for a while for rate monotone scheduling. The utilization comes down to where you could guarantee in any situation to always be able to make your deadlines, comes down to that little formula right there. Whereas the utilization could be no more than n tasks. So n is the number of tasks. And it'll be 2 to the root of n minus 1. So back with the 500 milliseconds, 300 milliseconds, and 100 milliseconds, and we do the calculations for the utilizations. So we plug that in, we add them up, we come up to about 95% utilization of the CPU. Well, we plug in the calculation of the same thing for three tasks. For three tasks, the best that rate monotone scheduling can give you is just under 78% of the CPU. So we easily went over what rate monotonic is guaranteed to give us. And you saw how easily that blew up. So if you really like doing math and limits and all that, you can run the limit calculation. And if you go from n goes to infinity, this calculation turns into the natural log of 2, which is just a little over 69%. So rate monotonic is guaranteed to give any scenario 69% of the CPU utilization, which brings us to sked deadline. It's not a rate monotonic system. It's a constant bandwidth scheduler with earliest deadline first implemented on it, which gives us a dynamic priority. And what that is, is every time, every tick or whatever scheduling event, it looks who has the next deadline, which deadline is coming up next. That gets the highest priority. So if we go back and implement the exact same thing that we just did with the three, you know, the dishwasher, the washing machine, and the nuclear power plant, you know, the washing machine starts off because it's got the next priority is the next deadline is the washing machine is at 800 milliseconds. So it runs once it's one. Then the dishwasher comes up, it's next because it's at the 900 millisecond period. So it runs. Then we're back with the nuclear power plant. Then we get back to the 800 millisecond where the period of the washing machine is now started. But instead of running the washing machine, the next deadline is a nuclear power plant, which is at 10. The washing machine started its next period, and that's actually 16, or 1,600 milliseconds later. And even at this point, the dishwasher is at 1,800 milliseconds. So 10 is still the highest priority. And we're all happy. So it could run, run, and continues. So it's get deadline. We implemented two new system calls in there. Because what was existing there before for changing your priorities didn't include deadlines and run times and periods. So we created a sked get adder. And the flags is there to help for future enhancements to the scheduler. And we also pass in the size. So it's similar to kind of like we kind of base this after like Unix socket or the socket system call or binding all the Unix where you pass in the size of the structure so you can make sure everything you're doing is correct. So the struct sked attribute can change in the future. And the kernel will be always backwards compatible. But if you want to know if your kernel supports a new feature that's added, you may have to add something new. And then the system call will tell you whether or not it's available on the kernel. The sked adder looks like this. So it's where we have the size of the structure. So make sure that this is the correct size that we're actually pulling in. And then your policy, that's the sked adder. So you can actually use this to change yourself back to use sked fifl or sked other or sked rr. Even that's why it has a nice there because if you're sked other, you could change your nice value using this call. And then the other fields will just be ignored. Sked priority again is for sked fifl and sked rr. But then we have the three new fields, runtime, deadline and period. That's for sked deadline. So when you write your code, what I always do is when I write a sked deadline task or function, I always get the information what the task query was. So I usually use sked get adder just so I don't have to figure out what this task is. Just give me what the status is right now and I'm going to make the changes of what I want and then write it back. The fields, the runtime and deadline and period are in nanoseconds. So everything is relative nanoseconds. So if you're doing it, although here's a little thing that you have to know. It's not in the slides. I'll let you guys know. So those are either watching the video or you guys won't be knowing about this, but those that just look at the slides, tough. I guess I can't exactly exact a level, but if you're doing anything more than a millisecond periods, like if you're getting to the nanoseconds, you need to make sure that you have hrtick, sked hrtick enabled. And that's in the slash debug, slash sked or debug, or was it sys? I believe it's in sys, kernel, debug, sked features. If you look in there, you'll see hrtick, sked hrtick or whether it's set. Look at that. Make sure it's enabled. Turn that on. Otherwise, everything is going, the resolution is going to be like in milliseconds. So if you want something better than milliseconds, you have to make sure HR high resolution timers are enabled for the scheduler. I mean, high resolution timers may be scheduled for your kernel, but if it's not enabled for the scheduler, you're not going to get the benefit from it. So, sked yield. I always love this. I've seen this used a lot in the past. I don't see it used as much. Sked yield actually used to be in the kernel and used quite all over the place. And one thing those real-time folks did was get rid of them all. In fact, we went through and got rid of all sked yield calls within the kernel and removed that call. So, sked yield can only be called by a system call now from user space. We still get let user space shoot themselves in the foot, but we don't want the kernel doing that. Sked yield, almost every single time I've seen sked yield used, it was a buggy code. I seldom ever saw it actually used properly for what it was actually made for. For sked other, it's another way of being nicer if you want. You could call sked yield just in case you want something else, give up my time slice for now and give it to someone else. And it does work for that. So, if you're on sked running some function or some program and you don't want this guy to be taking up all the CPU and batch jobs, call sked yield, that's fine, but nice works just as well. It's kind of obsolete in that use case. Sked FIFO and sked RR can use it, and the only proper time to use sked yield is when you have the exact or multiple processes running at the exact same priority. So, if you want to implement your own volunteer scheduler, you create everything at sked FIFO, do everything at sked FIFO priority, and when you're running and you say, okay, I want to let one of my other tasks run, you do sked yield and that will guarantee that another guy will run that's on the run queue and it just pushes you behind everyone in that FIFO. So, basically, since first in, first out, it takes you out and then puts you in the back of the queue and then you run again. But it only works if it's the same priority. And what we used to see was something like this. Like, sometimes they're like a try lock type of situation where you have a lock where you want to keep the lock order, grabbing lock A and grabbing lock B. But let's say the lock order, because I have, like, here, grabbing the F. So, basically, let's say that the normal lock order of your program, you have to grab B before you grab A. But for some reason, you grabbed A first and had to do some work, and then you grabbed B in the reverse order. But if you just take that lock, you could cause a deadlock because someplace else might already have B waiting on A. So, if most of your programs has B, waits on A, you don't want to grab A then grab B. So, what you might do is release B and then go back and try again. But I've seen people actually implement this and they did this. This is a code, actually, this is a real type of code that was done once I caught. I'm like, you've got to be kidding me. Because they're wondering why their system wasn't working. And what happened was they had multiple tasks with different priorities. And B was held by a low priority task. And then, well, higher priority tasks went up and called this code. And the A, B, release A, you know, SCED yield, which just rescheduled itself and then went through the loop again. So, SCED yield did nothing because the task it was actually waiting for was of a lower priority. But for SCED deadline, we actually have a new use for it. It's good. You use SCED yield properly. Because SCED yield is the way you tell the kernel, I'm done with my execution unit. So, if you're running, if you have a task to do every period and you're giving maybe, like, say, 500 milliseconds every 1,000 milliseconds, and you finish it in 200 milliseconds and you want to tell the kernel, I'm done, give up, you call SCED yield. And then, the kernel will actually just flush out the rest of your time and say, okay, and then you'll wake up at the next period. I have a, I forgot to put the link up here. If you look on, I believe in GitHub, but also if you go on to the kernel.org, RT tests, I'll have to post a text or send me an email if you're interested in this, inside the RT test where cyclic test lives, there's another program that I wrote called cyclic deadline that implements cyclic test. Now, cyclic test is a way that we use to test the jitter of real-time tasks where we would have a timer go off periodically and when it woke up, we looked at a timestamp and it says, okay, what's the timestamp now and what's the timestamp I expected to wake up and measure that and then it would record the jitter or how much off that was and give you the max, the min, the average. So, cyclic deadline does almost the same thing except for doing like the nano sleep that cyclic test does, it does a schedule yield. So, it sets up a period to the time it wants and then as it's running, it just wakes when it, it just says check the timestamp, okay, what do I expect it to be and then measure it and then schedule yield. Check the timestamp, it's just a loop of doing that. And that's how it's testing this deadline scheduler because it's not saying I want to sleep for this long. It just told this kernel, this is where my period wants to be and this is why I want to be woken up and then it just gets yielded and the kernel does the rest. My favorite machine, the donut hole puncher. Think of a conveyor belt going down with donuts that were made with no holes in it. So, you have your donut hole puncher punching around and it's all measured properly. So, now we have a period and we have a time slice that we have a deadline but the period and the deadline aren't the same because as soon as I run, I have a short time to make that puncher shoot down. Otherwise, I have lopsided holes and we don't want lopsided holes in our donuts. Especially if it's off to the side, you'll look at it and say who took a bite out of my donut? So, this is something else we don't always have. We can't have offset holes obviously. So, you can actually go in and put in runtime is less than or equal to your deadline which is less than or equal to your period. And when you do the calculation again, even though your period may be longer and you might have a lot of utilization, just for your calculations, for true utilization that's guaranteed, it's the deadline that matters. Even though the deadline between the deadline and the period, there might be opening up. But if you have two tasks waking up at the exact same time and they both have the same deadline, even though they both are sharing a lot, like you say, there's a lot of leftover utilization of the CPU because it's a small task. If they share the same deadline and they have close to the same runtime unit and that's more than one utilization, one of them is going to fail their deadline. So, when you do your calculations, deadline matters, not the period. But a lot of cases, period and deadline are the same, which makes things easier. SMP, multi-processors. The arch nemesis of real time. It screws up everything. In fact, a lot of times we always say we want to get, you know, if we had our way, the real time folks, we would get rid of 32-bit. We would get rid of multi-processors. We would get rid of modules. But people want these and we curse them. So, dollars effect for multi-processors. Let's say you have MCPUs and you have M plus one tasks. One of those tasks have a runtime of 999 milliseconds of every 1,000 milliseconds. By the way, if you look in the papers, usually they make the runtime, period and deadline all the same. So, technically, in the paper, it would be 1,000. But I'm like, come on, no one ever would make their runtime exactly the same as their period because, you know, you have schedule overheads and interrupt hands handling. So, I'm like, let's make it a little bit more realistic, tiny bit more. So, let's just say 999 milliseconds of every 1,000 milliseconds. Then you have M tasks. So, if you have five CPUs, you have five tasks that have only 10 milliseconds of 999 milliseconds. So, a slightly shorter deadline. They all start at the same time. So, when you implement this and if all the tasks have the exact same or those M tasks have the exact same period, they're going to all run. Now, if you have a good scheduler and they put them on all CPUs. So, since all these M tasks are going to be running on MCPUs because they have the shortest deadline, the 999, so they're going to run for 10 milliseconds. Well, that M plus one task needs only to give up one millisecond in that case. So, you are guaranteed for that task to fail its deadline. So, this is where things get tricky when you use earliest deadline first with multi processors. Because EDF cannot guarantee on all situations anything better than a utilization of one. Usually, you would say, if you have two CPUs, that would be a utilization of two, which would be 100% of two CPUs. So, EDF can only guarantee a utilization of one no matter how many processes you throw at it. So, there is limits you can throw into the equation to make it pass the give it more than a one utilization. So, it's not going to, we can't say multi processors are useless. Because you can achieve better than one utilization if you restrict the domain of your computation versus period. And we have two things you could do. One is partitioning. And one is global. Partitioning is where you say, okay, I'm going to bind my tasks to each CPU. And that way, if you bind everything to one CPU, you achieve a utilization of one. And then you bind something to a second CPU, you can achieve up to a utilization of one. So, right there, you have a utilization of two doing what's it called EDF. Now, the problem is for all situations, you could come up with something where that will not work. So, the whole idea that EDF cannot do better than one means that it can't do better one in any situation. But there are situations where EDF can't do full use of all the CPUs. That's as if you could break it up and partition it if you can. Global lets you say, okay, I'm going to let the kernel migrate these guys anywhere you go. But for all situations, one is the highest you can achieve. Sometimes you can't use EDF partitioned because there's no way you can accommodate everything to give you a full one on one CPU. You have to have migration. Although this above those utilization, that would be a trick to get that working. Even though it's only, what's it called, 1.6 utilization, that would be a really tough case to get working with EDF. But if you want to figure out how to partition things using multiple CPUs saying, okay, this group is going to have CPUs one through three, these CPUs are going to use utilization of two through six, another one is going to use all the CPUs, this becomes the bin packing problem. Everyone remembers your computer science courses, and you did NP complete problems, the bin packing, how do you pack like a truck and all that stuff. Same algorithm and it's NP complete. It's impossible to solve it. So in our lifetime, basically, if you threw in like 80 CPUs, there's no way you could figure out best calculations to figure things out. So EDF can't really partition makes it is very difficult if you do that. If you want to get the most optimal way of achieving your utilization. So we have looking at global EDF, which is what we implement inside the kernel. As I said, it can't guarantee utilization of one for all cases, but for special cases, we can. And the point is, if you have, let's just start saying, if you're deadline in your period are identical, I'm not going to play the whole trick there. So you have a deadline in period or the identical, so you have execution time. The utilization is a percentage of a task. So if you take all the utilizations of each task, so like I said, if you have, if you're using 100 milliseconds of every 1000 milliseconds, that's utilization of, you know, 10% or 0.1. And you look at it, you find the max utilization there. The utilization that you can guarantee is can be up to the number of CPUs minus the number of CPUs minus one times that max. So let's say we have eight CPUs, and the utilization of one of our max utilization is like that nuclear power plant that takes 50% of the CPU. We run it through the calculation, you know, of eight minus seven times 0.5, and we could achieve a max utilization of 4.5. Which is much better than one. But it's not eight. You have eight CPUs, you should have the utilization of eight. But we could do better than half the utilization if we limit our deadlines. So when you, this is where I start my rant on the current state of SCED deadline in Linux. When you start it off, the first thing we default to running on every CPU, we don't do partitioning. You can't do partitioning, it's difficult. And that's going to hurt using utilization of SCED deadline. We're working on it. I'm talking Peter Zilstra. We're coming up ideas to go because he said, doing partitioning, we always have to guarantee that the kernel, when you, we still guarantee that the bandwidth, the utilization is there, that you can succeed your deadlines from the utilization you give us. Those mathematical formulas are used in the kernel to make sure the guarantee is there. And the problem with doing any affinity, the SCED deadline tasks, you can't change the affinity. The affinity is always going to be all CPUs of the SCED domain. And we do that because to be able to change affinities arbitrarily would be that bin packing problem inside. And we don't want to do that inside the kernel. Running an NP complete algorithm is not something that people would like. That would probably cause your deadlines to miss your deadlines anyway. There's no forking of children or for other deadline tasks. You don't fork, can't have children. Your deadline tasks are fixed. Because that opens up a whole new can of worms. In the future, we may allow it if we have a good use case for it. Right now, we just say no. If you want a SCED deadline task and you want to fork something, basically whatever is running, it can't be a deadline. It's got to do the thread. You've got to clone, fork or clone. And then after you do that, then you've got to set SCED deadline. You can't just do it right away and then fork. That doesn't work. We are thinking of if you do it, it will just lose its deadline priority. So one answer is if you allow a SCED deadline task to fork, that task, because that changes as a whole new utilization, it just drops it to SCED other. That's one of the options we're looking at. And another thing is SCED deadline, you can say this is how much CPU time I want. So that means that the CPU is guarantee or the kernel is guaranteeing you bandwidth, that's the bandwidth, that's the bandwidth scheduler, that you'll get that utilization. The problem is on today's hardware, figuring out what that worst case execution time is, is extremely difficult. And sometimes when you find it, it's so much greater than the average. There's been times where I've done calculations or prime number calculations, that's what I do on my tests. I usually do prime number calculations for deadline to see how long it gets and measure it. And sometimes I figure out, okay, I want to measure how many loops do I get? How many primes do I get to satisfy one millisecond or even one second? And then after I got that calculation, I run it again and the time shoots to half or a quarter. So my worst case scenario, I'd have to say, okay, it really only needs 99.9% of the time, 250 milliseconds, but there's always those outliers that could be over a second. And that means what do I have to make my deadline that one second? So it's very difficult to calculate, get your application to use it just because today's hardware is so crazy. As I said, you can't do affinity, but you could create new schedule domains. If you want to do partitioning, which makes it a lot easier if you can partition and make sure that these tasks are running on this CPU and these tasks are running on this CPU, you can do that today. It just takes a little work. You have to go to CPU sets. You create your own set. You create another set. You put all your CPUs that you want to use, or let's say you put your CPU that you want to use, that's actually number three, into your set and all the other CPUs, like for this is four CPUs, I go zero to two, go into the other set, and then you have to set the MEMS memory, that's like for memory with CPU sets. Then there's load balancing, you got to turn on, you make it CPU exclusive, you do that for both, and then on top, at the top layer, you turn off load balancing there, and now you got two CPUs, or schedule domains, working. One schedule domain is going to be from zero to zero one and two, another schedule domain is just three. Now I could place a schedule deadline task into my set, so now that task is now bound to CPU three, and you got to make sure you push all the other tasks off that CPU, if you want. So it's not pretty, it's not nice. This is one of the things that's going to hurt utilization of schedule deadline in Linux today, is the fact that it's kind of a pain. I've implemented this all in C. If you want to look at and play around with this stuff, and you want to do it, find that schedule deadline, or it's called deadline test, and also cyclic deadline, I implement this in C and do all the work so you can partition, and it's not pretty, it's not nice. I've already mentioned this, about, it's hard to calculate worst case execution time, but there's a new, there's a set of patches out there that we're working on. Let's see if I have it on my next side. Yes. Grubb, no, this is not a boot loader. It's greedy reclaim of unused bandwidth, unfortunate name. So now when you hear Grubb, you got to say, wait, what are you? I got Grubb on the embedded device. Really? We use Umbu. So what it basically does is allows for unreclaimed, basically if you call schedule yield and you give up your CPU, that will be passed off to the other tasks that are using it. So now the utilization, you can put up the utilization of high and only use as part of it, and other tasks will, on a need, basically almost as a need for basis, be able to use it. There's a few other algorithms out there that use this, but the whole point is, let tasks get by, even though they don't, if they have that one outlier, and you only gave it so much percentage your execution time, because right now, if you have that outlier, the CPU, once you run up and you've filled up your execution time, the CPU will throttle you, and then you go off and you wait, and then you start again. And now you just, you didn't finish your task because you had some CPU problem where it took a little longer than you gave it. Now it will say, oh, do we have utilization still available? It will let you go beyond your utilization. And this actually works quite nicely. It's not in the kernel yet. Patches are going out. So expect it, probably maybe, hopefully, 4.10 will have it. If not, 4.11. Too bad, 4.9. If that's the long-term release, it's not going to have it, but maybe we'll back port it. Someone might. Here's a bunch of links. Top one, it's right in the kernel. Documentation, scheduler, schedule deadline.txt. A lot of stuff I talk about is in there. These are the papers I read before giving this talk. The lot of good information out there. The Anderson papers at the bottom. There's a lot of really nice, if you go to the Anderson link, there's a lot of nice papers about real time. You could be there all day, all week, all month reading them. So I got just under nine minutes left, and I guess it's question time. I don't know. No questions? Oh, should we have a microphone or something? Or just scream, and I'll try to repeat. So the question is, if you actually try to violate the constraints, you basically do the set attribute and you're violating the CPU, the set attribute will fail. It's linked. I mean, there's locks in there. It's serialized. You can't just have all these processes. Once you set up the schedule attribute, you forget it's only done it once. Once you start, once you start schedule deadline, you give it the attributes of what you need, and then the scheduler goes on from there. So the bandwidth is set up. You don't have to do it like every period or something. Well, yes. If the last process tries to set itself up and it's going to fail the mathematical constraints, that last process, that schedule set attribute where it tries to set itself will fail. Yeah, that's the whole point. Yes. So basically, it's just because one of them, yeah. Well, like I said, this is actually one of the few times where schedule yield actually works. But yeah, if you're talking about inside the kernel, you're doing it while you shouldn't be, inside the kernel, you don't want to do like schedule, what's it called, setting priorities within the kernel. It's your really specific case, which you may have. So basically, if you have one that's doing an interrupt storm, you want to make sure the other one gets in. But eventually, even though it's doing an interrupt, I guess it's just constantly runs. You always have it lower its priority if it does so many. I mean, that's kind of like what Linux does today with soft IRQs. Soft IRQ will come in if it starts doing too many soft IRQs, because it doesn't have a soft IRQ context where interrupts can go on, but it preempts anything else. Once it starts doing too much, it will pass off the information to Ksoft IRQD. So it will throttle itself. That's more deterministic. Right now, you're just, I mean, like I said, it's the easy approach using round robin. It's the easy approach. Like, okay, I'm going to use skedRR because that way, they have both of them at the same priority, and this guy will eventually get in, which makes sense. But that's kind of, to me, I think it's being easy, but still not deterministic. It's like, it's not for a real-time situation. You're just basically, you're trying to do bandwidth, kind of, with skedRR. Then why don't you just, why keep it, why is it a real-time task? Why don't you just use the skedOther, which would do the same thing? Okay, so you just have, yeah. Yeah, I mean, like I said, if you're just doing it just for bandwidth, okay, I'll let you go with this skedRR. If you want to be kind of, I would call that kind of lazy, just saying, just to let it go, but it's no determinism there. That's why I'm, I got to worry about skedRR because there's no deterministic behavior there. And if you, like I said, you have the issue where if two tasks go on the same CPU and another task, it's not going to be migration at all. You're not going to get full utilization of all, there's not going to be a fairness among those tasks. Yep. No, yeah, I'm saying, okay, it's, I said skedRR, that you could use it, but there's other ways of doing it, which I think is more deterministic. But I'll agree. I mean, if it works for you and it doesn't break and you don't need the deterministic behavior, yes, you could use skedRR, but I wish we would get rid of it because people think it's, sometimes think more of it than they should. So, yes, so I go over here. Why do I consider, why do I consider scheduling domains not nice? This is why I don't like it. Do I have to explain? No, no, yeah, yeah. I'm just saying it's not easy to set up. And currently, and I shouldn't be saying this, but I'll say it, currently, there's bugs in this in the kernel, huge bugs. There's times where the counting gets screwed up. So, and it's almost always on the worst. It's not like you, you're going, it's not, I've never seen it be where I got free utilization. No, I did once, where I got more utilization. The kernel let me get more utilization than actually I had, but then what happens is it gets screwed up sometimes where it, like you get lost or leakage, where utilization disappeared. And then when you go set everything up, it says, no, you don't have any bandwidth. And the only thing I had to do, I had to reboot the box, because I had no idea what screwed up in there. I found out there was, once I found the bug, I was able to get that bandwidth back, but that was a bug. And there's a lot of complexity when you do sked domains. If you look inside the kernel, the sked domains' code is not the most trivial of pieces of code in the kernel. And we're attaching priorities and bandwidth and these complex calculations to all this. So, we're working on it. Sked deadline is still in its infancy, so it's just born, let it grow. It's going to have growing pains. And right now, we're working on that. Anything else? Way back there? Can you have to scream? Yeah, you can. It's basically a bandwidth scheduler. So, if you just don't want this guy to run, so you could just have it, let the CPU throttle it, so you want to give something only a percentage. For example, what's a good use for the case for sked deadline, which I didn't bring up, I only have 40 seconds, but, is guests, guests and hosts. So, if you're doing virtualization and you have a bunch of guests and you basically want to limit a guest to only maybe 20% of the CPU and other tests, so you could spread guests around and only give, so if a customer pays 20% of the utilization, someone else pays 50% of the utilization, you could use this sked deadline to only utilize the CPU. And that's event triggered. So, it wakes up, when it runs, it only gets a percentage of the CPU while it's running and then when it goes to sleep or whatever. So, it can't be event-driven if you're just controlling that. That's actually one of the big use cases of it, which I should have brought up, but since I was doing this more from the real-time aspect, that's what I was going for. But yes, one of the things is the bandwidth, that's what the concept bandwidth scheduler is, and media, multimedia is very big. So, if you basically want to give TV or something where it needs a bandwidth, but it doesn't need all the 100% CPU, where it's basically frames are coming in, you could give it guaranteed frame packages thing. And a lot of the papers, if you look at it, talk about video, what they use, better use cases for how G-streaming and such. Yeah. So, you could, yep, a sporadic server for, that's another implementation. I got, yes, so, it got 14 seconds. For which one? DPPK? Okay. Anyway, my time is up, so thank you, everyone.