 Okay, it's two o'clock now. I know people are still coming in but this part the introduction part is not important anyway So I'll get over it. So we don't even bother listening My name is Steve Rosted Red Hat pays me to play with Linux My hobbies include working with the real-time kernel This tracing utility called ftrace testing utility called K test the make local mod config things like that That's what I kind of play on Not really much else to say about me Let's say back in 2007. I gave the talk to introduce the real-time patch. It was called inside the RT patch I gave it the Ottawa Linux symposium or less Obviously in Ottawa, and it was extremely popular there. No one knew exactly what this RT patch was they kept hearing about it They heard Linus Travolz. Just call us crazy people So people are very very interested and then last year I came and gave a talk about K test and people were asking me. Why aren't you talking about the real-time patch? It's been a long time since you presented it and I said, okay, you're probably right I probably should reintroduce what the real-time patch is about. It's becoming more popular a lot of it is now in mainline So I yeah, we do have a real-time patch But it's really focusing going on mainline is that it's not about if it's going to go in mainline It's when it's going to go in the mainline. That's the true question of today This is the slide the actual slide that I started with back in Back in 2007 OS So they asked me to change it so I updated it So it's my updated version of the slides and But things have changed within back then for example Well, I should call it understanding preempt RT because now it's It's no longer like I said the real-time patch is the preempt RT is the config option that you enable and since we expect it to go In mainline, I don't want to talk about a RT patch. I want to talk about preempt RT Well, I said things have changed for example Therein works it for Intel now. He doesn't work for IBM. So he was IBM now. It's Intel But that doesn't really matter because he's not part of this presentation anyway, so I Gave this exact same talk in Barcelona a few months ago at the Barcelona E ELCE you and Hello And free electrons posted it And actually it's kind of funny because that's Darren Hart right there and they I love free electrons they posted this and they said hey They not only that they post this a video of it of my whole talk is on video you go online see this whole talk there and On the blog they even said hey, we recommend seeing Steve Rostad's pre-time real-time patch I'm like wait wait. I'm recycling this talk now like people are coming to see me now like really you're going to be giving me Or showing this you don't need to come here. Just watch the video. So What should I talk about? Now I visited and we had the real-time conference in Raleigh, what's my right side out Raleigh, North Carolina where my brother my brother lives with his and my sister-in-law and They're into pumpkin chucking so You know, maybe we talk about pumpkin chunking But we want to be more real-time stuff. So what really interested me is really, you know trebuchets So actually I was like, wow, maybe it'd be cool to make a real-time trebuchet So, you know these how do you design it? There's level different way designs Well, maybe I can make a simulator that simulates it and So I started working on a simulator. I thought this would be cool to add like a real-time simulator and So I developed this thing whoops and said let's try to simulate it One thing I learned was after I started it That's a wacky trebuchet I'm way, this is like my new obsession. I'm gonna make this thing actually work. So, uh, let me kill that. Whoops No, it's not just friction. There's a lot of things the problem. There's a lot of things A trebuchet happens to be a double pendulum algorithm a simple pendulum is very easy when you attach another pendulum on top of a pendulum That's called a double pendulum and it causes a lot of calculus. So I decided to give up a mess and talk right back about real-time So Where do you get the real-time patch right now? I have a stable repository. I'm the maintainer of the stable tree I right now maintain a 3.0 RT patch, which will be I'm gonna stop maintaining it as soon as Greg Crow Hartman stops maintaining the 3.0 mainline Which is this year. I can't wait for that to happen It's becoming more of a burden on me because there's a lot of things that are broken about the 3.0 that I just won't fix That's fixed in 3.2 3.4. I have a 3.2 RT patch and a 3.4 RT patch Thomas Kleichner is still working on the 3.6 RT patch I believe he's going to come out with a 3.8, but he has he says I'll do it if I do it So there's no guarantee that it will come out at 3.8 But when he does I will be the 3.6 maintainer, but there's no 3.6 RT patch or mainline So I might still maintain it just Will just for temporarily Right now 3.4. I believe it's the long-term patch If three I don't know 3.8. It's not gonna be long-term. I know what's the next long-term 3.10 I think it's gonna be that Greg's going to do so we do a 3.10 So basically the idea is I will maintain a real-time stable branch as long as there's a mainline version of that same Brant or that tree is being maintained patches are there the wiki page is there You could get this slides up like when I'm done with it since I just finished writing or finishing tweaking this about 10 minutes ago I will upload these these slides right after this talk So what is the real-time operating system? Now the thing is I hate the term real-time because you have it's so overused. It's abused Real-time could mean anything. It means right now. Okay. That's what real-time is I Really wish they called it, you know a deterministic operating system, you know DOS It's exactly actually DOS was actually private best real-time system ever So what does a deterministic operating system do well? Basically, it does what you expect it to do when you expect it to do it. That's really what it's all about It's about no no surprises. No anomalies. That's what real-time systems about it's not about You know real-time because people thinks real-time is fast You know I was at one customer site and they said I can't wait to see you know get your real-time kernel up to see the performance It's going to give I was like, you know Real-time means Determinism determining is some kind of ads overhead, which is kind of the opposite of you know What real fast it's about so yes, we try work very hard. We do lots of tricks to get throughput out We do a lot of things to make this as fast our goal is to be just as fast on the average case as mainline That won't be but we're probably we are we are worst case faster than mainline Mainline has anomalies mainline. You'll see something. I could you know you could run this command in like 25 microseconds And mostly time and when you run on RT. It's 35 microseconds But every so often it may take two milliseconds to run that same command on mainline for us Maybe 40 milliseconds huge difference. That's what's about worst case fast times. That's what real-time gives you You want bill make make your deadlines you want to set up your system understand your system and that requires homework Real-time OS is only as good as what the hardware. It's running on some people forget this if your hardware Has you know cash issues SMIs going off you're going to have issues with your real-time response I mean if the system hardware stops because of the bus is now all congested and you're you have to wait for Things to happen the real software is not going to help you there and people forget that they sometimes blame us all the time We say well, let's look at our hardware. That's why we like to at red hat We have a certification of hardware that we actually look at the hardware We run a bunch of tests on it and make sure the hardware can actually do somewhat what we want We'll actually rate I'll give it say well expect maybe like you know 100 or 150 millisecond latencies Mills microsecond latencies and I'm those like it microsecond latencies Well, of course 100 100 millisecond latencies may be acceptable with some or one millisecond latency may be acceptable for them It's not acceptable for what we would do but it's there and believe it or not The mainline kernel is not will not guarantee one millisecond latencies There's several times where I could trigger three to four millisecond latencies with a real-time kernel even or sorry with the mainline kernel even with a lot of The stuff that came from real-time in there already. It doesn't have everything yet So what is the goal of the real-time kernel? Basically, we want to be a 100% preemptible kernel so What does that mean that means that we could react to anything at any time the interrupt comes in I know it triggers like you get a response from the network some device some sensor sends a Interrupt to the CPU we want that CPU to be able to stop what it's doing to respond to it if it's required to do so So we need to be able to stop at any time. We can't be processing, you know MP3 play play So when playing their MP3 player or something when you know it interrupt comes in We want to be able to stop going and work process anything that may be more important So that's what we want Obviously, you can't have a 100% preemptible kernel, but as Lita says we're crazy. We will try We haven't succeeded yet, but no one has so that kind of makes us feel a little better So how do we do this? Well, one thing is we try to disable or do not disable we're trying to disable disabling interrupts. Does that make sense? We try not to disable interrupts as much as possible and We also try to make sure that any type of any type of instance where preemption is disabled preemption means you can't stop Then you'll switch out this task and work on another task. We try to get rid of that throughout the kernel We want quick quick reaction times So When you install the real-time patch and hopefully when it's a main line, you'll get a menu that looks like this In your you know make menu config you'll see there's the full real-time pram kernel Yeah, let me explain some of these just to review I don't know if people do defaults, but I'll go over for those that you know just never thought about it What do all these things mean in the preemption models? No preemption that's the top one This is basically was the idea was for servers that don't need to do like not for servers that acts for web servers, but basically servers are doing Tasks batch batch processing of tasks things that you don't care about a reaction You just want stuff done just send the stuff out get the job done your compile Maybe compiling something that does just a lot of hard work, but you really don't care, you know the reaction for it You just want throughput Server is like don't disable preemption because whenever you disable preemption There's overhead to do a little sweat or sorry don't table preemption whenever you do a Context switch there's a little overhead there So sometimes just serializing everything is faster as usually some of the best throughput you get is by serializing all your tasks Let it run to completion do the next one run to completion do the next one server model like that is what we do No one actually enables this anymore that I know of Voluntary preemption this is what a lot of the distribution is actually default to Because they're still a little bit nervous about preemptible kernel, which is in mainline. It's been a mainline for a long time Voluntary preemption as you'll see a lot of places that say might sleep might sleep is like a debugging situation where If you have a preemptible kernel You don't there's some places you don't want to schedule So if you call something save you disable interrupts, and then you schedule call the scheduler utility You could break things because what happens is If you disable interrupts and you schedule out You kind of have to state in a kind of weird state because you're not supposed to schedule out Well interrupts are disabled actually worse yet if you grab a spin lock, and then you just schedule out That's even worse because you go away then someone that comes in goes to grab the spin lock He's only spin and and you could block or he could block you from coming back and releasing it deadlock so we put might sleeps in places where if you if you set it to bug option and It hits this might sleep it'll actually give you a warning say hey You know you are in a situation where you shouldn't be sleep scheduling But this what you just called could schedule because not that's the thing about debug or that's the thing about Kernel development you might be calling this function hundreds of times with interrupts disabled or spin locks held But if it never schedules out you don't care like it won't worry about it So this lets you know like this you know this might block on something might grab a mutex and block and you're basically screwed So what does voluntary preemption have to do what does it have to do about voluntary preemption? What's really nice about all these might sleeps? It's saying that this function can schedule we are specifically saying we could schedule at this point So what voluntary preemption means is when it gets to hits one of those might sleeps It will call a might rescheder which calls condition resched which will say hey if you know Something came in and wants us to wake up. Let's schedule at this point. We already know it's safe because we'll warn about it So voluntary preemption is just all these might sleeps turn into preemption preemption points within the kernel So that's how that works The preemptible ground was created by Robert Love His idea of rampable ground way. He's came out about is it was based off of basically S&P the multi processors because multi processors is the way you think it's a spin locks so if you have some critical section that cannot be touched well, it can't be re-entrant and It's great on a uni processor system with no preemption. You just Work with whatever data you have. There is no critical section. Everything is protected Well, if you have a critical section where if you know two Processes we're trying to access the data or modify the data at the same time You're going to corrupt the data So you protect it with a little spin locks everyone shouldn't understand what spin locks are Well, Robert Love realized was this actually this model also works say if we had a preemptible model where you actually could preempt the kernel These spin locks could actually represent places that we can't preempt So what the preemptible kernel does even on a uni processor system is been locked now becomes functional And it says just disable preemption while I grab the spin lock So all spin locks you grab just disables preemption and when you release the spin lock preemptions enable again anything outside that Spin lock can be preempted. There are some cases where that's not always a case And that's why there's a preempt disable function that you'll see throughout the kernel preempt disable preempt enable because we say well We're doing something special here. We're not grabbing any spin locks, but we can't if we schedule out here We could break things on uni on a uni processor system or a not uni processor system But when preempt is turned off these preempt disables are no lumps The preempt kernel basic basic RT now, this is where the RT patch now. This is not a mainline not yet Well, actually this may never be a mainline I'm just popping this up just because you'll see it if you download install the real-time kernel You'll see this big option. I'm going to explain this basically Thomas Kleitzner likes to debug the real-time kernel And since there's a lot of things that the real-time patch still does There's a bunch of things it adds and then it also adds Spin locks turning into mutex's which I'll talk about and The thing about this is he wants to make sure that there's something breaks. He knows what broke Was it the mutex's or was something else I add so what he did we added this config option Just so that he doesn't need to if something breaks and doesn't break under this convection config option He knows kind of like okay. It's the mutex's so this is sort of debug Debug what's called option basically ignore it Ways it too much time on that ready Now this is the thing that everyone here is here for well when you select the fully preemptible kernel This is the real-time patch. This enables everything that you want it gives you The almost preempt everywhere type of mentality What happens is all these spin locks turns into mutex's interrupt interrupts get pushed off into threads So there's really no hard interrupt context We also add a priority inheritance Feature to all the lock all the sleeping blocks in the kernel. We had priority inheritance Right now today priority inheritance is in the kernel, but it only works for few texas So if your system or if you're using like p threads and you have p thread mutex create and you give the attribute I can't remember the exact attribute name, but you can say prior inheritance or whatever That few text will get priority inheritance. So one thread high priority thread blocks on a lower thread lower thread p thread mutex it will Boost the priority of that lower thread Task holding the mutex and run there. So in user space this works But it's not in the kernel not without the real-time patch Let me talk a little bit about the sleeping spin lock. So what happens is when you enable this? Well, let me State that what's first thing I said is the pre-config preempt is a global lock What I'm saying is when you disable preemption You just said I'm the most important task on the CPU No one is more important to me It's like having priority 100 where you know 99s max 100 nothing to preempt you so That's this is bad when you have a low priority not important task disabling preemption It's now the most important task on the system. Well, it thinks it is So whenever you grab a spin lock that are the spin lock it disables preemption excuse me like Robert loves Talked about I talked about with Robert loves work and we don't want that So the way to do that is let spin locks sleep So we actually converts the spin locks into a mutex So instead of grabbing a spin lock or spinning we actually if you block We actually go to sleep now. There's something called adaptive spin locks, which Is what we really do which is if you block on a spin lock or a sleeping spin lock You check to see if the owner is awake because most likely spin locks in the kernel are very short time periods So instead of scheduling out and then scheduling back in that might be too long. You just actually oh the owner is awake Let's wait till the owners while the owners awake and still has a spin lock We're just my spin if the owner goes to sleep then we'll go to sleep too or the owner releases lock We'll get the lock, but a lot that speeds up things tremendously People and what's called Susie actually came up with that that was actually a great idea Now we have like two types of interrupt. Oh, yeah, let's have interrupt friends Okay, spin interups have our inner panelers have a lot of spin locks So if you turn the spin locks into mutexes and you actually run into a hard, you know interrupt context You hit you block on a spin lock You're going to sleep and now if you guys realize but sleeping in a hard real-time or a hard interrupt context is Not a good idea. So we have to push out all the handlers into threads. That's a must So once interrupts are threads, then that works actually makes it things. It's not really reasons to disable interrupts And again, this uses priority inheritance. It's not just for few texas anymore. I Add this is I added this from Barcelona preempt lazy. I want to talk about this another enhancement Remember how I said the real-time kernel number one importance is determinism Number two importance is throughput. Now these two are almost inverse relational to each other The more deterministic you have the less throughput you have How do we handle this one of the trip one of the things that we found out with the preemptible kernel? We're preempting all the time and then we have this timer tick that goes up in the for a sked other tasks Everything's kind of given a certain amount of time to run. So if you have a bunch of tasks get other, you know The CFS scheduler is going to say okay, give this guy some time to run this guy Sometimes it runs in this guy sometime to run what happens now is the tick will go off and says okay Your time is up preempt you so we stopped the guy at that time and run the next guy now What's the problem with this the problem is he may have been been holding a lock So we just preempted this guy holding a spin what used to be a spin lock is now a sleeping It's just mutex so we scheduled this guy out and schedule some other guy real-time tasks wakes up goes grab That same lock now. He's only block on it has to wake up this guy just went to sleep Give him the priority inheritance let him run release the lock and come down where if we just let the task finish While he had that lock finish and release the lock We want to have that contention this constant This what's called the CFS scheduler was causing a lot of contention of spin locks much more than a normal Linux kernel And it was killing performance. We still had the latency We still had the small latency. We still had the term determinism, but throughput on everyone seemed to be going down Tremendously so what we do now is with the preempt called preempt lazy So if the tick goes off and says oh this guy's holding a lock It just says hey this guy should schedule when he releases a lock, but don't schedule them out now Only time that's that's Conditional if a real-time task wakes up and or anyone that says I need to schedule now It will stop them then you'll run. It's just the tick-based CFS scheduler says, okay Your time has you know is up, but you're holding a spit sleeping spin lock Continue until you're done when you release the lock then go to sleep. So that actually Throughput went up tremendously once we implemented this priority inheritance okay Priority inversion is bad, but you can't get rid of it What we get rid of is what what we don't want and what we can get rid of is unbounded priority inversion Anytime you have a task running that when another task wakes up of higher priority and it wants to run But it has to wait for another task to finish doing whatever it's doing whether it's doing it process I interrupt or just holding a lock. That's a priority inversion That means that this lower priority task is running while a higher priority task wants to run But can't due to whatever reason You can't get rid of complete priority version But unbounded priority version means that there's a chance that this higher priority task wakes up blocks on a lower priority task There's lower priority tasks may continue forever. You don't there's no bound to how long that task will run and That makes your system Undeterministic you can't claim to be a deterministic Operating system if you allow unbounded priority inversion That's why the real the mainline kernel right now can't have that one two or three millisecond latency or jitter is due to priority inversion It's complex a lot of people have criticized priority inversion because it's a complex algorithm and we've actually tried to make Multi-owner for rewrite locks to have priority inversion with rewrite locks that just made it more just like exponentially more complex That's why we want another reason why we hate our W locks is because all our W locks now It's a single mutex in the real-time kernel, which kills our throughput. Let me explain what unbounded priority version is We have this task, which is a lower priority task. We have a b and c three tasks a is the highest priority b's second priority sees the third priority he's Sees running along and gets preempted by a but in the meantime c grabbed a lock a runs highest priority task It's that same block. Okay Go back and continue, but before c could finish and release the lock It gets preempted by b because b is of a higher priority and b runs Let's say b require it will run until a tells it to stop See what happens. There's no bound now b will just continually run and a is blocked forever Our highest priority task in the system can't go continue What does prior inversion give us her priority inheritance give us? See runs blocks as soon as we blocked here what that means is c will now inherit the priority of a so c is now running at a temporary priority boost So when b woke up it won't preempt because now c is about a's priority Which is higher than b's and b will have to wait so when it releases a lock You know this guy goes up a continues to run finish when a finishes now b can run That's priority inheritance and this is where everything's bounded You can actually measure this and see how things what your max latency will be Now there's some places in the kernel that we can't have sleeping locks. The scheduler is the main one Okay, you can't really have a sleeping lock for the scheduler And there's a few other places the timing code does this so we have this thing called raw spin lock Back a couple years ago at the real-time one of our real-time workshops We try to brainstorm, you know, how did what we should call this and you know Because we can't one thing at least our balls told us is The original way of doing spin locks was we had this magical macro that would magically turn some locks into sleeping Socks and some locks would not stay into sleeping locks and just by looking at the code You couldn't tell if this spin lock was a sleeping lock or not sleeping lock because we had this magic macro That was determined on the type of that lock and Lena said that's not acceptable because it either has to be a Lock that could always say I could sleep when you turn this feature on or it could be a lock that doesn't sleep We had this whole thing about what name what should we called it? I like to the name live lock and dead lock or goldly locks But we came up with raw spin lock So what we did is we converted everything that it's going to always be because before we're going to label what there's sleeping locks Maybe but we that was too many locks. There's only a few locks. It doesn't should never be converted So we said let's call those raw spin lock So if you look in the kernel today, you'll see a bunch of raw spin locks that really in the kernel There's no mean for having this raw spin lock mainline doesn't need it again This is another reason why I'm saying real time will be in mainline because we already have code in mainline That's only for the real-time tree So we are setting this up so when you see raw spin lock, that's telling you this lock must always Spin it can never be a sleeping lock so Thread and errors Now this side, okay interrupt when interpandler comes in a lot of times you'll have like say if your disk drive goes off It call calls interpandler and the interpandler runs well again interrupts a hard interrupt will disable interrupt Well when interrupt is triggered most likely to disable interrupts and then it will run on a not in a it can't be preempted A non-preemptible mode which means that this interpandler is not most important task in the system again And you know if you're doing interpandling for tasks that you don't care about it's going to block your real-time tasks So we don't want that we want to be able to you know push that off And let our real-time task if our real-time tasks more important than these interrupts We want to be able to let our real-time task user space task run over an interrupt handler, so Another some of the things that are kind of nice about making interrupts as threads You could prioritize each interpandler is sometimes some hardware supports that you've actually say this interrupts Handler is more important than this interpandler But some are a lot of architectures don't handle that give you that or it only gives you a very limited version now you actually go through every single interpandler and Prioritize what you want how he's put that let that's that came from the original path back in 2007 because update DB every time I was working on my working at home I was working to like 3 a.m. And update DB would kick off at 2 a.m. and I'd always know it because my system would just slow down to a halt and Actually, I installed the real-time kernel purposely so I could actually continue working and let update DB not bother me anymore today It really my machines are fast enough that I don't even notice anymore like kept it in there Here's what interrupt happens task goes off interrupt happens this device handler handles goes back to the task This is the interplay. It's what I call an interrupt latency This is the latency or actually the Yeah, I guess latency is good word, but You know the into the disturbance of your task is going to be this huge big that this device handlers really long Handler is you'll only see that in your task with threads We still need the hard interrupt handler. Okay, that's still a hardware feature We can't change the design of hardware. So we have to have software. So we're coming up with our priority high priority task interrupts Going come it's going to go up regardless and we go we wake up the device thread and come right back So this is the latency for us. It's really really tiny and it's consistent. It's almost Every time we have an interrupt handler go off. They're all the same length Then later on this got our high priority test size ago. Okay, I'm done Go to sleep now the device handler could go and work and finish up what it's doing There's some non threaded IRQs Basically the timer interrupt Which is mostly in charge of scheduling the real-time tasks that are going on So we don't want the you know the time if the timer up was a thread and you made it Low priority, you know, basically, how do you stop in schedule? Is there a question? Well, actually we want to get rid of that. I may talk about that later They say whether we want to get rid of taskless and soft IRQs Yes, they sometimes the soft IRQs run an interrupt context sometimes most most of time But they can't be running in a threaded context as well Actually it would help more Yes, in fact, actually I'm working on there's nappy, which means new API I'm actually working on something called a nappy, which I'm going to call it even newer API Which actually is converts everything to threads because the whole idea is I'll talk about right now. This is good. What happens with the nappy code It was basically interrupt will go off process it interrupt go off process it interrupt go off process it and actually It could the CPU was actually the bottleneck here Well, we found out was let the buffer handle it interrupt goes off kick off the soft IRQ Let it wait and then the buffer will handle all these Processes and then when the software you would wake up then it would say okay Now we just do them all at once or we almost and we go into a polling mode And we just like let's keep on just going in a soft IRQ context We just go into a polling mode now imagine if we made it into a thread where we get our first thing We just kick off the thread it goes into a like a permanent polling mode later Q fills up start going to polling mode and as a thread you're still handling the delay actually is Effective to have like the inner goes off post it off let it go Handle the into a polling mode because you can't do it in a hard IRQ soft IRQ You could go into a polling mode, but even then it's the polling mode of that could affect the rest of the system as well So yes, it might help your network, but it's not going to be maybe it's not going to help the tasks that need to start processing These tests so now you can actually control let my task Oh, you know let the interrupt happen, but the tasks that needs to process this start having it work as well so Yeah, basically we're adding the latency to the to the networking stack, right? Oh, yeah Well, the whole thing is you didn't the thing was you didn't want to do the polling in the hard interrupt context Because when you do polling in hard interrupt context that that stops all interrupts so you know on that CPU or you know whatever's maybe you just have like one CPU that's Committed or had affinity for that IRQ when that's running it's going to stop all interrupts So you've got to actually push off to a soft IRQ which is kind of an interrupt context But the hardware doesn't consider it interrupt context. It's only a software interrupt context That's why it's called soft IRQ So I'm actually be working right now. I'm doing some basic tests to see it There's some tests that show that this actually improves networking performance. So I'm we hopefully something will come out with that. Yes Well, the threads will have affinity the threads have affinity to it and actually in our t-task We actually when you set the IRQ affinity it tries to keep the threads with the same affinity We switch back and forth to that Also, there are some things that if you say it's called no, if you put in Some when you declare your interrupt request your interrupt You can say this interrupt must always be a real interrupt and it won't thread it. That's Don't do it. Don't think that. Oh, this is this will help me. It's mostly for like timer interrupts By the way thread interrupts is today in mainline You could actually right now We actually have two ways of doing thread interrupts one is per device where the device could say I want my task actually this one will be working on is making the driver itself say I want my Handler to be a thread so you don't have to create your own you have to do anything about creating work queues or whatever you want You could just say hey give me a thread dedicated for my interrupt and How do you do that request threaded IRQ instead of doing request IRQ? so that will give you a threaded IRQ and What's also nice about this you can actually have other you can have some that one interrupt line if you share an interrupt line You don't like some of the devices could actually have different Priorities for that one same interrupt line. So you can actually have one interrupt line that Handles three different devices and each device have you a separate priority on how that interrupt line works as they're all threaded We also have the big switch One big switch hand way where you could just say give me all interrupt handlers You can do is a mainline say give me all interrupts to be a Threaded interrupt, but if you have three devices that don't have their own interrupt Like threaded interrupt didn't say give me a threaded interrupt. They just said request IRQ Those devices will actually run Those devices will actually share the thread of the other devices that's on it. Yes Well, yeah, but today's systems can handle it Back then they really that was an issue and stuff so today threat and we also we we fixed Contact switching is much much faster today The overhead of threads is much lower. There's a whole bunch of things that the kernel has improved So sometimes looking back the way you did things is beneficial because the whole you know Paradigm of what you're working in has changed where the old some of the old ideas now are applicable today. I Mean who look at your cell phones How many people would use like look back at installing a full kernel with tracing and everything on it on a embedded device? You know look at how embedded device devices have changed See I said the request started I already talked about this. I said too fine. I've this I already talked about this When you'd say like when you flip the big switch We actually create our own little function that will just is what's called for all the IRQs that didn't say I want to thread it will be given a threaded function to use To enable the big switch you not only have to configure it in your kernel But you actually have to go into the kernel command line and say are the current the kernel command line parameter When you boot up the kernel you actually have to say thread IRQs To have it implemented so you can actually have it compile like on a production kernel if you want you actually have threaded IRQs The big switch enabled, but it won't default go that way unless the person put on the command line Hey, I want threaded IRQs Okay Things did not to do in the kernel local IRQ disable. I hate seeing this Really you should never have local IRQ disable in any driver. I Really get ticked off when I look in the driver code and see local IRQ disable because you shouldn't be doing that You're saying that my driver is more important than everyone else's driver. That's what you're doing and Well everyone thinks it is It's that old like it's like my patch is more important than your patch review it now You know, it's the same mentality Don't do IRQ disable in the mainline core kernel. We use it, but we try to we're trying to get rid of them We hate local IRQ disable. Sometimes I see local IRQ disable and I'm looking I'm like why what is it protecting? There's no comment No, nothing to see a local IRQ disable here and bunch of functions are called and then local IRQ disable I'm like, okay. What is it doing? Maybe they just said oh, I just don't know why this broke But if I do look like you disable it will it works now. I really believe that has some of the reasons And the thing is like I said, that's also probably means that there's SMP bug in there Yeah, it works because you just made it non-re-entrant for this one CPU But another CPU could come in and hit that same side. So a lot of times I see local IRQ disable And yes, a lot of times it's buggy. It means that it's not safe with multi processors So and like I said, it gives you a high latency and it really sucks for real time Use the spin lock IRQ save. This is the angel of it. We want we love to see spin lock IRQ save Custom preempt RT spin lock IRQ save doesn't disable preemption because the reason why you have spin lock IRQ Usually is because you have that spin lock will be used or somehow be associated with a interrupt That's using spin locks that's going nest within this guy And that's the case that if you don't use spin lock IRQ spin lock IRQ save or spin lock IRQ If you grab that lock the inner goes off grabs the same lock you just deadlocked But since interrupts are now threads, you don't have that problem anymore All those deadlocks just disappeared because you could sleep the inner go up grab your same lock But you could sleep too because they interrupt as a thread so it could sleep and then go back So all the in real time spin lock IRQ spin lock IRQ save. It's just a spin lock. There's no difference The interrupts are not disabled. Nothing else is done differently But what's nice about mainline doing this is you're actually kind of labeling giving the name the spin lock You're labeling what you're trying to protect. So it's actually good to do this Okay, the less obnoxious but still obnoxious Collar from instead of local IRQ disable we have preempt disable which is the same as exact same problems as Local IRQ save although you allow interrupts to come in you still don't let any task continue So you're like I just don't want any tasks to happen So I'm like preempt disable do a bunch of stuff preempt enable and a lot of times again I don't know what you're trying to protect. Why why do you have this preempt disable? A lot of these things are not commented so What's worse is like preempt enable no resched when you enable it because when you preempt disable if the interrupt comes in It says hey, you need a schedule when you call preempt disable it will actually call schedule But then you had this preempt enable no resched because for some reason you know that you don't need to do that check Usually this is called within a preempt disable So if you're doing preempt disable do something else preempt disable you already know preempt disable is already happening I don't know why you do it twice. Anyway Or preempt disable and you do preempt disable again. They do a preempt enable and you know you're in a preemptable situation It's sometimes okay to use the no resched Because you're not going to schedule anyway because you already have it's a nested preempt disable You could throw this in but this will warn if you do it if you don't if you use this when preempt enable no resched and you use that Outside of a nesting preempt disable it'll give you a nasty warning. So you actually throw this in there I don't know why I'm telling you guys this This is only because if you know you're going basically preempt enable no resched because you're calling schedule So if you do a preempt disable you do a bunch of work Then you're going to print enable because preempt enable will call schedule Right after if a interrupt came in and said you need to schedule out and you can't because you're preempt disabled So when you do preempt enabled it does a check. Okay schedule Yes, well, you don't want you don't want to do that if you're calling schedule right afterwards anyway So if you have like preempt enable and schedule what happens is you can say okay schedule out So you'll skip when the preempt enables car preempt enables call this is oh we should schedule out So it calls schedule you schedule out then we get scheduled back in the first thing you do is you call schedule again So the only time that you should ever use that is if you're calling schedule right preempt disable print enable schedule But you shouldn't be using preempt disable in the first place. So forget about the slide No, it has the same problems as low-caricule disabled. Yes Yes, yes preempt disable disable preemption on that CPU only so same same thing per CPU Now this is the one time where people will use preempt disabled They'll say preempt stable because I or IRQ disable and they'll do a bunch of stuff with these per CPU variables because hey I'm not preempted disabling. I don't want if I don't preempt disabled I might migrate to another CPU and now the variables I'm working with is not my CPU variables I started with so people use like low-caricule save this but this hurts real time So what we've been time for what we kind of added we added some Functions and that will be coming they're not in mainline yet, but they will be coming soon One thing you always do is you pin your CPU or pin like if you use a pin CPU thread So this task only runs on this CPU you can always use you know per CPU variables without worrying it because that task will never migrate But we added some stuff They said get CPU var actually disables preemption. So we don't like that. So we added get CPU light on non preempt RT It does the exact same thing as get CPU nothing different But when you enable preempt RT what it does is disables migration now. There's some cases where Now you have to wear it out. You can still schedule out So if you are going to modify some per CPU variables and you scheduled out something else may come in and modify those Same per CPUs variables. Well, I'll talk about that later Get local var is the replacement for get CPU var, but there's the same thing it just disables migration They don't migrate disable handler. That's kind of nice So say if you have per CPU variables that you have to protect from other If you have some per CPU variables that if you get migrated out Or if you get sorry if you get scheduled out and someone else could schedule then and goes to the same code path It might use those same CPU variables and mess up your whole situation Where in mainline preempt disable and preempt enable does all the protection you need There's no bug with SMP because it's a per CPU variable And you just want to make sure that no one else gets in there Well, we've added was this local lock which also local lock RQ local RQ safe Which is basically the same as local RQ save or local RQ disable or preempt disable depending on which version you use But in mainline, it's just like that. It either does preempts disabled or disabled IRQs But when you turn real-time on it creates this it creates a sleeping spin lock for you That's associated with the variable you're getting or with the name So you have to this thing actually pass in a name to it and you pass a name in it and it will associate It'll create a lock for you So when it goes into the section you'll grab that lock and then do your work and then but still you could be preempted And if someone else comes along another task comes along it goes to that same section It will have to grab that lock too, but you already have it so it will stop So it acts just like mainline doing a preempt disable, but using locks and allows More reactions happening. You're not stopping the entire CPU just to do your work Well, none what's nice about mainline. It actually helps you label the code for you. So it basically labels the The critical section so it's more documentation, which is a good for mainline RW locks don't use them. I'm not gonna go talk about it much Is that definite determinism there's problems with because as recursive locking you could there's two types There's unfair RW locks and there's fair RW locks also RW stems I think are fair with her so if you have a read writer lock, which means multiple readers one writer and Say if you have a bunch of readers, let's say it's unfair Which means that first come first serve or you know if you're waiting on it It's basically it's Everyone tries to grab the lock whoever happens to be first to get that gets the lock There's no queue you try again if you miss it you try again miss it you try again go into a loop Constantly trying so say if you have a large CPU system and you have a bunch of readers and they all read it A writer comes along tries to grab a lock, but it can't hold the lock while a reader has it Well, if one reader let's go of it Well, actually if one reader grabs a lock you go to grab the writer goes to grab the lock Can't because there is a reader on it another reader comes along it grabs a lock the first reader Let's go but because the first second reader came in the writer still can't have it You just basically what's called a live lock of the writer lock the writer lock writer will just sit there trying to go And it goes on forever very undeterministic I mean it could actually halt the system per se Then there's a fair system fair lock where you have a bunch of writer reader locks and What happens is when a writer comes in any new reader comes in they'll block they'll have to they can't grab So they block behind the writer now the problem with that is a lot of people are actually a lot of people I wrote code with the reader a locks into me go like oh, it's just a reader lock It never blocks so they'll put it before and after another lock Well, you just created the a bba deadlock associated with that because now a reader lock once a reader lock and block It's now untrue lock and you can actually have it where you do a reader lock Or whatever you have you grab your lock do a reader lock and block where someone else has the reader lock goes to grab The block that you have it blocks the writer never continues deadlock of the system No hurts great for power I just wanted that I throw this up just because if you really if your CPU goes into a real deep sleep when you Enable or no hurts and then you expect quick reaction times when we've had people complain Well, I got this huge latency Well, I looked at the eye like you have like no hurts enabled and you got real deep sleep going on It takes like two milliseconds to come out of the deep sleep sometime You know some systems are gonna take a long time to come out of your system Like I said real-time OS is only as good as the hardware if your hardware can't react to something because it's going into deep sleep You know really the best real-time Idol is idle equals pole. I mean you just have to up your AC a bit Real-time user space. Don't use priority 99 people love like say oh, I'm gonna throw a 99. Don't do that There's lots of you know 98 other priorities to use use them You know, you know, I've seen things systems is crash because of that. Don't implement your own spin locks in user space You know if you have a real-time task that depends off try not to use don't have it depend on like I owe that might take a Long time like be smart about it you can you know use memory messaging basically and M-lock all or M-lock whatever get your system. So you don't have to worry about page faulting in and Question actually before questions. I just want to see something. Oh SMI you can't see this. Let me get up it Darn it. Ah What's the control plays plus it doesn't want to give me a there it goes I was running at the time. I just killed it But if you can see I know probably can't see this but I was running cyclic tests During this whole presentation had one this is a laptop on my laptop. I'm running a real-time kernel So and I ran everything I ran my trebuchet program everything had a 231 Micro say max microsecond latency I guarantee you there's SMIs going on in this laptop almost every laptop has SMI's it usually handles thermal If you go in and turn off your SMIs you're most likely fry your laptop So don't do that But that was just the idea this I'm running the real-time kernel on this laptop So any questions?