 So my job today is that they put me after lunch, so I have to wake you up. And I'm doing it with a process-oriented presentation. Challenge, I love a challenge. Okay, so, nice we'll start it. I got a lot to talk about. It's funny, I started off with 15 slides and I think I have like 53 now, so. So everyone here is for the stable real time. Not stable or real time, but the stable real time, or real time stable, whatever. So for the upstream stable releases that you're all hopefully familiar with, everyone who's here not familiar with the stable upstream stable releases, okay? Yeah, yeah, yeah, BS. So everyone knows about we have the stable releases. Just overview of exactly some of the process that's going to, so after like three, 14 is released, you'll have a three, sorry, 14 is released. You'll have a 413 that gets stable releases for 13.1, two, blah, blah, blah, blah. When 415, RC1 is kind of released, the previous release, 413 is no longer supported. And 414 is now the stable releases, so really it's just back one that we do, the stable releases. This is the way we started. This is the way we liked it. But some people didn't think that was good enough. So we now have what was called the long-term stable releases. And every so often, you know, I think Greg has switched it to, was it once a year? Now he's going to make one release, a real time stable release or something. He's, before he just randomly picked one, hoping that people wouldn't just bombard that one release as, we got to get these features in, we got to get these features in, whatever. But now he's saying, okay, that's, it's been better lately. He's kind of like controlled people. So there's now, right now, as of today, well, actually, yeah, I see the 316 shouldn't be up there. Is it? No, yeah, it's still there. So 32, 316, 414, 449, 414. They're all projected end-of-lifes are the, those dates I have up there. So now the real-time stable releases. Currently we have what we call a development branch, which is right now on 414. Sebastian, not going to try to pronounce his last name right now, it's just, he works with Thomas Kleischer and he maintains the 414 development cycle. So all new code or anything that we need to fix or find problems with happens on the 414 release. And right now, it's been, when 414 came out, we kind of like started on that and been working on that for a couple of years. And like, as usually could take maybe one, two or three releases before we switch to the next real-time kernel release to be the development branch. Because we want to make sure that we stabilize all the real-time development or make sure that this kernel works for real-time. And sometimes it takes a long time, it takes two or three releases to make sure that happens where it's good enough to pass off to the stable team. And that's why 416 will come out, it's going to be the new real-time. So 414 is pretty much stable at this moment. But until 416 comes out, Sebastian will still maintain it. Once 416 comes out, he's going to stop work on 414 and say, Steve, take it. And I will pull it into the stable tree at whatever he did is what I will start working with and work on it. So I will be working on the 414 once that happens. So again, this is about the stable release process, not about the development process. As I mentioned, there's 414, that's how his last name is, I can say I can't pronounce it. So right now we have the stable releases are 4ix9ine, maintained by Julia, 4ix4ine, maintained by Daniel, wherever he is. 4ix1ine is Julia as well. And 318 is, Thompson-Ussi is going to be maintaining that, more as a learning process because if you're on the 318 kernel, anyone here using a 318 kernel at all? How about 318-RT kernel? You're using 318-RT, we need to talk. Because really I was told it's done, like we're not really helping out. So hopefully it still works. We would really highly recommend going beyond that, but yeah. I still also do 3-2, but I don't do any back port patches of anything that Sebastian comes up with because it's so far away that it really is almost meaningless to bring it all the way back to 3-2. So when 3-2 comes out with a new stable release, I'll check every so often and say, oh, 3-2 has updated the stable kernel, I'll forward port it or pull in the patches and all that. So keeping up with the mainline stable releases, so what do we do? So each of the R-Team stable releases, if you notice maps to one of the long-term stable releases. We don't usually care about the short terms, although the development process may work on the short terms and then go on and then switch up to the next one. But the stable releases, for the real-time stable releases, we don't care about the interim kernels. We only care about the long-term stable releases and that's what we work on. And we usually say that we will maintain that real-time kernel for as long as it's maintained upstream. So 318 keeps going forward and forward. It's like the one that it never wants to end. But we said we're done with it. It's just too much time consuming. But Tom, now you have a customer. So technically we should really sync up with the stable releases twice a month because Greco Hartman is a monster machine. He seems to pop out stable releases like a rabbit. Pops out baby rabbits. But sometimes we're also working, we're busy. We don't always keep up with the stable RTs. So we try to be good and back port all that. So how do we do this when we have a new one? When we say we're going forward to, like say the stable releases have a lot of releases that are out there and we're like, we got to catch up. We have to pull in all the fixes that the stables have. So the real-time kernels not susceptible to all the vulnerabilities that the stable releases have already fixed. So we say, okay, it's time to, now the way I did it was I would increment each one. So if I was like four, five, six releases behind, I would merge in each one individually and label and increment the RT count each one, one, one. When I do back ports from Sebastian's tree, which is basically he'll find things like, oh crap, this breaks real-time and it affects all the stable, it affects a stable tree. As long as it affects one stable tree, we have to pull back those fixes as well. So every time we update with a stable, we try to increment it. So what I do is I'll do a get merge of four, nine, 51, update the local version RT. Everyone know about the local version RT or local version files? If you want to give a little extra tag, when you do like the U name of your U name dash R to see what kernel version you're using, you'll see like dash RT one. So that comes from a local version file. So if you have a local version file in your kernel, this is for anyone that you want to add an extension to make it special. So when you boot it, you can actually see it. This is a different kernel. Create a file called local version, well one word, and put something in it and that will be appended to the kernel version number. So if you didn't know about that, now you know. So we use something called local version dash RT because it was kind of nice because these local version files that when the make process pulls us in, it just searches for anything that starts with the letter's local version. So we can put local version dash RT and have an update it. So I would edit it, change it from RT 40 to 41, commit dash A, make a tag on it, then go to the next release and so on and so forth. So when you look at that, if I do a get tag, I do grep495, I see all my releases and you'll see it's 41, 42, 43, 44. And I have it actually, this is I was way behind and caught up by hitting every single one of these. But it stopped happening. And so I looked at this and went, wait a minute, when I was writing these slides, I'm like, this doesn't work. Like this, something happened and I first blamed the wrong person and then we had a really long discussion with the right person. So what happened? Because when I first started doing this, I kind of did it this way, but then I was told, no, it's better to label every single one of them. But Julia wasn't. Hi, Julia. She didn't know the rationale for why I was doing it or that. And actually, when she asked me about that, I couldn't even remember what the rationale was to tell her why we did it for each one. Why not just merge it in and set it? So I'm thinking, well, maybe we don't need to do this. So yesterday we had a real time meeting, we had a discussion and actually, there's actually a rationale to do it, but there's a compromise. We don't have to do it for every single release. So when you look at this, if you do an increment with the RT, like I said, it's not always, this could happen. So if I go back, if you notice that it was 76, the next one was 4984. So to do this, you have to merge in, you start off with, get check out 49RT and it's currently at the 4976RT1, you do get merged of the 84 and you get these conflicts. Now you gotta go and look at the conflicts and work on, fix them, make sure they're right. Now to go back, and I looked at this, those are three conflicts that happened at three different stable releases. But right now we only see it at one. We only see it at the one with the RT base. So if something went wrong with that and we have to investigate it more, it's a little bit more difficult to figure out the conflicts when you're not at the place that introduced the conflict. So what we're planning on doing from now on, we've agreed that this makes sense, is instead we could jump, we could skip, but if anything has a conflict, we need to tag it at that release. So we tag it once, resolve one conflict, go to the next one, resolve the next conflict, go to the next one, resolve the next conflict. So it makes it a little bit easier if something went wrong because when we're resolving conflicts, a lot of times it's just us doing it. So it's only Julia doing it, it's only myself doing it, only Tom doing it, Daniel. So if we screw up, we may not know, it may work, we go around our tests and everything works, but then someone else goes out to the field and uses it and they say, wait, this thing broke and we got down to this conflict or it was this merge request. We wanna really know exactly where it happened. So we're planning on tagging it where the conflicts happen. So if you see random spots, that's why. A conflict happened here, we tagged it, we fixed it. So now it's easier to find. One of the discussions came up was should we add a tag just before the conflict too? So that way you can see if it works, that works. That's kind of up in the air, we may or may not. So that's not a requirement, we may do it, we may not. So going back to it, so what we plan on doing now is at each stable release, if there's a conflict, we're gonna tag it or at least we'll do that in the future. So we do a get merge for 978, solve the conflicts, get commit, increment the local version file, go to the next one, get solve the thing. When you catch up to all this, because right now when we even do resolve conflicts, not only about how you do it, when you resolve conflicts on one, I basically, okay, resolved, I'll just compile the kernel, boot it, run a few simple tests. Basically does it boot up, does it run, does it can handle some priority inheritance work? Yes it does, okay, good enough, go to the next one. And I don't spend too much time on it because once I get to that final one where I catch up to mainline stable, that's why I'm gonna do all my tests and my tests could take several hours and I don't wanna waste the time each time because this takes time and we're not doing this full time. So this is kind of off, we have other jobs to do so this is hopefully extra. So once you catch up to the final release that you're now equal to what's stable out there, although usually when I'm done with my test, Greg Rohartman put out another stable and I usually curse him for that. So we catch up, we run our tests, it passes, great. You know all our test passes is good, we're solid so we need to make a release. Now we also have to make a tar ball because actually here's probably a good question to ask the audience. Who here used the preempt RT patch, RT? Okay, who does it from get only? Who uses the tar balls or the patch cues? Okay, good, so it is, okay so we'll continue this. Thank you. So when I first started doing this, become creating the stable RT patches, all I would do is just merge the stuff and I used get cherry to create the patch quill and got this little patch cue and I kept pushing, I made my patch cue by using merge the stuff, use get cherry to give me all the changes that's different and then make the cue and blah, blah, blah. Well people are telling me that these patches did not apply, it didn't work. So I had this patch that I had to apply that's from Sebastian and for those that don't have good eyesight like myself, here's a little zoom in. So if you look at it, it did something simple, it just removed a function and I don't know why exactly, oh yeah, I had some sort of issue, boom, blah, blah, blah. So he found out that this wasn't needed, we could just delete it for the real time patch. I'm not going to go into the details of why this. So now you go to the thing is you go to the next release and one of the stable releases modified that function. All it did was change the work cue to make it a little bit different. And next thing you know, you go to look at what was changed and I just realized by looking at this, you know, Sebastian's the one that did this because this was before I went to real time or went to the stable release. This is actually Sebastian's work. I'm wondering if it will still crash with this update. So maybe this isn't even needed. That's a good thing we should probably check out. Take a note, see this is real time debugging while giving a presentation because I just thought about that now. Anyway, you notice the conflict right now with this. The fact that one changes, so if I try to apply the patch, I'm going to get this error, you know, hunk fail, blah, blah, blah, blah. That's because it doesn't match. So you can't just do a cherry pick of all the patches that are different. You need to make a cue that's based off of the latest stable release. So once we got up to four nine, like say four nine, 11 on this case, if you look into the stable get tree, you'll notice that there's a stable tree but you also see a dash rebase tree. The stable tree numbers, where you see RT1, RT2, whatever, they never are rebased. So if you pull from it and develop on it, you don't need to worry about it breaking your code when you do the next pull. They never rebase. We update them and that's it. So we make a rebase branch because as the name suggests is, we rebase it every time. So the rebase branch will always rebase and we tag it with a rebase. The difference between the two branches, if you do, if you notice get diff there, that actually if you do a get diff between the non rebase and the rebase branch, they're identical. There's nothing different of the code base. But how you got there are completely different. One is incremental, we applied a bunch of real-time patches, did a bunch of merges of stable, added some more real-time patches, then more merges and so on and so forth. The rebase branch starts from the stable release, so 4911 here, and it will actually have all the patches built on top of it. So when I make a quote queue or whatever, I use the rebase branch to pull, I just say give me all the changes from here to here because I know they all will apply nicely for when you guys download the code. So a little RT overview. Now RT is coming to mainline. It will be there shortly. Everyone says, yeah, yeah, yeah. By latest 2020, hopefully by 2019, possibly by the end of this year. Depends on several factors that's beyond the scope of this talk. But this is very important because people are gonna have to start understanding how RT sorta works or a little bit of the concepts because when you are working on the kernel and you don't care about real time, if real time's in mainline, that means that if you break real time, you gotta change your code to fix it. Yes, config time. You either enable it or you disable it. But it doesn't matter, you can have it disabled. And if you are developing a Linux kernel, you could break real time without ever turning it on and we're gonna hopefully have bots and stuff like that complain to you as soon as you push it to get trees or whatever, we're gonna start monitoring people and policing you folks. So we're gonna give you some ideas about things. One of the few things that real time does is it turns spin locks into mutexes. That makes the things much more preemptible. So a spin lock, don't expect it to be preemptible or turn preemption off. So if you need to basically make sure that this code here is not preempted, a spin lock is not gonna guarantee that that will happen once you turn into the real time world. It will guarantee that you stay on that CPU. So it disables migration. So in the real time place, if you do a spin lock, your migration will be disabled. So you can guarantee that you'll still stay on that CPU but it doesn't mean that something else can't come in and do something there. So you have to worry about that. Spin lock IRQ save will not disable interrupts but in the real time world interrupts are threads. So a thread interrupt could still come in there so it still needs that spin lock. So a spin lock IRQ save, if you do a spin lock IRQ save and access per CPU variables that the interrupt shouldn't handle and you say, well, I disabled interrupts so this guy shouldn't care about it. Well, it can still get to it unless there's a spin lock protecting that section in the interrupt handler. So a spin lock IRQ save does not mean that it's going to prevent a interrupt from actually happening in that case in the real time world. So you have to think about this. And by the way, thread interrupts are in mainline kernel. So that's actually kind of true regardless, although, yeah, spin lock IRQ save does it disables branch in mainline so it still protects it. Priority inheritance is there. So sleeping locks, the sleeping locks will get the same priority inheritance that few Texas have today. So here's my, I love, I always have to throw this slide in. I think I threw this slide in back in 2007 when I gave my talk at the Ottawa Linux symposium about understanding the real time patch and I just love this slide. So I have to reuse it in all my talks about real time. Just so those who here is unconfirmed about what priority inversion is. Ah, okay, you all. So can I skip it? So normal case, you have priority inversion, I'll be quick then, you know, priority inheritance will inherit the task. So task C gets the priority inheritance and then the guy has. So going back to this one, the old, I know I'm going to explain this, you guys already know this. A's highest priority, B's second, C's third. If C runs, gets preempted, A runs, grabs a lock that C owns. So C runs again, it gets preempted by B, goes on forever, and A never runs. So B not only blocks C, but also prevents A from running. So priority inheritance means that, once you give the lock up, or sorry, when A blocks on C, C will inherit the priority of A, so A runs. So when B wakes up, it can't interrupt or preempt C. So then it could finish, and once it releases a lock, it loses its priority, so A gets to go, runs, finishes, sleeps, and B continues. Priority inheritance implementation is hard. Okay, it can get really complex. It's been through several iterations. It looks nothing like it was when we first introduced it way back in 2004, or three maybe, perhaps. And one of the things that you have to realize in the real-time kernel, that what I do, is read writer locks, where you have multiple readers and a single writer. To do it, to keep the same paradigm as the Linux kernel, and you wanna keep priority inheritance, you have to have multiple priority inheritance, which means this, if you have a bunch of readers running at low priority, and a high priority process is running and grabs a writer lock and preempts, to get priority version from happening, because now it's blocked on every single reader until all those readers release their lock, the writer can't go. So multiple priority inheritance means that you have to update the priority of every single thread that that writer is waiting on. That gets complex. It gets actually exponentially more complex than the original real-time priority inheritance code. So there's three things that we could do. Well, Thomas will say two things that we could do, I guess Thomas hates the third one. And I implemented it once, but he always says I'm just crazy. First thing is serialize all readers. So we turned, in fact, actually that's what the real-time, I think, I don't know if we switched it back yet, for a longest time, we're discussing switching it to two. But the first thing we do is we serialize all readers. So if you have a reader write a lock, they become, it comes in mutics. So every reader will now serialize on every other reader. So if you have a reader write a lock across 60 CPUs and you have 60 readers go, they just went one, two, three and they all have to wait for the one that has the reader. That could be a long latency there, that causes slowdown, it causes performance problems, it causes a lot of bad things to happen. We would just love to get rid of reader write a locks, use RCU or find some other way of doing it. Because reader write a locks are horrible on the cache line for mainline kernels. So just because like, oh, it's a bad for real-time, no, it's also bad for mainline because of the cache bouncing around. You could actually kill performance with reader write a locks as well. So we like saying, get rid of it, but if you have it, the second thing we're thinking of doing is saying screw it, reader write a locks are not going to be converted to priority inheritance. If you use reader write a locks, good luck. You could have priority inversion and so be it. Third thing is, is they screw it, do it the complex way, my way. I've done this twice. I've implemented multiple reader priority inheritance, and it actually works rather well, but Thomas hates it because of the added complexity is a higher level than the Linux kernel should actually have, something that you need something stable on. It's almost impossible to prove that it's correct. But it's there, it's fun. Now like I said, preemption usually gets disabled by spin locks and preempt RT, it doesn't. So if you have per CPU critical sections, you have to be careful that you use the spin lock IRQ save. So some places we have, you don't use spin locks, we have a lot of places that you just do preempt disable and I've seen places where they do preempt disable and then they grab a spin lock or something, which will break RT. Because RT spin locks are mutexes and in the main Linux kernel, you can't do preempt disable and call a mutex. That breaks as well. So since spin locks are mutexes in real time, you can't do preempt disable and then call a spin lock. What we have instead is what we call a local lock. Because usually when you do preempt disable, you're doing it mainly to serialize some critical section, you have some data that's usually per CPU. And you know that the only way to protect it instead of worry about cache lines because reader writer locks are horrible, so what you do is you, one thing you do instead is you create a per CPU data and this per CPU data is just protected by a preempt disable. Because once you preempt disable and you don't have to worry about save interrupts, don't touch it, no other thread can touch that data because it's per CPU. So that's a normal common way of doing it, getting rid of reader writer locks. Like I said, reader writer locks are bad, preempt disable is good, well not for real time. So sometimes you get these huge sections of preempt disable which means that when you have a huge section of preempt disable, that means the latency of waking up a task that needs to run right away has to wait for that preempt disable to finish so that adds latency so we don't want that. What we have instead is a new feature that's going to come next at the mainline kernel, so coming this year, probably in a month or two, is local locks. Local lock is a way of replacing both preempt disable and even IRQ disable. You say local lock, you give it a name and in the real time or in the mainline kernel, that local lock is nothing more than preempt disable. Your code actually does not change or if you compile it, it's identical to what you have today. So come see, use local lock. Now why is this good for you? You don't care about preempt RT, you don't care about real time, you only care about your code, your mainline code. It annotates what you're protecting. This way you could say, hey, a lot of times, I don't know how many people have done this, I've done this a few times, looking at kernel, you see a preempt disable local IRQ disable and you say, why? How many people have done that? A few, okay. There's a lot of times you go through and you'll see preempt disable, you're like, why is this here? And there's all this code that goes on and it's because something here and here are related, nothing tells you why, so you have to go through Git logs, history, talk to the author who usually says to you, I forgot. So local lock is actually a way to put a name there and then you could actually say, this is protected by the local lock, this. So now you actually know when you see local lock with a name on it, hey, this actually has a rationale, I know I could figure this in the future, even when I forgot why I wrote this code, I know why this preemption was disabled but it's really a local lock that says, okay, this prevents other things, yes. It's a macro that will turn into a preempt disable, actually it could be, yeah, I think it's a macro or it could also be a inline, what's called static inline function but it's going to be like a macro and depending on the config option, it'll change. And on RT what it will turn into it'll be turning into actually a mutex with migration disabled. Yes, it's basically you can annotate what you're protecting. So that's the rationale of giving it to mainline even though it helps RT greatly, it's going to help mainline as well. So the local locks are a way of annotating blocks where that annotation, that name will be, yes. Yes, stay for the next presentation, Julia has some, yes. So stay, she's going to tell you a lot of the stuff that I'm talking about right now but she'll get a lot more detailed, yes. So, I'll just follow you after this talk. Anyway, so the local locks are good for that and it becomes a mutex or whatever and it's also a per CPU mutex so you don't have to worry about the cache line bouncing. So when you grab a local lock it's only grabbing a lock for that CPU. So if you schedule out, migration's disabled, you're always going to stay on that CPU but if something schedules in and goes to that same critical path they're going to block on that local lock until the other guy comes back and it has priority inheritance and all the wonderful stuff to get everything moving flowing just like a normal spin lock. But you don't have to worry about that. That's we take care of all the details just protect your sections with these local locks. Interups are the same way. Now interups are a little bit more. We have, by the time you'll have per CPU data that can be accessed by interrupts. Now in mainline, local IRQ save is good enough to do this because when the interrupt goes off you're going to have the same thing. You have the interrupt that goes into the that may access this per CPU data and you have to stop the interrupt from happening. So preemption disable is not enough because preemption won't keep other tasks from coming in it doesn't stop interrupts. So local IRQ save, so what we got now is instead of using local IRQ save you use local lock IRQ save and you annotate it. And then the interrupt itself will have to say it will only do local lock IRQ. Remember that's only, it'll make preemption disable in mainline but it doesn't matter preemption disable and interrupt is kind of a no-op. So this you still mark it and then you could say what it is. Another thing that we have too is we're going, I don't know if we're going to try to get rid of this. I don't know if we're going to keep it but if you're ever doing anything stable RTs you gotta look at this. We have something called local IRQ disable underscore no RT which basically says if you don't have RT configured it's local IRQ disable. If you do have our RT configured it's a no-op. But those are ugly. I think we're trying to get, we're afraid of adding those. They're just kind of ugly. So eventually you go and you do a merge and you do a get merge for your stable tree and you get a conflict. Oh crap, I actually have to think. Cause a lot of times stable releases if you do a get merge and everything goes fine there's no thinking. It's really just monkey work. You just okay get merge, just boom, boom, mostly scripts just get there. Yeah, yeah, do this. You just read Facebook while your automated processes going on and when it's done like okay you commit and do everything. But then if you do a get merge you say oh crap I actually have to think now. Conflicts are actually good because there's a lot of times, not a lot of times there's a few times that a merge happened where it didn't conflict and that actually, but it had a bug in there. I'll talk about that later. So let's say you're dealing with conflicts. I do a get merge of four, nine, 35. It gives me boom, boom, boom, boom and says oh crap I do a get diff and I see all this. For those with bad eyes, a little zoom in. And you'll see that oh, there's this head which is remember the head is the RT branch. So head I have this SIGQ free current first. Below that is reschedged fresh blah, blah, blah and this is kind of a trivial one. Like I said this is an easy one. I can actually tell right from here how easy it is because I can see that I changed it looks like I probably changed the SIGQ free from into the main line. And below down below you'll see this disappearing worn on once tasks come think here. So the way I handle this I'm not sure if Julia does this but the first thing I do is I look I do okay you get diff four, nine, 34 the vanilla kernel with my kernel that I was started at before I got the conflict. I wanna see what did I change? What was exactly the change that the real-time kernel performed? And sure enough looking at this, yes it renamed the SIGQ free to the SIGQ free current first. So there's a reason we did that, okay simple. Down below there was a worn on once and okay it was we added one. So then I go and say okay let's see what happened with three four, nine, 35. So I diff the exact same source so I go four, nine, 34 to see what four, nine, 35 did and let's look at what it did. Okay yeah and then some stuff in the top and then okay I added this little resched timer just above this no big deal not too hard. Down below it added a little red sked timer equals false. Okay so there's a little conflict there it's pretty easy. So you know finally this is the final result. Yeah I do the rename add this do some changes. Easy conflict resolution. A little more non-trivial and this actually goes back to the bug I think happened before. So I did this you know you go up to four, nine, 61 get diff I see wake up all locked wake up all okay it moved. So I do the same thing diff the RT code against the stable release that it was based against okay I renamed wake up all locked or wake up all to wake up all locked good. Then I go and look at what the upstream kernel did compared to the same baseline and it moved the wake up all outside the raw clock. So obvious answer right well you know it's the obvious answer so you know it's wrong why come on it's a pretty easy one get out exactly. Two German friends left no the yes it's no longer locked wake up all locked is in main it's also in mainline but it's also nice in RT and because it's okay it's a wake up queue the wake up queue has its own lock it's a sleeping lock mainline uses this for optimization because if you could say hey I have a lock that protects a work queue why do I go and use the work queue lock as well. Well here's the thing if you go back and look at what it did you notice that there's a raw spin lock there. Now a raw spin lock does not get converted into a mutex it stays a spin lock just like it is in mainline it disables preemption it's a spinner it doesn't sleep. Well wake up all has a queue or wake up has a wake up all has a queue lock that's a spin lock but what we know about spin locks they turn into mutexes grab your raw lock and then a mutex and then it really say a raw lock is bad bad in mainline bad in RT. So we had to convert it to a wake up all lock because what wake up all locked is telling the colonel hey I had this work I had this queue wake queue that I have my own protection don't waste the energy of grabbing another lock to do it. So we prevent the lock but you're right. So the correct, well yeah here's where I talked about the spin locks. So all I did was just move it out it would be wrong because the lock is now not protected at all so when I just move it I actually keep the exact same change that the upstream did just move it out. So this actually this actually removes one of the patches from the real time colonel by adding this in the real time colonel doesn't need to modify it. So dealing with no conflicts like I said sometimes things are very subtle and this is actually a little bit of a misnomer because my slide shows a conflict and thank God there was a conflict on this idea because if there wasn't a conflict if the original code went in first before another code went in we would never have solved the breakage and it would have just merged completely nicely and we would never know an issue happened and luckily the code that got brought back had a pre code brought back that the real time patch. The reason why I bring this one up is on 4.9 this was fine. On 4.18 we back ported this patch and they didn't do the update because there was no conflict. The conflict here is this. So if I go through and did a get diff 4.965 to 66 I see what it did. It moved this little block of have RT push IPI and I wrote the code for this so even on 3.18 I knew the problem was going to be there. So it moved it down here. If I look at what did the real time kernel do it added that one line. That one line is very important because in the real time kernel the IRQ work doesn't actually happen in IRQ. Well it does because IRQ handlers are now threaded. So IRQ works are now threaded. Well the scheduler has its own IRQ work that would be kind of destroying the point of using IRQ work if it required scheduling. So this is a way that does the real time task push and pull logic uses IRQ work in the kernel. So if the scheduler calls IRQ work that gets scheduled out the real time tasks aren't going to emerge in until that IRQ work which is not a real time task gets scheduled in to say oh this real time task can be moved over here. Kind of kills everything. But no one would know that unless you actually knew the code. So sometimes you gotta actually look at some of the code or look at main line upstream say okay what did they do make sure it works in the back point. So this like I said this is one of the hard things that I have to be very careful about. And this is where I kind of explain what it is. So next is back porting RT patches. Stable is the easy part. Like I said that's a lot especially if there's no conflicts it really is mundane. You just go through run through a bunch of tests you get a lot of Facebook reading in at the time. Or Google plus for some reason. So one of new RT patch or document or when new development comes out the we look for stable RT tags. If a stable RT tag is in there Sebastian will say when Sebastian's writing code and he doesn't commit there's certain things that he says okay this is broken and past kernel releases. He'll put just like in mainline when you see see stable at vigor.curl.org the real time code has the same idea of doing stable RT at vigor.curl.org and we will scan for these and any of these we will pull in and that's very that's useful. But I also found out that he there's a lot of patches that should go be back ported that you know Sebastian doesn't really think is important but when I look at it and I know how my use cases I'm like no I think that is important. So a lot of times a lot of changes in upstream in the back port patches we back port almost a lot a lot of patches and some things we just don't back port. Matters how much of a change of it is a big impact. We may not back port it sometimes we add new features to the real time patch not so much anymore because we're getting ready to push it to mainline but a lot of times we add new features to the real time patch those never get back ported. So how do we synchronize among all our ports? So we right now like I said we have four nine four four one and three 18 that are right now dealing with back ports. So we're working on using Google Docs so we write down all the anyone of us of the stable maintainers can add to this saying hey here's the commit name here's the upstream commit ID that's in Sebastian's tree that has okay and then I'm gonna say I applied it and I'm going to and then other people we can all go and look at this and say hey so and so pull this patch in maybe I should look at it too and then if it's not applicable because sometimes it only affects like some like the one I said you see the NAs there that was one of the patches was only applicable for four four didn't affect for one or beyond so the other two it doesn't matter. So this is a way we synchronize among ourselves to use this. Still have updated four one. What's wrong with it? It doesn't seem to work and there's not many it's not really because like this shows you that there's really not much more work going on with the upstream kernel we're right now working about getting it ready to go into mainline. So right now the focus is that so we're actually having less and less issues with mainline kernel. So it's getting really close. So what do we do to backport it? Usually what I do is I always save a branch from what was the last time I backported. So say for 14 or 15 or 13 was the last time I did a backport from Sebastian's branch. Now he's up to 24 RT 19 and I'm like okay let me go see what he's changed and see if I want to backport anything. So what I do is I first thing I do is I kind of left a step out here is I go to my tree my stable tree say it's 49 RT I will branch I will make a copy of it like attempt branch copy of it then I merge oh wait sorry let me go back. I don't start with my tree I'm working on 49. I actually take the tree that I last backported the 4, 14, 15 RT 13. I pull that this is the last time I actually backported patches from Sebastian. I make that into a temp tree. Then I merge 4, 14, 24 into that tree. So cause that matches the stable release that he based his stuff on. Cause I don't care about anything that he pulled from stable. I only care about what he changed. So to have a proper comparison I need to start make my tree equivalent to his stable to see his differences. So what I do is I go and I start with my 4, 14, RT or 4, 14, 15, RT 13 make it to pull into a branch pull in 4, 14, 24. It has a bunch of conflicts. You know what I don't care. I save them all as is or I go in and delete them. I don't care about the result. I only care about commits. As long as it goes up there I just want to know what's the difference that he has. So I don't have to actually even spend any brain power fixing the commits. Those, they're irrelevant. And then how many people here know about Git Cherry? Okay, how many people, the rest of the people don't know about Git Cherry. Okay, so what Git Cherry Daniel now knows which he's like darn it. I wish I knew about that a long time ago because he actually almost rewrote Git Cherry. Git Cherry I've been using for a long time. It's an awesome utility. So here, Git Cherry will take the branch. So right now remember I said, I'm at, technically I'm at, let me go, oops, ah, geez. So I'm at 4, 14, well actually I'm at 4, 14, 24, RT13, technically because I merged 24 there. So I'm at 4, 14, RT, 24, RT13, and I do a Git Cherry of head. This shows me all the commits that are in the RT19 that are not in RT13. It shows me, boom. You'll see there's 18, 19, 20, boom, boom, or whatever I was at. So our 14, 15, see you'll see all the commits it's added. I only care about, I don't care about his version commits. I only care about what he added. So I take these out and then this is what I use to build a quilt queue and I will now look at each individual patch and examine it myself. So this is more of a manual work of saying if it has a stable RT on it, I pull it. It's going to be part of the things I pull in. If it doesn't, I look at it and say, should I pull it? Sometimes yes, sometimes no, sometimes I try and that's it. Thank you. Questions? So wait, so wait, you're saying that if we have a conflict when we pull in, yeah, we tag it. Well we should, well I always did, Julia will from now on, so yes. Well like I said, no, I do, I mentioned it and you might have missed it. I said that when I have a conflict, I usually pull, pull, pull, pull and I tag each one but now I really only have to tag the one that has a conflict on it. So I pull one in, have a conflict on it, I fix the conflict, I'll build and boot the kernel, run a few tests just, I basically spend maybe, after the things built maybe 15 minutes on it, it's smoke tested. And then I go on until I hit, I catch up the mainline, the stable and that's why I run my full stream. If I screwed up, usually a lot of times if I screwed up that conflict one, the full run will actually find a bug. And then I go back to, yeah, if I hit a bug, I say, wait a minute, I fix it back and now I might actually revert all the way back to that first one and say did this and then run the full test suite on that first one. I've only had to do that once. Anyone else? Wait, why, okay, so wait, you revert everything, pull everything in, wait, wait, you, you. Yeah, the merge conflicts will still be there. Oh, wait, wait, you get, you get a state, you pull from my stable tree or? Okay, so wait, you're trying to, why not revert, you can't do reverts or, I mean, you can't do rebases on your tree or something? Okay, so basically, okay, so let me ask you this. Okay, so this will be recorded, no one knows what you're saying. So it's just like me talking on the phone. So anyway, basically I think what you're saying is you have, you use the stable tree, you're pushing off to other people, maybe adding your own stuff or something to it, and then you're, but other people rely on your work, so ideally what you do is you're reverting everything out and you're coming back and pulling, it's once you have the conflict to get the mainline stable. What happens, have you tried instead of reverting, trying to do a pull from the mainline, the real time stable right there, because it might actually fix everything for you. So instead of pulling back to the stable, just pull from the next RT or stable and it will actually have the updates that fixes the conflicts for you. Well, okay, if it works for you. No, I mean, like I said, it's right now I can say if it works for you. Do it. Yeah, yeah, yep. Anything else? Okay, thanks again, bye.