 As you probably know, a lot of people know I'm Steve Rostin. I maintain the real-time trees that, you know, when Sebastian and Thomas are done with one and they work on to the next one, I go and I take over the, what's this thing on, nope. So right now I'm currently maintaining 3-2, RTE, that goes on to May 2018, Ben Hutchings, who is the maintainer of the Debian kernel, asked me if he maintained a 3-2 kernel, would I maintain it as well, and I said, sure, as long as you do it, I'll do it. I didn't expect him to do it for years, but I'm keeping up with my promise. It's actually one of the easiest ones to maintain because it has a few things that actually backport to it. So if something doesn't backport well or if it doesn't apply, I just don't bother with it. But that's actually going on, the last I look is going on until May 2018. 3-4, it looks like, I believe it's done. I backported stuff to it and then I looked at the end of life and realized it was already over. So actually it got updated, even though stable stopped maintaining it, I maintained it, but I think I'm done with 3-4. 3-10 is still going on until, I suppose, October 3-12, 3-14, I believe it's just ended as well. I think I backported as well that, although it says it's done in October 2017. Let's see, if there's no more releases, I'm not going to be backporting it. 3-18 is still going on until January. And then right now we have 4-1 and I believe 4-4 is the long-term, at least long-term kernel release, which is supposedly going into 2018. 4-6, I'm not going to support, as Thomas already said. There's no support for mainline, so I'm not going to bother supporting it. 3-8 or 4-8, if they start working on 4-8 and switch to 4-9, I'm not supporting 4-8 either. So when 4-9 becomes, if it becomes a long-term kernel and they decide to go to 4-11, I'll maintain 4-9 until it's, well, it's done. I have a lot of scripts to do the maintenance. I posted a link right there. I copied everything over. This isn't my main repository. I kind of modified things. The throw where my really nasty scripts are. Basically, when I found out that I was doing something over and over again, I'm like, this could be scripted, so I just wrote a quick dash script and did that. And I'm going to go through real quick what I do for each release. This is for every single thing. So if I do this for 4-4, I do this for 4-1, I do this for 3-18, all of them. First thing I do, I have a cron job every night that will do a get remote update on my repository to pull down Linus's tree and the stable tree. So in the morning or something like that, or when I eventually get around to working on stable and I'll decide to do a get tag, grep, like right now, for example, I'm just saying 4-4. So I do a get tag, grep v4-4. It also has my tags in there. So I see where I'm, OK, oh, they added two more releases because Greg adds one every week. So if I missed two weeks or three weeks, I have to do three or four of them. So first thing I do is I edit the local version RT to update the version of RT file because every time if it's going from like 3-4, like 19 to 3-4-20, even though it might be 20, 21, 22, I will make a dash RT tag for each one, even though I'm not even going to go through the testing. So actually, if I'm behind by 3, I will merge the first one, fix any conflicts. If there's conflicts, then I kind of do a little small, quick run, let's see if it builds and boots. If there's no conflicts, I just basically push it down to, I have this little script that I use. I do a commit release, which will actually do the, actually get check out, commit, and even put the Linux tag in there. So I don't have to worry about typing Linux, the version or something like that. The commit release will actually do that for me. And then I just push this branch because the machine that I work, do this on is not the machine that connects to kernel.org. So I have to push it off to my kernel.org or the machine that I talk to. So I just push this branch, it looks at the branch I'm at, knows where it goes and pushes it. Then when I go to that main machine, I will write tag, I'll pull that in and I'll say tagRT and it looks at the log files and it parses the log files and says, oh, do you want this tag? I'm like, yes, do it. So I don't type anything other than tagRT. And then I'll do the get merge, the X plus one. Then I edit the version file again for the next one. And now here's where I start whites. Once I get to the final, so I'm equal to what stable has, I'll run a K test, who here's not familiar with K test? Few people, K test actually lives in the kernel code. It's in the kernel proper, it's under tools testing K test. In fact, actually K test is what created the tools testing directory in the first place. It's a script that's allowed, it will do build, boot, and it'll build your kernel, install the kernel, and boot the kernel on your machine, make sure you can even have it run tests for you. It's completely config. There's lots of examples, config files down in there. The config files here are actually on that link that I have. So if you want to look what I do, this is where, like I said, I tell people, I read Facebook more because I'll be working on code, and I'll stop and I'll just say, okay, K test, and then go read Facebook while it does everything. And then I'm like, oh, K test is done. I'll go back and work. Then what I try to do for at least 4.4, the major, the closest ones, I try to make sure it works cross-compile. So I have a K test cross-compile, which we'll actually go through and make sure it will do a make def config plus it will add the config preempt RT full to it, and it will boot all these architectures. If the architecture doesn't support config preempt RT, usually it won't, that config will just be unset by the make, because it'll do a make old config with the settings. And then it'll just make sure that nothing got broken in the meantime. A lot of times this takes a long time to run. This is why I was like, hope to have some extra machine power, because sometimes I skip this step, and every time I skip this step, someone complains, hey, your last table broke power PC. I'm like, damn it. So then I run K test on my test box, which actually compiles and boots to my test box. And what I do there now is that's where I SSH to that test box, and I'll put it on this next one. I'll run cyclic test, and then actually I make a screen file. I run cyclic tests with these options. Then I'll do another screen. I'll do hack bench and another screen. I'll do or just CCJ20, Kineural build, and a loop. And actually that's a script too. I didn't add that one. It actually just says run stress, and it does the hack bench and the DCC, which is equivalent to basically what RTVAL does as well. The only difference is RTVAL just does a kernel build. I actually do a distributed kernel build. So it's actually pounding at the network as well. Then I go back to my other box. And I'll type K test in ctest.com, which will run on another box. It's going to build six different configs, which it does all the options. It does no preempt, what's called lazy volunteer preempt. And it hits six different options, six different configs. I might want to make sure that the kernel still boots and works on each of those configs. And one of them is like even a debug config. So to make sure they work on I386, because a lot of times things will break on I386 in 32-bit or 64-bit. So it runs six configs and boots a kernel with an I386 environment, and I386 environment. And then it does six the exact same type of features for a 64-bit environment for Intel. When all the tests are finally done, oh, I think I even put in there that one of the configs is it does a Make All Mod config. Another time that it takes about 35 minutes to run the Make All Mod config. So I've got two minutes to do a Make All Mod config. Wow, I need the power. And this is a distributed CC stuff. It's a three-boxing. I don't have these super boxes that you guys have. And I don't have SSDs on all my boxes, only some of them. Yeah, but it takes me about 35 minutes to run the Make All Mod config. So I skip that step. And usually when I skip that step, I get complaints that one of my stables broke Make All Mod config. It's like every time I skip a step, it's almost a guaranteed it's my break. So I try to add this. Now, remember, I'm doing this on every single release that I do or try to. So when all the tests are all done, I'll go back and I'll go and I'll kill the compile. I'll kill the hack bench. And I'll go kill cyclic tests and see what the results are. It shows me the results. I expect my machines are slow. So my machine usually are about 40 microsecond latencies. It is about 40, 45 is the max latency. And that's what I expect to always be. If it shoots up, there's been a time where I screwed up merge, and it gave me a few hundred microsecond latencies. And I went, oh wait, this is new. And when I looked in, I said, oh, I screwed this up and fixed it. So I try to make sure it still, everything works. Then on that same box, I'll run my rtmigrate test dash c, which will give a pass fail to make sure it works so the migration still works. I have a mutex test and a mutex hammer that's kind of like the PI stress tests that you guys have. I wrote these before those existed. Do they do different things? Oh, well, the PI mutex test basically makes sure PI works. It goes in and makes it. I forgot exactly how I did it. This is a while ago I wrote it where I made it so it actually does an inverse priority. So if it doesn't work, the thing will never stop. It just you run it and it will just go forever. Yeah, so I want to make sure that's OK. Well, that's what I think that's what my mutex one does. It just basically randomly does it. The PI mutex test is a very short one. It does it in different ways. There's a few examples of ways it different tries it. But it basically uses barriers to make sure it forces the ordering that it will actually hit the rate or the problem if PI is not working. So it's not just randomly trying different orders. It's actually sets up an environment that it will fail if PI is not working properly. The mutex hammer one is kind of like that. It just sets three things. It just hammers things like crazy. So it eventually fails. It'll do it. But that just runs crazy. And then I have my CPU crash script I run, which is stress CPU hop plug. When I first gave that to Karsten, he basically said you should have told me I should have pulled out the fire extinguisher. It took out every one of his boxes. It's basically takes a CPU hop plug. Basically it does a binary shutdown of your CPUs. First it will shut down the CPU zero if it's CPU zero is not available, the CPU one. It'll start with the CPU one. And then it'll do CPU two. Then it does CPU one and two. And then it does CPU one and three. And does it binary all the way up in a fast loop. And by the way, 4.4 basically could crash. I do run this five times. In fact, I had to short it down to five because if I did it longer, if I ran it in the loop longer, pretty much it kills every box still. So CPU hop plug is broken. There's bugs in there because if I stress this too long. But I just want to make sure it works. I almost stopped doing this on 4.4. 4.4 barely can make it through at once. Sometimes it makes it all five. But for some reason, something changed in 4.4 that just made things really not work very well. OK. What, 4.4 to get better? Actually, last time it ran fine. So maybe it has. I haven't had it lock up, but it was recently it locked up. So I mean, I could easily just do a seek five, run it five times, and see if it survives. This is only four CPUs. So it's not even like it's got a lot of CPUs. So it runs through all the combinations real quick. Then when I'm all done and I'm happy with it, I'll update the version thing where I might commit and push the branch again. And then I do the tag RT, which does the special tag for the tag. And then I do a push branch. On my main machine, I have a separate script called push branch. So I actually have to put a, I have another script called this branch, which will tell me which branch I'm on. And it will push my branch up to kernel.org. I have a script called makeRT. What it does, it basically creates the, it makes the prologue that you see, it will make a nice little prologue of what will be pushed out. And make patches is what makes most of the patches from real time from the get trees. And it will put it into a temp directory for me. And then I check out a rebase branch where I do a get rebase of the .41, find out whatever there is, examine, make sure I have the right branch that I'm committing to. And then I kick that off. If there's conflicts, I'll go and fix them. And usually, the only time there's conflicts is if there was merge conflicts in the first place. If there's merge conflicts, I know that the rebase is going to have conflicts too. So I just try to remember what I did to fix the merge conflicts. I'll mix these conflicts. Then after I get down to the bottom, the local version always conflicts because that's the one thing I don't save. So I modify that, commit it. I have another tag, RT rebase, which does the rebase tag. And then I push that one up, force it. I have to do the dash of f because rebase is a rebase. It doesn't, it's not, I have to add a dash f there. And I also add the tags. So now my tags go up there to kernel.org. And I run another script called makeTarBall, which uses the rebase to create the patches, tarBall, for everyone. Then I go up over into my special directory that has everything, that has what's up on kernel.org. I move my patches down to the older. And I move the temp patches, or the patches that were creating the temp, back into my directory there. And then I run a script called moveOldPatches, which goes up to kernel.org, moves all the patches that was in the directory into the older directory. So if you ever go to on kernel.org and look at the stuff, it'll move there. I have a script called signPatches, which will actually use the GPT signatures to sign everything, sign all the tarBalls. Because to push something up to kernel.org, there's a special way of signing things. You have to have in a tarBall, but you can't sign the tarBall. You actually have to decompress the tarBall. So if it's compressed with XZ, you have to uncompress it, sign it, and then compress it again, which I find is a little bit of pain. And then I upload patches, which is the thing that does all the scripts to push it. And then I send out my email to all the kernel lists. That's just a stable release. Now what happens then? I'm going to say, OK, Sebastian's been doing a lot of work now. And I want to make a release candidate. Because I'm now going to start pulling stuff from Sebastian stuff that's up in the main, like 4.6, or 4.9, or 4.8, or whatever you guys are on. And what happens is I have a 4.6 branch that has the last time I did a pull from Sebastian. So I check out attempt directory copying the 4.6, or Sebastian's, or up the mainline versions. Yep? OK. We're going to get to the same discussion about how much they would be. OK, I'm almost done. No, OK, I'll try to be quicker. So OK, I just want to go through this one. So let's see. I'll go click down. Yeah, because I merge in whatever. First thing I do is this is the script that shows you how I figure out your patches. I make a copy of the previous pull. I merge the stable release that you're on. And then I do what's it called a cherry against your patches, which will show me all the patches that are not in mine that are in your tree that are not mine. So that's how I figure out what commits. And then I have a script that actually makes them into a quilt queue. So then I go through and I'll say I do all the quilt push, fix the conflicts. Anytime I fix something, I always make sure I keep the version for that. Do the get quilt import, make a next directory for it, push out the RC release. And I have a make pre that does what's it called the little prologue for you. Now you go to quilt mils, and I do the sign patches, up quilt patches, everything else scan. And then finally, when it's after it passes the RC release, this time, oh, by the way, and then I do all the testing, I do the exact same testing that I do for the stable releases. Mainline, I'll do tag RT, what's it called? Push this branch. So basically, I'll just do the same thing once I got everything else. Scripts, if you know the only difference is make RT, if I put in two versions, it determines because it will say, oh, this is not a stable release. So I actually have to do a compare. So when you see the short log, so it does the short log information, and then to make the patches, it will also do the incremental patches, the jump from one release to the next release. You forgot to mention when you read Facebook. Oh, yeah. On this part, no, only on the K test ones. So I even have a Sherry Pig script here. So all these strings up here, just to give you an idea. And then I have a script. Now, this is something I know people don't like what I do, because I have this script called remove old patches, because the RC release, I don't keep up there. I delete them. Once the RC releases are done, I delete them. I actually don't delete them at home. I actually move them into an older directory. So I have every single RC release that ever existed. So do you think that the RC releases should be, that should stay up there? I don't know if I want to keep the patches or make a git repo for the RC release. Does anyone care about the RC releases? Someone say yes? Oh, there you go. So you want to keep the RC release. Someone told me that there might be something for RC release to, for some spec or for some, you know, to prove history or something. He took a picture of me, so I have to take a picture of him. Someone did up here too, but I don't believe it. Well, yeah, I mean, maybe I'll just keep them and start. I mean, I could load all of them up too, but that would be part of discussion. So here, first off, who here uses the stable releases? OK, yeah, I got a couple of hands. I almost should have said, who here uses RT? People are just here to say, hey, I want to see what this RT thing is about. So I guess there's a few people who use RT. I mean, OK, maybe I should do the, let me go back to the very first slide, because I should probably do the auctioneering. OK, who here uses 4-4? 4-4? Do I hear 4-4? Yes, yes, yes. How about 4-1? 4-1's OK. OK, how about 3-18? Anyone 3-18? No, we're on 3-18. We're on 3-10. You're on my personal machine. No, I don't give a shit about your personal machine. I said who uses it. I should say, who really needs it? Well, 3-18, why do you need 3-18? Or do you install just all of them? So am I just maintaining 3-18 just for you? So anyone on 3-18, besides just for personal? Oh, 3-18. OK, we do have 3-18. 3-14's over. So it's over. Sorry, it's done. 3-12. Anyone 3-12? So I can actually stop supporting 3-12 then? Assuming all your users are in this room, yes. Yes. Well, I'd say 3-10, but that's us. 3-10 is Red Hat. So I definitely have to maintain that one. And 3-4 is over, and 3-2 I know is used by Debian. So OK, I could still 3-18 is supposed to be the end of life in January. So I'll even just keep it up. I just want to give an idea of who still uses this, because I kind of, you know, it's a lot of time. It takes me at least two days to go through all these guys full-time. I mean, about two days of work to get all these kernels done before I go out. And that's why sometimes I'm not always up there, because if I have other work to do, and it takes up two machines of mine to run all this. So I usually hold off until I kick them off at a lot of settings off at night, but still. I think that was it for discussions. Yeah. So is there anything more that should be done stable? Is that basically satisfied people for what I'm doing kind of for stable? I mean, I'm hoping to get some more CPU power from Thomas so I could get those all-mod configs done in two minutes. And also, actually, that'd be great. I'd love the all-mod config and the cross-compiles, those two things I would love to have done. And that would save a lot of time. I think trending would be valuable. Trending. So you currently test for, do I get worse than 40 microseconds? But we don't have any idea, for example, if it was 38, 39. So we are integrating it into our small test form. So you get 10. That's part of the stable too. Yeah, OK. He can basically push his release candidates into our revenue story, and then we do the smoke tests right and we track it on the website. So that'd be good to integrate as part of the release process, is you don't want to see a trend that. Well, I can say. We track him away all the Facebook time. You know, the funny part is, yeah, no, you are. You take away my Facebook time and you take all these quotes. I'll plug in on the test site. You can do Facebook on the site. Yes. But no, actually, believe it or not, there's been more of a trend before it was around 60. It's actually been getting better. That would be good to know also. Yes. No, it is true, because I remember I was actually shocked that I don't break 50 anymore. I mean, I was like, well, my runs, and they run, some of them, because I let these run, especially if I do it overnight. So that's like an eight-hour, eight-nine-hour run. And I full load on the system. And the system's like an old box, OK? So it's probably more deterministic, though, because it's an older box. You'd get a little bit nice. You can't get one of them. Yeah, so release candidates, I only do when I pull from Sebastian. For stable releases, I just do the merge. I do my thing tough, and I'll just pull that out. Right. Yeah. I mean, it just shouldn't be in the package and be done with it. Yeah, because the release candidates are where I could have pulled in the wrong patch, because sometimes Sebastian says, no, no, no, that doesn't go to stable. That's why I do that, just to get the OK. Yes, this should go to stable. But when I'm pulling from already stable, it's already been done. OK. Yes? I guess it's a little odd that the tests are being on on this. Well, we're working on that. Yeah. This was actually kind of last minute pulled. When the stables came out, there was no coordination on what we were doing. Thomas has asked me, could you create stables? I'm like, OK. And I said, what am I going to do to make sure my tables are actually working? So we're trying to be a little more professional now. Yes. Mike, where's the mic? It's still there. You have to turn it off. Bottom? Oh, there's an on switch right there. Yeah, you found it. Yep. Hasan. Can you explain the difference if I just took a stable kernel from kernel.org, and I took then the matching quilt series and played it on there? Do I get something different than what's in the stable tree if you do it's a bug? Because, oh, one thing I forgot to show in my slides here. After this, because the patch quilt comes from the cherry pick, this right here, the make tar ball release. That make tar ball. That's the patch quilt. That's going to be the quilt queue. And before I do this, I do a get diff against the stable against my get repo. So the only thing that actually, there should be no diff. So that's one thing. So I get diff and it comes up, they're equal. If I, the commits are not the same, but the actual end result is exactly the same. At least get tells me it is. So if it is different, that's a bug. So, and you got to let me know when I got to fix it. You don't have any fixes that are related to the changes in the stable. If there was, you know, there's the quilt patch queue it comes out from Sebastian and there it is. And then there are changes made on the stable branches. Do you ever fix things that break there? Do things break on the stable? In RT, do stable fixes introduce questions? Did he find something that's broken? You find things. Oh, that's something that's broken. Yes, I will, in fact, actually there's two patches that have to go into three, two. So if it's something that's only specific for a stable and it doesn't, usually I like to wait if the fixes will go into mainline. Cause I don't like to pull, I don't like to pull in stable fixes that are marked to go into Greg Hartman's or the stable tree, the mainline upstream. I'll wait until they have it up there and I'll pull that in. But there are some times that there's a couple of things that broke that were RT specific within a stable tree. Those, someone sends me a patch, I will pull it in and that's usually, I usually add them to when I pull from Sebastian. I'll add stable specific changes as well. So if I just used the stable release plus Sebastian's patches, I will miss some of these changes. How do you add his, how do you use a stable release plus his patches? Oh, that's, that was by. No, no, no, I'm saying it. No, he's usually, he's billing his stuff and development stuff. But I can download the patch queue for his? For a particular stable release. Yeah, yeah. No, he only works on the development branch. I work on all the other ones. So when he's done, when he moves to another development branch, I inherit what he was working on. Another question, which actually pointed out something, shoot. I thought of something about that when you said about fixes. I can't think of what it was. A little concerned too. Oh, one thing is the patch queue in the queue in my rebase. If there's something that's back, I never, like if I revert something that becomes actually another patch. So if you ever look at the patches in stable they always grow. They never get smaller. I don't fold, I don't merge things. I don't revert, like if a patch is added that shouldn't have been and it was released. There's gonna be a patch that undoes it. So that change, there'll be two changes in the quilt queue that basically become a no-op. So the rebases are, although it's still a rebase, I sell them ever merge anything. So I try to keep it. I think that's the most stable way of doing things. So sometimes if you look, if you try to figure out, oh, how many patches are on 4.4 or 4.3? Because people ask me, oh, because they want to compare different, like how RT is doing. You can't look at my tree. You actually have to look at when stable had it because those changes will actually grow. So it's actually the number of patches within the stable trees are going to be actually bigger than when they originally were being worked on in the development branch. So don't go and say, oh, look how much it shrunk. But from my, you have to actually look at when to know the actual times. I believe you did use your patch set when you, yeah, you didn't use what was in mine. Yeah. I didn't use your stuff. Yeah, I noticed. I have reduced it. Okay. Is that it? Thank you, Steve. Thank you.