 All right, hello, my name is Julia Cartwright. I'm also a national instrument with a grotian. So today I'm going to talk about using Coachinella to find nested execution context violations. But before I get into that, I want to make it known that, yes, I'm going to be talking about Coachinella, and yes, my first name is Julia. But I'm not Julia the Wall. I mention this only because about once every six months, someone pings me on IRC, asking me a question that only Julia the Wall would know. And I'm wondering if she also has people confusing us as the two Julias that work on the kernel. So I wanted to start off by asking how many people here are on Linux RT users and follow the Linux RT users list. Because I imagine it's most of you, I would hope. If you're not following this mailing list, I would suggest you do so. It's pretty low traffic, lots of interesting problems that, yeah, low traffic, OK, that's a relative term. But it's not that bad. And yeah. So I wanted to talk about a few. If you follow it long enough, you'll see that there's a clear pattern in problems that people are running into with preempt RT. And I'm going to talk about the problems that are hit and using static analysis tools like Coachinella to resolve them in some meaningful way. So this is an example of one such bug report that came out of the mailing list. So here you can see that bug sleeping function called from invalid context at kernel locking RT mutex. And then if you follow the stack trace, you see that addStrikeBio function was called. It tries to acquire an RT spin lock. But because it's running with IRQs disabled, that's an invalid operation to perform with interrupts disabled. So from there, if you were going to actually dive in and figure out what the hell is going on here, you would start looking at this addStrikeBio call, which is in the RAID 5 code. This has since been fixed, but this is a few months ago. And this is also just one representation of bug. There's a lot of bugs that get filed. I think this was just related to using RT spin locks in places that they shouldn't be used. So if you dive into that RAID 5 code, this is what it looks like. In this case, you have addStrikeBio. That's the function that actually is manifest in the call stack. It calls this lockall function. And in lockall function, you have local IRQ disable, followed by acquiring a bunch of spin locks in a RAID. This isn't the exact code. This has been changed since then. There's also a similar unlockall function that I've not put here just to make it clear. And so clearly there's this pattern. You have an explicit local IRQ disable, and then you're acquired spin lock from within that region in which interrupts are disabled. And so because I'm imagining most of you already understand why this is a problem, I did want to put this summary up here in case you weren't clear. This is the most concise way I could express this, is that code executing within a local IRQ disable like your local IRQ enable regions runs with interrupts disabled on a local CPU and invoking schedule either implicitly or explicitly when interrupts are disabled is a context violation. That's what the config debug atomic sleep is gonna tell you. And then thirdly, spin lock implicitly invokes schedule on RT with sleeping spin locks. And so therefore code executing within an IRQ disabled region must not use spin lock. Any questions about that? That's a lot of words up there. Sorry. So that was one example. I'm gonna set that example aside. I'm gonna look at another example that's very common to happen on the RT users list. This is a very similar related issue. The debugging information is a little less verbose because debug atomic sleep wasn't enabled in this kernel config, but it's very similar. You can see that this MSM GPIO IRQ AC function is called and it tries to acquire this RT spin lock and the RT spin lock is invalid to be invoked because interrupts are disabled at this point, which I just happened to know because if you look here, we're in the middle of interrupt dispatching and interrupt dispatching happens in hard interrupt context. So if you look at the code for MSM GPIO IRQ AC, G MSM GPIO IRQ AC, you can see that it looks like this. Here's the function definition up here and then you see that it's doing some pointer manipulation to get to this P control object and then it's acquiring the lock and releasing the lock using the spin lock IRQ save and IRQ restore flavors. Now this case is a little bit different than the prior one. This is a violation because I happened to know that MSM GPIO IRQ AC is invoked in hard interrupt context but there is no, if I just looked at this function, that context is not made explicit by the use of a local IRQ disable or local IRQ enable. So that presents some interesting challenges for Coach Nella, which I'll talk about. So this is my variation of that same problem summary in this case where instead of the explicit local IRQ disable, we just happened to know that code and interrupt dispatch is invoked in hard interrupt context and it's similarly a context violation to acquire a spin lock within that region. One conclusion I draw from this is I don't think driver developers really understand when to use spin lock and when to use raw spin lock. I don't think that's well understood. And so often the proposed fixes to these bugs that pop up is we'll just convert this to a raw spin lock, right? Because that's what you do and that's not good for RT. So I think there's some gaps here. Let's go to, does someone have a comment? No, okay. So my next was a, this happened just a few weeks ago. This is also on the RT users list. Is Arnaldo here? See, I've never met him before, okay. So he posted this and this has been a long ongoing thread on the RT users list. This was code in some Infiniband driver and those Infiniband people are crazy, but it was broken in RT because they made explicit use of this preempt disable. So the way that they were doing their locking is they were acquiring some lock and then from within that lock region, doing a preempt disable and then doing a spin unlock with the intention that preemption would be, continue to be disabled outside of the spin lock protected region and this is broken RT for a number of reasons and one of the reasons is why are you doing, why are you using preempt disable? But this is another context violation or can be and this is not a unique problem to this driver. I decided to go like, well I'm gonna look in the driver's tree and see what interesting uses of preempt disable are there. So this is one such example. This one I don't even understand why the preemption is disabled here. There's, it's actually kind of scary to look at preempt disable in driver's tree. Yeah, so okay, a main line, this handler is invoked in hard interrupt context in which preemption is implicitly disabled, but then they decided to disabled, make preemption even more disabled for some reason, I don't know. And also why disabled across the task list schedule, it doesn't make sense. So that's interesting, I would like to be able to detect these cases and kind of analyze them further. So here's my second point here is that driver developers don't understand when to use preempt disable. I think that's pretty clear. And a more general case which I presented evidence is that I don't think driver developers really understand RT. And so as a bit of an aside, I decided to look like the evolution of these API, like the usage of these APIs in the driver team, driver tree over time. So on the x-axis here is the kernel releases since the get, it was the first I'd get and this is kind of difficult to read, I think for some of you. But this top line is the usage of local IRQ save in the driver's tree. And then this green line is local IRQ disable. And down here you can see preempt disable kind of like slowly rising in this red over time. Similarly that there's local BH disable kind of slowly writing up. And this big spike here is the only API including this list that implicitly messes with execution context which is a raw spin lock. And the raw spin lock wasn't introduced until 2633 into mainline. So you can kind of see that slow growth. So anyway, I thought that was interesting. So there's some things we can do if we think that like driver developers don't understand RT, there's some tools that could be used to help them out. Historically we've gone to runtime tools. We've used locked up as has caught a lot of issues that exist in mainline but to help RT. I think there's an education gap clearly that needs to be overcome which is what every driver developer should know about RT. And I've already engaged some of you for comments and feedback about what that education should be like. But for the sake of this presentation I only intend to cover some of the stuff I've worked with in Coachinella. So if we go back to that first example I gave where we had a spin lock, I guess before I get even further how many of you have used Coachinella or know what Coachinella is? Okay. Two questions. There's two questions, okay I'll ask one. How many of you have used Coachinella? Okay, that's more than I thought and how many of you don't know what Coachinella is? We'll go that way. Okay, so Coachinella is a static analysis tool. It's produced by Julia LeWall and her research team. And it effectively allows us kernel developers to do some static analysis effectively and look at control flow graphs and determine whether specific conditions are held. So in this case, applying that tool, a tool like Coachinella to this problem set is what I really wanna look for is I wanna look for in a single function if I ever invoke local IRQ disable and then I take a spin lock, then that's a context violation. I can determine that statically. As long as I haven't invoked local IRQ disable and then local IRQ enable and then acquire to spin lock. And so how you express this relationship in the Coachinella language, which is really difficult for most of you to read, I imagine, but hopefully you can find my slides later on and dive in. But how you express that wanting to see when a spin lock is acquired is you use this, so the top here is kind of boilerplate. The bulk of it is right in this region here where I'm saying, okay, I'm looking for a local IRQ disable or a local IRQ safe when it's not followed by local IRQ enable or local IRQ restore. So this basic list dot dot dot expands to any statements in that control flow. Yeah, I don't know, we'll see if I can, sorry. Well, this is your fault for sitting in the back, Steve. My name is. Let's see, I don't know that I can. I'm getting older. It's like me. Okay, I don't know that I can't. Sorry, Steve. I would happily talk to you afterwards if you want to see that through the details. So if you run this, if you, if you, there's two, there's actually more, there's four use cases of the Coach Rinella tool, but this is report. This is the report mode of Coach Rinella, which is just like, give me all the matches. And if you look at this, there's 38 violations of this rule and 414 RC5, the latest RC release. And if you do that same analysis on RT, this is the latest RT release. There's still 22 violations in the kernel tree, which I'm not intending you to look at all of these in details, but you can actually see that we still have some work to do in RT if we want to actually eliminate all of these. And some of these are non-trivial to fix because it requires some understanding of the reasons why a driver developer has made the decision to use a ClioQ disable, but some avenues for further analysis. So I'm going to talk about the case that's a little bit more complex, which is this MSM GPIO IRQ AC function. So if you remember this one before, in this case you have the function name here and in this control flow I've acquired spin lock, except there's a problem. If I wanted to apply the previous patch I had developed, I have nothing to match on. I have no local IRQ disable, like explicit context function in here that I can write a patch to match against. So instead I have to get a little clever in how I construct this patch. So in order to do that, I'm going to use two features of Coach Nella that I don't think are really very well-wide understood. Coach Nella allows you to specify dependent rules. So you can say, okay, here's a match, and this one depends on this one. And any meta variables which are matched in the parent, if you will, is available as a value to the child. And I'll show you what that looks like here. And the font's going to get smaller and Steve's going to kill me, but. So the strategy in this particular case is to declare two rules. One rule which matches functions execute with interrupts disabled. And then a second rule that depends on the first rule that then uses this typical control flow analysis to detect when a context violation occurs. So there's two separate and very distinct rules. Now the first one's kind of interesting because it's like how do you match functions with execute with interrupts disabled? And there's no clean, easy way to do that. So what I did was I looked at, well, okay, so I know that any IRQ chip implementation is going to implement this IRQAC callback. And I want to be able to look at any function that is pointed to by this IRQAC member in an IRQ chip struct. So basically I'm going to construct a Coachinella patch that will generalize this pattern, generalize over the name of the IRQ chip, and the name of the IRQAC function, okay? Because I want to be running this across many different IRQ chip implementations. And so that rule, that rule one is actually fairly easy to define in the Coachinella language. I say I have, I'm generalizing over these identifiers, which one is the name of the IRQ chip and the other is the name of the function being pointed to. And then if you ever, Coachinella, if you ever see the situation where you have some static IRQ chip object that points, the IRQAC member points to this underscore, underscore IRQAC meta variable, then please bind the value of that function name to this IRQAC meta variable and Coachinella. And then if I talk about my strategy, my strategy is then to use that in the name of that function that being pointed to to implement my control flow analysis. So from here, if I, this is my rule two, this is rule two, it depends on rule one. And you can, if you use identifier rule one dot IRQ dot underscore AC, this is the meta variable that was bound by rule one. So effectively I know that this IRQ, it makes available that IRQAC meta variable in my subsequent matches. So then from here, the IRQAC function I know is gonna be bound to, for example, MSMGP, IRQAC. And then from there I can do the typical, okay, well is there any colors that's been like IRQ safe, spin like IRQ, spin lock, and any control flow within this IRQAC function. And then, so if you run that against mainline today and RT, there's actually five violations that exist and these are non-trivial to fix. Turns out when I first did this pattern, when I first started doing it using Coachinella, I solved like 30 of these problems. And there's five remaining that are, well I haven't looked at them in great detail and they were non-trivial to fix. Yeah. Does that only detect direct uses of that in the function directly called by that or the whole call stack? Only in the functions that are called by it, yeah. So it's one of the limitations, yeah. And also because I had to choose my anchor point as being some arbitrary, like I happen to know that IRQAC is invoked in hard-interrupt context. Right now this patch is written is only covering that use case. There are many functions that are implicitly invoked in hard-interrupt context. If that function called another function that did this, it wouldn't. Yeah, unless it's available, because I think Coachinella actually works at like a compilation unit by compilation unit. So if it's like in an inline function or something, then yeah, it would be caught. But if it has to cross a function call boundary in code, then it's not gonna work. So solving these that I mentioned before using raw spin locks and converting to raw spin locks is one thing you can do. And so I'm gonna talk about how for a lot of these cases, just actually doing the raw spin lock conversion is just mean, like stupid, busy work. And so I wanted to learn more about Coachinella and actually have Coachinella generate me patches. So I'm gonna talk about using patch mode of Coachinella to actually give me a patch for all the IRQ chip implementations that are using spin lock, convert them to using raw spin lock. Now I'm gonna say this with one caveat is that, like I said before, converting to raw spin locks is not gonna just magically solve all of your problems. You need to be very deliberate about when you use raw spin locks. But it was a good avenue for me to learn about Coachinella patch mode. So this was the strategy we had before. We had rule one, which is match functions which run with interrupts disabled. And then rule two, which is use that match function and see patterns within its body. So if we go back to this example, I want to, if I want to be able to change this spin lock to raw spin lock, then I also need to understand where the spin lock is defined. And I can get that statically from this data here. If I can parameterize across this struct MSM pin control, like basically any type, and a member name. So if I have those two pieces of information, then I can use Coachinella and have it convert to a raw spin lock. And then also the straightforward control flow analysis to rewrite the actual spin lock to raw spin lock. So basically I need to generalize rule two, like I said, to generalize over this pin control type and the member name. So the way I can do that is to augment rule two. So I need to capture the type and member name of the relevant lock that then I can use further into rule three. So rule three is going to use the type name that I've captured and the member name that I've captured to actually generate the diff hunk for rewriting the spin lock to raw spin lock. And rule four uses the type and member name to generate hunks for updating the spin lock collars. So that's on the calling side. So I'm not gonna go in great detail here, but I basically, in order to augment that, this is good to study in your notes later. To augment that, I needed to declare this new type T that I'm gonna bound to, bind to, and then these two new identifiers. One, the L in particular is the lock name, the lock member name of type T, or in the struct defined by type T. And so from there, I just modify this to the actual function match to look at, well, anytime you have a pointer to a type T, and then you, yeah, so yeah, generalizing over that, every time you have a pointer named X to a type T, and that type T has a member lock known as L, then match this match. And so that gives me enough context. So when this match hits, then I know type T, I know the lock member type L. And so from there, then it's pretty straightforward. This was a trick that I kind of just discovered. I didn't see any examples in the Coachinella patches that do this, but you can actually rewrite members just by putting this T here, where T is like the type that I've matched in rule two. And then somewhere within that struct definition, you have this member known as L, and you can use the minus plus in this semantic patch language to actually generate your diff. And then from the control flow side, you already kind of saw this where I was matching spin lock, but instead of using, instead of just matching, I'm gonna generate a diff, so there's a minus and a plus. That's pretty much it there. Limitation, one was mentioned about how it doesn't, well, one was mentioned that this is limited to a single compilation unit, so across function calls that doesn't really track. Secondly, it doesn't eliminate all the hard work. In fact, you could look at this and say, well, you could just use set and just fix this. That's true, I could have, but Coachinella's fun. Right now, I'm only looking at IRQ chip callbacks, but maybe this is an avenue where if we get a better enumerate patches, like callbacks that we know implicitly are invoked in specific contexts, then we can further augment these patches to match more. So with that, I know I just kind of brain-dumped a bunch of stuff, but if anyone has any questions about what I did here or where I'm going or other things you think that might be useful for this sort of analysis, that'd be great. Yeah. I think I have a lot more work. I have a lot more work in one way, yes. Please use the microphone for questions because the guys on the stream and YouTube can understand the non-mic questions. So I think it's awesome, first and foremost. I also think that with something like this, you're getting a lot of work, potentially a lot more than you can handle. I'm not disagreeing with you, yeah. But these problems just don't, I mean, they're there, right? So they need to be taken care of. The problem is that each of these, I mean, I told you that I ran a patch again and generated patches across like 30 different anti-RQ chip implementations and fixed those up, but each of those required me to go in and audit every critical section for each lock. Is this really doing minimal bounded work and so on? Which that takes a lot of the effort. I think Thomas mentioned something before. It's like, well, you could, I don't forget the case we were talking about timers. Like knowing when things should be at your timers or not. It's still the same thing. It requires analysis. It doesn't, other questions? So is anybody else concerned about the number of raw spin lock usages in the kernel? So we'd be looking at this problem the other way. So most of that spike, at least the spike in the past few releases with me actually fixing the bugs that I caught by this tool. I keep an regular eye on the usage of raw spin locks in the kernel. So actually it's not, I didn't find anything too bad in the last couple of years. So there was one user using an arc spin lock in some driver code, which was anyway broken. So, but that's just because people didn't know anything about locking and they used the first file, they find randomly selected and copy something. So yeah, you can't avoid that kind of problem. But in general, yes, the amount of crap in the kernel is insane. So this is going to be not only for RT. I mean, we're doing a lot of cleanup and overhaul in the kernel all the time and we just have to keep that going if RT is accelerating that to some extent, fine. But yeah, it's a huge task, but we have to do it. No way around it. Yeah. Having better tools to identify problematic cases I think is useful. It's also the question if some of these things should not start going into the process side of how patches are submitted because I think Cauchy check and sometimes not even check patches being run on some of the patches that are in the RT. So that might need to be fixed on the process side. Any other questions? I'm happy to show you my tiny text afterwards if you'd like to see it. And show you how these tools work in action too if you're curious. Great. Well, thank you so much.