 how we can test them better. So if you have no idea what any of those words mean, don't worry, we will get into it. We will go over the background of transient execution stuff, look at what that means and how it works, have a look at how security vulnerabilities are discovered and reported and fixed and distributed. Take a look at how exactly these vulnerabilities are tested currently and how we could potentially do better. So let's dive right in. Oh yeah, let's talk about me briefly. So my name is Russell, I'm a Linux kernel developer. I work at IBM, so I'm in charge of kernel hardening for IBM's power CPUs. So I do Linux kernel stuff to make the kernel more secure on power. I'm based in Canberra, Australia. So the conference, assuming the schedule has not changed, is currently 5 a.m. local time for me here. So yes, if there is the time for questions at the end, please be gentle because I am half awake. So let's get into some background. There's a decent amount of background. Some of this will be revision for some of you, for others it might explain what the word salad and the title means, so let's just get into it. So we need to talk about speculation. So speculation is something that all modern high performance CPUs for the past 20 plus years have done and it's pretty critical to the way that they operate. So I found the easiest way for me to reason about why speculation exists and how prevalent it is, is to take a look at some designs. So this is a Xylon Z80 from 1976 and I'm not a hardware expert myself. I don't know what all of these bits and bobs mean, but the overall idea is kind of straightforward. Instructions come in, they get decoded and stuff happens and that's kind of it. It's kind of a simple design. You have stuff come in, stuff go out. You can kind of pinpoint where things are happening at each stage. This is well before the kind of the prevalence of out of order execution and speculation. This is a micro architecture diagram of Intel Skylake, which is a few years ago now. And this is looking at a single CPU core on the die, not even talking about multiple CPU cores. And this might all look like gibberish and it certainly does to me, but the point is, within a CPU core, there's a number of different components that do different things. There's a lot of them actually and a ton of stuff happens out of order. So each component wants to keep working. They don't wanna be waiting for another component to finish its IO before they get to execute their thing. They wanna be doing stuff, right? Everyone wants to keep busy. So what happens is that CPUs pretty aggressively look for stuff to do that they may have to do in future to get ahead. So if you're waiting on some IO, you don't wanna just be sitting there doing nothing. You want the other components to be predicting ahead to get some more work done. If you're waiting to check, if the thing you got from IO is above a certain number, so you're wondering about an if statement, then the CPU will just predict true or false and start executing down that path in case it's necessary. So this is where we get into transient execution. There's some confusion about speculative versus transient and what that means. So this is an Intel diagram. And the term transient execution came along to specifically refer to mis-predictions. So when a CPU makes an educated guess and that guess is wrong, it is going to execute instructions that are not committed. It's going to execute instructions that do not end up being visible to the user. Those are spec, those are transiently executed. So something is an instruction is transient if it is executed without the results becoming architecturally visible. So you won't notice it in your program. It can only be observed through side channels that can reveal micro architectural state. That's why it's not noticeable, right? Like you shouldn't notice. It should be all completely invisible to you. So the whole transient thing came from Intel. It's a good change. Before this, every speculative thing was, you know, a spectre-like or spectre-variant 412, 8.Q. And that's not the most descriptive or helpful. So having a term to specifically refer to mis-predicted speculation is quite nice. So that's what transient means. That's what transient execution is. Is a relatively contrived example I came up with to see how exactly transient execution happens. Not that there's particularly dire consequences here, but basically we've got a loop that iterates 2048 times. Within that loop, there's an if statement. And that if statement is true 2047 times. But the very last time, it's false. So as far as the program is concerned, this if statement block gets entered 2047 times. There's a counter and the count will be 2047. What will happen within the CPU, however, is that the branch predictor will say, well, this if statement's been true 2047 times. It's probably gonna be true the next time. We're gonna predict it's gonna be true. And so the contents at that final iteration will be transiently executed, maybe, most likely. As far as the user can tell, nothing's happened here. But there are micro architectural state changes. In this case, there is an out of bounds value that is read that will be stored in the level one cache as a result of this mis-prediction. So this is an example of how you can induce CPUs to do transient execution. You can deliberately mistrain them. And it's not just obviously the branch predictor does a number of different things within CPUs that make educated guesses that you can attempt to bamboozle. So some vulnerabilities over time. I basically just wanna highlight that a lot of people have heard of Spectra meltdown and not a whole lot else. This is a class of vulnerability, right? These vulnerabilities are the gifts that keep on giving. It's not so much a specific vulnerability as much as these are concepts and methods. So if you think of software, in software, what happens is that there's a line of code with a bug, there's a buffer overflow, whatever it may be, that line gets fixed and no one could do that, you call it a day. These aren't quite so simple because these are concepts that can be broadly applied. And it's hardware, it's based on microarchitectural details of hardware, which can change from generation to generation for no good reason, or for no deliberate reason, I should say. So just a random handful, I pipped out the original Spectra meltdown. There's STL stored a load, there's level one terminal fault, microarchitectural data sampling and recently had branch history injection, which was CVE 2022-0001-0002. So we're not out of the woods. These things keep popping up and they don't go away as easy as a simple software fix. Over time, hardware changes, the way that hardware reacts to these things changes, the way that mitigations work changes as there's special instructions and whatnot. So it's a bigger picture than just, oh yeah, Spectra meltdown was a thing and now we're fine. We're still not fine. And anything, any singular process still has a lot of issues, essentially. Going between processes, we tend to be pretty good. You'll enter the kernel and exit the kernel and that's a great place to do a bunch of flushing, a bunch of mitigations, but within the same process, it's very difficult to prevent all speculation because you don't have a nice place for mitigations to be applied. So how do these problems get fixed? Well, hardware revisions as I kind of alluded to, whether that's a deliberate micro-architecture change to prevent a certain thing from happening or there are new instructions to say, okay, well, this was mitigated in software by doing this, we're gonna make a specific instruction to make that go faster. So that's the kind of thing that happens in hardware revisions. There's also firmware or micro-card updates. These are kind of like hardware patches. A lot of stuff you wouldn't expect can actually be changed without changing the silicon, which is just hardware vendor magic, but that's the thing that can happen. Kernel patches are obviously a pretty big one, so most kernel mitigations will tend to be flushing, so you enter the kernel, you exit the kernel, whether that's you take an interrupt, you make a syscall, or you're doing a context switch between processes, the kernel will flush some states so that the whatever micro-architectural details could potentially be visible on the other side are gone. So you can't leak information, you can't induce bad behavior, you'll get a clean slate when you come back. So that's the way that kernel mitigations typically work. And finally we have compiler features, which is an area I'm not particularly in touch with, but compilers like Clang and GCC have a number of hardening features for spectral execution, so they will essentially generate code that is less susceptible to these problems. They will generate code that will actually prevent you from speculating out of bounds or jumping to a bad place or what have you. So how these fixes get distributed, and this is where things kind of go out of control. So the matrix here is really, really big. Hardware vendors obviously are in charge of the hardware and also micro-code. This sometimes is not just limited to Intel AMD, the manufacturers of you buy your server from someone, maybe they're in charge of the way that your micro-code firmware updates are delivered to you. Fixes obviously go upstream, so when there's a vulnerability and there's a kernel patch involved, that will end up in a build on kernel.org, it'll end up on in Linux 12 volts, Git tree, typically out of band. So these aren't typically, if there's an Intel vulnerability, the patch is typically not going to be sent to ArchX86 and black-shattered 10 revisions. It's gonna be sent to security at kernel.org and it's going to go in out of tree without public review because at the time the patch is being developed, it's still under embargo. The public disclosure date has not yet been met. So typically you'll see some commits just kind of sneak into the final source tree that weren't public before and there's some people on Twitter that will point these things out and be like, hmm, wonder what this could be and then a couple of days later you learn about this new vulnerability. So that's how things get out there and ultimately a lot flows downstream from kernel.org but not many people actually run their kernels from there. So most of the puzzle falls to distros and distros control a lot of the puzzle for how these things go out to users. They can be involved in firmware, so things like Phwapt can control how users actually end up consuming these things. Distros often have packages for microcode as well. The kernel obviously, each distro essentially forks the kernel. They have their own kernel version that will have a number of kernel versions depending on how much they're still maintaining. They have not just the kernel, so they'll have a lot of back ports including functional back ports that kind of change the landscape from the upstream stable kernels. They also will have a different config too, you know, def configs or other distros. So essentially what the point I'm making is that each distro needs to be able to be quite confident they've got everything right because there's a lot of moving pieces and every time they apply one of these things, it may not behave in the same way. It may not have the same consequences as in the situation that it was written for. Distros also control tool chains. So whether your client or GCC has something in it or whether that's a default feature or not can be something that's decided by a distribution. So if you're building something in user space and you want to be mitigated against something or other, your capacity to do that or to do that conveniently may depend on the distribution that you were running. Finally, there's back ports that go into distro kernels that I wanted to mention because distro kernels, especially long-term support, enterprise-y kind of distro kernels can be very old. At least the base of them is very old and if there's been significant change in specific areas then it can be quite difficult to actually back port security fixes to them. Something like interrupt handlers or whatnot may not be the most change-friendly area of the kernel and if it's changed significantly in the decade since that kernel version then it's not exactly the easiest to apply mitigations to it which can have consequences in how they behave. So you end up in a situation where the matrix is too big. You don't know if your CPU, whatever revision it is of whichever skew from your vendor you have, what microcode you're running, what distro, what version of that distro, what kernel of that version of that distro are you vulnerable to whatever exploit and it's quite hard to get a good answer about that. The most you can kind of get is a trust me bro, typically. But even then, no one can tell you trust me bro because there is no conglomeration of these different parties to all combine together and say you're good. So it's a tricky question to answer. So let's back up a bit and look at how these vulnerabilities get discovered and reported. So all right, a researcher finds a problem. You're a security researcher, you're having a great time. You find a problem. You are probably gonna test it on your laptop or whatever, wherever you found it. You'll probably round up a few other CPUs from around the joint, test it on ARM, test it on Intel and AMD, see how far you get. Then you disclose it. So you let those parties know, hey, there's a problem that you need to fix. Here's my proof of concept. Here's me explaining it. Let's figure out a way to fix this. The affected parties will coordinate a public disclosure date. So if let's say you find a vulnerability in an Intel chip, that's going to require patches to Linux. Well, that's gonna be more than just Intel's involvement. They're probably gonna talk to distros. They're probably gonna talk to board partners or whatever, depending on the reach and try and say, okay, let's all work together to make sure we're ready to go for this public disclosure date. Hopefully they do actually fix it by then. That would be nice. And finally, there's a public disclosure. Hopefully the fix is already out there for people to use and we can call it a day. So looking a bit more specifically at some of these points and essentially what can go wrong. Who gets notified? And is everyone that should know actually in the know? This is a good question because the issue at this stage is that it's essentially at the discretion of the researcher. If you're someone who is looking at these things for a living, you've been in the space for a while, you have a good understanding of which parties need to know. That doesn't really matter because there isn't some body dictating who gets an email when there's a problem. It's up to the discretion of the researcher. So if the researcher hasn't dealt with this kind of thing before, they're not aware of how far reaching it is. They might only reach out to the one vendor they tested it on. And for some vulnerabilities, that's fine. I mean, if you look at something like Intel, something like Intel SGX, which is a really Intel specific concept, then maybe that's fine. But for these speculative execution issues, essentially anyone manufacturing a high-performance CPU is potentially affected and you can't know that they're not unless you ask and point it out. So oftentimes there are hardware vendors that are not notified that should be, that only end up finding out on the public disclosure date. You could say, well, you'll send an email to security at kernel.org, which is a private mailing list for Linux kernel maintainers so that you don't announce to the world that everyone's machines are compromised. But ultimately these are hardware vulnerabilities and the way you mitigate them can involve Linux but that's not the be all and end all. So there's not always going to be an email there which is gonna tell probably everyone that makes CPUs because they all employ kernel maintainers. But that may not happen and that may not have the right context to fix the issue in broader context like specific distros or tool chains or whatever. And then finally there's distros and the main thing here is just that you're never gonna catch all of them and whether or not they fix it in time is highly variable based on the distros. So distros do not just say, cool security patch, I'll apply it straight away and call it a day. Distros don't do that, they don't do that at all. They have their own schedules, they have their own priorities. So even if a distro is contacted whether or not they'll actually fix it is quite variable. Public disclosure comes around and then the next question is you read an article saying there's this new vulnerability, it sounds very scary, how do you know that you're actually safe? And it's not that easy. Most of the time there's not a public proof of concept that you can just kind of take and run if there is a proof of concept that's not really in the form where you can do that. It typically comes down to who does your kernel have blah commit. If you're an end user that doesn't know how to check that within your distro, specifically you have distros that don't publish it, get history, that you can't really check for a specific commit. You have to look for the specific code if you can get the source package or whatever. And if you're not one of those major distros then you can't really have a good idea. You're probably gonna try and backport the patch from upstream but again, you don't really have the tools to make sure that actually worked and did the right thing. So there's some issues there but once everything's resolved and we can call it a day, what happens next? Well, the security researcher has done their thing. They've got a lot of props for finding a cool bug. They're probably giving it a cool name and logo and they're off to the next thing. The kernel maintainers, they've applied the fixes. The fixes work. They've either got a proof of concept to test or the people that reported it have verified that it works and they're moving on. It's mostly problem done here. And the thing I wanna highlight is that potentially, not always but potentially, the capability to verify that the vulnerability is actually properly mitigated can disappear at this stage. The people who can know for sure that something is working or not are the security researcher and the parties that they're reported to, especially CPU manufacturers. They'll have the people to know, yes, this is a problem. Yes, this fixes it. But how can everyone else know? And how can they check? Because those parties aren't gonna be continually running regressions on different distro versions and different new hardware revisions. And if they are, how do you know what they're running and if it's working? So there's the potential for essentially the open source community to know if we're vulnerable and not entirely in the hands of these parties that are potentially moved on to other things. So there's an issue there. So let's have a look at the way we're currently testing. Most of it is trust me, bro. And the kernel has a whole, a whole CISFS directory of trust me, bro, which is CIS devices, system CPU vulnerabilities. If you don't wanna type that in and grab it, the LS CPU utility will read it for you. So this essentially for a number of different vulnerabilities tells you what the kernel thinks about itself. So it will say things like, yes, I'm mitigated in software. Yes, I'm mitigated in hardware by this thing. No, I'm not vulnerable to this. Yes, I'm vulnerable to this, et cetera. And this is good. It's don't get me wrong. It's great that it exists, but it is limited. The first thing is it is not comprehensive. It doesn't contain every vulnerability. These entries are functionally ABI. So a lot of things parse them and you can't really just add and change entries trivially without there being consequence. Issues here include entries getting overloaded. There's a lot of things that you could consider specter V2 that have different mitigations. So things like specter V2 might say you're safe and you have this mitigation and this mitigation, but that can only cover one part of it. You don't really know. As I mentioned, there's vulnerabilities that don't have entries. And there's also vulnerabilities that only affect some architectures and not others, but there's a similar one for that arch that can get overloaded or not there, even though it's kind of the same thing. So there's a single essentially SysFest entry for everyone, even if that may only affect a specific vendor. So it's very far from comprehensive. I say it's not exactly perfectly generated here. It's essentially a maze of if this flag and this thing on this version, then we can say this, which is fine. That's how code works, but it's far from a proper flowchart handed down by a CPU manufacturer to say this is a specific case in which you need to apply this specific mitigation in the kernel. So there isn't a clear plan. It's kind of just the people who work on those arches put that in the kernel and hopefully it's good, but potential for new CPUs on older kernels and then it's misreported or firmware ships the wrong flag to say what the CPU supports and now it's wrong or either it's vulnerable when it says it's not or it's misreporting that it's vulnerable, et cetera. So there are potential issues there. And there are things that pause this that you could consider tests. So one great one is Spectre Meltdown Checker, which is an awesome script. It mostly supports Intel, but it'll work on AMD and also work on ARM. And it'll look at a bunch of things, including Cessifest, but also what's in your kernel, what's your kernel version, what microcoder you're running, yada yada to figure out, okay, out of all of these many CDs, which ones are you potentially vulnerable to? And it'll suggest how to fix them. So very cool. Another type of testing, I know that this is what we do for PowerPC in our kernel self-tests is our mitigation successfully applied. So we can look at the code patching and say, yes, the instructions are in the right place. If we're applying a mitigation on context switch, we can look at the context switch pathway and say, okay, are the instructions that flush the state actually present? And then we can do stuff to observe that. So this is kind of the lowest, you know, the easiest level of observing microarchitectural state, but if you are reading, you're reading a bunch of stuff, it's getting loaded into your cache. That's quite fast. You can use something like Perf to look at the cache hits and misses events. You can then, you know, induce mitigations. Maybe it's happening on context switch or on kernel entry and exit. You know, you make a syscall and then you perform the same operations and you're seeing a whole bunch of cache misses because the state has been flushed when you enter and exit the kernel. So there's ways like that that you can pretty easily observe that the microarchitectural state has changed. That is in the side channel because you can't tell what was there. You can't do anything useful with that, but you can do things like simple checks for, you know, has there been a state change? And so do we have actual tests that actually test the actual exploit? Exeter is yes. So this doesn't tend to work the same way as kind of one and done mitigations. If you are used to working in security and in kind of user space land, you'll go to, you know, exploit DB and look up the version of your software that you're trying to own and there'll be a proof of concept there. These are more conceptual than that. So we have a few resources that are quite good. The first that I wanna mention here is transient.fail, which is this cool website that has like an ecosystem diagram of different kernel vulnerabilities, sorry, different transient execution vulnerabilities that will give you a description of them and kind of map out how they relate to each other and give you a proof of concept, which is quite cool. There's Google Safe Side Project, which is sadly abandoned, which was a series of literate demos that showed you different transient execution vulnerabilities and how they work. So, you know, you would have code, you can run that would actually do it, but also, you know, it was very well commented and documented to kind of explain how these mechanisms work. So you can understand it better for yourself, which was super cool, but sadly, no longer code still up, just archived. And then depending on the security researcher, you do sometimes get proof of concepts from individual vulnerabilities and the recent Spectre branch history injection cases, a good one there, because they've got some good proof of concepts up for everyone to go and play with. So, how exactly could we change how we test to do better? Which is essentially the thing I'm getting at here. And the first is that I think we should maintain exploits as tests. I think that we should have exploits that we treat quite similar to other things we used to test the kernel that we can use to check for regressions. And a key part there is that we should have something that non-security people can run. We have all these different CI systems. We have people running stuff like LTP. There's a huge ecosystem of kernel tests. And I think that we should have something that can run in that vein. And the people running it don't have to understand it, right? Anyone should be out of test for regressions. We should have something that has security people we can hand off to people that run a whole bunch of automated tests and keep us confident in perpetuity that we're mitigated. Because different CPU micro-architectures happen over time. You'll have hardware mitigations that are introduced. So, okay, cool. You turn the software mitigations off because you don't need them anymore. But what if there's a problem in them? Or what if the firmware reports? It has them, but it doesn't, et cetera. So there is a need to test for regressions. And there isn't a great way to do that right now. So we should have something that we can treat as a pass-fail test like our other kernel tests and be able to hand that off to folks. And then we'll just deal with the bug reports. So, I'm not just, isn't just a pie in the sky idea. I've started working on it. I'm still in the relatively early stages, but yeah, I'm building a test suite for transient execution exploits that is essentially trying to be more of a pass-fail, trying to be something that can run in CI and produce reliable results that is for something that has a lot of variance, which is speculation, essentially trying to attain statistical confidence when we run that there isn't a problem. So that's what I've been working on. It's based on Google Safe Side. So it uses the memory oracle and cache side channel concepts from Safe Side, but I've rewritten it in pure C from C++ in that case, trying to have something that's a bit more palatable to a kernel developer audience, and also myself. And yeah, the goal is to be something that we can hand off to testers, get integrated into CI systems and be able to have that same kind of confidence that our kernels are still secure like we can with other things. So that is the goal. I would really love to hear any feedback if there's something like this that already exists. Obviously I would like to know if there's ways that hardware vendors or distro people are already testing these kind of things that is more in depth than what I've heard of, then I would love to know that. If you have any thoughts or suggestions or concerns in general, I would love to hear it. So that's all I had today. Thanks a lot for coming along. Time permitting I'm around to answer questions. If not, please reach out. I'm really curious to see what people think about these issues and very open to comments or suggestions or feedback. So I appreciate your time and I hope you have a great conference. Thanks.