 Right, it's 20 pass, so I better kick off, I suppose. Hi there, my name's Colin King, and I work for Canonical, so Canonical is the company behind Ubuntu, and I've been a kernel engineer for Ubuntu now for nearly 12 years, and part of my work really is dealing with day-to-day bugs, finding issues, fixing them, and one thing I really don't like is bugs. I don't like things breaking things, and my heart, my passion is to see things being fixed. So some of my work, I do static analysis in the kernel, and I find bugs and fix them, and other parts of my work is I run regression tests on the kernel, make sure that we don't have any faults before we ship a kernel to Ubuntu users. So without further ado, I'll talk about StressNG, which is a tool I wrote specifically to stress test systems and find bugs. So StressNG, I started about six years ago hacking on StressNG. It's really kind of from the original stress program, which had four little stress tests in it, and that was all right, but I wanted to expand the functionality, so this is stress next generation, and at the time we were playing around with laptops and trying to see why they were getting hot, and Intel had brought out the Intel thermal D, and I wanted to see how good that was at keeping machines cool. So I kind of started hacking on StressNG to try and make processes get really hot and toasty, and that way to see how good thermal D was at keeping the system cool and not overheating. So that's where it started from. But like all things, when you start writing a program, you kind of see another kind of use, and you think, oh, I can extend that, and it's grown into a bit of a monster now, which seems to do a lot more than I originally intended. So the whole idea, really, of StressNG is to stress test systems. So originally, as I said, it was making CPU get hot, but as time progressed, I've made it so it stresses system calls, so that's all the kernel system calls I can get coverage on, really. And interfaces, that's device interface, opening devices and doing all sorts of legitimate things, but in weird ways to see if I can break drivers, and also exercising ProcFS and SysFS. So really it's a kind of stress tool. And a lot of bugs we see are where things are done concurrently or done in a slightly strange way. So StressNG does pummel, hit the system as hard as it can to force out bugs. So it's not really a real world use case, but it's interesting to force out bugs. And there are quite a few. So StressNG has grown, and it's got lots and lots of tiny little tests, and the philosophy is like the UNIX philosophy, do one job and do it really well. And StressNG is just full of little stressors, which do one thing and try and do it as well as they can. So it's small, but lots of things. And because of that, when you have a little stress test, which just spins around maybe on one system call, or maybe just hits the processor in one little way, you can use that as a little microbenchmark. So caveat here, benchmarks are benchmarks as in they are good for testing things and seeing how well they work, but don't treat it as gospel truth that when you get an up performance number out, that is 100% truth. It's just a good idea of showing you relative to performance. That's what StressNG can do. So some stress tests are really good at measuring some things, and others are a bit more wobbly because you're talking to a file system which has more jitter. So it's kind of treat it as kind of see how it works for you, but don't blame me if there's lots of jitter. So caveat MTOR. So one of the, well, I have to look at the screen here because it's not my screen here. So one of the other things is because it's got lots of little stress tests, you can use those stress tests at benchmarking latencies in kernels. So, for example, there's like, there's a juke stress test. There's lots of jukes, but you can run a real-time stressor with that, which meant instruments, the latency of doing the juke corn returning, and then you can see how much jitter it is. Or, for example, you can do virtual memory accesses, the VM stressor, which does mem mapping and spitting pages and pulling them back. You can run that with the real-time stress test, and that will again give you latency timing. So lots of little tests can be used in lots of useful ways. Another test is system burning test. So you've got your new server in the lab, and for some reason it panics. What's wrong with that? Maybe it's memory. Well, you can use stress energy to hammer the CPU or maybe to exercise all the pages in the system with lots of patterns being written to memory and then read back. So you can do soak tests on your hardware. So you can find out dodgy bits of memory like that. So the other thing is we use this in the kernel team to do regression testing. So we run 200 or so stress and G tests on all of our kernels before we ship them, and all it does is just exercises various interfaces just for multiple millions and millions of iterations over a fixed length of time. And if it pops out at the end of nothing breaking, then we're quite happy about that. So it's a very simple regression test, but it gets a lot of coverage in the kernel because I've targeted stress and G to look at lots of ways of using system calls. Yeah, and then the other part is kernel coverage. How many of you have used G-Cov on the kernel to get kernel coverage? That's cool. So the whole thing about G-Cov is you can enable a feature that when you run the kernel, every jump and every track through the kernel can be instrumented. And at the end of your testing, you can then suck the data out, instrumented data, and generate nice little graphs and see where the kernel has been touched. So as stress and G, I run my tests and look at the coverage, and that way I can see which parts of the kernel might need further testing. And that's kind of useful because as we get new kernels, there are new code paths which haven't been exercised before. So I keep my eye on that and try to invent new ways of stressing the kernel. One other feature is it's got a verify option. So if you're just stress testing memory, you can say verify the memory so you can write a pattern into memory, then you can exercise more bits of memory, then you can go back to the original pattern you wrote and you can read it back and sanity check that. And lots of stress tests in stress and G like file system tests, where you can write patterns down to the device, you can read them back and sanity check it. So the verify option is a good way of sanity checking that your system is working correctly. But it does slow the stressors down. So don't use that as a benchmarking tool because when you do sanity checking, there's an overhead to check stuff. And the other cool thing is I've been doing searches online for who's been using stress and G and I found a lot of people in academia are using it for setting up different environments like cloud environments and loading up certain nodes and practicing like noisy neighbor kind of experiments. And so it's really encouraging to me that seeing people using stress and G for research to see where systems are busy and what kind of ways you can alter the scheduler or ways you can circumvent noisy neighbor type problems. So it's kind of an all-purpose tool which seems to fit quite a few use cases for people. So as I said, it's got lots of stress tests over 200. And I call those tests stressors. I'm not sure that's a real word, but I made it up. So as I say, it's covered the CPU instruction and data cache twiddling. It exercises the floating point unit, integer arithmetic. And on x86 processes, all sorts of esoteric constructions. It's like bit shuffling. And it depends what version of the compiler you have of GCC. But I've written code so it tries to exercise the CPU in many different ways. And that can hopefully throw bugs in. You can see if your CPU is working correctly or it's really good just to stress test it and make it hot and see if your thermal domains are working correctly. What else have we got? Yeah, device driver tests, block tests. I perform lots of eye octoes and really do the kind of stuff which users probably wouldn't do in real life. But it's good to exercise device drivers just to see what breaks. And I've seen quite a few drivers on Linux and other operating systems, too, which don't actually handle some of the correct semantics correctly. And because of race conditions, you might have lots of threads opening something up and then closing them and then duping and doing all sorts of kind of things like that. And you get reference counting bugs on a race condition because there's no locking and suddenly, bam, the kernel breaks. So we found, I think, over 20 now weird corner cases in the kernel. And that's good. I've either fixed them myself or sent bug reports to upstream and said, no, this doesn't look right. And you can provide a stress NG stressor, which reproduces that issue easily. So it's not like this goes wrong in some weird way and I don't know why. We've got stress tests which actually breaks it in a very clear way. Yeah, so I don't want to go through that in too much detail because it's going to bore you to death. But you can see there's lots of different types of coverage which stress NG can touch. So I kind of think it's a versatile tool. And I'm very grateful for people who've given me comments about saying, well, maybe it's just stress virtual memory this way. I've had lots of feedback from different people. But I'm still looking for more contribution. So if you've got any great ideas or even patches, I gratefully receive those and I'll try and make stress NG a better tool. So when I started stress NG, it was a Linux-only tool. It's all written in C, so it should be really portable. But as you know, things aren't really that portable. Because when you put something onto BSD, there's some system calls that Linux has which BSD doesn't. And sometimes the semantics are slightly different. So to make stress NG portable, I've tried to put a very, very thin abstraction layer in. Mainly it's kind of shims, macros, or if necessary, a small function core. So it's very lightweight. And this has made stress NG portable to all the BSD systems. And I think it's one of the ports, you know, the BSD ports repository, you can download stress NG. I've got it working on Solaris and it crashes it. Minix as well. And Android, there was a user who said, I like stress NG, but can I run on Android? And I figured out, oh, you can just cross a compiler, statically link it so it falls into libraries. And you've got this massive image, but it runs on Android phones. And it's good for testing those really old Android kernels. Yeah, on the Mac, that was dead easy. Haiku, one of these experimental fun operating systems. It's got POSIX interface, so it builds on that quite happily. GNU HURD, which is kind of dying, but it still works and crashes that. And yeah, I made it portable, so you've got GCC it will build. And it doesn't matter which version of GCC, because stress NG will detect which features your compiler supports, and it will try and build it as cleanly as possible. And if your compiler doesn't support the features, it will just silently ignore that functionality. So you might run a stress test, and it will say it's not supported on your platform, but at least it's built. It builds on Clang, and even builds on TCC. So I tried to make it portable, and this means you can get stress NG, build on Red Hat, build on Fedora, build on Slackware, it builds anywhere. And if it doesn't build anywhere, let me know, and I'll try and fix it, or even better, send me a patch. The other thing is, stress NG uses quite a few support libraries, because who doesn't? And if you haven't got that support on your system, maybe the library just isn't available, stress NG ought to detect that when you build, and it will just leave it out, and the tests which rely on it will tell you that it's not supported. But at least you've got something building which will work for most of the other tests. So portability is great. And here's something that I want to show you. So Minix, I thought Minix was really rock solid, and then I found out it wasn't quite so. I broke the file system with that. Track and Flight BSD, and, yeah, K-pupicity. BSD kernels have been really good, actually. File bugs and they get fixed, but it is possible with very simple tests. I think one of them was just doing dupe. It was just doing lots of dupe calls, and it was multi-threaded, and I think there was a race and something just exploded. I'm not sure if it's one of those dumps there, but, yeah. And even on Open Indiana, Solaris. Solaris has been around for ages, but I managed to break it quite quickly with Stress NG. So Stress NG is not there to be a denial of service attack tool. Please hear me right, but it's there to basically hammer your system and see if there are any kind of corner cases where it breaks. Now why is that useful? Well, if you've got a device, an embedded device, and you're using an old kernel, run Stress NG. You might find their bugs it pulls out, and there hopefully should be upstream fixes which you can backport to your kernel. Or, supposing you've written your own device driver, yeah, it looks right, does what I want, you know, frame buffer or maybe a bit of radio work or whatever. Run Stress NG on it, and it will try and exercise the interfaces, and you might find some breakage there. It's good to find bugs before you ship a product. So give it a spin. You might not find any problems, but who knows? And I say, because it's portable, you can just build it on your platform, you know, and if it's not packaged, it's easy to build. You just need to compile it and the source code. Right, so enough of that kind of sales talk about it. How do you actually use Stress NG? Well, it's a command line tool, and each of the stressors has a little name. So for example, I want to do a mix of IO operations here. I want to do reads, writes, seeks, and I want to have lots of those kind of active. They're a bit like, let's have a really stupid database which does lots of activity on the disk, that kind of simulation. That's what the IO mix stressor does. And so you specify IO mix, and then the number after it is the number of instances you want to run. So in this case, I just want to run one instance of the IO mix stressor, and I'm going to run it just for 20 seconds, and it's got the verbose modes, so it'll give you lots of information about what it's doing. That's a simple example, but you don't have to just run one stressor. You can run multiple stressors and multiple instances of those stressors. So the next example, we're running two CPU stressors, about two instances of the CPU stressor, and four instances of the virtual memory stressor, both at the same time, and we're going to run that for five minutes. And as I say, there's over 200 stress tests, so you can mix and match. You can say, I want all these virtual memory tests just to run here for an hour, and see what happens if it blows up, or you can pick IO stressors and mix them maybe with CPU stressor and see what happens when you get that mix, so lots of flexibility. And for convenience, instead of having to specify the exact number of CPUs, you can also specify zero as the number of instances, and the stressor will look at how many CPUs there are online and use that number of CPUs. So that's quite handy, because I'm regularly testing machines, and I don't go, oh, that's how many CPUs is this machine, I've got eight threads. I don't even understand, I just put zero in, and I know it's going to populate every CPU with a stress test instance. So that's kind of simple, okay? Another type of test, I don't use this one so much, but it's quite handy, it's the seek option that sequentially run every stress test. So here I've got seek zeros, that means on all the CPUs run every stress test one after the other. So first of all we'll run the first set of stressors for a certain amount of time, and then it will run the next lot across all the CPUs, and then the next lot right to the end. So here it'll run each stress test on each CPU for 30 seconds. And the TZ option is show me the thermal zone information. So it will dump out how hot your system's got. So depending on what type of thermal zones you've got, it will tell you, for example, the CPU package perhaps, or maybe the ACPI thermal zone, or maybe different thermal zones, depends on what your system supports. But it's a good way of seeing which stress tests make your system run too hot. What else have we got here, page 40. Yeah, so page 40 is interesting. You can force that minor or major page faults, and there are a couple of stressors designed specifically to generate page faults, so you can see how that happens. That's those two examples. And I've got another example here where we run 32 timer stresses, and they're all firing off timer events a million times a second. So theoretically you should see 32 million interrupts a second. I probably can't actually do that in real life on some of these small devices, but it's a good way of hammering the system and just see what happens when you push it to the max. It's not a real-life situation, but you never know what happens. So it's a good way of seeing if you break the system. So I've got another example. Sorry, I've got, it's just a full of examples. I just want to give you a taste. I won't go through every stressor because that could take all afternoon. This one, so a lot of small embedded systems might be running without swap. So what's the question if you've got a leaking application which just keeps on memory leaking? So you can run these stressors like the break stressor that keeps on expanding the S break region, keeps on expanding that and it touches the page each time. So it forces a page to be, a physical page to be pulled in. We've got stack stressor running at the same time and that basically expands the stack on a thread. That's where on a trial process and touches the page, just keeps on burning pages. And then the big heap stressor and the big heap stressor does a mix and match of expanding the heap using malloc or doing big mallocs which expands, well, which basically does big memory maps. So this stressor here keeps on sucking up memory and touching pages. And hopefully the kernel um killer will kill off the stress and g processes rather than something else. And I've seen this run along and system D gets attacked. And, you know, things start to break or demons in the background just disappear and we haven't got the correct rules running to make that respawn. So it's, you know, really could test seeing what your system's like when it's under a lot of memory pressure. And the core thing about stress and g is when you're running it, if any of these stressors get ummed by the kernel, stress and g monitors that and it will respawn that stressor. So it does its best to keep itself alive for using the various methods of changing the um adjustment levels. So, you know, it tries its hardest to keep itself alive and pummel your system to death. Yeah, okay, the next bit is I've got illustrations of how you specify sizes. So, you know, certain stressors, you say, I wanna run a VM stressor and I wanna use two gigabytes of memory so you can specify two gigs or you put two meg or your numeric number or you can even say, actually, I want to run it with 50% of the system's physical memory. I don't know how big that is but I know I wanna do 50%. And a little bit further on, I've got a malloc test there, I've got 120%. So I'm basically doing over commit there. I'm saying, I don't know how much memory the system is, but I want 20% more over commit. What happens when I run this stressor? Do things break in kind of weird ways which I didn't expect. Yeah, and like the file system there, I've got a hard disk, it's a stupid name, HDD, but that basically creates big files but you can say, I wanna use a certain percentage of the file system to create massive files and see, can write and do some horrible IO on that. So yeah, so you can specify sizes by percentage or by size and bytes, kilobytes, megabytes, gigabytes. So I tried to make it flexible. Yeah, what is next? Oh yeah, so some of these stress tests have got lots of tweakables, lots of little options because defaults are good, but it's good to be able to override the defaults. So here I've got a stress test where I want to allocate huge amounts of memory. I wanna lock closed pages into memory if I can. What else is there I'm doing? I wanna, for each page which is allocated, I wanna populate it into memory. So when I do a mem map, populate that physically rather than wait later for it to be set using copy on write. You can put MAdvise options through and you can specify the way that you're stressing the VM stressor, so I've got VM gray there. The virtual memory stressor, you can say, I wanna stress it with different types of ways of writing patterns into memory. There's about 20 or so virtual memory methods that you can use. So you can fill it with gray codes or maybe do zeros or zeros followed by all ones or roll to rotate bits. So there's lots of little options in these individual stress tests. So yeah, that kind of gives you an idea of kind of the flexibility in each individual stress test. And they just say there are quite a few options in StressNG, so the manual's quite big. It's about a 60 page manual. I've tried to detail everything in detail, but yeah, there's a lot of options. So that sits down for, it's quite big. And here we go, if you don't specify a method, the stressor will go through all the methods inside it to exercise, and I'll describe that later. Okay, just a quick break. Here is, I found on YouTube someone had a video of a Raspberry Pi being run with StressNG and it's quite instructive to see how hot it gets and seeing the heat dissipate across the board and it kind of flares up, gets really hot and then cools down, but it's quite a nice little demo. So if you go on YouTube, follow that, you can see someone using StressNG aggressively making something a bit toasty. So as I said earlier, we use StressNG in our lab to do performance measurements to see if performance is regressed. So StressNG has a metrics option and certain stress tests work really well with a very stable measuring throughput of a stress test. So for about, over the 200-odd stress tests, we use about 20 or 25, which are very reliable for measuring throughput. So when we put a new kernel in with new fixes in, say, linear spectra meltdown things where you do a system called boundary crossings, you can actually see the performance drops off and that way you get a good idea of how performance is impacted by certain fixes. And as I say, the kernel team at Ubuntu have this monitored for every kernel which goes out. We constantly look at trends just to make sure there are no outrageous performance regressions going out the door. So in this case, as you see, there's a metrics option here and it will show you the number of BOGO operations done in a stress test time. So a BOGO operation is a bogus way of measuring things. It's like, how can I explain? In a stresser, it does an iteration on a test. So some tests, it might be just doing one system call. Some tests, it might be doing a whole load of system calls and other things. So a BOGO op is one iteration of those tests. So some stress tests, it can do millions a second. Other stress tests, it might do 10 or 15 a second. But it's a known way of exercising something in a measurable way. And from that, we can compute the throughput in terms of BOGO operations based on the real, on the walk clock time, that's the real time. That's time to start the test or finish the test. Or in terms of use time and system time combined. So the two ways of measuring throughput. So stress and G gives you that ability. I can't see it on my screen, so I have to lean back here. Oh, yeah. So with the metrics, there's a YAML option to dump the performance metrics. So you can dump that and then come back later on and look at the tests based on a certain kernel or a certain release of stress and G. So it's a good way of just tracking everything rather than dumping it into a plain ASCII file. It's something which is machine passable. So we can build a database of performance regressions OK, so one of the other things I've been doing over the last few years is looking at performance of the next kernel in terms of low latency or the generic kernel, as we have an Ubuntu. So it's useful to see how the kernels perform in terms of low latency or even real time performance. And because stress and G has got so many different stressors, I thought it'd be useful to basically put a cyclic test in stress and G. So you can say, I want to run the cyclic test against a certain known stress test, and that way you can see the latencies in the way you exercise the kernel in that particular mode. So this is equivalent to the cyclic test in the real time folk use. I think it's just called the cyclic test. It's the same kind of idea. Here, you can say, I want to generate a distribution stats every 250 nanoseconds. I want to use a nanosecond clock timer as my timer to see how the latencies change on that. I want to fire that timer every 20,000 nanoseconds. I want to use the round robin scheduler. And I want to do all of that while exercising the virtual memory with four virtual memory stressors. So you can pick and choose which stress you want, plug it in, and you'll get a set of metrics. So you can see here, 25% of all the tests happened with a latency of 3819 nanoseconds. And the 99.99 percentile, it's 46,000 nanoseconds. So you can see the spread of latencies across a collection of tests there. And again, there's another set of data there showing you it's bucketed everything into 256 nanosecond buckets. And it's a frequency count. So you can see at 3,750 nanoseconds, there were 3,543 time-away cups occurring. So that's kind of the, is that modal? I mean, oh, whatever. You understand what I mean. It gives you a nice distribution graph. And you get lots of stats. And you can see the distribution and where latencies occur. Now, as I said earlier, stress tests have different methods. So here's an example. Here's a made-up example. Sometimes it's good to have the computer science he tests. So this generates a big tree in memory. So it's good for testing memory handling and lookups. But this stress test generates a tree of millions of numbers in. But you can say, actually, I just want to use an AVL tree. It's specifying the tree method. But you could pick from things like splay trees, red-black trees, or your binary tree. So that's what the method option is. It's just another way of doing finer control of a test. And if you don't specify the method, stress and tree will just go through all the methods one by one until the test is finished. So if you just do tree without the tree method, it will do all the four binary trees in round robin approach. And you can run it for like 10 minutes. It's a nice, general kind of feel of how well your memory works with different trees. That's an example. And if you don't know which methods are available, you don't have to go to the manual page. You can just say the stress test method, which is available. And it shows you all the methods which are available. So you can see some stress tests have got lots of different ways of running a stressor. Yeah, so the CPU stressor is the one which has got the most methods. It's got 75 different ways of exercising the CPU. If you don't specify the method, it will go through all those 75 options. And that way, you get a really good mix of CPU stressing. But here, I've got an example where you run the CPU stressor, and it says, do a fast Fourier transform. So those who do signal processing, you know what that's doing, lots of cash hits and lots of floating point operations, multiplies, and in a nice butterfly pattern. So it's a good way of seeing a specific CPU stressor working there. So yeah, now I said earlier, there's a verify mode. This is useful if you want to do something and make sure that the stress tests are actually running correctly and you get sane results. So the CPU stressor, when you put the verify mode on, it will check that every arithmetic operation after doing a loop looks correct. So when you do a Fourier transform, it makes sure that the result is actually sane. Or if you're doing a queen search, it'll make sure that's being computed correctly. Or if you're doing lots of floating point double arithmetic at the end, it will look at the result and say, yeah, that's what I expect. But the verify mode on costs time because it has to do sanity checking. So do not use the verify option when you're doing benchmarking, because it will just skew the results. Yeah, so the bottom there, burning test. I want to see if my CPU is actually doing what it's meant to do. See if Intel's not sold me a dud, for example. Run it for an hour, and it will pop out. If something's wrong, it will tell you that a computation's failed. OK, so that was a very, very whistle-stop quick tour of what stress and G can do. But it can do a lot more. It's kind of mix and max. Try it out with different tests. And it's really the design there to break systems before you ship them. It's designed to stress test things. And hopefully it will find bugs before the system goes out the door. That's my hope. And if you find any weird ways of using stress and G, or you've got some extra tests, or you find bugs, please get in contact and send me patches, because I want to see this used and a successful project. So I've got a quick start guide. There's a kernel project page on open2.com. It's GPR2 license project on GitHub. So you can clone it, start working with it, send me patches, please. I've written a comprehensive manual, which describes every option. It's quite a deep read, so sorry about it being long. But at least everything's documented, because documentation is key. And if you're interested in the type of kernel coverage results, we get running stress and G. I've got some examples there at the bottom link. So hopefully that gives you a flavor of what stress and G does. Thank you very much for being patient and listening to me. And any questions? Thanks a lot. It was very interesting, actually. Nice tool. My question would be, do you have any stress tests, especially for multi-core systems? Like, to stress, how efficiently you may use your multi-core system, and if it works at all? And probably benchmarking that. OK, so you want to see how efficient something is running with a certain spread? Yeah, so either something like, typically we use an embassy multi-bench for that. But do you think something similar could be done with stress and G? So to be quite honest, stress and G hasn't got anything to measure power. But I have in the past run stress and G with a multi-meter test to laptops to see how the Intel shed is working. So I've just used it, the variety of tests which I think should be used to exercise various components and used ways of measuring it external to the machine like a multi-meter. So still, I mean, typically we have now multi-core CPUs. So how can I stress all of them and preferably with some loads which require some IPC between cores? OK, so if you want to do inter-process communication, stress and G has got a whole lot of IPC testing. So it's got pipe stressors, shed memory, semaphores, system five semaphores, and the kernel semaphores, which, what are they called? This case, but there are IPC tests you can run there. But there's nothing specific to say, run something on here, run something here and glue the two together. Thank you. Is there any plan support for stressing the GPU? Yeah, do you know what? I've heard this question a few times and my knowledge of the GPU is zero. So I've not written any stress tests and I've not really gone down that route because I'm a little bit hesitant at writing code which will be working this year, but in two years' time, we might have to be thrown away because APIs have changed or because I haven't written the code correctly. So it's a great idea, but I haven't ventured down that route because it scares me a bit because my knowledge is zero there. But feel free if you want to contribute, please do so. There you are. Okay. How common is to get RCU stalls if I'm really stressing, running StressNG with sequential options in a KMU? Sorry, can you say that again? I didn't quite catch the catch. So how common is to get RCU stalls if I run StressNG in KMU? Okay, so if you run StressNG inside KMU, the general behavior will be like if you're running it on real hardware. So it depends which stressor you're running. Just run all of them, sequential and cyclic. Oh, if you run all of them, it's StressNG you should complete unless your kernel is a bit dodgy or you've misconfigured it or you haven't got swap, it might take a while to get through. Yeah, it takes, that's fine. So it's not very common on, let's say, ARM64 or X86 KMU instances to actually have RCU stalls. It should work because before I ship StressNG, I actually run StressNG on, well, all platforms I mentioned, so BST boxes, a whole lot of systems and they're all done in KMU. Okay. So it should work and it shouldn't break. Okay. But if it does break, that's kind of cool because we found a bug, so. Yeah. Thanks a lot for your work. I've just looked up and StressNG knows how to play with user fault FD. Did you find something weird during building this test? The only weird thing I found was the documentation, which was a little bit esoteric and I couldn't quite get my mind around it. So it did take me a while to get it working in a way which was reliable. And I'm not sure originally if it's because I implemented the test very early and there were possibly bugs or the documentation was lacking at the time or maybe I just didn't understand the things. But as it stands now, it's quite a reliable stress test. But it's one of those APIs I haven't fully covered. It just does a minimal amount of stressing with the API. So it can be expanded. Currently, man on user fault FD is quite good and it even has the example of usage. You just copy from manual, paste and compile and just run it. So it describes it very well right now. That's cool. That's good. Because a lot of the time I implement these by looking at how the kernel expects, looking at the kernel documentation and trying to marry the two together. And what's really fun is when actually run it through coverage, I see gaps where I haven't covered the API. I go, that's really interesting. I didn't know, you could open this device, read, you know, read, write, as well as read-only, simultaneously and you can do this all at once. So I've learned a lot on the way on how this system calls and meant to work or maybe not meant to work. And that way I've abused them as well because if you see how things work, you go, oh, I can actually men map onto that. I didn't know you could do that. What happens if I do that and things break? So sometimes not having the documentation is really good to help dig into understanding how APIs work. But I'm glad there's more documentation on examples, that's good. Thanks again for your work. Oh, my pleasure, yeah. Anyone else? Hi, can StressNG also be used in conjunction with preempt RT? So for example, when does preempt RT end with its performance or whatever? You should be able to run it with that. Try it out and see what happens. I can't answer any more than that. Theoretically it should. Did you ever break real hardware, was it? I've never set anything in light. I've never actually melted any chips. So that's the other thing, don't be scared of running. I've got an expensive board, I'm gonna brick it or I'm gonna set it in fire. StressNG doesn't do that with Intel hardware. When you get to the past the thermal zones and you hit TJ Maxx and the CPU, the CPU will just shut down. You shouldn't see any kind of that funny blue smoke popping out your machine, so. But if you do, let me know, that'll be a win. Okay, how are we doing? There should be no issue unless you're a thermal driver developer. Sorry? There should be no issue with that test unless you're a thermal zone driver developer. Hopefully this shouldn't be. You never know. Okay, thanks, I think we call that a wrap. Thanks very much everyone for listening. Thank you.