 Thanks, Christina. So as Christina mentioned, my name is Brendan. I work at Google. And I'm going to be talking about KeyNet testing strategies. I'd also like to introduce two fellow panelists, David and Daniel. They work with me at Google on KeyNet. And they will be assisting me with answering some of your questions. With that, I'll get started. So first off, as the name implies, KeyNet is a unit testing framework. And I'm going to touch on that a little bit more. But this talk came out of, I was hoping to talk about how to write tests in a number of different situations. And there were a lot of different areas that I really originally wanted to cover, some of which including hardware testing and otherwise just testing code with a lot of difficult to handle dependencies. I realized, as I put this talk together, that if I wanted to cover everything that I originally wanted to, this talk would probably be several hours long. So this talk, I'm just going to focus on the basics, on the best practices for writing basic tests, establishing the necessary context for that. And if we have some time at the end, depending on how many questions we get, I'm going to cover some of those other topics, give just sort of a brief overview. And we'll either cover those other topics in either additional Linux Foundation events or we'll maybe post some follow-up videos afterwards. So I mentioned unit testing. I was going to explain what that is. So I've talked about, this topic has been a topic of discussion on the mailing list for a while. And I don't think it's super important for the purpose of this talk to outline exactly what a unit test and integration test are a functional test. Sometimes call an end-to-end test actually is. For the purpose of this talk, we're going to say that an end-to-end test or a functional test is a test that runs in user space and tests the kernel via accessing syscalls or special files like device files and proc files and such. Those tests are very important. I'm in no way saying that those tests are not important. I actually think that any new device file, or especially any syscall call, should absolutely be accompanied by those kinds of tests. However, KUnit focuses on the in-kernel tests where you test kernel APIs, things like data structures inside the Linux kernel, like the linked list, things like that should actually say print K, not print F. And that's what we're going to be focusing on today. So again, the other kinds of tests are important too, but today we're going to be talking about in-kernel tests. So if at any point I misspeak and say unit test, just think in-kernel test. So why are testing internal APIs important? I think that there's two questions that you can ask yourself to decide whether testing an internal API is important or not. The one is, do people use your APIs? If other people do use your APIs, then it would probably be really important to them to know if your APIs change, in which case you should probably test them. And if not, that means there's probably not a lot of people who understand how they work, in which case you should probably write tests to make it easier for other people if they ever do need to go in and change your APIs, that it's easier for them to do so, or if you become unable to maintain your code at some point or there's just too much work, that it's easier for other people to deal with it. The other question you can ask yourself is, do your APIs change a lot? If the answer is yes, then testing helps people understand how the APIs are changing because testing helps document behavior. If your APIs don't change a lot, then you should also still probably test them because people are going to be that much more surprised if they do change, and tests will help reflect that. So a common notion is that tests provide stability, and that's true. But tests also provide other functionality, tests or documentation. When I look at a code base that's well tested, I've in the past worked on a lot of code bases outside of the Linux kernel. And I'll often look when I'm trying to understand how a API works, I'll look at the tests for that API because they often show all of the different edge cases, and they usually have minimal examples of how it works. Plus, I know that if people are regularly running tests, those tests are always going to be up to date. In other words, documentation gets stale, tests don't. Of course, you should keep your documentation up to date as best you can, but there isn't really anything to validate that that is actually the case. As I mentioned before, tests also show how code behaves. And another thing, which is particularly important for internal testing, if you sit down and think about how you would go about trying to make sure that something as big as the kernel was well tested by only using user space APIs, I think you'd quickly realize that there are an insane amount of different possible states that the kernel can be in. And your tests would have to try to manipulate the kernel to get into every single possible state. However, if you test lower down functions, the amount of state within that call stack is much smaller. And the potential state goes up combinatorially with a number of functions in that call stack. So the only way to achieve high levels of coverage realistically is to test internal APIs. So now I'm going to go over a brief how to use KUNA. I'm going to go through this pretty quick. At the end, actually during this, I'll wait till the end. At the end, please ask me questions if there's anything that doesn't make sense. I'm going to walk through some of this stuff really quickly and code afterwards in a live coding session. But I'm going to go through this kind of fast because I'm assuming that everyone here has kind of a basic grasp on building the kernel and stuff. So to start off, KUNA has two parts. I know that sometimes confuses people who are new to KUNA. There's the internal testing library. They're basically the code that lives inside the kernel, which you actually use to write the tests. And then there's also this wrapper tool called KUNA tool, which is written in Python, which basically just helps you build and run and get results. These two things can be used independently from each other. I do know there's people out there who don't like KUNA tool, and so they just write their tests and then they build and run their kernels manually. But there are some people who like the convenience of KUNA tool. So just keep in mind that you're not bound to using KUNA tool. You can still use the other KUNA facilities. Just keep that in mind. That's sometimes somewhat confusing, I think. So if you want to run KUNA tests and you don't really know where to start, a good place to run some tests without having to do anything. If you have a freshly cloned Linux kernel tree, you just type in this command tools testing KUNA. KUNA.py run, and you will build and run tests and get something that looks roughly like this. I think this mostly speaks for itself. It basically says this particular test passed and is part of this test suite. So these are the results you get back from KUNA tool. But if you were to run your test without KUNA tool or if you were to examine the IndieMessage blog, you would see this somewhere in it, which is actually how KUNA gets its results out of the kernel in this semi-structured, semi-human-readable format. We call it KTAP. I'm not going to get into the details of KTAP, but KTAP is the standard test reporting results, I guess, standard for the Linux kernel. So the next thing I probably want to mention is how to configure KUNA. KUNA, just being part of the kernel, is configured like any other part of the kernel of a KUNA config. So here's an example of it. We call it a KUNA config, but really all it is is a min config, where we ran, I forget the command off the time I had, but basically it's a min config where if you pass this into, I forget which make function it does, but it'll generate a full valid config for you. I usually just use KUNA tool to do this. So KUNA tool will take a .KUNA config, this min config, and turn it into a full config for you. So it's just basically a really convenient way for you to just talk about the configs that you care about for your test, make it a little bit more portable for you. But also, if you don't want to use KUNA tool, like I said, you can also just use make to generate a full config for you, for it from this, or you could just copy them into a .KUNA config that you already have. Yeah, I already covered that. All right, so now I'm going to attempt to do a live coding session. Oh, yeah, I mentioned, sorry, before I get to that, do anyone have any questions so far? I think I might have gone through that a little fast. Let me check the, OK, there are, no, it doesn't look like there's any questions. OK, so I'll start the live coding session. Hopefully this works well. All right, so you decide you want to write a test. And for the purposes of this example, I wanted to start off with something that everyone's somewhat familiar with. I realized that when it comes to the Linux kernel, it does so many different things. That's kind of hard to do to come up with something you can assume everyone's familiar with. So we picked something that's pretty easy that hopefully everyone's familiar with. Math functions in particular, we're going to be trying to test the greatest common denominator function. I assume that everyone here knows what greatest common denominator is. If not, I think it's a simple enough concept that in between while I'm talking, you could probably just go over to Wikipedia and look at it, look it up really quickly. But there is also a greatest common denominator function in the Linux kernel. And it just takes two inputs and then returns the greatest common denominator between those two things. So let's say you want to test this function. How would you go about doing that from the very start? Typically, even I and I think David and Daniel probably do this as well. I like to try to find if there is a test that is really similar to the test I'm going to be writing. So that way I can just try to copy that. If there isn't really anything similar, then you can still just go over to there's a inside of lib slash kunit. There is this kunit example test. And this contains some of the boilerplate that you can then use to build a test from. So I'm going to copy that over. And now we need to start, well, I'm going to go ahead and delete all these comments so that way it's easier to see what's going on. I'm going to walk through what I'm doing for each part. So there's a little bit of comments for us. OK. So the most basic part, like the fundamental building block, the most important part of writing a Kana test are the test cases. So here I'm going to call my test case GCD test. I'm going to write some other test cases. But for now, I'm just going to call this one GCD test. Oh, yeah. And I also need the header file for GCD. But anyway, so I guess that's the first thing is this is just normal kernel code. You're calling kernel code for mother things. So it behaves just like the way you would expect any other kernel code to behave. So I'm going to I'm using this cheat sheet. I'm going to copy over this header file. And now I'm going to try to. So I'm going to I find the best way if you're building up to write its test. Start off with just a single test case that is really like basically as simple as possible. It's effectively just like a sanity test. So if you try to think off the top of your head, what's like the simplest base case for GCD? This thing that comes to my head is doing the GCD of 0 and 1 for math function. Putting in 0 and 1 is usually kind of one of the first things you start off with. So let's do that. Put in GCD 0, 1. And of course, the GCD for anyone who doesn't know, the GCD of 0 and 1 is 1. OK, so this is going to be your initial test case. We're going to do some other ones. But first of all, let's get this test case set up. So Keynet has these things called init functions. I'm not going to talk about them too much. But basically, they're supposed to be helpful. The idea is you have test cases grouped together in switch, which I'm going to mention that a little bit. But they can share initialization functions, which will set up state that can be used by each test case. But in this case, GCD is really simple. So we're not going to use that here. So I'm just going to leave that here. OK, so then the next bit of code here, this is basically where you register all of your test cases. So for now, the details of what's going on here aren't super important. Just keep in mind that all the test cases that you write, you need to put them in this array here. And then you wrap the function name in a Keynet case. So that way, I don't mess up and waste it any time. I'm just going to copy that over from here. But basically, you just name the struct something. It doesn't really matter what it is. And then you just copy over the function name of your test case from here. In a test case, the only thing that you have to do with a test case is it just has to have a return type of void and then take this context object pointer to a struct Keynet. Lastly, once you've put all of your test cases in this object, you then need to create a suite. A suite is just a group of related test cases. And it has a name. As I mentioned before, it can have an init function, but we're not going to use that here. It can also have an exit function, which is just the opposite of a init function. It runs after the end of every test case. And then you put the test cases here. It's based, a test suite is literally just a collection of related test cases. And then finally, you register your test suite with the core Keynet, which will then run them using this macro here. All right, so that should get us going. Now, obviously, just like any other kernel code, you have to add it to a make file. So not really anything to talk about here. I hope everyone here already knows what make files are and how that works, so I'm not going to talk about mentioning that any further. But similarly, since we're configuring it as a test that needs to be configured to build in, we need to add a config here. Not too much to mention here. Only a couple of points. One is that if you can try to make your test try state, I know that I like to, for the most part, run my test as built in. If you build your test into the Linux kernel, they'll all just run right before the user space is brought up. However, if for some reason that doesn't work for you or some people just like running their tests as modules, that can be loaded later. You can also configure them as a module and then the test runs just as soon as the module is loaded. For most tests, it won't make a difference whether you run it at startup or whether it's run as a module later. So to be convenient to other users, just try to make it try state if you can. Obviously, if it needs to run before user space is set up or it needs to run after user space is set up, then you can make it module-only or built-in-only. But generally, if you can, try to make it try state. The most interesting thing here is the default can-it-all tests. This is something that was added fairly recently, I think, second half of last year. Basically, if all the dependencies are met for a test, so somebody can specify this can-it-all test and then for any test, which can just be... They don't need any dependencies configured, they can just be turned on, they'll all be enabled. There are some CI systems upstream which use this. So it's pretty handy if you add this in because some of those CI systems will just automatically pick up your tests. And also it's convenient for people who are doing things like building and maintaining distros because if they want to run all their tests on a better associated with a .config to have, they can just turn on can-it-all tests and they'll be able to run all the tests that they care about. So try to do that. And I think everything else here is pretty straightforward. Okay, so hopefully if we did all that right, I should be able to run or to build and run this test now. So I find the easiest way to get a setup, build environment for running can-it-it is just to use the can-it tool. Here I have these additional flags, timeout equals 300, that's basically just if it gives your test a certain amount of time to run. I've not seen this time out in over a year, but I guess it's possible you could be working on a new test and there could be an infinite loop in it or something. So I still add it in because it can be helpful. So this is going to build a .cunit directory which it's just a build directory and by default, it's going to build it as a UML kernel. To be clear, can-it-it is architecture agnostic. You can build can-it for other architectures. Any architecture, at one point last summer I went through every single architecture and every architecture I was able to find, that I was able to build, I was able to find a tool chain to build it, I was able to get can-it-it to run on it. So to be clear, I'm just using UML here because it's fast and convenient, but you can use can-it-it with any architecture. So this is just setting up a build directory, building a UML kernel and building it right now and then it's going to run it instead of the results. And then I'm going to be able to go into this .cunit directory, this build directory, and then I can go and edit the configs that it drops there. Just a convenient way to get started. So it's going to take a couple of seconds more. I can take a couple of questions now while we're waiting. Is there any questions? Doesn't look like it. So, okay, oh, wait. Yeah, okay, let's just... All right, so we can see there are some tests that are configured by default that pass. So now I'm going to go in and I'm going to edit the .config and the can-it-config. So, first off, it added a couple of config entries to my can-it-config that I thought were useful. I don't care about that one, so I'm going to drop it. And okay, we can see that our, the new config we added is here. So I'm just going to add that to this config here. You actually don't have to add it to the can-it-config. You could just turn this on from here and that would work. But this way, if you get into a bad state, you can just delete the .config, or the site of the .config, and it can be regenerated from the can-it-config. So, also if you then want to save this .config somewhere as a mid-config, you can use for configuring your tests. That's also handy. So, useful to add it here, but not strictly necessary. You just need to turn it on somewhere. Another thing to note about this, I've seen this trip some people up in the past, is that if you delete something from your .config, and you're using it to rebuild a .config, if it's still present there, as I think is kind of implied by my previous statement, it'll get built. It'll still build that config. Can-it-tool only verifies that the configs that you put in it are in the .config. It does not care about configs that are not specified. And obviously, if there's conflicting ones or it for some reason can't turn the config on, it'll just complain at you and refuse to do anything. So, it can be handy. Some people find it a thing, but I generally find it helpful. So, I'm gonna rebuild it now that we turned our test on. We can see it's regenerating the .config here as opposed to generating one. So that means that it is not creating a new .config from scratch. So, we can see our test pass. There is a warning here, which I did wanna mention. So, most of our expectations, the expectations that Can-it uses, most of them only work on primitive types. But when they're performing comparisons on primitive types, obviously, if you're performing comparisons on primitive types, which are not the same primitive type, some unexpected behavior can happen sometimes. So, to address that, there is this macro that, it's not, we didn't write this macro, but anyway, there's a macro on the Linux kernel for things like that, which you can use to type check, or it'll make sure that two types are of the same type. And we use that. The warning is a little bit verbose and it's not, if you look at the top line and the bottom line, it's not super clear which one it's talking about. But anyway, if you see this, just look for the type check warning. And it's just warning you that the two types that are being used here don't match. And the reason in this case is because this is a literal, so by default it's just being treated as a signed integer, whereas GCD has the return type of unsigned long. So we can fix this warning just by adding a UL here. If you run it, the warning should go away. So, anyway, I just wanna mention that because there are some people who got confused and annoyed by that in the past, but try to fix your warnings before you submit them, but obviously the test will still run. Cool, so now we have a working test. Now we can add some test cases. I think I'm at the 15 minute mark. So now we can add some test cases, some more interesting test cases. So a good principle to follow when you're trying to add, I think somebody's unmuted. Does somebody wanna ask a question or otherwise can you mute? So a good principle to follow when you're adding new test cases is to try to test all the various edge cases and the various conditions that are interesting for your test case. So you can imagine for a math function, zero and one are obviously interesting inputs typically to a math function. Testing negative inputs might be interesting, but in this case, since it only takes unsigned integers that wouldn't, they'll probably get converted to signed in or to unsigned. So maybe not super interesting to test here. There are other things we might be interested in. So we don't actually have a case right now that tests where it actually finds a common denominator between two numbers which are composite or which are going to share a greatest common denominator greater than one. So we might, that might be a good place to start. So if we add one here, it's really easy to just copy existing test cases and we can call this one composite or something. And let's see if we take the number, the inputs two and four, the GCD should be two. So we can try that. I'm gonna need to add, register that down below now. Okay, another thing might be interesting is testing two numbers which are co-prime so they don't share any, but the greatest common denominator they share would be one. So we can just test two numbers or just pick two numbers which are prime. Why am I, I should have just copied that's what I don't know what I'm doing. Sorry. Anyway, so I'll just pick the numbers like three and five or something. Okay. All right, so let's run these and see if they work. Cool, they do. So I think you can probably see this is going to start getting repetitive if I start testing all of the, if I copy, if I make a new test case for each and every set of parameters I wanna add in. So this kind of gets to the next point of when should you create a new test case and when should you maybe test multiple different things inside of a single test case? The best advice I can give here is treat your test cases as normal functions just like normal functions, your function should be easy to read and that's really the most important thing to follow. Make your test cases as easy to read as possible. And really, especially in test cases you should err on the side of making it even more readable than regular code. And all of your code should obviously be written to a reasonably high quality, but because tests aren't concerned with things about like performance and such, really do everything in your power to make them as readable as possible. So if you find yourself writing a test case where you're doing a million different checks in it and your test case is several hundred lines long, it's probably not a very good test case. You should consider maybe splitting up the different checks if you find that there's groupings of checks that are testing for similar related properties. Another thing is it's okay to use helper functions if you find that you're doing similar setup in different functions or in different test cases, maybe pull some of that logic out into a helper function. Obviously in some cases, you can just use an init function which will set up all the state for you. So yeah, just for the best thing to do is just try to make them as readable as possible. In this case, because these checks are so simple I think it's probably okay that we can combine some of these checks in a single test case. So what we can do here is we can put the parameters that we're putting into the test function that we're putting into the function we're verifying into an array of structs which will group the parameters together and then we can iterate through that struct and run that check multiple times within the same function. So if we have time during the Q and A, I might show off parameterized tests but I'm not gonna talk about that right now instead I'm just going to show how you can do multiple checks, doing this within a single function because it still kind of generalizes to other useful things where parameterized testing might not work super well. So I'm gonna drop these other test cases and I'm going to add in some test parameters. And again, I'm just gonna go to my cheat sheet because I don't wanna get the test definition wrong since we're kind of running low on time now. So copy this over. Okay. And then there are a couple of questions in the question and answer box when you do this. All right, let's see how I see Q and A. I don't know if you can see it and then I'll read them out. Yep, I can see them. You can test zero and zero. Oh yeah, that's a good advice. I'll add that in when we're doing this now. Closing, the closing brackets at the end of the test is that a canary to mark their brand. No, so in this case, yeah, you do need to add that in at the end because since this is added in here, it may not have access to this compilation unit. And I mean, the way that this check is done is yeah, it searches for a null entry at the end. So yeah, you'll get a compile error if you don't add this in here. So yeah, that was a good question. Yeah, did I answer? Yeah, okay, it looks like I answered that person's question. So Brent and I have a question a little bit higher level. So if somebody, a device driver writer or developer is trying to decide to go with the user space test or use K unit, are there any tips on how one would decide which way to go? Yeah, so I think that the first question you should ask yourself is, are you adding a user-visible feature? Like, are you adding a syscall or are you adding a new something like a device file, some kind of, I don't say virtual file is that then comes as regular files as well, but are you adding some new way for the user to interact with the kernel? If that's the case, I think you absolutely should write a user space test. I mean, especially for syscalls, I think that's actually required now, right? I don't know. Yeah, yeah, so in that case, you should absolutely, you know, you're required to someone's gonna yell at you until you need to do that if you don't. So in those cases, you might still want to write internal tests. And I think a good way to look at, to decide whether or not that's appropriate is to just look at your code coverage. If you have a lot of code, it's not realistic to think that you're gonna hit all of the interesting code paths from a user space, just like what I mentioned earlier about the, you have this combinatorial explosion when your stack traces get, your, yeah, your stack traces get deeper and deeper, you're not realistically going to create every single possible interesting combination of input. You're gonna get a lot better coverage if you test some of the code internally, or rather it's easier to get better coverage if you write some of those tests as internal tests where you can call internal APIs directly. Now, in which cases would I only write an internal test? I think if you're only writing an API that is going to be consumed by other internal users, I think it's, in that case, it's kind of obvious that writing internal only tests is probably the most appropriate thing to do. That's a really good question of Shua, is there anything, I think you might have some thoughts on this. Also, David and Daniel, do you have any comments you'd like to add to that? That was the type of thing in the chat, but I'll come on and say that we have seen, there is some benefit to using K-Unit for device driver testing to KITR. We've seen that we've looked at the coverage we get from user space testing, I know which of the all these tests haven't been up to you, so I can't point to them, but they will often lack basically any coverage of any air condition, like, you know, if we fail to do this read or we fail to, you know, underage to this module and stuff like that, for some drivers that never gets code coverage of user space testing, because it's a bit too hard to reliably force that air condition, whereas you could, if you wanted to refactor the code and user of K-Unit and like pass and directly say, like, you know, the most simplest thing you can say, like a Boolean, like pretend that this failed or something like that, you can do something more advanced, which Brendan will probably go into later about interaction, we could trick the code into actually getting the error code and make sure that it cleans up things properly and stuff like that. That's been lacking the primary motive there for people trying to use K-Unit for device drivers so that they can't get that last bit of coverage to verify that they handle errors properly. Yeah, that's a really good point, Daniel, thank you. David? Yeah, one thing I would add is a good sort of rule of thumb is you don't want to have to like write a new kernel module or something to expose things to user space just for testing, if you could just write your test in kernel. So it's a lot nicer if you're testing things that are in kernel to have your test be in kernel rather than have some kernel module that exposes some internal implementation detail to a debug FS file or something that you then peek out from user space to test. So that's one good rule of thumb. The other thing is just think about the level of the thing you're testing. If you're trying to test what a function does in the kernel, then K-Unit and a sort of internal kernel test is probably the right level to be doing that. You can easily test things that are just not visible from the user space. If you're trying to test a user space interface, then having your test in user space makes sense. Hmm. Yeah, that's also really good advice. Shoo, is there anything you'd like to add to that? No, this is complete. And yeah, one comment to make is lock depth. So for example, if you have a locking routine which are internal to kernel, that would make perfect sense to say, hey, let's miss kernel list type of things that you would want to do kernel only test. But if we have a syscall type thing that exposes, it's a good idea to write a user space so we can look for regressions. Definitely, that's helpful. Thank you. And I think maybe we can add a document on all of our thoughts to the kernel, I mean, you know, testing to say when to use what. So think about for us to guide people on. Yeah, absolutely. I think that's actually something I think we might have talked about that before, but yeah, that's something we absolutely should do. Yep, cool. Thank you. Yeah, thank you, great question. So yeah, I'm gonna go ahead and finish up this example here, I'm mostly through it. I'm gonna go ahead and go over and because we've already taken some Q&A questions, so I'm gonna go ahead and finish this example up. But yeah, okay. So I added in these test cases here. So yeah, in this case, obviously we just need to loop over this. So I'm gonna save everyone the effort and just copy this from here. And I think probably everyone at this point is probably convinced that this will probably run. So I won't waste anyone's time with it. But yeah, I wanted to go over some other potentially interesting inputs. I think somebody mentioned a really good example of one. So I wanted to call it out here. Adding one in where there's zero, zero. I actually don't know what'll happen if we put in zero, zero. I'm pretty sure it should be one. I'm pretty sure that's mathematically what should happen. But I've not actually tried that one. So that one might be kind of interesting. And some other things you might wanna consider like, okay, so two and four, like two is actually a multiple of four. So it might be good to try to add some inputs in which are not just multiples of the other number. It's gonna kind of vary on your function. And I think this kind of gets like a deeper point of actually, in this case, I didn't actually open up at any point and look at what the GCD function looks like. I'm just kind of blind testing it. But you should actually open up your code and actually make sure that you're covering all the conditions that your function that your function has. I'm not gonna do it here because it takes a little while to set up and maybe I should have prepared it before this presentation, but a really, oh, oh, huh. Okay, well, actually this kind of gets that another interesting point. So I'm pretty sure that was that first test case that fails. So I'm pretty sure that was that first test case that fails, but looking at the zero message, it's not actually super apparent that that's the case because we can only look at the results here. And obviously there's multiple things that have a result of one. So one thing you should, this kind of motivates my next point of you should try to make a result not just the test cases themselves readable, but when the test case fails, you should make the output readable. And in this case, I think you probably saw below KUNIT expect equal prints out what was passed into it, but it's not able to evaluate this argument and then this argument, but not evaluate the entire thing. Like that's just not really feasible to do. So in instances like this, it can be handy to use the KUNIT expect message variant, which I actually have that here. This KUNIT expect message variant, all it is is it just takes additional parameters at the end of the test case, additional parameters at the end, one required parameter, which is just a print F style format string and then whatever arguments that format string takes. So if we copy that over, I also need the format string and we should get a little bit more interesting of an error now, or I guess a more helpful error message. Okay, so yeah, we see that that failed on the GCD input of GCD zero zero. So that's actually interesting. I suspected, this was really actually not staged in any way. I expected the GCD of zero and zero to be one because I don't think that zero is a, I mean, in most instances, and like the integer math we're doing, I don't think zero is a valid denominator for anything, but evidently the way that the GCD functions written that actually apparently is. So that was a good suggestion. Yeah, so I'm not going to go into this any deeper. I think we've covered a lot of useful territory. I can extend this example better to show parameterized testing if people are interested in that. But I think people probably get kind of the gist of what you should be looking for when you're thinking of test cases to try. So let's see. I did want to go over really quickly what we discussed because I know that was a little somewhat free form. I just want to very quickly review the points I tried to make or hopefully made. So first off, when you're trying to write your first test or you're trying to write a test and get it established, look for a test to copy, places you might be able to look for these. Obviously look in the directory that you're working in. There might be a similar test there. Also look in menu config that can help you discover some relevant tests. If all else fails, you can copy boilerplate out of Kana example test. Get the test suite working. Very first thing you want to do is get a very, very simple test case like as basic as possible that runs the code that you're interested in working. I usually find that writing your very first test case, I don't mean very first test case, period. I mean, every time you're writing a new test suite, the first test case is usually the hardest because that's the one where you're building up all the necessary context in order to get the thing that you're interested in testing working. So just start with a really simple sanity check. Once you get that running, adding additional test cases is usually more or less to some extent copying that first test case and then making it do more interesting things. We have a couple of questions if you want to hear. All right. Excuse me, there we go. Any comments about coverage or measurement for a new test? So I have a slide at the end where I'm going to sort of touch on this, but I don't believe in trying to achieve specific coverage numbers. I have heard as a rule of thumb that coverage of 70% or more, it does not mean you have good coverage, but having coverage below, certainly below 50% is usually a bad sign. That probably means you need more coverage. Probably below 70, it's something you'd want to consider. Above 70, you can actually have really high coverage, like 80% coverage and not and still not have coverage on the appropriate places. And there are, there is sometimes code where trying to get coverage for it is not helpful. Like in some cases you have code which is generated in instances where you should not ever be trying to test generated code. It should be the responsibility of the library maintainers who write the code that generates the code to test that code. So basically the short of the answer is there is no correct coverage number. Looking at coverage is helpful. Looking at coverage, coverage can be a very, very helpful tool, but I wouldn't focus on the numbers. The most valuable part of coverage that I've seen, and this is actually what's in the slide I have, is coverage, things like GCov, they can generate coverage reports where you can actually look at directories and see the coverage number for that directory. And then you can actually dive down into a particular file and it'll actually show you what lines in a function were executed. And that can be really super useful to try to decide whether you've actually covered all of the interesting paths and getting back to what Daniel mentioned before, making sure you've covered those error paths. Because there's a lot of times there's really common error paths, but they're hard to hit over the course of normal testing because if you don't control the code on the other side, then it might be hard to make that error condition come up when you want it to. So yeah, coverage is useful. The coverage tools are extremely useful, but I would focus on looking and making sure you've covered the interesting test cases. The coverage numbers are, they can be helpful-ish, but don't get to, you know, oh, I have a high coverage number, it's good. Oh, I have a low coverage number, it's bad. It's more complicated than that. Okay, next question. Does KUnit run tests associated with LKM? Currently, I do the tests and run the module as a driver and check on UML execution, but if I could use run as an LKM, I take it LKM. Sorry, is that Linux kernel modules? Sorry, Marcus, if you wouldn't mind posting the chat. Do you mean Linux kernel modules? Or do you mean something else? I'll just wait until Marcus clarifies that. But... Yeah, sorry. Yeah, Linux kernel module. Okay, Linux kernel modules. Yeah, I think I mentioned earlier in my talk, but I did kind of just briefly gloss over it. KUnit can run as a module. So all you have to do is you can just in your config file specify that you'd like your test to build as a module, and then you can load it when you have everything set up the way you would like it, and it'll just run the test as soon as the module is loaded. If your test then, if you then put it in a position where it depends on running as a module, you do want to, in the config file, make it module only, which is something you should generally try to avoid if possible, but if your use case demands that then, yeah, you can make it a module only test. And then you can get your environment set up and then you can load the KUnit module once your everything's set up and then it'll run as soon as the module's loaded. Yep. So one thing I'd add, sorry, Brennan, one thing I'd add quickly to that is that the KUnit tooling does, so the KUnit tool Python script there will not load modules for you to run KUnit tests. You have to use, you know, make menu config or whatever to enable your test as a module. And then what you can do is use KUnit tool, KUnit, you can run KUnit.py pars to parse the output that you get from manually mod probing that module and then you'll get the nice tooling output from that. Yeah, if you run it, load a kernel module with KUnit tests in it, the tests will run. The output will be put to the kernel log. If you've enabled it, it'll also be put to debug FS and you can use KUnit.py pars to get the nice colored test output from that. Or you can just read that a KTAP formatted output manually if you prefer or put that into some test system you have. Yeah, yeah, that's a good point. Yeah, this is something we've actually been debating recently and we're trying to think of different way, like what's the easiest way to set that up? So maybe this is a discussion we should move to the mailing list, but yeah, currently you just have to manually do that. So that's good information. Thank you. I'm almost at the top of that so I can, I guess this is the last question that you want to take, maybe. Okay, all right. So this last question then. So I want to let Daniel address this because this is kind of, this is definitely mostly within Daniel's house. Sorry for putting it on the spot, Daniel. Yeah. So currently Kate is not having support for mocking anyway. We did send an RSC and actually my seller who's in the chat helped us out with that. But it is stalled because we haven't found a test case to use it. We were trying to focus on basically off-structs, faking design, adding like, you know, helpers to say, create a function with it, can you say expect call with this integer value and then turn this thing. So that's sort of there in the works, it's just stalled. So at the moment there is nothing specifically to support that. We actually would like to encourage people and actually clarify that we're using the term mocked a bit more so I suppose there's a big difference between a mock and a fake. But some people are using it changeably but in our context when we're talking about Box and Kate and we're referring to things that do not know how to return values on their own basically. You have to tell them, give them this input or turn this output and stuff like that. And we'd like to avoid people using it because it's going to be useful. But yeah, so support doesn't exist. We don't really have a timeline for it. We would like to get it in but it's just it's been hard to find something that satisfies everyone's use case because some people can pay the points overhead of using an off-struct or add just you know, a function pointer from the direction so we can change out what the implementation is. Some people don't. And in the original version of Kate that Brenton proposed the Alpha Master branch that's called now we actually used think-time hacks basically to step out functions. But this is potentially already complicated and it will add more magic to your build and test system that most people probably don't want to pay but it would avoid one time overhead that some people have told us they want to do so it's hard for us to reconcile the trade-offs here. So I guess the it depends on what you want to do you can always do stuff yourself at the moment. If you can use off-structs yourself I would recommend making a fake off-struct that has, you know, the method you call. If you can't, you'd probably have to rely on using some sort of compile-time direction you know, doing if-depths on that and just have like some magic, you know, config value for your test and say like if this is defined instead call my test function which I'll only define in the test otherwise it doesn't get built and it's not compiled at all under normal builds. The answer to your question is a bit ranting sorry. I should probably write something that's a bit more concise but similar we just haven't got onto doing that. Yeah, so I think that that was a good answer, Daniel but I just want to mention here that this was kind of one of the special topics I was thinking if we didn't get a lot of questions we didn't spend a lot of time on it so I'm going to do a summary briefly and this is one of the this is probably one of the main points that I decided it's so in-depth there's so many things to cover that it should probably go into a different talk. There are definitely things that you can still do with Canyon in its current setup it's not like built any kind of built-in mocking feature but there's a lot of different strategies that you can do to achieve your goals. Daniel actually wrote up this really good document we'll need to we'll need to share but yeah, this is probably something watch out for more information from it because we'll either do a follow-up talk here or we'll just post a video to YouTube or something like that so sorry I know that's not a sorry our answer can't be more complete Thomas but it's a it's a big topic so it was an enforcement a little bit too much for here I got the timing wrong you still have half out almost so yeah okay good yeah so awesome yeah so sorry about that Thomas but I guess you'll have to stick keep you keep posted and I think that once I finish up with the review the overview maybe that seems like a good topic to dive into as an overview of of alternatives that we can do so stay tuned and we'll get to that one final note that is what we don't have is and what appears to really be impossible to develop is a single overarching mocking implementation within KU that solves every purpose and what you really need to look for is what actually can't you do the way your code is laid out currently and it may be oh I need to have a fake you know block device structure or I need to refactor my driver a little bit to introduce some function pointer based interface between you know the code I'm testing and the actual hardware that I can then intercept so in a lot of cases you know a bunch of C macros that a K unit can provide is not really what you're looking for anyway you know if there's some way you can can refactor your code or develop some you know fake implementation or something that's probably going to be better than any generic mocking system and what you need and what we hope to make easier and provide better guidance around helpers for those cases where what you really need is I need to find some way of intercepting calls to a particular function but most of the time you can actually do that or what you're actually wanting to do much more easily by just just implementing a function pointer based interface to something or putting an if def somewhere you know a generic mocking frameworks not actually what we've discovered having tried to implement this over and over again is what everyone actually really needs yeah well said well said thank you David okay so I'm going to go ahead and finish up the review of the different things from from that and then we'll go back to Q&A and depending on what questions how many questions we'll get we can dive into some of those other topics like the yeah so okay it's already covered getting your suite working so once it comes down to writing test cases once you have your you know you have your initial one you're adding some more in there's I've heard the saying in the past test first test middle test last oops I should probably be presenting the slide basically you know test short inputs test long inputs test medium length inputs try to try to cover all of the you know interesting cases try giving your function invalid input try to exercise those error paths in your functions and then kind of going back to that question that I forget who asked it was but going back to one of the questions is code coverage is very helpful it can help you look at your function and make sure that you've covered all of the various interesting conditions in your test again like you want to make sure that you covered those error conditions sometimes the easiest way to make sure you've done that is to run coverage when you're running a test and then look at the coverage report and see did you actually execute all the error paths over the course of all of your different tests so that's the code coverage again it's I wouldn't focus on a particular number but it still can be a very valuable tool and I think probably the most important thing to do when you're writing test cases is to make them useful such that they cover an interesting variety of inputs but probably the second most important thing is make your test readable the most important thing about good test cases is that it's easy to understand what the test cases are doing so that way people can improve them and modify them over time when people get an error it should be easy for them to understand what they did wrong or what went wrong so with that there are no hard and fast rules just like there's no hard and fast rules what your best the best way to write a function is really super long several hundred line test cases are really bad just like several hundred line functions are bad you know having a McCabe's complexity of like 50 is going to be really bad and generally with test cases you even want to keep your McCabe's complexity even lower there's usually not a lot of reason to have if statements and stuff in your tests so try to avoid those but also having over a repetitive test cases can be bad too like we saw in our example so in something like that consider using parameterized testing or if that doesn't work like I did in the demo you might consider trying to create a for loop that loops over some different inputs in different cases but generally speaking just follow the coding practices so I guess we've already been taking questions let's see if we have any new ones is there anything anyone wants to ask before we dive into I'm going to give an overview like I said Daniel wrote up this really nice doc on what to do I'm going to say it's not a doc on how to do mocking per se but it's a document that kind of covers like what you can do when you want to do mocking so if I don't see any questions I'll just go ahead and dive into that another topic we can cover is on managing state and as I mentioned before another thing we can also talk about is I can show how to convert the tests that I wrote in the coding session into a parameterized test so once we get done with the mocking stuff we can go into one of those if people are interested so actually I'm just going to go ahead and present the doc that Daniel wrote I already looked it over and made sure there wasn't anything that is not safe to share but and Daniel go ahead if you want to dive in if at any point I'm talking about things and you have more that you want to add feel free so okay so the first thing which I know Daniel mentioned when he was giving his response is runtime indirection he mentioned specifically you can use that so a lot of times you'll have a function like I think the person who asked the question asked in the context of hardware I might be mistaken I might be conflating that with something else but a really common use case is it has to interact with hardware in some way so one thing you can do is you can if the performance is not super, super critical you can wrap all of the code that talks directly to hardware like make those functions as simple as possible and then you can take those functions and then rather than call them directly in your the function that you're interested in testing you can instead pass in a function pointer to what would typically be a code that the hardware accessing function and then you call the function pointer from inside the test like obviously if you have multiple functions here you'd probably put this like Daniel said inside of an abstract but this is just kind of to try to be bare bones like minimal concept here so then when the code is functioning normally send data funk function that maybe all it calls is like some sort of like right long into a register or something something that's very easily visually validated and then when you're testing this would get replaced or maybe called directly by the testing function and instead it would pass in some kind of dummy function some sort of fake function that either doesn't do anything or maybe it's a mock and it just pretends to do what you expect the hardware to do in that case and this is actually a really powerful idea you can actually a lot of hardware behavior is actually fairly like the way it communicates with the driver a lot of the the communication is actually like pretty simplistic and so you can actually build up a state I mean behind an obstruct and then simulate hardware like errors coming from hardware and things like that so obstruct is one way and actually with the class mocking thing the class mocking that we implemented there's a RFC out on the list we sent it out a long time ago and like David and Daniel mentioned we might do something with it, we might not but that's basically all the class mocking really does it kind of just helps you in these scenarios where there already is an obstruct it just helps you build up that obstruct but there's really no reason you can't just do that without KUnit or without the mocking functionality so I think I kind of already covered most of these pros I mean this is a pattern actually one thing I should mention this isn't really new this is a pattern that's heavily used by a lot of drivers in the Linux kernel so there might be some people who are a little concerned about performance but there's already a lot of places where the kernel does this there's definitely some places where you won't want to do this but most of the time I'm gonna say like 95% of the time it's fine and that's really the right way to do it so getting into the cons well like I said 95% of the time that's the correct thing to do but 5% of the time it isn't the there are instances where the runtime overhead of calling of an indirect call is going to be too expensive in which case you're gonna have to do something more complicated again this is like it's probably not even 5% of the time it's probably like 1% of the time so you know really think hard whether your code is actually that performance critical it probably isn't but there will be instances where it is another thing I think is kind of clear from this example here is that there's a lot of boilerplate in doing that class mocking if we were to write a library for it might help cut down on it a little bit but honestly there's gonna be a lot of boilerplate no matter what so jump in super quickly on the performance critical note it's also worth noting that there's a difference between something being performance critical in production and performance critical while you're running your tests so that's true my code is really performance critical in a lot of cases you can just if deaf out you know have the indirection only exist when you're testing in that case it's not as much of an issue you're not going to be benchmarking your unit tests necessarily that's a very good point that's kind of a hybrid approach between there's actually another thing that Daniel mentioned here in the stock which is covers sort of like the other side of that hybrid but I'll point it out when I talk about that but yeah that's kind of what David's talking about is a really good idea because it's kind of like the best of both worlds between trying to redirect at compile time and trying to redirect at runtime so yeah good point is there anything else you wanted to add you turned your mic off so I'm guessing now okay so another thing you can do so I mentioned compile time there's also link time that Daniel pointed out here this is actually another place where not the RFC that we have out now but in the K-Unit Alpha Master if anyone's familiar with that which is based on the version I think the first RFC of K-Unit that I sent to the Linux kernel it made use of this and that was not super popular as I wrote it but nevertheless you can still use the strategy and employed here and this is really helpful when you have a function which for some for some reason you can't just you can't do in direction and have that function called you can't have that function passed in as like an abstract or a pointer or something what you can do is you can make it a weak symbol and for people who aren't familiar with weak symbols weak symbols are basically symbols where weak is a flag basically a flag for the compiler to tell it that it's a weak symbol by telling the compiler that you're basically saying if you see this and this is the only symbol that you see use this function definition however if you see another if this function is defined somewhere else without the weak symbol use that one instead so it basically provides you a way to depending on whether the other symbol is visible or not and that can be controlled by just using macros or something it'll control which function definition is used so basically in the context of a test environment like when you're running test you can configure the macros or can't configure in such a way that the normal definition which accesses hardware could be replaced by some kind of mock or fake definition which doesn't to make it more test friendly and yeah so you can see here you have your weak definition which is the real definition and then in a test file this test file would only get built in if you're building in the test and presumably in that case you wouldn't care about the real implementation so this would end up getting linked in and called in all of these call sites instead of the original function so that's one option um yeah um this does carry some pretty significant drawbacks in the case of the doing the runtime indirection with an obstrect or pointers that has the advantage that you can really can easily control the scope of doing that so you're basically saying I want this function to call these fake functions instead of the other ones but all of the other code in the kernel will remain unaffected however because you're letting the linker do this and more importantly the linker is going to replace all of these call sites and make them all point here instead of the real original definition so basically this is something that wouldn't work with something like kmalik which I don't know that anyone had ever tried to mock out kmalik but argument's like let's say somebody tried to do that you wouldn't almost know code in the kernel including a lot of code in kunit wouldn't work if you tried to do that because there's a lot of stuff like a lot of infrastructure like very basic infrastructure in the kernel so this basically the big problem here is it has a really large blast radius so it's very powerful but it also has a lot of drawbacks to it do we have any questions no I think another thing you can do which also has a very large blast radius is using ftrace I'm not we don't really actually this kind of more or less those which are gives an example but ftrace is a cool feature where you can take a live kernel and at runtime you can patch in function behavior this is totally unassisted like you can use this separately from kunit now this is very powerful and it does have the benefit that you can sort of turn on these fakes and mocks only when you're ready to use them in a setup kernel but again despite the fact that in terms of from a temporal standpoint it has a smaller blast radius than doing the link time thing it still is going to change every single call site in the kernel so in that sense I guess from a spatial standpoint affecting the entire binary it has a very significant blast radius and thus has the same potential issues as the link time variant and also ftrace when I've looked at ftrace it's not I don't find it particularly intuitive to use it's complicated and it's also very architecture dependent so that's another major drawback to ftrace it's definitely not as easy as the other ones so the last one which is kind of what David sort of touched on compile time and direction well I mentioned his was kind of a hybrid approach between compile time and direction and runtime what compile time and direction is is basically you can add some macros in and sprinkle them in the functions that you are you know causing testing issues that make it difficult to test the code maybe because they communicate with hardware or something and then the definition of these macros can change depending on whether it's configured for a test or not now again this does have the limitation same limitations as everything except the runtime variant and that these macros are going to get changed everywhere from when you compile the kernel all of the associated call sites will change but it is a little bit more flexible than link time because rather than forcing you to use a particular function which of course in the other case you could wrap the function that you're interested in and yet another function so that way you can kind of limit the scope to just being the code under test here I think it's a little bit easier to do that in some regards and it's kind of easier to drop things in where like depending on some actual information you can rather than just call a function you can actually have it return from the function and basically prevent it from doing a hardware access or something like that and like David mentioned you can combine this with the runtime in direction with using something like an obstruct where the call site might be when it's compiled without testing it is a a fixed call that calls directly to some function but in the case of test when it's running the test context then it calls the abstract and in which case it could choose which one it does so that gives you kind of the benefit of it has the performance of it has no performance penalty but it has the benefit that it's still very heavily limits the blast radius when you're using it in testing because you still are able to control it with the abstract in the test environment so that's kind of an interesting approach to doing it oh yeah if you skip to the very bottom of the dock and you'll see examples of this sorry I didn't quite catch that if you skip to the very bottom of the dock like the very last section on the last page there's examples of what that is more practically about the idea you can have a wrapper that you only have for the test if you scroll down more there's an example of how you decay specifically to avoid changing community things which is the heads up only a few minutes left on the meeting sorry I lost track of time so we do have a question on Q&A are these examples available someplace no unfortunately we need to send those out so we'll follow this up with sending some of those examples out so sorry about that okay so I guess we only have a couple minutes left probably not enough time to cover anything else in detail are there any questions that anyone has is there any comments that any of the other panelists would like to make on anything that's been said so far I see some new messages okay yeah 0 is the correct result by the way the GCD 0 is oh good to know okay so the function is not wrong I didn't suspect it was sorry she was saying something I was just saying thank you David and Daniel this is great thank you for having us questions I just want to say thank you so much to Brendan for his time today and thank you to all the participants who joined us as a reminder this recording will be on the Linux Foundation YouTube page later today and the slides will be on our website as well we hope you're able to join us for future mentorship sessions have a wonderful day thanks everyone