 The Linux kernel has a highly active code base. Its mailing list has 30,000 messages a month. There's 6,000 commits a month, which translates to hundreds of commits a day. If we look at just one version of the kernel, 5.13, there's over 13,000 commits in one version alone. This isn't surprising. There's lots and lots of users of the kernel, targets lots of devices, lots of architectures, different software distributions, and there's billions of devices around the world. But all this change means things can go wrong, and so someone's got to test these kernel code changes. It's open source, so there's lots of testers, lots of maintainers, lots of different practices. What we can see here from an LWN article that lists some of the most active bug reporters of a fairly recent version of the kernel. And we can see that some of the most active reporters are test robots, and this makes sense because with so many changes, it makes sense to have robots that are just continuously testing the kernel. And some of the top bug reporters are robots. So a couple of examples, some of the most active reporters are Intel Zero Day. Intel Zero Day is a whole infrastructure of machines that run continuously on Linux Next in about a two-hour patch window. And it runs a suite of tools, compile testing, boot testing, performance testing. Another continuous test infrastructure that uses a slightly different approach is the syspot fuzz tester. And this also runs continuously on the kernel, trying to generate as many inputs and hammer away at subsystems as much as possible. But with all these testing approaches, either robots or humans have in common, is that before they can actually build a kernel for tests, they have to pick a configuration file. And this is because the config file decides what code is gonna be included in the kernel binary. But if the config file doesn't actually cover the changes from a specific commit, well, there's no way that the new updates to the kernel can be tested. The patch submitter and maintainers may have some idea about what config file to use, but automated test tools have a lot harder job for this. And even so, both humans and robots miss bugs due to using the wrong config file under test. And there are studies that show that even really simple bugs like compile time errors get committed to the kernel only to be found by the hapless user who happens to stumble on a broken combination of config options. So an example of this is a memory bug that was introduced by a commit from several years ago. And if you look at the change, look at the code in the commit, the bug is only a present if you turn on this very specific option, this media controller DVB option. Years later, syscaller, the fuzz tester, had the capability to find this bug in mid 2020. But because the config file that the robot was using had this option disabled, no amount of its testing could ever have found this bug. And it wasn't until months later when the maintainers of the test robot were updating the config files for new versions of the kernel that they happened to enable this option based on new defaults that they had. Unrelated to whether this commit happened to need that config file. So by chance, they had the right config file to test this patch. And by the end of the year, syscaller had found and reported the bug, which was backported and was repaired and backported to the stable Linux. And what this tells us is that if we wanna test changes to the kernel, we need the right config file that's actually gonna include the code that was modified by the commit. But in general, it's hard to find the right config file for a specific commit even by hand due to the large complicated build system and all of the indirect dependencies that there are between configuration options. The patch may only have a few dozen lines and these may be controlled by a variety of different combinations of configuration options being turned on or off. So looking again at syscaller and Intel Zero Day, they have different approaches to picking their config files for test. So syscaller has a bunch of hand-crafted configuration files or sets of configuration options that are really tailored for the runtime environment that they're testing for structures on, the subsystems they wanna test, which testing tool they're using. And so they're really focused on having the right configs for their test suite. And Intel Zero Day takes a different approach. So they use the built-in configuration options provided with each kernel version. So this includes the dozens of default configurations for each architecture. They use the all yes and all no configuration options. But most of the configuration files they use are randomly generated, configuration options generated by ran config, generates hundreds of these for each patch window that it runs multiple times a day. So that hopefully it can not only cover more code, but just catch all of the different interactions between configuration options, catch more bugs, catch more performance issues. But typical config file generation strategies don't hit most of the code in commits. So here we looked at the version 5.13 kernel when building for x86. And we measured just how many commits, how many committed patches were covered using the default configuration and also using a randomly generated configuration file. We see that with these configs it's unlikely that we're actually gonna hit a commit by chance. Def config only covered about 15% of the patches, ran config only about 20%. And you can try to generate lots and lots of ran configs but they're not uniformly distributed so you don't have a very good chance of hitting the patch even with lots and lots of ran configs being generated. So you might be thinking, well, why don't you just use all yes config? All yes config tries to turn on as many options as possible so you'd expect that it would cover lots and lots of code. And in fact, it actually does cover lots and lots of the commits, although it doesn't cover everything. The problem is that for a realistic bootable kernel for doing performance and run times testing, it's unlikely you'd wanna use all yes config. First, the binary is large. So it may not even fit on memory restricted devices and it takes a long time to build. So in our tests, three hours versus 20 minutes for a default configuration which cuts into testing time. But the real issue is that all yes config is really designed for compile testing. So if you do all yes config for x86, you're gonna end up having some ARM code being built. And when you do build it, it may not even work for your test infrastructure. So this is why Intel Zero Day and Syspot use other configurations besides all yes config for their runtime testing. They need to match the actual device architecture and system that they're using the kernel in. And they wanna make sure that the specifics of their test infrastructure are met so they're gonna actually boot and run the kernel. And similarly with all mod config, it may be faster to boot, but you kinda kick the can down the road to runtime where you have to figure out what modules you need to load that may contain the code from the commit that you need to test. So the key problem here is there's lots and lots of reasons for testers to select a config file, but these usually don't provide a guarantee that the changes in a commit actually get built for testing. So our solution is called KRepair and it's meant to help automated testing ensure that changes to the kernel are covered. And the way it works is it takes whatever config file, the test automation or the user wants to use, and it also takes the patch file from the commit. It then figures out how to update the .config file so that that updated .config file will actually cover the changes from the patch, but leave as many of the other settings that are needed for testing in place. In other words, instead of testers trying to figure out what's the best way to pick config files for coverage, KRepair tries to take care of that. It diagnoses why a config file doesn't cover a commit and then repairs the config file to do that. And so our hope is that, particularly for automated testing infrastructure, we'll be able to find issues due to changes sooner by covering the code from commits as soon as they're made. So to understand how KRepair works, let's first look at what makes it so hard to just pick the right config file to cover a commit. The build system consists of hundreds of thousands of lines of code, it's really a substantial piece of software in its own right. And it's the build system that controls what code gets included or excluded from the resulting kernel binary. The build system is complicated, code is spread across several languages and it has pretty arcane behavior. But if we want to know how to build the code from a patch, we first need to look at the build system. At a high level, the build system takes a config file and turns it into a kernel binary. And it does this by slicing out a subset of the code that corresponds to the selected options. So let's open the hood on the build system to see the components that control this process and what role they play. The first component is Kconfig. And Kconfig defines dependencies between configuration options. And checks that the config file given by the user is valid. For instance, it makes sure that if you wanna have a USB based file system that you can't build a kernel unless you also include the USB subsystem. And Kconfig has about 150,000 lines of these dependency specifications and it uses those to validate whether the config file is correct. If that passes, if the config file is valid, then the config file goes on to the make files and the make files look at the options and pick which source files to build for the kernel. After that, the Cpre processor uses those options to pick individual lines of code within the source file using if-defs, the preprocessor's ability to do a conditional compilation. And finally, once all this code is selected, the compiler and linker turn this code into object code and link it into the binary. So when it comes to understanding how the build system picks which code to build, we're really only concerned with these three components, Kconfig, Kbuild make files, and the Cpre processor. And so we can think of these as working together to take a config file and selecting out a subset of code to build. And these components are exactly the cause of why a config file may include or miss changes from a commit patch. Let's look at an example of this. So here's a commit from the Linux kernel from a couple of years ago. It's formatted as a patch and it shows which lines are added or removed. For instance, removed lines are in red, added lines are in green. And notice here that on lines five and six, the change lines of code are controlled by a preprocessor if def. What this means is that these lines, this line of code that's changed will only be built into the kernel binary. If this configuration option GIC non-banked happens to be enabled in the config file. So if this is our commit that we want to test then our config file needs to enable this option. But there's a little wrinkle here. Complicating this is that the change that you can see to this line on line 13 is in the else branch of this if def. This actually means there's no single config file that could ever cover all the changes in the commit. We need to have two config files. One with the option enabled, one disabled. Now, if all we had to do was just turn on and off this option, this would not be such a hard challenge. But this option is not the whole story. There's also the make files. And the make files also conditionally include certain source files. So this is the make file for the IRQ GIC file that was modified by this patch. And it's controlled by yet another option, arm GIC. So even if we have the right configuration option for the source code, we still need to have the right configuration option for the make file. If this option is disabled, the file is never included and the code from the patch can never be built. So now that we know that there are two options that control inclusion of this commit. And that's not too bad, but again, this is still not the whole story. Probably the most complicated part are the K config dependencies. And because if we want to turn these options on, it's not enough to just edit the configuration file or go to the user interface, the menu config to say turn these on. And that's because these options depend on a whole host of other options to be enabled or disabled in order for this code to be compiled. So in this case, the two options that we want to enable ARM GIC and GIC non-banked, these are not even user selectable options. They have to be selected by enabling other options. They're selected indirectly. We can see that by the select instruction here. So for instance, if we want ARM GIC to be enabled, we have to first enable ARM GIC PM. If we want to enable ARM GIC PM, this depends on another option, PM, and so on for all the other options that control the code that we want to change. So in short, it's really, really hard just to figure out what configuration options need to be set in general so that your config file covers a commit. So our solution KRepair does this by first taking the commit from the user, looks at the build system code, and figures out what requirements there are to including the lines of the commits patch. Then if the user's config file doesn't already cover the code in the patch, it starts removing the options that are preventing that coverage. And finally, it automatically adds these options back to the config file so that the patches coverage requirements are met. And what this results in is a repaired config file that covers the patch. But we don't want to make huge changes to the config file. We don't want to just have all yes config because of its issues with testing. Testers have their own reasons to generate config files. We want to target these changes to the config file so that the repaired config file remains as close to the original config file as possible while still covering the changes in the patch. And this we believe will enable the most utility from KRepair. So now I'd like to turn this over to my co-presenter, Najeeb Ildaran, who's gonna show a demo of KRepair in action, finding a repaired config file for a particular commit from the Linux kernel. Thank you, Paul. Now let me continue with the demonstration of KRepair. KRepair is a part of the KMEX tool suites which includes a set of other tools that work on Linux's build system. You can easily install it from source or through Python's package manager. You can see the link to our GitHub repository in the terminal or you can check our conclusion slides for the same link. It requires installing some additional dependencies for more precise results for which you can find the directions in our repository at the top readme file. We implement KRepair's algorithm in a tool called KLocalizer. So in today's demonstration, you are going to see me running KLocalizer to demonstrate KRepair. Now let's start with actually running KRepair. First, we need a patch file we would like to repair a configuration file for. Git can easily produce a patch file and we can give a commit change for creating this patch file and one can use this functionality for testing, let's say, the commits from a day, a week, or from even a release of the Linux kernel. But for simplicity today, I'm going to use a patch produced from a single commit. Now first, let's check out to this commit. Then we can use git show to create a patch file for this commit. As you can see, this command produced a patch file in unified diff formats and KRepair can work on the such format of patch files. After obtaining the patch file, then we need a configuration file we would like to repair as it might be the most extreme case I'm going to use ono config here. So what's extreme about ono config is it almost builds nothing which includes the patches that I'm going to use today. One subtle point about KRepair is that the Linux source that KRepair will use needs to be at the exact state when the patch is applied. This is because it analyzes the patched source files as well as the build system files from the Linux source so that it can figure out how to do this repair. Now that we have the configuration file to repair and a patch file we would like to repair the configuration file for, we can now actually run the KRepair tool. So for this, I'm going to use the tool named KLocalizer as I mentioned earlier and to specify the patch file, we can use the include option. Then to specify the configuration file to repair, we can use the repair option and KRepair will run on this patch and configuration file to repair. So what happens under the hood is that KRepair analyzes all components that determine the inclusion of source files and lines. These include kconfig files, makefiles, and cpreprocessor code. It is fairly fast because it does this analysis on demand and it caches as much analysis result as possible. On the other hand, some results will take longer because when it cannot find the results in the cache, but in any case, KRepair will typically take less than five minutes to finish running for a single patch. So as you can see, KRepair finished running very quickly in this case and it's outputted a configuration file that is repaired to cover the patch file. So in this patch, we had a single configuration file which is enough to cover the patch. However, there can also be multiple configuration files which is going to appear in our next example and I'm then going to explain why we are talking about multiple configuration files. Now that we have the output configuration file, now let's check how much Repair happened to make this coverage happen. So I'm going to copy commands that computes the difference between two configuration files. And as you can see from here, this Repair changed less than 300 configuration options which is out of 15,000 configuration options in total. Now that we have the configuration file, the Repair configuration file, we can actually test whether this configuration file actually covers the patch. And for this, we have another tool in our tool set that is called coverage. So let me write this command for checking the coverage. I'm first going to specify the configuration file. Then I'm going to specify the architecture we are using. And then I'm going to specify which patch file I would like to check coverage for. And finally, I'm going to specify some output file. Under the hood, what happens is that this tool builds and checks pre-processed files for determining whether the patch is covered. It is now finished and we can check this coverage report to see the results. And as you can see here, it says that the Repair configuration indeed included every line in this patch, which is good for testing the patch. Now that we are done with this example, let's continue with another patch that is comparatively more challenging. Here I have a patch file that I prepared earlier which modifies mutually exclusive lines of code from different branches of an if-def directive. Now let's try running CarePair the same way as we did. This time, I'm also specifying the architecture I would like to try for repairing. If we don't specify an architecture, CarePair tries each architecture one at a time until it finds a suitable one. This time, different from the first example, as you can see CarePair fails. It says that the constraints are unsatisfiable and it failed to find any configuration to cover the patch. As we saw in the patch earlier, let me print it again. This patch modifies mutually exclusive lines, which means that it is impossible to build the changed lines of code by using a single configuration file. To handle this case, we just need to change some options in CarePair. So earlier we had this command. Instead of using include to include the patch file, we can use include mutex, which tells CarePair that we acknowledge that there might be mutually exclusive lines and we accept the possibility of having multiple configuration output files. In fact, we can always use include mutex because if CarePair can cover the patch with a single configuration file, it will have the identical results as using include. So as we can see here, CarePair started sampling some configuration files and it is sampling two configuration files. This way, these two configuration files handles the different branches of the if that directive that our patch modifies. On top of these options, K-localizer provides a lot of other options for advanced usage. For example, one can also space file files and list of lines with include options. Let's say that we'd like to repair our configuration file for this patch, but we would like to also use this configuration file to test the interaction of this patch with another subsystem. And for this, let's say we would like to compile some certain lines from another file. For this, we can use as many include or include mutex options as we'd like. Here, we can specify list of lines, including range of lines. This way, we can run CarePair again to include this additional file in the build by the repaired configurations. So if you'd like to learn more about K-localizer's advanced usage, you could check the manual page or the documentation we have in our repository. We also have some other tools like coverage or some bug finding tools like Kismet in our repository that have been useful to analyze and do things with the Linux's build system. So as you can see, CarePair finished running and the output configuration files will compile both the patch file and the specific file and lines be specified with the include option. Thank you very much for your kind attention to this demo. So I'd like to thank all of the members of the team who worked on CarePair, Najeeb who's a co-presenter, Jeho, Julian, and Julia for all their hard work on this project. But in conclusion, automated testers need to pick the right config files to boot, test, and run, making sure that these config files cover patches as hard and CarePair automatically edits config files with relatively small changes to ensure patch coverage. This work was supported in part by the National Science Foundation. So I'd like to give my thanks to them. And if you'd like to use this tool and try it out, please visit our GitHub page and provide any issues or feedback. And I hope some of you will find some use out of this tool and I'm happy to take your questions. Thank you.