 The Linux kernel works on lots and lots of devices, billions around the world. Everything from routers, refrigerators, and mobile phones, all the way up to laptops, desktops, and data centers. And what facilitates this reuse is the kernel's enormous automated build and configuration system. If you've ever had the pleasure of configuring your kernel to build, you might have seen this menu-based system that allows you to explore all the different options you have for configuring your kernel. For instance, if you wanted to have USB file system support, you would need to go in and make sure that you have USB support turned on first. Of course, most people just use an existing configuration file or just have no need for building their own kernel. And while this reusability is a great boon for computing infrastructure in general, for maintainers, this configurability adds additional challenges. For instance, if a maintainer gets a patch from a developer and wants to make sure that that code is tested in particular or wants to make sure that that code is compatible with all the different ways that that kernel can be configured, well, it's not so easy to do that. There's lots and lots of configuration options, lots of possible configuration files. And developers will typically submit a .config file. But this possibility for mutual exclusion and indirect dependencies of configurations on each other makes maintenance a challenge. There's a couple ways currently of dealing with this extreme configurability. There's the all yes or all mod config that attempts to turn on every possible configuration option. But this can never actually turn on all the code, particularly across architectures, just because of mutual exclusion between options. Intel has its zero day testing infrastructure and this will just build lots and lots of predefined configuration files, but also just generate lots and lots of random configurations to hopefully try to find that one configuration that is broken in the kernel. And as you can imagine, this takes lots and lots of energy and lots and lots of time because there are just way too many possible configurations to test all of them. In fact, there's about 15,000 different configuration options and just an enormous number of ways to combine these. Now, a lot of these can be turned on at the same time, but not all because there's just mutual exclusion between these options. And trying to figure out how changes in code or how bugs are influenced by what configuration options turn them on requires a maintainer to try to just trace through all the possible dependencies that these configuration options have on each other. If we look at, say, mobile device manufacturers, they'll often just fork a kernel, tweak it, try to get it working on the one configuration file that they need and may not even bother mainlining their changes. And of course, there's lots and lots of reasons for this, but one issue is the trying to make sure that your changes work in the context of the entire kernel's configuration isn't always necessarily very easy. Now, years ago, the kernel developers created a new language and configuration specification system called Kconfig. And this was intended to help developers define and manage all of the possible configuration options that are available to the kernel, but also make sure that they can define all of the constraints around using these different options so that people using the kernel won't end up building a broken or invalid version of the kernel. And it turns out there's just a ton of code in the current Linux tree for describing these options. There's about 150,000 lines of this Kconfig language. And they define both a mix of dependency specifications and configuration option definitions and also text and prompts for that Linux configuration menu system. To see what makes working with this building configuration system so tricky, let's open the hood on the build system and take a look inside. So first, we've got this Kconfig configuration language, which contains all of the definitions of the configuration options that are available for selection as well as constraints among them. And when a user has a configuration file that they want to use to build their kernel, they first pass this in to the Kconfig system, which makes sure that the options they've selected are valid and will hopefully build a valid kernel. Then Kconfig will pass along these option selections to the build system, which will pick which source files to use. And then to the C preprocessor, which will slice out individual lines of that source code. And finally, they'll hit the compiler and link here to build the kernel binary. And our whole goal to helping maintainers work and trace through these steps of the build system is to use automated tooling that can help automate some of these challenges. For instance, figuring out which code depends on which configuration options, automatically generating configuration options that will test changes to the code. Or in the case that we'll see today, find some kinds of bugs that are in the build system itself that are specific to certain combinations of options. And our focus today is on the Kconfig part of this problem. How can we develop a tool that can automatically work with these dependencies and answer questions about its behavior? And since the configuration system is so massive, and there's just so many possible configurations, our goal is to try to do this statically just by analyzing the source code of Kconfig. And so as a kind of organizing challenge for getting this kind of analysis right, we also worked on a bug finder for a specific bug in Kconfig called an unmet dependency bug. It's not a show stopping bug. It just means that there's some inadvertently conflicting dependencies that we can actually find automatically using the tool. So let's first take a look at how this Kconfig language and system works. Here's an example of a configuration option defined in a recent version of the Linux kernel. And it first declares the name of the option, in this case touchscreen ADC. It declares it to be a particular type, in this case Tristate. And Tristate just means it's either turned on, turned off, or turned on to be included as a loadable kernel module at runtime. There's also a prompt, which contains a textual description for the user of, say, MenuKconfig or a command line based tool. There are dependencies. So it depends on says that in order for touchscreen ADC to be enabled, we also have to turn on these other options. They also have to be on in order for us to even be able to enable this option. And conversely, turning on touchscreen ADC can force other options to be on. So if touchscreen ADC is turned on, this IIO buffer CB is just automatically turned on. And this is useful for defining, say, configuration sets for a particular architecture or a particular chip. And all three of these aspects of the language actually work together to define the constraints on this option. Even the prompt itself defines some differences in behavior on how this configuration option is treated, because you can have invisible configuration options that the user is not allowed to select but get automatically selected by the select construct. This undement dependency bug actually comes from the K config manual itself. It says that this select construct should be used with care. Select will force a symbol to a value without visiting the dependencies. What that means is that an incorrectly written specification can actually cause configuration dependencies to be circumvented, potentially leading to a broken kernel build. And this is the bug that we challenged our analysis techniques to try to find automatically in K config specifications. To see an example of one of these unmet dependency bugs, here's this touchscreen ADC configuration option with a couple of other options that it has interdependencies with. So first note that this configuration option called IIO buffer CB, it has a dependency on two other options, IIO and IIO buffer, which means that IIO buffer CB shouldn't be able to be enabled unless its dependencies are met. But at the same time, whenever touchscreen ADC gets enabled, the IIO buffer CB gets force enabled, regardless of whether its dependencies are met. And the thing is is that just looking at this is not enough to tell us whether there actually is a potential for an unmet dependency, because there could be some other dependencies on IIO and touchscreen IDC that ensure that that force enabling can never happen. So the general rule for determining about whether an unmet dependency exists is if it's possible to disable the direct dependencies of an option while also enabling this select construct, what are called reverse dependencies. If both of those can happen at the same time, then it's possible to configure the kernel with this broken specification. OK, so let's take a look at how we tackle this challenge of finding these unmet dependency bugs automatically. Our approach to solving this is to take this question of, are there unmet dependency bugs in this K-config source code into a classic Boolean satisfiability problem? Because if we can do that, then we can leverage these existing solvers that can tell us an answer to Boolean satisfiability problems. So our first step to doing this is to model the K-config language as symbolic Boolean logic. And the way we do this is we turn, for instance, the dependencies that Touchscreen ADC has on other options into its equivalent Boolean formula. In this case, Touchscreen ADC can only be enabled if its direct dependencies, IIO and input touchscreen are also turned on. So we model that in logic by saying that, well, if Touchscreen ADC is turned on, that implies that IIO and input touchscreen must have also been turned on. Similarly, we do the same for IIO buffer. If IIO buffer was enabled, that implies that IIO must also have been enabled. But when we get to an option that has one of these select reverse dependencies, it's slightly different. We have to model the fact that it can be enabled both when its direct dependencies are enabled, but also it can be enabled if its reverse dependencies are enabled. So we model this by saying that if IIO buffer CB was enabled, then that must have meant that either IIO buffer and IIO buffer were both enabled, its direct dependencies, or its reverse dependency, Touchscreen ADC, which has a select word. And both could also be true as well. Once we have all of these formulas modeled for the whole K config specification because these dependencies can cross over any possible files, we then take every combination of options that have a select reverse dependency relationship and turn them into a formula that describes that condition of the unmet dependency. So in this case, the formula is getting much larger, but the idea is that if the direct dependencies are turned off, IIO buffer CB and not its direct dependencies, but its reverse dependency, Touchscreen ADC, is turned on and all the direct dependencies it has, then if that formula is can ever be true, then we have what potentially is one of these unmet dependency bugs. So here's the architecture of our tooling. And it's actually a bunch of separate reusable components that we have. We first actually took the K config language parser from the Linux kernel and then used it to kind of simplify some of the syntax. We pass that into a tool called K clause, which converts all of the syntax into these logical formulas. And then the Kismet tool will take all combinations of options that have a reverse dependency relationship and construct this verification condition to check whether there is potentially an unmet dependency bug. And then finally, we'll use an off the shelf solver to answer that satisfiability question and say, is it possible in some configuration for this unmet dependency to happen? And then after that, we'll actually generate a .config file that we can use to test against the actual build system to see if we got it right, if the tool was correct in its alarm. Let's take a look at how we evaluated our tooling. So we took a somewhat recent version of the kernel. It's a little old at this point. Looked at all 28 architecture families because each actually has their own entry point to a K config specification, though most of the dependencies are actually shared between them. And then we ran our Kismet bug finder on each of the individual 28 architectures K config specifications. And anytime if we got duplicate alarms, we just do we do duplicate at those. So first, we evaluated the precision, how many true positives we got from this and how well that compared to any false positives. And so we found 151 true positive unmet dependency alarms that we confirmed by generating test configuration files automatically. And it turned out that we got this with 100% precision, which means our model was really, really accurate for the K config language. But precision doesn't mean it's not the whole story for accuracy. So we modeled Boolean and Tri-State options, but we did some tricks to make non-Boolean stuff faster. We didn't model it completely or soundly. And so that meant we were open to false negatives. And so we also evaluated that by seeing if we could find any negatives that Kismet missed. One hard part about this is we don't really know the ground truth. We were dealing with 150,000 lines of K config. We don't really know how many unmet dependencies were there. So we generated and tested around 11 million random configuration files using the RAND config tool and actually found that there were eight positives from that process not found by Kismet. And we don't know what the ground truth is, but we were heartened to see that Kismet found many, many more than this random process and missed eight compared to the random approach. So in terms of performance, each K config specification for each architecture took between 37 and 90 minutes or so to run and analyze between 10 and 12 or so 1,000 select constructs. And so doing all of these 28 architectures takes a little less than a day, but if you can parallelize this, then you can get this done in an hour and a half. And the kind of main takeaway here is that compared to random testing for an hour of doing this, even in parallel, 28 parallel threads, we found many more of these unmet dependency bugs. And so we get this recall and precision trade-off for this under-approximation that we had. And it's potentially a useful complement to run config for these kinds of bugs. We submitted some of these bugs as patches to the kernel maintainers. We haven't done all of them yet because we don't want to automatically submit stuff. We actually try to make bug reports and talk with the maintainers and make sure that this actually is a bug. All so far have been confirmed as real. Many have been mainlined. So now I'd like to turn it over to my PhD student, Najeeb, to walk you through actually using this tooling. So take it away, Najeeb. Now let's move on to the demonstration of Kismet. I will now use the Linux version that we used for our earlier experiments so that I can walk you through some bugs we actually patched. Kismet is a verification-based static analysis tool to find unmet dependency bugs. Given an architecture, it will analyze the select constructs in the K config specification and create detailed reports. It will also create test cases that exercise these bugs and it will do validation based on these test cases. To run, we can simply pass the name of the architecture to analyze. For a single architecture, it will take from 30 to 90 minutes to finish. And for all architectures, it will take about a day, but this procedure can easily be paralyzed. Since it will take long, let's continue with a simpler case and analyze a single construct. In addition to the architecture, we can specify which select constructs we would like to analyze. And for our case, let's pass one. When run, Kismet will check this construct in several phases for optimization. If this is a bug, it will generate a test case and validate the bug. And after the analysis finishes, we will have a TXT file containing the aggregated results. Here we can see the contents. And this file will include the performance measurements and the information on each pass done throughout the analysis. As you can see, we will have a few different passes and we will have the results for these passes. And we will also have a CSV file containing the analysis results for each select construct. In this file, since we tested a single select construct, we have a single line of results. And the results say that there is an unmet alarm, which means there is an unmet dependency bug for this select construct. Also, it will include the path to the test case, which we can use to replicate the bug. Let's now copy the test case. And let's copy this config file and try to run this using the build system. When we run the test case, Kbuild will warn about the unmet dependency. Here, you can see the unmet dependency bug warning. And when we compile, we will end up with a build error for this specific unmet dependency. To speed up, let's compile this in a parallelized manner. So we actually see some error messages, but we miss them because of the parallelization. Let's run once more without parallelization so that we can see the error message more clearly. As you can see, there is a compiler error. And when we further investigated this build error, we found out that the unmet dependency bug was the root cause. And this was also confirmed by the Linux maintainers. And our fixing patch is merged into the main line kernel. In our experiments for the specific Linux kernel version, which is 544, we found that about 30% of the unmet dependency bugs caused build errors. So far, we sent 38 reports or patches. And 19 of these reports or patches were confirmed by the Linux maintainers. And the remainders are pending for approval. Out of these, 15 patches were committed to the Linux kernel call base. Since this is a manual process requiring discussions with kernel maintainers, we are still investigating, reporting, and submitting patches for the alarms we found. Thank you, Najeeb. And that concludes our talk. I just want to give a shout out to all the students who worked on this, including Najeeb and Jeho and Julian, who helped get this stuff solid and evaluated. In conclusion, configurability of the kernel is a boon for reuse. But it's a challenge for maintenance. And the build-in configuration system that manages this configuration process is complicated. Our goal is to try to automate and analyzing these build system components so that we can provide useful automation and tooling for maintainers. We talked about k-claws today, which models k-config as symbolic logic. We talked about kismit, which uses that modeling in order to verify that there are unmet dependency bugs in the kernel k-config specification files. All of our tooling is free and open source. It's available via Git at this link on the slide. So be sure to check it out. There's some other tooling in there as well. And special thanks to the National Science Foundation for funding part of this research. And thank you very much for watching this talk. Have a good day.