 Thanks for joining my name is Darren Hart. I'm the director of the open-source technology center at VMware and today we're going to talk about managing Linux kernel configs specifically with config fragments. So just as a general idea, how many folks maintain a Linux kernel config for a project for a product for Okay, whoa, whoa, okay Awesome and So today we're going to talk about some tools that are available within the Linux kernel to try and minimize the amount of work and effort around that process All right, so This worked before here we go Linux is used broadly throughout the industry from phones to servers to laptops to super computers and The requirements of each of those is very different. So this is this is the embedded track And so in that space we particularly see a wide variety of configuration types Embedded is a lot less boring than PCs and servers and so we see just a lot of diversity in the way that we create Configuration fragments so on the left for anyone that's relatively new This is what a few lines of a typical config looks like for the Linux kernel You'll see in the lower area here 3,456 is the length of that particular dot config This is the largest number of unique config symbols in the Linux kernel And it's not really going down so over this period of time except for the last couple We've seen a continual increase in the total number of configs Now those of you that manage your configs are saying whoa, whoa We never use all of those we use deaf configs for a given architecture And that really cuts that down a lot and you're right takes it all the way down to here, which is still a lot And then you can say okay, but I only do that once and then I only have to do a diff Each time for the for the deaf config is it changes. Okay, sure still smaller 500 lines of difference between each one. Okay, but a lot of that's context, right? So really in terms of actual changes, you're only talking about a hundred or so sure and That's just to maintain the default you have made no decisions you haven't made any selections of configs yourself This is entirely every single kernel revision just to maintain a default config So we can do better than that But before we get into that let's talk a little bit about the K config language so K config is a mechanism that we use within the Linux kernel to be able to Create dependencies and such throughout the various config symbols in the kernel On the right is an example K config file all the config symbols within the kernel are created from these K config files You'll typically find one in any major directory or subsystem And they're all created with a few general rules or a few general types First you have the name config and in the symbol name, which is the all caps bit that you see and typically how we refer to the symbols Everything has a type and these are generally Generally you see yes or module or sorry module or built-in, right? So all these types are derivatives of tri-state or string tri-state is built-in module or No, and then string is just that it's a string and then the other types are derived from those a bool Eliminates the module and then you can create hex or or int and numbers from the strings The prompt is the part that you read in menu config prompt is the user Visible user visible description of what this symbol is for if it doesn't have a prompt it doesn't show up in menu config But that doesn't mean that it's not there It means it gets selected or something like that in another way and we'll talk about select in a minute There's a setting called default and this This tells the configuration system whether or not the config should be enabled by default Unless it's something that everybody uses that's typically no Depends is the way that we satisfy Mutual requirements within the k-config language So if you want to write an i-square C driver you need to have i-square C enabled if you want to write If you're writing an acpi driver you need acpi if you're Creating drivers that only run on x86, then you might depend on x86 But it gets more complex than that so on the example on the right you'll see depends on acpi wmi or ACpi wmi equals no So ostensibly that's a little confusing right it says I depend on either You have this feature or you don't have this feature So that's a yes, right? So depends means more than just this has to be there The depends statement means that if you want to build this Driver or this feature You have to you and you depend on this other symbol then you cannot be If that other symbol is built in then you can be built in our module But if that other symbol is a module you can't be built in because you will be missing the symbols at boot, right? so what that is saying is if symbol foo Then you have to follow up on whatever depends on that means or not symbol full foo, and then we don't worry about it It's a little bit confusing, but the reason I highlight it here is It's illustrative of the complexity of the k-config language and then finally Select so select is that hand grenade that we throw into our configs that just breaks everything So when you do a select it's like a reverse dependency without dependency tracking So you can select another symbol, but it won't go through and make sure that those selects that that symbols Dependencies were satisfied so you typically only use select for something that has no dependencies and even better if it doesn't have a prompt Because what that means is the user is not going to be able to go select it on their own So always be careful if you're defining selects or using well both defining and using selects Okay, so if you'd like to know more there's excruciating detail available in the documentation, okay? so we're talking about managing Linux kernel configs and So before and we're typically all of you doing that are doing so in revision control And if you're working with revision control, let's review what a good get commit is a Good get commit should start with a problem that should be defined. Why why is this change necessary? It should describe the intent of the of the developer This is the problem and this is how I'm going about fixing it. This is what I intend to do Then the changes should address the problem in a way that is consistent with the stated intent and Then number four is the most important part, right? It doesn't do anything else and We'll talk about that In this context, okay, so let's take an example Let's turn on the Dell SM BIOS driver So all of you are familiar with menu config on the right you type and make menu config you open up that menu You search through it you check a couple of things you go to exit and it says would you like to save your changes? You save course and it writes it to the dot config so the Diff created by this should indicate to me that I've made that I've enabled those two drivers Right, that's the only thing I wanted to do here. That was my intent was enable Dell SM BIOS so my Changes should reflect that and nothing else Huh? Okay, so in green you see the three that I wanted and then there's context. That's cool But then there's all these other things And you might and so things like the Dell WMI descriptor is set to a module. I didn't turn that on But then you might say okay, but all that other stuff is just comments, right? No, because when we these are symbols and we check to see if they are defined the pound symbol is not set is Effectively no, right. This is saying I explicitly am saying that I don't want this thing and so the reason that that's a problem is Let's say later you've got this Dell laptop You just turned the Dell SM BIOS on so you want to be able to come in later and turn on the Dell WMI LED So you go and you look at the config and what you realize is hey some developer in the past intentionally disabled the Dell WMI LED they turned it off But you're trying to turn it on so now you need to know why did they turn it off You want to know that right because you're not just going to change what they did intentionally With something else unless you know because if you turn that on to make your laptop work You might break theirs Because you don't have their laptop to test So this is where the managing a monolithic kernel config gets to be a real problem because you're recording not only your intent but also the artifacts that come out of Turning certain things on in the kernel config Now this was these 57 lines for example though keep in mind that was to turn on one driver So presumably you're gonna have 100 that you explicitly set for your project or your product. So now you have 5,000 lines of Diffs that you're managing to turn on those and then any time that the depth config changes your context changes And as your context changes you have to rewrite these patches, too. So this gets to be a pretty ungainly Thing to maintain. So what can we do differently? So let's start with what I want my diff to look like So on the left you have an example of what I would like my diff to look like I have the three lines that I explicitly changed and nothing else And I save this to a file that's called Dell SM bios dub. What did I call it? Dell SM bios WMI.config So if I do that Then I can use this is what I call a configuration fragment and To be able to apply this The Linux kernel build system knows how to merge Configuration fragment. So if you look at the commands on the right I do a def config and then check to see what the Dell SM bios is. You can see it's is not set and then I do a make def config Dell SM bios WMI.config and so what that will do is merge the config fragment Into your larger doc config for you and then we do a we do a test Afterward with grep and look for SM bios and we see that both Dell SM bios and Dell SM bios WMI are Configured and now if we build the kernel will have that support Now the the change is something that is independent of the def config So not only do I only record the three things that I wanted to change But unless the unless the later kernel versions change the name of those symbols or add Dependencies that you thus have to satisfy you don't have to change that file ever so you're able to maintain the Specific things that you wanted to change in a separate file Build the config and then build your kernel and so this makes your your config management a much more explicit process, right? That's good for you. That's good for people doing debug later. That's good for people doing code review It's all a lot more explicit in what you're trying to do Okay, so what does it actually do so? The the last one I just piped all this to dev null because it's noisy But if you look at what's happening here, I just run the same command again based on def config add my configuration Fragment so it says it's using the doc config is base and then it's merging the Dell SM bios WMI config into the main config and then it goes through each line one at a time and it says Hey, this config fragment is overriding the previously defined value of symbol foo Its new value is going to be this and then it just walks through each one and in your Config fragment and it'll do this for every config fragment that you specify in the command line So you can have a hundred of these and it'll just please don't have a hundred But it'll go through each of these and and give you a report out And then finally it'll save that to your doc config and now you're ready to build This is dependent on a script called merge config so while I was at Intel I spent a lot of time working on the kernel management in the Yachto project and This idea was kind of born from the way we managed that in the Yachto project and I felt that you know Config fragments which we use in the Yachto project are a great idea that should probably have an upstream counterpart Maybe we can leverage that so with a few other Developers outside of Intel we wrote the merge config script And we merge that upstream and since merge config as Yeah, merge config has become a part of the Upstream build process so MIPS uses it power PC uses it. This is not an additional thing This is upstream Linux. It's available there now and it's used as part of the default build system So if you're not interested in explicitly generating the config and building your kernel and you just want to manipulate config files You can use merge config directly It's in scripts kconfig and it's got a few options there a couple of things that you can do My examples previously were all about using def config, but if you're working on embedded you probably have some sensitivities to Resources you don't want a big kernel. You don't want a ton of modules. You want to keep this minimal so you might start with all no config So that that's the dash in Option up there so you can start with all no config which is all no Right, and then you can add I need this architecture. I need this machine. I need these drivers Okay So a little bit about config Organization so as you can imagine if you did a config fragment for every driver you wanted you'd have Way more than you probably wanted to manage But if you did it all in one then you kind of start having the same problem that we started with so The yachter project which I think does a really good job at the way they recommend doing Configuration management breaks things up generally like this so I'm going to use the same example There's certain policy decisions that you make when you're creating your Linux kernel for your projects you might decide on which file systems you use or which crypto Algorithms you're going to support which networking stacks those are all policy decisions that you make and that can be one configuration fragment another might be the architecture You could use a def config, but maybe you don't want to use that entire def config Which is usually more like this works on my server or my my general PC or something like that If you have a very targeted device your architecture definition might be considerably smaller more restrictive So if you're pairing this with a no config having an architecture fragment is a good idea And then from there if you have board specific things so which which buses do you have and which specific drivers are on board? That's where you can put those kinds of configurations And then if you're working on a device that is not locked down, but might have users adding or removing things to it That's where the general drivers work. So things like USB webcams are a really good example. Like which ones of those do you want to support? and Then finally looking forward as I was creating this presentation. I noticed a few things Creating your configuration fragments is currently manual So you go in there and you go through menu config and you search and you find the one that you want And then you go type it into the file and you say what you want it to be and It'd be really nice if I could go into menu config and have it Record which things I check yes module on and then just when I exit it says do you want to save your config? Maybe it could instead say would you like to save your fragment? That would be really nice, right? It just generates that for me not all the things that changed just all the things I intentionally changed. I don't care about all the things that got checked by default. I'm accepting the defaults These are just the things that I changed. I Wanted to write that on the plane because Dave Hansen shamed me into not having had a 30,000 foot commit But my flight's got all screwed up. I think of that done. So maybe on the way back And then another thing I noticed was merge config can silently fail So if you add something that has it doesn't know about Rather than saying hey, you tried to config this to that and it didn't happen. It's quiet about it And I noticed that last night and I didn't actually Nail down exactly what's happening. So that's something we can work on fixing and Then something like make audit config Where you pass all your config fragments after it where it can go through and then verify Are any of these things that you've asked for not in the dot config? It seems like it would be a pretty useful tool for some of our CICD processes when people are changing the configs. So those are my thoughts on what we might do in the future And that's the content that I had to share with you today that leaves us with a little over five minutes for questions I've got a live System up if you had specific questions or you wanted to see us in some things worked we could go through But if you have questions, I've got some folks with mics that'll happily walk them back down So we've got some quit. Let's start going through questions here on the left and Suzanne over to No Up here in front So what I wondered about a configs That how like doctor uses them isn't it couldn't be in some cases the order be Significant and how does doctor decide in which order it uses those conflicts? Okay? Does the order matter? Yes, the order matters. So each each subsequent one can override the prior I would recommend that when You have these that you apply them in this order as well But this is also why I think that the make audit config is Something we should work on because then you can because something I forgot to mention was your your fragments can indeed be Contradictory and if they are something you should be aware of yeah good question, and that's not doctor specific. That's upstream, but yes Hello, do you think you could sorry one sec Bart there's a question Oh, let's take the mic over to him in the back Go ahead. Do you think that this kind of merge conflict system could be used for something else is a similar conflict system? Say build route build route busy box. Yeah, anything. That's k-config based. Yep, right next to you Okay I was having the question whether you could only enable the conflict symbols or also disable them Oh, yeah, good question. You can also put pound config symbol is not set. That's the same thing as saying no Yep Hi, turn is there or do you have any easy way to trace back when a Dependency's not met because when you when you merge it you say it says Something didn't get enabled, but it was on the config right, but it only tells you like the last thing That wasn't enabled for it to be enabled. You know what I'm saying? Yeah, the whole dependency chain The it's a really good question. I think it's something we'd all like it's something that the doctor project has struggled with too And one of the problems with that there's a term that we use That describes the type of prop that the type of problem that a dependency Chain is and I don't remember the term but the the problem is You might have Well, you will have a number of depends on this or that and then that gets Made more complex by each of those having it depends on this or that and so if you say enable this and it says I can't because the dependencies aren't met the What we'd like it to do is say please go enable this Unfortunately, that is please go enable this and this or that and this and that parentheses or this and that unless your arm And then do this unless you're x so no I think the best that we can do there is do the make audit config And then maybe provide some how-to's on how to do that spelunking basically through the Config's unless somebody else has a brilliant idea on how we can fix Bruce These things I say well what I use right now in the octo. It's the Python K config Lib and it now can up to 418 plus parse all those dependencies So when it fails so you don't make it in the final config It is does a pretty good job of going back and saying you miss this dependency Selected or that and you can do that spelunking in Python and dump it out. Oh, okay I'd like to catch up with the after and see what's possible there. Thanks in the back Can you create umbrella fragments that include other fragments? No, right, I don't I don't think we have any kind of an include system right now config fragments are strictly fragments of Dot config so nothing more than config symbol or pound config symbol is not set is that anyone know differently We considered that and what the the octo solution to that is a One higher level language that wraps that that allows you to create those includes But in terms of the mainline Linux merge config tooling we do not have that What about the options that get? enabled by default when you Enable or disable some other options. What about those so that is a really good question And that is what I was trying to illustrate Here so that's basically what all those red things are and that's the additional context that you didn't intentionally Change, but that got added to the dot config and so The approach that I'm recommending is that don't worry about the defaults What because you didn't specify them let this let that be managed by the system only explicitly state what you care about Which is in green and then if you when you go back and you look through your config later You might find that hey, this is disabled or enabled things that I cared about and then explicitly add those into your config fragments Yeah One different question Could you use these fragments to build a graph of the driver dependency at runtime? Sorry one more time Could you use this fragments as some hint to the corner when it probes drivers that we have something? Exactly present on the platform you would like to run this kernel on oh It's a good idea. I don't think we have anything like that now, but yeah Yeah, thanks for your talk just a small notice. There is a tricky word imply for in K config so When when when some option Enables or disables another option if it was not explicitly set it is not depend but imply and I Wanted to ask whether your script while switching on and off the options Report about changing implied options, so it isn't it is the dependency, but it is a weak Dependency it's a weak dependency. Yeah, so for people wondering Go look here. It'll talk about imply. I was trying to decide do I add imply or not? I thought it was probably a little more special case This doesn't handle Select or imply any differently than If you were to just change the config manually right because all this does is merges Explicit statements, so it doesn't it won't report out on anything on what what use if you selected anything or implied anything Okay, yeah We have three more minutes We're okay because we're we are past But if we're I don't think there's anything conflicting in the room, so unless they kick us out I think we have one up there and then aren't up here for the next mic. Go ahead Yeah, I think the defaults are actually a problem because when you bump your kernel version The defaults may change. Yep. So just doing that Yeah, your intense is Is has well is is is not applied anymore And I don't think there's any good solution for that except for comparing the full coffee, which is horrible So you're He makes a good point, which is when you do change your your base kernel version. You're going to get a bunch new defaults this is much less a config fragment management problem and more of Hey as maintainers, we need to be doing a better job at not enabling things by default that we shouldn't I got Corrected a few months back about having way too many default yeses in the platform drivers x86 You might notice a patch from Linus that no no no no no no no The equals no not the anyway So that is something that as kernel maintainers We need to try and do a good job at avoiding that bloat of the default yes But otherwise you do need to do a compare if your kernel size grows Hopefully you're watching that in your CICD and you're able to come back and say what do I need to turn off and unfortunately Yes, that means adding a number of config symbol is not set All right, so I'm missing a feature to turn on a symbol more but not less so for example we might want to test DRM drivers and Enable them as modules on arm, but I applied the same contract fragment on x86. We're setting DRM from yes to module Would just lose the console so we don't want to make it we don't want to make it Module on x86. We want to leave it at yes an arm. We would Want to make it module if it doesn't already on but we don't want to build it in because that just loads the kernel so That to me sounds like something you because it's architecture specific You'd want to include it in an architecture fragment as an explicit statement the idea was that We have a couple of those and it's always a Fragment that is in principle architecture independent So we have a fragment that we want to use for running K self-test so having a K self-test fragment should not depend on the architecture We don't want to have the whole matrix of all architectures and all K self-test tests Yeah, but just having some way of saying apply this fragment but if it says Equals M and it's already enabled then don't turn it off. Don't make it less. Yes That's a good suggestion You you mean patches welcome. Yeah, no, I mean let but catch me after I can't actually write everything down right now Or send me email because that's that's a collection of things. I'd like to I'd like to have okay Haven't kicked us out yet anyone else, okay Everybody thank you for coming. I appreciate the questions. Feel free to well. It's not up here now But feel free to email me DV heart at view more.com If you have any other thoughts or questions or suggestions, thank you everyone