 All right, I think we're go. So welcome. I'm George Wilson. I'm the IBM Linux Technology Center security team lead and security architect. And this is my colleague, Elaine Palmer. So I'm Elaine Palmer. I work in IBM research in the security department, and I focus a lot on secure boot, particularly for our big mainframe systems. So today, we'd like to talk about establishing trust in Linux keyrings as trust built in imputed or transitive. Standard disclaimer, Nina did a good job earlier of going through this, so I will know it. So the agenda, first we'll talk about some of the goals that we have, then we'll give some background and definitions of some of the problems. We'd like to talk about trusted sources in the Linux kernel, a bit about kernel keyrings, and then some recent activity. So the goal of the talk really is to share some of the things that we've learned along the way, because neither Elaine nor I are keyring experts, we're not prolific contributors to the kernel, but we do use the keyring for various purposes. So for instance, secure boot, key management, self encrypting drives are various purposes we want to use it for. And we'd like to just share some things that we've observed and encourage some discussion around some of those topics. So what we're going to focus on is really the dot keyrings, the built in kernel keyrings that are created and consumed internally by the kernel, and notably, I mean, user space can't go really do a lot with the dot keyrings except see what's there. What we won't cover is other keyrings, you can use the keyring for a lot of things. We won't be talking about user space, we will not be talking about TEEs or devices, and hopefully we can stay away from secure boot because we've talked about that a lot over the years. So now some background definitions and problems. So trust is not a yes or no binary function, either on or off. It's more of a stepwise function or a continuum. So I might trust highly the firmware that boots my system, right? It's supplied by manufacturer, it's signed, I trust the manufacturer, I might trust the kernel less, it's got more code, it comes from different people, I might trust user space even less, it's a lot more code, it comes from a lot more people, may have different certificates to verify its authenticity. Or I could view it as a spectrum or continuum, I might trust, you know, a distro certificate to verify my operating system, but I might not want to trust it for other things. So problem number one, if keys and business relationships aren't known at kernel build time, how do you establish trust later? And there are ways to do it, but that's, keep that problem in mind. Problem number two is it's difficult, if not impossible, upstream for machine owners and end users to load their own keys and sign their own modules. This changed in 5.18, but we have some criticisms of it and some ideas for further improvements. But the situation has dramatically improved upstream. Those had ways of doing this, right, but there wasn't a good upstream way of doing it. So 5.18 has got goodness in it. Problem number three is there's not a one size fits all security model. A lock down appliance kind of system is very constrained, you can control what goes on inside of it, you can even create policies that are very specific to the system and control your updates in a very constrained way. A general purpose system might need to get various certificates to verify various pieces of the software. In a circular economy, you know, people may want to take the keys that exist on their machine, destroy them and put somebody else's keys in their place. For critical infrastructure, you may require multiple signers, you may want to test the kernel and packages before you put them into production and only permit those things to go into production. So we'd like to introduce a new sort of term, and it will be useful hopefully later in the talk, to impute. So to impute is to attribute something to another thing, right? So people impute wisdom to owls, or owls wise, I don't know. People impute value to Bitcoin. Is it worth what you think it is? Maybe today, maybe not tomorrow. Antique dealers impute value to antiques, home buyers impute value to houses. House might not be worth what you think it is. So we attribute value to things, but that value may not be accurate, justified, or in any way valid. So we want to apply this concept to trust. So sometimes we may want to trust things that are associated with particular, you know, attributes that, you know, someone or something has. So for instance, I trust the Acme baby formula company because they have a great reputation, right? My stockholders advice, he's a really personable person, seems really knowledgeable. Are those things true? Maybe so, maybe not, maybe they're true for me, maybe they're not true for you. And this is versus transitive trust where there is some trust relationship established up front, and then later that trust is extended to other things. So A trust B, B trust C, therefore A trust C. So an example of this would be, you know, I trust my car dealer, had good service there, my car dealer trusts the parts supplier, therefore I am trusting the parts suppliers to apply good parts. So then we'd like to talk about sources of trust in the kernel, clarification. The kernel has to trust what it's told to trust. There are reasons to add more keys are made solely within the kernel based on what it already trusts. That could either be a good thing, you could have chaos castle here or you could have the strong fortress depending on what that route of trust is and how secure it actually is. Moreover there's no online connectivity, so you can't go check whether certificates are valid at boot time, generally. You can't go check for revocations with OCSP. The kernel could be disconnected from the internet, it could be on a military aircraft or something, or it could just have no network connectivity whatsoever, it could be a kiosk. So you can't go verify these certificates at boot time, generally. So there are three sources of trust in the Linux kernel we'd like to describe. So first of all the built in keys that are just usually distros include these in the kernel image. And they're there and they anchor trust for other things later or they can be used directly themselves. Trusted trust is I want to trust something because I know where it came from, I trust the source, I want to bring it into the kernel and it all depends on who or where it came from, the origin of it. And I as an individual need to make a determination whether or not to trust it. And then there's transitive trust where something that was built in or trusted because it was imputed trust is automatically trusted because the built in or imputed thing signs it, right? So it depends on who's vouching for it and that could be once again either good or bad, you could have chaos castle or happy castle there. So then we'd like to talk about kernel key rings. And with this I will turn the talk over to Elaine. Thank you, George. Thank you, Elaine. Let me have a show of hands. How many of you are familiar with the machine owner keys or mock keys? Pretty good. Okay. So we'll talk about a subset of keys within the kernel here, specifically asymmetric keys. So following these models of trust, the built in keys, as George said, they're built into the kernel, done at build time, there is an option where you can build your kernel with a space to put the keys and then fill in the keys and then sign the kernel, but essentially they're still built in. And an example of that is the built in trusted key ring. For the imputed trust model, in this case, keys are provided by the machine owner, might be an administrator, and in that case the machine owner imputes trust to those keys. And examples of that are platform key ring and the machine key ring. We'll get into those in a minute. And then for the model of transitive trust, so any key that's signed by a key that's already trusted and already in the kernel will be trusted also. So if a CA's key, a certificate authority's key is trusted, then everything signed by that CA will also be trusted. And an example of that is the secondary trusted key ring. All right, so let's go through some of these key rings. This will go through some of the key rings as they were up through Linux 5.17. So starting with the easiest one, the built in trusted keys key ring, what are those used for? Keys on that key ring are used to verify other keys, kernel modules, and Kexec kernel images, and where they come from, they're built in. So those are the easiest to follow. We add more keys. This is a secondary trusted key ring. Those are used to verify other keys, kernel modules, and Kexec kernel images, just like the built in keys. They are linked, the built in and secondary key, trusted key rings are linked together, but adding a key onto the secondary trusted key ring has to be added using transitive trust. So you can't add one unless it's already been signed by something already on the secondary key ring or by something in the built in key ring. Some more key rings. So the DyDyma key ring has a different set of keys in it. Those keys are used to verify Kexec kernel images. They're used to verify kernel modules. And different from the others, these are used to verify files. Part of the integrity subsystem will verify signatures on files. So this is where it gets, Dyma gets its keys from to do that verification. You notice it does not verify other keys. So the buck stops there, the Dyma doesn't carry on with more keys. The way, how do the keys get into that key ring? Well, it uses the transitive trust model also. So in order to get a key onto the Dyma key ring, it must be signed by something either on the built in key ring or something on the secondary trusted key ring. Now let's go to the bottom right with the blacklist. Blacklist is sometimes called a key ring, but it's not really a key ring. It doesn't hold keys. What it's used for is to block file. So for example, if you're trying to check the signature on something and you want it to always fail, you can put that file digest into the blacklist. Or for certificates that you want to block, you put the to be signed field in the blacklist associated with that certificate. How does that get populated? Well, it can be built in. Some distros, for example, will use this as a way to revoke keys. They can build in a revoked key in the blacklist. Or following the imputed trust model, keys that are in the UEFI prohibit list, the DBX, or the mock DBX, or by root users adding things onto the blacklist, it follows both models, so built in and imputed. We have one more to talk about up through 5.17. And that's the platform key ring. And it's used for one and only one purpose. And that's to verify kexect kernel images. That's it. Where do those come from? It follows the imputed trust model. And those keys come from UEFI Secure Boot from firmware or machine owner keys. So far so good. All right. So let's talk about recent activity on these kernel key rings. So starting with 5.18, there's been an addition of a new key ring called the dot machine key ring. The purpose is it's linked to the secondary key ring. And it does exactly the keys on the machine key ring have the broad usage as other keys on the secondary key ring. And meaning machine keys on the machine key ring are used to verify kernels or kernel modules or kexect kernel images, same authority, same powers. They come from, they're using the imputed trust model. So you might have a system management system or your machine owner keys. Those are what are reported or pulled into the machine key ring. And that is controlled by a configuration variable and set at boot time. You'll notice this big gray box over here on the right where the IMA key ring is disabled when the machine key ring is configured. And we'll go through that in just a minute. All right. So this new machine key ring has its advantages and its disadvantages. Where the upper right, as George mentioned, distros have needed this functionality for years. And they've been doing it through their own patches that they keep carrying year after year. So the good thing is that this upstreams finally some of the functionality that's been carried for years. The middle one, it now supports the machine owner's authority to extend that trust. So if the machine owner wasn't involved in the build, then the machine owner now can have a say so and extend their authority, for example, to kernel modules. The middle bottom says, we've got a flexibility now where we can have runtime additions to the secondary key ring with keys that have been added dynamically. And the bottom right is that there are controls on this machine key ring so that it can be whether or not it's even allowed is configured in at build time and whether it's enabled is set at boot time through EFI variables. Those are the good things. On the bad side, the scope of this new machine key ring is really quite broad. And if we go back to Linus's some of his original requirements to talking about adding keys, his quote is, we want to enable machine owners to add their own keys and sign modules they trust. So if we go back a slide to the diagram here, what you'll see is that the machine key ring, because it has such a broad scope, it can do all of the things at anything else on the built and trusted key ring. It's not restricted in any way. So it adds functionality that we need, but we think it goes too far. So the other thing is that the machine owner keys can be a large number of keys. We don't know the original source of them. And by loading all of them onto the machine key ring, there's no fine tuning of what goes on and what doesn't go on. So it's all or nothing. So what we would propose is that as a slight variation on this, where what is loaded onto the machine key ring are only CA certificates that are in the machine owner key ring. And that, if we did that, then this picture changes so that the machine key ring has CA keys only. And it's still linked to the secondary key ring, but it's only got CA keys. And then we can enable the IMA key ring. And let's go through the positives and negatives for those. So upper right. By separating the CA keys or certificate signing keys from code and data signing keys and those authorities, it encourages good key hygiene. If you go look in any kind of key management book, you know that you really should restrict your usage of keys to either key signing or certificate signing or code signing or data signing or TCP. Whatever it is, network communications, it's best to restrict the usage of your key. So by having this restriction, we would say that anything on the machine key ring has to be a certificate signing key. As far as separation of duties going down a bit, in having IBM works a lot with the finance industry and their dead set on keeping separation of duties, they must do that. And by doing it this way, we can support where an administrator or the owner of the machine is the one that loads the parent certificates but is not involved in the code signing or the data signing or signing of more keys. So we support separating those duties. The limiting the machine key ring, it has the same positive as the other where you can control it at build time and enable it at boot time. And, oh, I missed one, where you can, by having two levels, a parent certificate and a child certificate or a key signing certificate and a code and data signing certificate, you can more frequently rotate the child certificates so that your machine owner controls the parent, but the other keys can change at will more frequently. So I think we have gotten through the background and the proposal before we get into comments and questions. There are places that we've pulled this information from there in the slides. On top of the references, we've had to tap the brains of Mimi Zohar, who's here in the audience, Nina, who spoke earlier, one of our certificate experts in IBM, and David Howells has an excellent video on video tutorial on key rings. And for getting the machine key ring code in and written and upstreamed, Eric Snowberg and Jarco Sakinen. So let's go back to questions and comments. Yes. Right. So the problem goes, the question was, why is it that IMA is disabled when the machine key ring is disabled when the machine key ring is present? And that's a choice made by Mimi and others working on the IMA subsystem. And the reason is that broad expansion of scope from with the machine key ring, let me go back to this original picture. Originally, or as it is written now in the kernel, the machine key ring has everything on there has broad scope. It's as broad as everything on the secondary key ring. And what that would allow, for example, if you go to, let's say, a UEFI variable that gets loaded onto the machine key ring, because of that scope, it can end up linking to key. It can even be used to verify files in IMA. It's too broad. Mimi, did you, sorry, the concern is that you're, what was, yes. So let me repeat that. The concern is that by loading in the machine keys from anywhere or from this, using this imputed trust, then it will end up influencing what goes onto the IMA key ring. And that was unintended. Is anyone here an expert on, for example, what restrictions we would want to request on those keys that go into the machine key ring? We know, for example, that it should be a CA equals true. And the key usage should be used to sign certificates. And both of those should be critical when you're parsing the certificate. But there might be other things that would be helpful. And perhaps there's some opinions there. We thought about things like policy, to get some selectivity to the certificates that we're loading. And now that gives you something else that you've got to go manage on your machine. Is it sufficient to control this through CA certs? Or is that not granular enough? I think that's kind of one of our questions. I think there have been discussions of perhaps adding key ring policy in the past, but there's really no policy now other than just the signing relationships between things. So not advocating it necessarily, but that could give us the ability to say, I don't want these things. I want these three of these eight things on my key ring. Not sure that's necessary or helpful, but that's one of the thoughts that we had. We definitely want to do the separation between the CA cert and the code signing certs. You need some way to go delegate changing those code signing certs easily. And in CA certs facilitate that. So we think it was a mistake to allow code signing certs there on the machine key ring. Which brings up something I didn't mention. The reason that this having two certificates is viewed on the negative side of this slide is because although it's a positive to have these two certificates, in common practice, we've found that often the code signing certificates are self signed. They have the CA bid on, and they're used to sign code. So there's only one certificate. And asking to add two certificates in place of where one was used in common practice might not be well received. Question? The theory was pretty hard. But it allows you to only specify as a certificate. I think in general, expanding that quality to be more granular and allow you to have a more expressive statement about what the administrator has done to make a lot of sense. Back on what you were asking about CA certs, the other one I would consider a pathway. Because making sure that the certificate is what includes these certificates to make sure that it's not going to happen to the URL for the certificate which will come with an issue of these certificates you'll have with that certificate. I should try to summarize that for the recording, but. Come on, Kelly. Let's see in summary that more granular control is absolutely required of the key rings and what can go on the key ring, perhaps by policies. Just simply putting it on the key ring is really not sufficient to get the intention of say the administrator or whoever built in the kernel. It's not enough. And what was the second one? Oh, if we wanted to try to restrict answering this question, how do we identify or restrict those certificates we don't want as we could exclude those with a path length of zero? Did I miss something? Thanks. Also, you're in violent agreement. I guess that's it. So corner us at break or somehow you have our email. Yeah, and we can kind of maybe plot a little bit and see how we can realize this.