 Hello everybody and welcome to my Linux Security Summit talk about patch attestation. My name is Konstantin Rabitsev. I am in charge of kernel.org infrastructure at the Linux Foundation. I've never done anything like this before. I'm not a professional recorder, so I hope it works out. There probably will be some awkward moments when I'm rapidly trying to fix or switch between windows. So hopefully there'll be a few of them and not too many. All right. So let's get to the topic of our presentation, which is patch attestation with patat. Patat is the tool I wrote. And before we talk about the actual tool, let's go and look at what the state of patch attestation is without it. So RFC 282 attestation for mailed-in patches. What options do we have these days? There's two mechanisms of doing it. There's an end-to-end, which means it's developer to developer, developer to maintainer. You sign the patches on your workstations. The developer who reviews them verifies them on their workstation. Or there's the main level, meaning you don't do anything on your workstation. You submit it to your SMTP server. The SMTP server signs it using the DKIM standard, which we'll talk about in a moment. And then the developer basically does nothing to verify it because all the verification is done by your receiving SMTP server. And if the DKIM signature doesn't match, chances are the mail ended up in spam anyway and you didn't see it. So for end-to-end attestation, there's two mechanisms. There's OpenPGP and then there's SMIME. OpenPGP is about as old as each other. OpenPGP was introduced, I believe, late 80s or early 90s. I don't remember which one it is, but in the ensuing 30 years of life, it didn't really solve the problem of key distribution because key distribution and bootstrapping trust is really a hard problem. And there is no easy sidestapping of that problem. Like, we can create a certification authority, but then you have to trust certification authorities. Obviously, if you don't want to trust anybody, then you have to worry about key distribution and management on your own. And that is the crux of the problem. Another problem with OpenPGP is that the mail client support is still subpar. There are still many mail clients that do not support OpenPGP or there are those that do support it, but don't do it on a sort of the mobile application or anything like that. Anyway, if you've ever tried to use OpenPGP with signing messages and verifying messages, you know that it's hard to get it configured right and it's hard to get the tooling to do the right thing when sending and receiving mail. Now, SMIME is another standard for doing the same thing as OpenPGP signatures, but it really is relegated these days to the corporate world. There used to be public certification authorities that would provide you an SMIME certificate for signing a mail, but there used to be two of them two years ago. Now it's just only one that does this and they do it in a really strange fashion. If you try to get a certificate from them for signing a mail, they will send you the certificate, the public key, and the private key, which is totally at how it's supposed to work. So I don't recommend you use them because that's kind of really sketchy. Anyway, there are vendor-specific end-to-end mechanisms for signing and receiving and verifying mail, proton mail to the NADA, but they're all within the vendor itself. They don't scale outside, so we are not going to talk about them at all. So the main level verification is DKIM. Let's take a look at that in a moment. So patches on OpenPGP, can we use current OpenPGP standards for signing and verifying patches? Well, yes, but nobody does it because there are important problems with it. One, it breaks code review. If you use the inline attestation, I think you've seen some of that there is going to be dash dash dash begin, PGP sign message dash dash dash, then the body of the patch, then there is dash dash dash begin signature, and then there is the base 64 encoded signature at the very end of the message. And the problem that it introduces, and not just those headers and footers, but also the fact that if there are any double dashes in the message body itself, it will add a dash space before that so it doesn't get confused where the actual signed content starts and ends. And that, of course, becomes a problem because Git itself uses triple dash as a mechanism for logical separation of commit content. So there can be anything that's above the triple dash goes into commit, anything that's below the triple dash is just auxiliary information for the developer to review the patch. For example, it can contain history of the patch, it can contain the short log, any sort of information that the maintainer may find useful, but not preserve in the actual commit. So that if it's in the inline sign message that will get turned into dash space dash dash dash and tooling probably just choking that believe me. So mind based data station is a way to solve this problem ish. By default, it does not sign subject. It is not in the signatures do not include the header metadata like subject and from and date. And that does go into the git commit. So this is the information that we really kind of want to be included in the signature. So we are there are ways of doing it with git you can you can copy the the subject and copy the date and copy the from into the body of the message and get will look at that but almost no tools do this and it continues to be a problem. And of course, the male user agents and archival and everything that can strip the signatures. I've seen this happen all the time that I recognize that the bgp signature is a trusted and valid content they may quarantine it or do anything like that. So the SMTP relays may may cause problems verifying this. Also does not attempt to solve key distribution, not going to talk about it too much. There is a Web key directories that sort of stepped up to try to solve this problem, but it continues to persist. Decay matter station is created to combat spammers really. So when gmail.com sends an email to you, it wants the to to verify to your receiving client to verify that the mail actually came from gmail.com just so the spammers don't pretend to be anybody from a large company like Microsoft or Google or or Facebook and so forth. So Google and Microsoft and Facebook will add a signature the the decim signature header that will contain information about the message including the body hash and it will sign it with a private key and to get the public key after query DNS to retrieve it at a specific specific domain record. It covers the it's actually pretty good. It covers the header metadata. It says the there's an h equals which includes which headers are included in the signature. It includes the body hash which is actually the entire body canonicalized and then that's this that's the hash of the body. And then the the actual signature includes all the data that precedes the B equals characters in the header. So they will include the body hash because the headers and then the actual header content will be part of the signature. So it really is pretty cool. It's an open standard. It is widely used. It's easy to implement. So it is I think we should rely on this to a certain degree. So it's decim good enough to attest patches. It is but with a few caveats. You know nothing about the infrastructure doing the signing right. If it's a small developer mom and pop shop you probably don't the key is probably not on an HSM anywhere. It's probably somewhere in there on the main server that is doing mail processing accessible to the postfix so whatever mail transfer agent that is doing all the signatures. So if that server is compromised then all of the email you know is suspicious after that because you don't know if the key was stolen or anything like that. The DNS zones when you're looking up the public key you're doing a DNS query. Of course DNS zones are still not signed. The vast majority of them isn't even if they are signed. The DNS sec is turned off verification for most of the very most of the systems that are trying to perform the verification. So kind of you hope that nobody's done anything nasty for your DNS lookup but you can also never be sure about that. Now the problem is that the relaxed body canonicalization. So if you look at the c equals relax, relax that tells you that the body was canonicalized using the relaxed standard and relaxed standard one of the things that it does it will kind of collapse all the multiple white space into a single space. Not a problem for patches for languages like c or or pearl anything like that but for languages syntactic white space like python obviously this is going to be a problem because a return true with a different indent could return you from a verification function and that will be a nasty vulnerability. So we can't really rely on DKM at least when it's not using the relaxed standard for canonicalization to really attest patches because it doesn't quite do it for us. It's better than nothing. If you look at the tools that I provide before for example it will use DKM verification all the time regardless if it's simple or relaxed canonicalization because simply it's better than nothing at all. So we can I suggest that we adopt DKM for our purposes. We're going to use the standard almost in its exact form. We're going to twist it slightly. We're going to instead of canonicalizing the body using the relaxed or simple canonicalization we're first going to pass it through git mail info and git mail info one of the things that it does it will look for things like a from or a subject in the body of the message to substitute its own from and subject. It will look at the same for date. It will parse the metadata of the message and the commit message and the patch and prevent present them separately so we can actually see how git sees them. So we're going to take this information that git mail info provides us then we go into canonicalize using it simple and then we're going to not use the DNS for key distribution but we are going to use our own key rings and I will talk slightly about that in the future and we're going to use open pgp instead of rsa or ed259 directly because it just makes sense for us for as a kernel development community because we already are using open pgp quite extensively for a lot of other things. So this is the example with open pgp shot of 56. We use open pgp for actual signatures. We use shot 56 for creating the hashes of the body and each of the headers. You'll notice that it looks exactly almost like the decim signature. There are minor differences there that for example we include the length of the message. I won't mention why there's reasons for that. You can read the code to figure it out. There is we would say we only we only include the from and the subject hitters not we don't care about any other headers because they don't become part of the git commit so we can ignore them and we the b content is the output of open g of gnu pg specifically that we include in a direct form and that includes the date of the signature. We can't actually use the date header from from from the email itself because interestingly git send email will change the date to the latest so if we sign it and then pass it through git send email the date header will stop validating I have no idea why it works this way it's really maddening but there are probably reasons why it's this way but so we're not going to sign the date header but we do include the date of the signature in in the open pgp standard itself so we do check those to make sure nothing untoward is happening. Okay so skipping to we obviously not going to do it by hand I propose that we do it with a tool I wrote called patat silly name serious business it means potatoes and a lot of languages and I fully appreciate that and apparently in Norwegian it means false which is really strange to me but it just happens to be that way doesn't really matter you can clone it from git curler org there's the url right there it can clone it from github because that's your thing you can be from there you can just install it from pip with pi pi using pip install patat it's there it's really only a few hundred lines of code and of course we don't do our own crypto we just rely on new pg and on pi knuckle if we support the ed259 key directly won't mention those in this presentation because we're trying to keep it short it is for python 3.6 it's the earliest the latest the oldest python version still supported it is intended to be used via sort of you set it up once and then you forget that it's there it's invoked via send email validate so if you do use git send email the idea is that they will enable you configure patat once and enable it via the hook and after that you all the patches are just going to be automatically signed before you uh you don't even have to think about it twice before doing it and I think everybody should sign their patches because why not it's just an extra header all should sign I'm going to have a demo session uh this is sort of the overlook of how it's going to work you can install it on your own and test it out or you can just follow me right here I'm going to switch to the terminal there we go and I already have patat pre-installed that's the latest version as of today maybe later when you're playing with it and I already cloned the Linux repository here and I created a commit that you can um I can look there and it pretends that we just released uh kernel 600 that's it's not don't don't yeah don't think too much about this so let's do git format patch patch one out take a look at this commit uh this is the what uh the patch looks like it's really very short I just changed the uh version and patch level to six zero now how do we make this uh signed with patat well first of all I already have a pgp key generated just to for the purposes of this example it's right there and it's got the signature sub key d25519 we can use that directly so let's say git format patch one we just pipe it straight wait before we do this I'm just going to show you my git config just we have the signing key right there just what you would use for assigning pgp uh signing a git commits um directly so this is the same it's user dot signing key specified if that's specified then patat already knows how to find the key that you want to use for your development work so we're going to do format patch one student standard out and I'm going to pass it directly to patat sign and if we look at the output here you will see that we gained two developer signature developer key headers developer signature contains the information about the uh the actual attestation it's got the signature it's very short because we use a edt59 key uh the developer key is really for the purposes of just information we don't use the information directly from this header to verify the key uh verify the signature because that'll be silly obviously because we don't carry we can we don't trust the contents directly from the same message we're trying to verify but if you are interested in how to import this key into your key ring or do any other work about it this is information that you will need to obtain it right so it's got the denities patata example.org it's an open pgp key the fingerprint is right there so you theoretically should know how to retrieve that from a key server or how to ask the this person for the public key not what go into it right here so we can pass this straight to patat validate right because it's our own key where you have the public key for it it says the pass if there's anything about the body or the headers that were signed that was different we would have gotten a bad sake so obviously we don't want to pass all the format patch to to to patat sign so let's just enable the hook and there's a convenient command to do this uh install hook you'll say that it installed the send email validate hook if we look at the contents of that you will see that it's just uh literally just one command patat sign dash dash hook and then just whatever is the um hook gets on the um as an argument and after this so let's say get format patch one let's say o to send so you created that patch we will now do uh get send email and I say dry run because we don't want to annoy anybody and then we say to send star right and you already here see that patat was already invoked it was signed and if we control see now if we look at the message as it exists on disk you will see that there is a signature header present there so let's say uh test the local host because it doesn't really matter for dry run where do you want there we go um and if we look now if this wasn't a dry run it would have just gone out but if we look at um uh to send oh one you'll see that the headers were added even though they weren't there before this was done by the hook so any patch that we now generate with format patch and then we send with get send email will automatically get those signatures and we wouldn't have to really worry about patat invoking it manually or doing anything like that pretty cool huh all right so you signed you sent the patches how is the maintainer supposed to verify this patch that it's uh hasn't been tampered well tools like before will do it automatically for you so let's get uh case cook is a good person who is doing uh using patat before all is signing so we're going to just demonstrate it right here so we're going to retrieve his uh message that he sent a few days ago and is uh if you can see that but before we'll do two attestation verifications one you will verify the dkms signatures are correct and we can see that they are correct here the dkms signatures passes but it will tell me that there is no key to verify case cook at chrome or so i don't have it in my key ring now i could ask case to send me the key and import it into my key ring right gpg import or there is a way to deal with this other ways there is a repository that i provide on kernel.org that i track i call it the kernel.org key ring and you will have to trust me unfortunately there's no way around it like i said bootstrapping trust is hard so if you want to trust me you can just clone that repository and use it with before let me look at git config and say for key ring look at this repository that we just cloned and this is uh the url will be in the presentation you can see that there is a dot key ring right if we're doing a key ring open pgp chromium.org key case cook and there is a default key that points to the one that we have in file for case cook so now that we have this key ring src there right we're going to rerun the same command before am and you will notice that now before says oh well we already have the key in the key ring not the open pgp key ring not the new pg key ring on disk but we used the the key ring that is in the repository and there that key matches and the signature passes and we can just say this got signed by case cook at chromium.org and this tells us that after the message was sent by case via all this mtp servers via you're gonna have to get stored on lorker.org anything else that's been done to it we can still verify that it has not been modified between when case sent it off and when um we retrieved it all right switching back to slides let me go back here this is what we did uh the key ring management is still kind of nascent i'm not going to go into it too much i'm just going to mention how it works briefly uh there will be a command for before that will allow you to simplify management of the key ring currently the only thing it does it can show you the keys that were used for signing a particular message thread so test that out by yourself if you want so before we'll use a global new pg key ring so if you already have a bunch of keys in your key ring you don't really need to worry about re-importing them for before use we will always look in the key ring new pg key ring if we don't find the key elsewhere but what i suggest is that we stick pub keys into the git repositories themselves right it's not as crazy as it sounds right the there is it is a bit of a circular how do you use the git repository to verify the git repository itself but the distributed trust is hard and this kind of allows us to set sidestep the problem uh git repositories are distributed they're also auditable you can uh for kernel.org we have uh the audit trail for all the git commits so if there's something bad happens you can we can look at the uh the audit trail and figure out who added which key where at which point uh we can uh the git commits themselves can be signed and verified so for example if if there's a maintainer maintaining with a bunch of sub maintainers or a bunch of contributors that that they work with they can verify um they can verify their own signature on the git commits it'll be one person that's kind of acts like as an introducer to all of the other people who are working with that sub maintainer this is a hard problem key ring management and key management and it's it's complicated so i don't want to bog everybody down too much so i just wanted to point out that if you do have um the key in your new pg key ring that you don't really have to do anything else it will be automatically used it'll be automatically verified um if if before finds the patches signed with that key now sub maintainers can keep their own key rings in a separate ref in a repository for example if you have uh if you're a sub maintainer of a specific driver and you work with 10 contributors who send you patches for the drivers all the time you can just create your own ref and there's information for patat or before you can look into it how it's done and you can create your own rest like ref refs meta key ring and you can just keep their keys there for before to use this is a way to sort of centralize your key ring management so if you have copies of the repository in other places it's easy for you to not have to re-import the keys in a bunch of other places you can use the curl at org's key ring like i've just demonstrated with my uh with my example or you can use the fault through a combination of all of the above you know before we'll look into any key ring src that you specified before it looks in your in your new pg key ring so you can play around with it i i recommend that you do this this is this is kind of uh i think a cool idea that allows us to simplify key public key management for contributors and for our sub maintainers so whose keys should be added to the project key ring uh it doesn't really make sense for one of contributors obviously so if somebody sent you a patch that you've never worked with this person before uh you can just always you should always assume this the code is malicious and you should be reviewing with extra zeal right but uh maintainers don't really scale so if there's somebody that you work with on a on an ongoing basis day in day out right you want to uh be able to sort of have a quick way to verify that the patches that they sent actually did come from them from this particular person that they haven't been tampered with uh and root anywhere so uh this the key ring as per management as proposed by patatin before allows you to to add this as an extra sort of protection around your workflow um doesn't say you should stop being vigilant constant vigilance as always but uh yeah just keep keep this as an option especially if you work with a whole bunch of different contributors and you want to make sure that the patches you receive are coming straight from them thank you questions or comments send them to tools at linuxcrown.org um patches are welcome if you if you use patatin you find that it doesn't do something that you need or before or anything like that if you do sign the patches that will get warm and fuzzies uh so i will feel very good about that so be so free to do that and that really is all there is to say about it so thank you very much for being part of this talk i believe there is questions and answers you can ask me in the chat i actually have no idea how it works i hope that somebody introduced this um topic and uh please ask away or you can find me and uh on the mailing list or um hanging around thank you very much have a good day