 Hi folks. I'm Matthew Riley. I'm a software engineer at Google. Today I have the privilege of presenting along with Marina Moore. Marina is a PhD student at NYU, well known for her contributions to the update framework and the notary project. So the topic of the day is software supply chain security. At least from where I'm standing, it seems like we've agreed we're going to solve software supply chain security with signatures. To be short, code signing is nothing new. Just to call out a few examples, Microsoft had authentic code signing for Windows binaries back in the late 90s. Devian has secure app since the mid-2000s. Of course, we have some more recent container focused efforts like Docker Content Trust and Cosign. Now, something we noticed about these newer efforts is they seem to treat signatures as an end in and of themselves. Look for signed containers has become looking like for the padlock in your browser. If it's there, you're safe, right? Not quite. It's not quite where we are in part because on their own signatures don't mean anything. I mean, okay, there's one law of physics thing that signatures mean. A valid signature means that the private key and the signed data were in the same place at the same time once. But that's it, that's as far as the math can take us. If we want to make decisions based on valid signatures, if we want to use signatures to improve the security of an ecosystem, we need to rely on policy to answer questions like whose key was used, how did they protect it, and how is validations supposed to work? Policy is what makes signatures useful, and coming up with beneficial policies is the hard part of any signing infrastructure. That's what we're hoping to spend our time talking about today. First, I'm going to tee up the problem by looking at how policy defines an ecosystem and what sort of questions a policy might address. Then Marina will look at how this applies to real systems that chose policies based on their specific use cases. So we said policy is the set of rules that make signatures mean anything. To put it a different way, policy allows signers to predict how verifiers will behave and vice versa. And to grossly oversimplify, every policy decision that we're going to make falls on a spectrum. Or on the one end, we make signatures easier to create, and on the other end, we make signatures more meaningful. At either end of that spectrum lies peril. If we make it so easy to sign things that anyone can do it for anything all the time, then validators don't learn enough to tell good software from bad software, and they won't bother validating. On the other hand, if we make signatures mean a lot by making people jump through more hoops, then we reduce the number of signers in the ecosystem. We reduce the coverage of signatures, and validators can't rely on software being signed. At either end, we kill the ecosystem. What we're looking for instead is a trade-off, a spot somewhere in the middle where we can meaningfully improve security for everyone. All right, so we've talked about how to think about policies and trade-offs at a high level. Now let's get into the nitty-gritty of some specifics of what a policy needs to answer. And we'll start with one of the most foundational questions. If I have something that's signed, who signed it? How do I match a signing key to some real-world identity? Now, again, there's a real trade-off here. If I make this too hard, if I require ID cards and lawyers and contracts, then only big, well-resourced companies are gonna bother signing anything. And the long tail of software, the left pads of the world, are going to remain unsigned. But if I make it too easy to get a code signing signature, if I don't require any proof of anything, then we end up with spoofed identities and a lot of signatures that don't really tell us much. It turns out signatures have turned our software distribution problem into a key management problem. That problem is different, but it is not meaningfully easier. Different ecosystems end up solving this in different ways and the solutions tend to cluster into one of a few buckets. Some systems rely on a central directory or arbiter of identity, which is trusted to make statements like this key belongs to this name. This is sort of like the DNS approach. Other systems ship with a hard-coded set of keys, which are either used to sign things directly or delegate authority to other keys. This is like the web PKI model of certificate authorities and root CAs that ship in an OS store. Finally, some systems choose to provide a weaker guarantee. Rather than check that things are signed with the right key, they assume the first key they see is correct and they just make sure that the key stays the same over time. This works as long as your first request wasn't tampered with and as long as the entity does not change keys over time. Okay, so let's say that we have a way to map keys to real-world identities like people or companies. Now we have a new problem. A company doesn't sign code. Some computer at a company signs code. Wherever that signing happens, we need to keep the key protected so it is still meaningfully the company's key. We could bring software to a developer's desktop to sign it before release. Ideally, this would not mean that every dev has a copy of the company key. Maybe instead they have individual keys stored on hardware tokens and we have a delegation or a thresholding story for those keys. Even then though, we can only trust the signed software as much as we trust the sketchiest piece of software on that dev's machine. If we sign on some central infrastructure instead like a continuous integration server, now we have a chance to run more rigorous security checks in a more controlled environment. We could require that everything be built from head on a protected main branch that enforces code review. But now we have to have a machine key and that machine key is subject to machine compromise. It's especially true if we're using a persistent build server rather than throw away VMs. These examples also demonstrate another policy decision. Do we wanna keep our signing key somewhere that it can be accessed and used by automation or do we want it locked away where we need to run some sort of ceremony to get to it? We can trust those offline keys more, but we can't use them as easily, which might lead us not to assign every build or every commit. Spent a lot of time talking so far about how to get signatures right. But no matter how hard we try, something will go wrong. Your signing key will accidentally end up on GitHub. A developer's UVA key will walk off the desk over a holiday. Or maybe you kept control of your key but you signed a bad build because your source repository was compromised or your build server had malware and you didn't know. Our policy needs to have a story for how we find out about mistakes and how we fix them. In practice, the most common way to find out that your key was used to sign something bad is for a user to find that something bad on their machine and send it to security at your company.com. In some more elaborate setups, that key might be on a protected server that generates audit logs or we might have a binary transparency log setup where we list all the artifacts that we've signed that validators should trust and we can go through that list to see if anything is out of place. Once we find out about a problem, our policy choices will determine how easy it will be to fix. For example, how many good signatures do we need to invalidate to make sure the bad one doesn't pose a danger? Can we revoke just one signature for a release? Do we have to revoke a key and invalidate everything that it signed for however long it was valid? Or do we need to burn everything down and start with a new root of trust? And if we do need to do that, can we? We also have to think about what happens to users while we're fixing this. How quickly is a bad signature invalidated? And then if someone tries to use a container with one of those newly invalidated signatures, what happens? Do we fail fast? Do we show a reasonable error message? Or do we just prompt the user and say press Y to make this your fault? Dealing with mistakes is by far one of the hardest things to get right in a code-signing ecosystem. And that's because by definition, you're starting from a bad state and you somehow need to get back to normal without creating any, or at least many, new surprises. All right, we've talked about policies and the questions they need to answer. Before we get into covering some real systems, I want to quickly recap what we've seen so far. We saw that a signature only proves that the private key and the data were together once and that everything else, all the meaning we hope to imbue into that signature comes from policy surrounding how we map keys to identities, how those signatures should be validated and how we heal the ecosystem if things go wrong. That policy needs to be universal, agreed on and relied on by all the agents in the system. And it needs to find the right trade-off, the right equilibrium, where it's easy enough to deploy and strong enough to be valuable. Finally, to bring this back altogether as we go looking at real systems, remember that any useful policy needs to start with a full threat model of the system that we're trying to protect. We need to recognize the valuable targets, how they might come under attack and how we will use mitigations like signatures to frustrate those attacks. And with that, I'm gonna hand this over to Marina. And we're gonna hear about how these decisions have played out in real life systems. Thank you, Matt. So as you've heard so far in this talk, signatures are just one piece of the puzzle. In this section, I'm gonna discuss some of the policy trade-offs that really complete this picture. Each of these policies is based on a threat model for a specific project. And the examples I'm gonna use are from software distribution and I've seen these through a lot of the work I've done on Tuff. So first, a little bit of background. Tuff, or the update framework, is a framework for secure software updates. It uses cryptographic signatures to sign metadata that provide information about various pieces of the software update process. Each piece of metadata is signed by a role, and a role is just a key or a collection of keys controlled by a person or a collection of people that's responsible for a specific piece of this update process. And really responsible for a specific piece of the metadata in this update process. These roles can delegate some or all of their authority to other roles by signing metadata with the public keys of a role that they're delegating to. And Tuff makes some policy trade-offs itself, like some of these signature policy trade-offs that we've talked about, like specifying how delegations can be used for revocation and also things like timeliness and consistency of different things in the ecosystem. But it leaves a lot of policy decisions to individual implementations, and that's what I'm gonna talk about next. So as our first case study, I'd like to talk about Uptane, which is the automotive variant of Tuff. This is pretty unique because automobiles, sorry, are a very controlled and safety-critical environment, and this really affects the policy trade-offs that are made in this implementation of Tuff. Because cars are physically in a factory before being sent out onto roads, Uptane can do key distribution just in the factory, in person, which is pretty unique in the world of software. It also has a pretty interesting approach to the trade-off between online and offline keys in that basically it uses both. So the image repository uses two different repositories. The first of these is the image repository, which contains a library of all of the images that are currently approved for use in any car, and these are signed using offline keys. And then we also have the directory repository, which uses online keys to specify what images a particular make-and-model of car or specific vehicle should install at any given time for various reasons. And the vehicle then uses a variation of threshold or multi-signature signing to ensure that any image that's installed on that vehicle was signed by both the director and the image repositories. And this gives Uptane both the security of using these offline keys, as well as the flexibility of online keys, which is a pretty cool system. So the next case study I'd like to talk about is PEP480, which is a proposal for end-to-end signing of Python packages with Tuff. The first half of this proposal, something called PEP458, is in the process of being implemented in warehouse, and by the time you're watching this, it will likely be in practice in warehouse, which is the PIPI server. The key piece here that informed the trade-offs was the importance of developer usability. We really want end-to-end signing to be adopted by as many developers as possible to secure as much of the Python ecosystem as possible. And as we know in supply chain security, that every package has dependencies. So the more of these packages we can include in our end-to-end signing solution, the more of them will have broad consequences. So for key distribution, PIPI hosted a public key generation in signing ceremony, which was live-streamed and also recorded to establish public trust in this generation process, the key generation process. These public keys are posted online, publicly available in as many places as possible, and they'll also be distributed in PIP and other warehouse consumers, so they can automatically be used as well, but you can verify using the online sources. So warehouse controls these root keys and then uses them to delegate to some intermediaries, what Tuff calls the target's role. And then these intermediaries delegate to individual developer keys for specific packages. So does this in two ways. For established projects that are known periodically, these are delegated to using an offline key controlled by warehouse in kind of a manual process. But between these manual processes, we still want to have this end-to-end signing, and so new projects can automatically be added using online keys that are just controlled on the warehouse server and no manual interaction needed. So the final case study that I'm gonna talk about is Notary V2, or specifically, the Tuff Notary Subproject of Notary V2 that implements Tuff on registries. One of the main goals for many of the Notary V2 implementers is real-time availability and scalability. This case study also has the particular challenge of dealing with ephemeral clients. And for ephemeral clients, trust on every use is the first use. So if using trust on first use, it's kind of an infeasible way to manage your key distribution. So instead, for this key distribution piece, Tuff Notary suggests distribution during container provisioning or using existing secret distribution mechanisms like Spiffy Inspire. Also in Notary V2, real-time use requires the use of online keys. But security risk of online keys can be mitigated through delegations from offline keys. That's kind of the way that this works in many of the cases of people using this. And so this works by having an offline key that lists the online keys that'll be used for your day-to-day signing. And if anything goes wrong with this online key, if it's lost, if it's stolen, if something happens to your server, this online key can then be revoked by using the offline key that is not on your server. So you can use this offline key to then create a new online key and use that for any future signing. So the main takeaway from all of these examples is that policy decisions really depend on your threat model. And you really can't make these decisions so you know what your threat model is and what particular things you're protecting against. So if you'd like to use signatures, first look at why you wanna use them and ask yourself which policy trade-offs make sense for your particular project and the things you wanna gain through the use of these signatures. And then finally, I wanted to include a friendly reminder that once you have these signatures and policies in place, make sure that you implement the other side of this where you actually verify the signatures and enforce the policies so that they're actually used to improve the security of supply chain instead of being used just to check a box. So thank you all for watching and please let us know if you have any questions in the chat. And our contact information is listed here if you have any further questions or if you missed this live Q&A session. Thank you everyone.