 Okay. Hello everyone. Thanks for having me. Thanks for coming. My name is Lukas Püringer and I'm a researcher and engineer at NYU's secure systems lab where I work on supply chain security projects. One of those projects is tough short for the update framework. And I should have given this talk today with my dear colleague UC from BMWare, but he had to cancel his trip to Valencia at the last moment. So I will try to cover for him. The title of this talk could be read in several ways, updates from the update framework. So one way to look at it is that tough is a framework to deliver secure content or to securely deliver content can securely deliver content and allow secure updates. So we're going to look a little bit at content delivery, why it's important, why it's important to protect it and how tough can help you with that. And the other way of understanding the title and this was the original intention is that tough is a CNCF graduated project. And I'll just give you updates of what happened in the or under the umbrella of the tough project in the past year. All right, but let's start at the beginning. So software updates, software distribution is a very crucial part of the software supply chain. At the edge of it, it is where things get multiplied. So whatever comes out of your software supply chain, eventually it lands on many, many devices. And those devices could be computers, it could be container clusters, it can be phones, IoT devices, your light bulb, whatever. So that's why it's also very attractive for attackers to attack somewhere in the supply chain or at the content distribution infrastructure because a single compromise can have huge impacts. In fact, thousands, millions of devices. So I guess we all agree that we somehow have to protect the content and the content delivery infrastructure. And we have to do that at scale because the software supply chain is rarely write code, build it, distribute it, but it's really a supply chain of supply chains. So you pull in all these dependencies and so you can't just focus on your package that you maintain, but you also have to think of the entire dependency graph. Because again, one single supply, one single compromise somewhere deep down in the dependency graph has severe consequences. It affects all the packages downstream. And yeah, who knows who works on this package? I mean, do you know it for all your dependencies as a maintainer? Maybe this is a sole maintainer without two-factor authentication enabled. Maybe he's or she's frustrated, working for free. And everybody relies on that package. So, and not even the maintainer of packages know where all their dependencies come from. So we really can't expect the end user to have a good picture of this entire dependency graph. So the solution is signing everything. We are already doing that. We're signing a lot of things and it is a good solution to protect the integrity and authenticity of artifacts, but it has its own problems. Signing alone won't be enough. So you also need to know who you trust to provide a signature. And you need to do this at scale, as I said before in this graph. And even if you figure that problem out, you need to know how to deal with a compromise because signature keys get lost, they get stolen, and that is fatal. And revocation has its own set of problems. And when you deal with a revoked key and you have a new one, then you need to start with the trust decisions again. And then there are additional properties that you care about, which are freshness and consistency of your repository artifacts. So the good news is that the tough framework provides ideas and mechanisms to deal with these issues. It has built in protection for freshness, consistency, and integrity. It allows you to delegate trust at any desired scale. And it has built in compromise resilience. That means both it reduces the impact of a compromise. And it allows you to recover from a compromise in band. Let's take a closer look. Tough uses cryptographic signatures for this by signing the content itself. It protects its integrity. And by signing the entire repository, it protects the consistency. And it has a multi-tiered signature exploration scheme to make sure that the content you download or update is fresh. So that is really the easy part of tough. You probably wouldn't need your own framework for this. But here it gets more complicated but also more interesting. So bear with me. This graph you see in the middle defines the roles that are or is like shows the roles that are defined by the tough framework and the delegations, the trust delegations. And there are several different trust delegation mechanisms prescribed by tough. So first on the top part of this graph you see the root delegation. This is where we separate responsibilities for a repository by roles. So there's a dedicated role for the freshness of the repository. There's a dedicated role for its consistency and a dedicated role for the integrity of everything. And it all goes back to the root of trust, this root role on top. And we'll see on the next slide while this delegation is very helpful. And the second delegation mechanism separates responsibilities by content. So this is the lower bottom of the graph with the arrows. The integrity role in tough can itself delegate to more integrity roles. And by that you can kind of name space your trust in your repository. You say I only want this role to provide signatures for let's say in the case of PyPI and Python packages. I want this role to provide signatures for the Django project, this web framework. And I want a different role to provide signatures for the cryptography project. So I don't want each signature be trusted for everything. And then there's a third mechanism for delegation that works within a role. It means that you can share responsibility. So even for a specific role like the root role for instance, you can say I don't want to put the trust in one key, but I want a signature threshold. So that was all about trust delegation that also helps us to reduce the impact of a compromise because it allows us to balance out availability, risk and responsibility. So by separating things into different roles like a role for the freshness of the repository, one for the consistency and a completely separate role for root of trust, I can make so that roles that have high responsibilities can minimize the risk of a key compromise and make them by making them offline roles. So for instance, the root role is only responsible for delegating the trust to the other roles. And that means I don't need it so often and I can sign it with offline keys. Roles for freshness and consistency on the other hand are in a repository that gets updated very often are high availability keys. So I will likely use online keys for this, but I don't give them any more responsibility than that I have. I don't use them to delegate trust to other roles for instance. And so maybe I should have said this earlier, delegating trust to a role basically means listing the keys for that role. The root role has the public keys of all the other roles and the keys that the other roles use to sign so this allows us to use the delegating role or delegator to just change the keys and thereby rotate keys for a delegated role or delegated. I know this is a lot and I'm happy to talk more about it after the talk if someone has questions about this. It took me quite a while to understand it and I'm still trying to make it easier for others to understand. So we've seen how tough works in theory. Let's take a look at what tough actually is at its core. It's a specification. Then there is a process for updating that specification, the tab process and then there is a reference implementation and apart from that there is a broader tough ecosystem. So the specification defines all those roles that we've seen in the graph, what responsibilities they have and above all the metadata format that is used to represent the role. And it also explains how to manipulate those metadata files for the roles in a repository to some extent and in great detail it explains how to consume that metadata on a client. Given that the specification is very dependable, so a lot of organizations or companies use tough, we can just go ahead and change it. But also a living document because new use cases arise, we have this tough augmentation proposal process that we use to discuss new features and slowly adapt them in the specification in a responsible manner. And then the last thing at the core of the project is the reference implementation. It's a Python implementation that offers a client updater and a metadata API. Client updater can just be stuck into any software updater and basically works off the shelf. The metadata API is more flexible for repository side activities. I'll talk about that later what that means. The reference implementation is very symbiotic with the specification. It's almost a little bit like secondary literature. So we designed it with readability in mind and also spec recognizability so that when you look at the code you're like, no, okay, this comes from the spec. But it's also production code quality. So Datadog actually uses our reference implementation. And I personally am very happy that we recently released 1.0 of the reference implementation which followed a huge refactor where we removed a lot of legacy problems and made the code overall more modern and more easy to maintain. So that's the core of the project. And then there is more of the project. There are several libraries in different languages that implement Tuff. There are several organizations and companies that use Tuff. Datadog uses it. AWS Boll Rocket uses it. Google's Fuchsia OS uses it. Six-store uses it. I'm going to talk about that later. Then there is a variant of Tuff called obtain which is it uses like it's specification compliant but adds extra things to work with the requirements of the automotive industry. And then there are a bunch of projects which are not really tough anymore. They are not specification compliant, but they were inspired by a lot of concepts of tough water. So let's dig even a little bit deeper what tough in practice means. I've already talked a lot about client and updater and also the repository side. So this is a notion that's really important for Tuff. And yeah, so far we've learned that Tuff is mostly a collection of metadata files, metadata files that lists keys, delegation relations, content and signatures over it. So in reality, this means that we need a repository that somehow maintains this metadata and the client that knows what to do with it in order to guarantee all those security properties that we discussed earlier. And one problem in practice is that the repository, the way you create the metadata there varies a lot from project to project. So you can just use the Tuff code and stick it in and then it works. You have to really think of which roles exactly you mean you want to use. But the good thing is that the client is relatively easy to use. And I will show two use cases where Tuff is used or almost used. So PyPI is the Python packaging index. I don't know if everyone's familiar with it. That's where you usually get your code when you do pip install. And we've been trying to integrate Tuff with PyPI for a while now. There are two different approaches or two different stages of securing PyPI with Tuff. The first one only signs content in the repository that gives you the freshness and consistency and of course integrity properties that come with Tuff. It also allows you to like gives you a mechanism to rotate keys with a safe route of trust in band. But it doesn't give you the fine grained namespaces for different projects. And it also doesn't allow you to have offline keys for the content signatures because content gets uploaded loaded every second. So these keys need to be online to sign it as soon as they're uploaded. The second approach, these two approaches like the first one I just talked of is defined in PEP 458. We've been working on that hard in the last year and made some good progress. The second one which bases on PEP 458 is called PEP 480. And this brings you these extra guarantees with namespaces for different projects and with offline signing keys for the artifacts in the repository like the packages by making developers sign the code and the repository delegating trust to the developers. This has a bunch of usability problems, how developers manage the keys, how the trust negotiation between the repository and the developers gets established in the beginning. So we're still trying to figure this out. And community repository that means repositories where packages get uploaded by random people are really a hard use case. There are easier use cases like SixTor. They also use Tough to protect the root of trust of their certificate authority and of their transparency log called Recor. So in their case, they don't use the Tough targets role, which is for protecting the integrity of the content to protect any actual software, but they use it to protect their keys, which gives them the guarantees that come with the root signature thresholds. It allows them to easily rotate keys with root if they get compromised and also has these freshness guarantees so that the attackers can replay stale metadata or stale packages. Yeah, those are two projects that the PyPI project is something that I've been working on a lot. SixTor is something that the SixTor people have been working on a lot. They are featured with a booth here at the conference. And both is still, I mean SixTor is not so much work in progress. It's actually used in production, but there is still more to come. They're trying to expand that lower part of the delegation tree with Tough. And there we're almost at the end of the talk. There are many things happening in the near future. Two of them I wanted to point out are the repository playground. This is something that UC who couldn't come today is spearheading. It's basically the attempt to create a new prototypical community content repository with real workflows and best practices that can then be easily adopted by by real life repositories like PyPI or MPM. Because the problem of repositories like PyPI or MPM is that they are relied upon by many, many, many people. So they are not so happy with experimenting with things. And we can't really do path finding life in the repository. So we try to make a prototype that can then be adopted maybe by several content repositories. And that prototype should mostly be based on Tough because Tough really solves a lot of these problems already, but not exclusively. And another project which is related to SixTor is the attempt to use ephemeral keys that are connected to OIDC identities, which are identities from email addresses, to use those for developer signing of the Tough targets metadata. Because at the moment the targets role delegates normal local key pairs. And yeah, there are a couple of UX pain points with that, how developers can manage their key material. And with the SixTor approach, we could maybe leverage the usage of these keys that only live for a very short time and are connected to the email addresses and authenticity providers that already exist and everyone uses when they log in via Google or via Facebook or via GitHub or whatever. Yeah, that's it from my side for today. There are many ways to get engaged with the Tough community. There are many ways to reach out to us if you have questions, if you think that some of these ideas are good for your project, we're happy to help brainstorm how it can really be used. As I said, it is being used in many places. It looks very different in each place, at least the repository side. So it's a bit of a barrier to get started with it, but it's definitely worth it. And I have a lot of references on the next two slides. I will make these available online for further reading. Thanks. Yeah, and I think we have another 10 minutes or so for questions. So I'm happy to take any. There's a microphone, if someone wants to... Can I give you the microphone? I think it's recorded and might be nice to... Do you think or do you have any opinions for against Cosine and Notary V2? What was the second thing? Cosine and? Notary V2. Oh, I think they do different things, but I'm less familiar with Notary V2. I don't know it. Not anymore. So Tough was used in Notary V1, but it's currently under redesign. And I don't know what they came up with at the moment. Marina knows better about that. So I can't really give a recommendation, but what Cosine does looks really cool. The problem with Cosine and Sigistor in general is that it doesn't have this fine-grained trust delegation I was talking about. So you can name space who you trust for providing what. You can tie everything back to a certificate authority, but you have a signature for a container. You see that it is in some transparency log. It ties back to a route of trust, but you don't really know if that signature was allowed to... Like you have to do that separately. You have to somehow check the email address and see if it looks like the person attached to the email address seems like someone who should provide a signature for that container. But yeah. So I think, I mean, it's a super cool tool and the ephemeral keys solve a lot of the UX problems of signing, but I think it needs more, especially about this trust namespaces. Anyone else have questions? Can you speak more about how the trust namespaces work in a practical example? I don't know. In the Python repository, is it at the package maintainer signs and then as well the repository signs? So you have multiple layers of trust there? Yeah. So in the current draft of PEP 480, it would be a trust on first use that when you register with PyPI that you upload public keys that will then be audited by the PyPI maintainers and at some point get into the delegating metadata that delegates the trust to the developer. But it's a bit hand wavy currently in the PEP and this really needs to be figured out in a better way. I think that's a reason why it has not yet been implemented because it's not so easy. Yeah. Anyone else? I guess we can take this again. Yeah. Thanks everyone for coming.