 Good morning everyone. So, fun to see so many people here, even though it's Friday and the last day of the conference. So, welcome to this presentation. We will discuss how we can use the update framework to secure your production delivery systems. So, I'm Fredrik Skolman, work at Staff Engineer at GitHub. My team is primarily focusing on supply chain security and artifact integrity. And I'm also very active in the open source community, especially around the update framework and sick store. Yeah. Hello. My name is Kyle. I work at testify sec. I'm an open source engineer there and I'm an author of repository set for tough that we will present today here. Um, and also I'm maintaining the in total archivista and I'm, I'm trying to be very active at the tough community as well. So we're going to start now. So to begin with, I would like to present an expiring example where the update framework might be something for you. So usually when you think about securing the packages from a registry, you might sign at the registry side. So the where fire can make sure that the package they are getting from the package is the one expected. But there are a lot of pieces missing there. And I think we two can do better by signing earlier on in the stage. So for instance, we can sign at the time we're building the package. And by doing that, we can also collect a lot of other important metadata about the build, such as which repository did you build from, which get commit, which workflow file did you use and so on like what were the actual build instructions. So this is typically called like the build provenance. So it would be preferable if we can sign not just a package, but also the build provenance that describes everything around it and ship that to the registry. And then when you're pulling down the package, you can pull down the build provenance as well and verify that. But that creates some challenges in how you're going to manage the signing key in the CIC system. It's slightly more complicated than just having a registry with a single key. So that's a little bit how I want to talk about it and how the workload identity can be used for that. So for NPM, we fixed this. It was GA last year, I think. So when you're building a package with NPM, you can add a flag dash dash provenance and we will automatically collect all the context from the build and create the salsa build provenance with that and sign it together with a package and push it to the registry. And this works very good for open source software because when you are signing with CIC store, as a consequence of the signing operations, a lot of information about your repository and so on will be put on a public append only ledger. But if you're using a private package or maybe building some inner source scenario, you may not want to leak that information to a public ledger. So for that, you may want to host a private six store instance and, yeah, what will it take to run one of those? So I'm not going to talk about operating the entire six store because that's a fairly big thing and it's not really relevant. But what I want to focus about is how do you secure the trust route for your six store instance. And the trust route is typically what you would think about metadata describing the services and the certificates, the entire certificate change, the public keys and so on. And this is a highly secure or is a highly critical piece of information. So you need to be very sensitive about how you are protecting this because if you, for instance, managed to get a certificate chain for your testing environment into the production trust route, you may end up in a situation where you can actually verify a test bill properly and put it into your production. So you need to have extremely tight access controls for the trust route. So this is kind of similar, you might think, to like managing a trust route for the WebPKI, but it's very different. For the WebPKI, you have millions of untrusted intermediate certificates and other identities that are floating around that you may need to care about. Assigning a code signing certificate is slightly smaller. The use case is very different. You usually have a much smaller number of certificate chains. You know about them at time and you don't change them as often. So it's slightly different from running at the web scale. And also the client has to be tightly integrated with a trust route. The client has to be able to learn about new certificates or which certificates are not valid anymore and preferably the client should be able to verify that everything it got is the expected one and it should, for instance, be able to detect if no updates can be delivered. So there are some functional aspects here as well that's very different from how you might think of code signing a forties. And the security is also very different, or it's not very different, but it's very important. And it's primarily two things you need to consider and that is securing the delivery of the trust route to the client and you need to protect the trust route at rest. So there are various attack vectors in here like you can have an external actor, an internal threats. Could be honest mistakes. But important is that when the trust route is deployed and ready for consumption, you have to be able to verify that what's actually being deployed is the expected one. So it can't be tampered with once you have deployed. You need to continue and monitor for correctness after it has been deployed by the system. And the same for the client, like of course you need to be verify that when you're pulling down the latest trust route, it is also the correct one and the complete integrity is sold and it actually was approved for deployment. So there are some options we can think about and let's talk a little bit about X509PKI and how that can help and then we'll dig a little bit more deeper into the update framework. So X509PKI is a known standard. It's been out there for a long time. It's proven to work kind of good. But also have features for certificate life-side management in terms of various protocols for accessing certificate stores over different APIs. And there are concepts such as certificate revocation list or other online operations to be able to detect what certificates are not valid anymore. The problem is that it can be quite challenging to operate PKI system based on X509. There are many things to consider. It can be hard to work with certificate revocations list because you might get partial updates that you need to combine at the client side to sort of get a view of the complete state. And it's the PKI itself. You need to sort of have good controls over what's being deployed and how to secure that. And I think it kind of boils down to you need to write and operate a lot of custom code to manage such a system. So with that I would like to introduce the update framework, which is a framework for securing updates over insecure channels. It's used in many systems such as agent delivery systems and there are adaptations for the automotive industry. The framework specifies the behavior of the client and the interaction it should perform and it provides very strong and good security guarantees around the service side repository. And all this makes it very suitable for managing a trust route. A tough repository consists of a lot of metadata files describing the entire repository but it also describes all the different artifacts that's going to be delivered to the client. And this is pretty good because it allows you to always provide a complete state to the client so the client doesn't need to merge files or things like that to understand what's the current state it should operate in. And a very strong guarantee you can get with tough is that you can have threshold-based signing. So that means that you can designate a team of signers that will be working to approve and endorse any changes to the system. And with threshold-based signing you can configure in such a way that you can for instance require free persons to agree for any update to be delivered. And that mitigates a lot of attacks such as single actor trying to change the content. And those keys that are used to sign off the entire tough route are easy to revoke and rotate in the event of let's say a key loss or if some key is compromised. Tough itself also provides a lot of security threat mitigations in the client interaction and it has a lot of good protections. Two examples would be freeze attack and a rollback attack where an attacker might prevent a client from getting updates or the attacker might try to replay an earlier scene response to trick the client into entering a state where it's using obsolete data. But the strongest thing I want to push for again is that with threshold-based signing you can have very good confidence that the content you are actually using is the one that was approved for delivery. And with tough you also get a good benefit in terms of client implementations. There are a lot of client implementations in different languages for different platforms such as Linux, Mac OS and Windows. When you think about tough, you can kind of think of it as your favorite operating system kernel but you can think about the different repository implementations such as distributions where there are tailored for specific needs because depending on your use case you might have different needs. So there are different implementations out there that focuses on specific areas. Sorry. So I want to talk a little bit about tough on CI. So tough on CI is an open source implementation of repository automation. So it's very tailored for use case where you don't have that much or that many frequent updates and you won't have a good security posture and use hardware security models for all designing operations. And relying on HSM keys is very good because when it's pin code protected you enter a security mode where it's something you have and something you know. So it's very strong in protection against threats. Tough on CI works by relying on a Git repository and when you add content to it, automation kicks in and detects for changes and understands what has to be done to continue to get everything signed and into a working order. So the automation detects that open up a pull request that guides all designers through what has to be done by providing comments that says what has to be done. So you can see a simple screenshot where repository automation says that yeah, this new update is now signed by the expected signers and we are ready to proceed to publish this. So it's very easy to use. So an overview of how it looks is that you might have a team maintaining the trust route, the certificates and the content that goes into repository. So when there are new certificate chain you would like to publish. You would extract it from let's say your Clyde KMS store or whatever it's stored. Then you add it to the repository and the automation detects that there are new content to be delivered and it opens up PR and notifies all the different signers that there is an action to be taken. They need to sign off this new change. When the threshold of signers has been reached, the PR is now ready to be merged and the system notifies you that you are now ready to proceed. Once you merge the PR, automation detects that and prepares an update and stores it on a content delivery network where it can be stored, sorry, where it can be accessed by the clients. So to operate a system like this, you typically would like to have two teams. One team that is maintaining the actual content and prepares all the updates and any tooling that you would like to have around it. Then you would have a designated team of individuals that are designers that verifies and reviews all the changes and endorses that by signing. And so acknowledging the updates. When you're dealing with hardware security module or HSM keys, it's very important that you think about the key management and your disaster recovery options or what is happening if a key gets lost like what's your incident response. So I won't go much into detail about that, but Kaira will talk a little bit about how you can do it. Because it's very important when you're dealing with a tough repository that you do not lose access to the keys that are used to sign it. Because if you're losing too many keys and you can't sign a new update, you're in a bad situation because you may lose access to the trust route. And of course, once you have it up and running, you need to have constant monitoring to make sure that no one is trying to, for instance, change the content. So it's always the expected one out there. So to sum it off, operating a tough repository today is not that difficult and there are a lot of tooling out there that can help you to achieve your goals. So if you have a system where you don't have that many frequent updates, Tafone CI is a great tool and I definitely think you should give it a try. If you have a scenario where you have more frequent updates, I will now hand it over to Kaira and he will present how you can use Taf for that. Thank you, Fred. Yeah, I will talk about another implementation of Taf, that's RSTaf. So RSTaf is a generic application and the primary goal was also to make the generative adoption easier. So the project is early adopted by RubyGems and PyPI. I say early adopted but it's a work in progress, so we are implementing it together. And the project is under OpenSSF and of course it's an neutral governance structure, so it's a project where everyone is welcome to contribute and help us. Well, my talk would be a little bit different because he gave us a lot of concepts about Taf and how it's important to secure the ways to do this. And I want to go in more operations with Taf. I will give an example using RSTaf to secure a self-managed repository. In that case, I will showcase of a JFrog repository and RSTaf helping to secure this repository. So this is an example from an organization, so let's understand the organization problem. So it's a software company on secure industry. They provide different artifacts, so they have server applications embedded in the software and client application. And they want also to secure the artifact distribution and have a well-defined release process and incident process. So the challenge on this, the first one, was to define the release process together with this user and how they will release the internal and external artifacts because they have artifacts that are used internally and the artifacts that used to build the product that are shipped to the customers. So we need to secure the release process using Taf and integrating it with CICD and JFrog. And they want to also secure the distribution and they have different use platforms, Linux, Windows, macOS. And of course, very important to them, it's document the incident response process, what they do if something goes wrong. So the organization structure, the process flow, it's the most common that everybody is familiar, developers push comments to get, CICD building and tests, and release artifacts to the repository. I want to highlight a bit today repository because there is something that is interesting because they have a kind of two repositories. One is internal repositories when they release the internal libraries and toolings that are used to build the product to the customers. And those, sorry, the products, the installers and the data go to the repository that is exposed to their customers. So how they do the internal artifacts, it's important because they release very often. They have more than 200 release a day of their libraries and tool internally. So this is used also by the developers to change those libraries and tools and also use it in the CICD. So the internal repositories use it in that way. And the external artifacts that are the product, they use the, of course, they use the internal libraries and tools to build this product, and then release it to their artifacts, then it will be shipped to their customers. So when this is shipped, it's available to the users to download. That's, let's say, a GA release. Well, the solution implemented here uses rstuff, tougheam, gotuff, and they are experimenting tough.js also. This is to enumerate how many libraries we have out about tough, so it's easy to integrate. So let's see what we did. It's just adding rstuff along with JFrog Artifactory in a way that will be part of the process, but we don't change too much their process. That is good because it impacts low for them. And why rstuff in the self-managing repository? So the first reason is that rstuff, actually tough, it's artifact agnostic, so they have different tools and libraries that they release in different languages, like C++, Java, Go, and other languages. It's artifact agnostic, so tough and rstuff, or tough on CI doesn't care which language you are using. And it's released a process agnostic, so it's easy to fit it to the existing process. So we don't change too much the way that they release their components. And of course, rstuff can be deployed on private or public cloud and on premises as well. So just a quick introduction about toughy because I will use this here to show the case as well. They use this to show the case. Toughy is a small generic tough client that's written in Go, using the Go tough. It's another tough library and it's supported by Mac, Windows, and Linux. And it supports also multiple repositories, so you can configure and use different repositories. I will show it in a quick demo. So how was the implementation? So the first thing was the tough ceremony, defining what are the tough administrators. So they have the CTO engineering and release managers involved. And as Fredrick mentioned about the root metadata, how it is used there. So they have key owners for the root metadata that sign the root metadata. And they use the online key to sign the target metadata that is used to store the artifacts. And they have some management process internal like what happens if the head of product changes leaves the organization, they know how to rotate this. So I will describe it more later. And also process for online key. If they're online key, the KMS key rotates, how I update that. So the second part, it's integrated to the internal release where they have the internal artifacts. So how it is done, the process is still the same but we introduce tough. So basically the developers push the code to the Git and when it's the time for those tools and libraries when they release it, they have the version tag and it will build and add the artifact in JFrog repository and also add this information to the tough metadata using an API call. And this will be signed by the KMS key. So and for the product, it's a little difference in the process because the process is not started by the developers. The process is started by the release and product manager. What they do, they define all the libraries tools that will be used to build the product in agreement with the developer leaders, then they push the release. And it pulls all the, from the internal artifact repository all the definitions of libraries and tools that used to build and when it's built, they create a pre-release and they sign this pre-release with the HSM keys as well, the UB keys. And this is validated by the CI CD and it is released again to the tough metadata. So there is one thing here that you see it's a little bit confused because it's not what we want with our stuff, actually what we want to simplify it. When the product is built, they sign directly the tough metadata. So we skip a lot of process here and when it's signed, it will be available to their users to consume it directly. So we want to simplify this. This is a feature that will be coming soon, the possibility to use offline keys in the target metadata. So now it's the part that is really important is to use the tough client everywhere to secure the users outside of the organization, the clients, customers, and also secure the developers internally for an internal attack. So I want to highlight the internals because I really like the solution that they applied here. First they have developers, as I said, they develop different languages and toolings and libraries. So they have MacOS, Windows, and multi-platform. And also in the CI CD, of course, they have multi-platform and we need to help them with that. So what they did, the IT department deploys in our machines the toughie with the trusted root. So they can use this to download the artifact. So what they do actually, they did something very interesting for me. I want them to share with us. They create a VS Code extension that is basically a package manager for the internals. And when developers want to download some library or tool to work, the source code, they use the VS Code to search and install it. So it's a very interesting solution. I didn't see it yet, but I want. And they use the tough also when they want to build the product as well. So in the CI CD, they use this to download the source code. So the product for the clients, the solution is still on development, but it's an interesting case. What they do is they create a very light installer that the user can choose what product they are allowed, of course, by license, all those requisites. But they can choose what they want to install or download. And they use this to download. And they are using that case toughy, but they want to change it to... because this application is Java. So, and once they have the product, what they have in that product, they have the opidators that it's using tough as well. For these, they have different use. They have toughy, go tough, and now they experiment tough.js as well. They use this... Every time that the user want to predate this tool, they use this to pull from using the tough metadata and securing that. So I couldn't have more details from the user direct... this case direct. So what I did, I wrote... I did a quick demo. I did this yesterday and today just to exemplify how they do this. So here I'm using GitHub and JFrog and R stuff as well. So I'm releasing a new version of this demo package. It triggers the CI CD in GitHub. So I approve the release. Once it is approved, it will add the artifact in the JFrog and in the tough metadata. You can see through the runners that will be very simple API calls for R stuff as well. And who is familiar with JFrog, you see next that it's a similar... a traditional call that is not changing how you use JFrog's artifact. So, yeah. I did also a quick demo that it shows, of course, the artifact in the JFrog, but here I'm using the tough... I have the repository configured and I will use this to download the artifact. And behind the scenes is using tough clients to do all the verification and metadata before fetching the artifact. Yeah, that's it for this demo. Let's see. So last thing that's quite important is the incident response. So how they map this. Of course, this is not their document. It's just an example that I wrote to understand how it could be effective in any organization. So we have some administrative... It's not a security incident, but it could go to that direction if not there. But what happens if a root key admin leaves the organization or change their role? So they have the actions that they should do. What if DevOps member with access with the key MS leaves the organization so they can rotate the root metadata, resign it. They don't need to go to each client and give the new version. It's done by the tough metadata. So... And also here is some incident response examples. Like what if the key MS key is leaked? What if the ICD pipelines have some access leaked? What happened is the tough API? So here I create some ideas of how they could do this, but of course each organization is to map it to their purpose. Yeah, and we are on time, and that's it for our presentation. Thank you. We have time for questions. I think four minutes is left, so there are microphones somewhere if you would like to ask us something. Hey, so I must admit that I have heard tough mention often, but often skipped it a bit because it seems like... I'm going to say something stupid, the more complicated six-door. So my question is probably a bit, when would you put up a scenario where yeah, you should... tough is the right choice for this and above six-door, not as to they want to split up, but more like when to use what? When we think about the secured supply chain, I would say that the tough is the... we are familiar with the secured supply chain like the salsa graph. It's in the end of your supply chain. I mean, when you want to distribute any kind of artifact that needs to be trusted by the users, I think that's the point where you should consider to have. Okay, well, is nobody else going to ask then do you also use tough to sign the data like S-bombs and stuff like we use six-door often? Yeah, it's a good use case, adding the artifacts of the S-bonds in the tough metadata to create a trust between the users. If I get your question. In six-door, it's very normal to sign S-bombs and stuff agitations. So is that also a normal thing to do in tough? Yeah, so for that case, in six-door, we only use the update framework to actually provide the root certificates and the public keys for the infrastructure. And then, via that trust route, you can then use Fulso, which is a certificate authority, and create a signing certificate that you use to sign your artifacts such as your S-bom. So you wouldn't really use tough for that in the six-door example. So tough is more like an enabler to provide the trust route to the six-door clients, and then they use the rest of the six-door infrastructure to create signing certificates to sign what is ever produced in the CI-CD pipeline. Thank you. First of all, great talk, well done. Yeah, just following on from this guy's point, given that it's artifact agnostic, doesn't that mean that if I choose to, I can secure my attestations, S-bonds, that sort of stuff with tough without six-door. Is that correct? Yeah, you have your own trusted root metadata to secure your attestation. Of course, you could also use the six-door to sign your attestations, but you could create another layer of security. Okay, thank you. I guess we're done. Thanks for coming. Thank you.