 Hello, everyone. Thanks for coming today. My name is Hayden Blausfern. I'm a software engineer on Google's open-source security team and one of the maintainers on the Sigstore project. And today, we'll be talking about how to run your own Sigstore securely. If you're at this talk, you might have heard of Sigstore before, but if not, I'll give a really quick overview of Sigstore before we dive into how to set up one privately and securely. Sigstore is a project under the open-source security foundation that opened SSF, which is under the Linux Foundation. And its goal is to simplify code signing for artifacts. If you're familiar with the Project Let's Encrypt, it had a similar mission but focused on increasing HTTPS adoption to secure traffic for websites. And it was very successful in doing so by providing easy-to-use tooling and free services, in particular a free certificate authority, in order to issue SSL certificates. And Sigstore aims to do the same but focused on code signing. We enable this by providing free and publicly available transparency log and certificate authority services, which you can also run within your organization too, which we'll talk about in a moment. And one of its other goals is to remove the need to manage artifact signing keys. We found that this is one of the pain points for adoption when it comes to doing code signing. Users frequently don't have signing keys, so they don't really know how to manage them. So let's take a look at how Sigstore enables signing without having to manage keys. The first service within Sigstore is Fulcio, which is Sigstore certificate authority. And it issues certificates that bind public keys, which can either be ephemeral, meaning from a single signing event, or long lived, if the artifact owner maintains some signing key. And it associates this key with an identity that comes from an ID token obtained through the OpenID Connector OIDC protocol. An artifact owner will fetch one of these ID tokens and then give this to the certificate authority, along with a public key. Fulcio will verify that identity token since it's signed and extract the identity from it and include that in the certificate. And this identity could be an email address. This identity could be a service account. It could also be an identity that's associated with a workflow run on a CI platform, such as GitHub Actions. The public instance of Fulcio will also write to a certificate transparency log. And we'll talk a bit more about the role of transparency within Sigstore in a minute. But roughly, the purpose of this log is so that you can have an immutable issuance log, which holds the CA accountable to everything that it issues. You'll also see this within WebPKI that publicly trusted CAs will issue certificates for domains and include these in certificate transparency logs so that domain operators, or in this case, artifact owners, can monitor the logs to see how their identities show up in them. The second component of Sigstore is Recor, which is our transparency log for storing artifacts and their signatures. After an artifact owner signs an artifact, they will upload the certificate that came back from Fulcio and a signed artifact, a signature in the artifact hash to Recor. And then Recor will return back a proof saying that this entry was included in the log. And if you're familiar with transparency logs, note that I am glossing over exactly what this proof entails. But roughly, it's a cryptographic proof. The artifact owner can then take that signed artifact, the certificate, and the proof and upload that somewhere, let's say to a package repository so that any verifier that wants to use a package is able to verify the signed artifact. They can verify that using the public key that comes from the certificate. They can verify the certificates, matches a identity that they were provided out of band that maps to that artifact. And then they can verify the proof to know that the entry was actually included in a log. And the benefit of this is much like with certificate transparency, it allows the verifier to know that an artifact owner can be monitoring to see how their identity and their artifact shows up in the log. Maybe they have some requirement that an artifact will only show up once in a log, for example. This is an overview of all of six-store, putting all the pieces together. And the main thing to take away from this diagram is that I've also mentioned cosine. This is the command line utility that six-store has created that enables artifact and container signing and verification. And it's not the only tool within the six-store ecosystem too. We also have some command line utilities such as six-store Python or six-store JS that actively being developed that allow you to interact with the services within six-store. So one question that comes up a lot is do I actually need to rely on the public instances of six-store or can I rely on the private ones? Like I mentioned, we operate free to use public instances. And these have a SLO of three nines. We have an on-call team staff to monitor these services. And we recently, back in October, went general availability or GA for these services declaring that these can now be used in production workflows with more confidence. But organizations may have other requirements and want to have their own private instances of them. This could be for performance reasons. Maybe three nines is not sufficient. And the organization needs to have higher availability. Compliance is also another reason. For example, maybe the organization has data residency requirements and requires that the data that gets generated as part of the system live in a geographic region near theirs, whereas the public instance operates out of the US region. Another compliance reason may be that there's a requirement that the organization control routes of trust. For example, the certificate authority signing key or the signing key that's associated with the transparency log used for generating proofs. Another one is privacy. This is one that comes up fairly frequently. The identities that are included in certificates are written to these public logs. And these identities, like I said, could include emails. It could include service accounts associated with private projects. It could include workflow information associated with private repositories. And organizations may not want this information publicly available. So these are all reasons that organizations might want to operate a private sick store. So for the rest of this talk, I'll give some recommendations of how to set up a secure instance of sick store within a private organization. And this setup should give you the same guarantees that we have with the public instance. Note that I'm not going to talk about how to actually deploy sick store, for example, how to harden your Kubernetes environment or set up ACLs appropriately. They'll all mention at the end a Slack channel where if you're interested in that, there are some folks within the sick store community are working on how to simplify deployment for sick store too. So to start, let's talk about self-managed PKI or public key infrastructure. And there's two things I'll discuss here. One is managing artifact signing keys in contrast to ephemeral signing keys. And another is managed private CAs in contrast to the full COCA. When we talk about artifact signing keys, we typically talk about three problems. Distribution, storage, and compromise. Distribution is the process of distributing the map from a signing key or a verification key to an artifact. Frequently what happens is we end up relying on trust on first use or tofu. And we see this, for example, with SSH. When you first log into an SSH server, you're prompted to remember a key fingerprint. And if the server's key changes for any reason, you're presented with a dialogue. But frequently, users simply click through that and ignore it. And so what we want to do is design a system that removes the need to rely on trust on first use. Another consideration with managed artifact signing keys is storage. Where do you actually store the signing key? Are you going to put it on a flash drive? Do you encrypt that flash drive? What happens if you lose that flash drive? Are you going to put the key in a cloud project? What happens if there is shared access to that project? And how do you also audit access to that signing key to make sure there's not any unexpected access? And this comes to the final point, which is compromise. Inevitably, compromise does happen. And how do you mark a key as no longer trusted? Revoke that key. Within WebPKI, the way this is done is relying on certificate revocation lists. These are lists that CAs maintain that have a list of certificates that should no longer be trusted. But certificate revocation lists are very difficult to work with. They can grow indefinitely. They're difficult to ship to clients. Clients have to figure out some cadence to fetch these. And they're also hard to search once they grow really large. So this is another thing to consider when managing artifact signing keys, which is what does your revocation story look like? And are you able to efficiently work with really large revocation lists? And this is why Sigstore defaults to ephemeral signing keys. The idea is that for a signing event, Sigstore or the command line utilities is going to generate a single signing key that will be thrown away after the signing events. And then we associate that verification key with some stable identifier that comes from an identity token. But private organizations may not want this. They may already have an existing system in place for managing artifact signing keys. And Sigstore works perfectly fine with artifact signing keys. For example, when I mentioned the recore, I said that we can upload certificates. We can also upload artifact signing keys, too. I wrote a blog post a little while ago called Adopting Sigstore Incrementally that kind of talked through this problem. It assumed that an organization doesn't want to go from a point where they have managed signing keys jumping all the way to what we sometimes call keyless signing or this identity-based signing. And one of the intermediate steps was that you could issue identity-based certificates for long-lived keys or for managed keys. Folio doesn't enforce ephemerality. It doesn't have state and track if it's seen a key before. So the question really comes down to, what do you want your verification policy to look like? Would you prefer having keys mapped to artifacts or sorry, identities mapped to artifacts? We have found that identities are a little bit easier because we expect that users already have an identity to use. And very frequently, that identity is already publicly mapped to the artifacts, maybe by association, maybe by the identity being specified in a repository. But if an organization already has the right policies in place, there's nothing preventing them from using artifact signing keys. Switching topics, now let's talk about the CA side of things. Many organizations already have managed CAs. These could be through things like Step CA or through a cloud provider such as GCP or AWS. And Sixdoor will work out of the box with certificates issued from any CA, not just Folio, as long as they conform to the Folio certificate profile. This includes things like setting the identity of the subject alternative name, and this example here I've shown in email, including an extension that specifies an issuer, which is the OIDC provider that issued an identity token. And optionally, including an extension that this here is called an SCT, which roughly is a proof that something showed up in a transparency law. You'll hear me say key management a lot throughout this talk. This is a common problem, not just with artifact signing keys, but also with the services that are managed and have signing keys. And this is something to consider with managed CAs too. If the signing key for the CA gets compromised, a attacker could mint certificates for any arbitrary identity by passing any sort of authorization checks. Something else to consider is rotation. CA certificates expire, or in the event of a compromise, you may have to rotate it. And so as an organization, you need to have some policy in place that lets you rotate out certificates in a secure way without giving attackers an opportunity to inject their own CA certificates and own trust routes. And at the end of this talk, we'll talk a little bit about how we do that within the production environment of six store. If you don't already have a managed CA, another option is simply to operate a private instance of Fulcio from the open source repository. The same key management considerations are important here. Fulcio has the concept of signing backends, which are roughly where the signing key lives. In this example here, I show it's within a cloud environment, within KMS. And once again, you need to make sure that access controls are appropriate and that access to that key is locked down. We'll talk more about transparency logs in a moment, but I think it's worth noting here that I also believe that it's critical that you operate a certificate transparency log, even within a private environment, even if you won't have these certificates public. Because this will give you an immutable issuance log such that an auditor can verify how identities are showing up in a log and make sure that there's not anything unexpected. And this is beneficial too in the case of compromise of a CA signing key. If a CA signing key is compromised, either an attacker will have to upload that certificate to a transparency log, effectively forcing attackers into the open, or they'll omit the certificate from a transparency log at which point verifiers should simply not trust the certificate. So now let's talk a little bit about private transparency logs. And before we do that, I'll talk a little bit about what a transparency log actually is. If you're familiar with blockchain, you might have heard of this data structure before, which are Merkle trees. And this is a representation of a Merkle tree on the right here. Roughly the leaf nodes of the tree are hashes of what's going into the tree. And then the parent nodes are hashes of the concatenation of their children. And this construction allows you to calculate certain types of proofs over this tree. And one of them is called inclusion proofs. These are cryptographic proofs that allow you to show that an entry actually is in the tree. Another type of proof is what's called a consistency proof. And a consistency proof says that between any two points in the log points defined by size, let's say size two and size 10, we guarantee that the tree remains append only. Meaning that between those two points, what's in the tree was not modified or removed. And over time, this gives you the property of immutability that trees can only be appended to and not removed from or mutated. More accurately, the mechanism lets you detect any sort of mutations over time. There's many applications of transparency logs. We saw certificate transparency as one example. A binary transparency, this is things like record. And key transparency is another one that I want to mention, which is useful within the end-to-end encrypted messaging space that it give you a publicly auditable mapping between keys and identities. Within Sigstore, we have two primary uses for transparency logs. One is that Fulcio writes its issued certificates to a certificate transparency log. And also that Recor entries are appended to a transparency log, which is sometimes a superset of what goes into Fulcio, but there may be use cases where you only want Fulcio or only want Recor. These are all questions that I've heard at some point when discussing with retainers or community members, which is do I actually need a transparency log if I'm going to be operating in a private environment? And I'll try to convince you that I do think it's necessary to operate one. One question is, well, what if I already have an existing system for audit logging? For example, maybe you're on a cloud platform that already has audit logs. One benefit of operating a transparency log is that it allows you a few things. One, it allows you to move a transparency log into a different environment, let's say a different cloud provider, so that you're operating a different trust boundary and it reduces the risk of compromise either due to an insider risk compromise or even compromise of the cloud platform itself. Maybe a insider is able to mutate the audit logs, maybe they're not immutable, maybe the cloud platform itself has a bug or has some compromise. So with a transparency log, we can move it to a different trust boundary and have additional confidence that's in the event of a compromise only either the log or the platform where things are signed gets compromised. Another question is, what if the artifacts that are being signed privately will ever be released publicly? And I mentioned this one because if you're operating a private transparency log and not making it public, but then you take an artifact that's been signed privately, even if it's on a private transparency log and release it publicly, the problem is that as a verifier, I have no way of computing an inclusion proof to check that an entry's actually in the log. And I also can't calculate consistency proofs that the log remains immutable. And I mentioned this because I think it's worth considering when deciding on whether or not you need a private transparency log. If your artifacts are gonna be released publicly, it's likely that it's valuable to rely on the public transparency log. Another question is, well, can I just use a database if let's say I already have an existing audit logging system? And I think what this really comes down to is do you want an immutable record? A database is not going to be immutable. And once again, it's prone to insider risk or compromise. And so to me, I think transparency logs are really critical because they provide this immutable record. Now one detail that I think's worth noting is how you actually get this property of immutability. And that's by relying on monitors. Within the transparency log space, monitoring is a little bit of an overloaded term. I've said monitors a lot so far, referring to identity monitoring where you are querying the log periodically, asking for every entry in the log and doing introspection on the entries in the log to look for certain expected identities. This type of monitor, the expectation is that it's there to calculate consistency proofs. So periodically it will query the log and say what's the current state of the log. It will then query the log again some size later and calculate a consistency proof between those two points. And if the log is not able to present a valid consistency proof and the monitor is not able to verify it, then there's an expectation the monitor should alert the ecosystem to a potential compromise. But another very important role of monitors is to prevent what's called split view attacks. And this is really where everything kind of comes together. So split view attacks are where the log presents different views to different callers. And each view can look consistent, but the contents of what's being presented might look different. For example, maybe the log is malicious and chooses to show nothing to some verifier, effectively causing a denial of service. Or maybe they choose to omit entries. Maybe the log's even secretly adding entries in unexpectedly. And this is why it's important for monitors to do what's called gossiping, where the monitors share their current state with other monitors. And if all monitors see that they currently see the same state in the log, we can have some confidence that there is not a split view attack. And it's important that we operate more and more monitors in order to feel more confident that a split view attack is not occurring. Once again, like how I mentioned that it's important that a transparency log be operated in a different trust boundary, it's also important that the monitor run in a different trust boundary than the log. Because if the log becomes compromised and it's in the same trust boundary as the monitor, an attacker might be able to shut down the monitor and you wouldn't be notified when something malicious is occurring. I've left a couple of links here to some of the monitors that we're working on, one of which is in the six-door organization called the Recor monitor. This is something that I, along with Purdue University, have been working on. This can run on GitHub Actions, which is useful for monitoring public transparency logs, but you can also operate this yourself. Another monitor was created by the team behind Trillian. Trillian's a scalable Merkle Tree service and this is actually what backs Recor and certificate transparency. If you're interested in talking more about monitors, I'm happy to do so after the talk. So next let's talk about time-stamping. This is a really critical component of SigStore. That is because the certificates that get issued by Fulcio are very short-lived. This is in contrast to SSL certificates that are typically issued for 30 days, 90 days, a year, it could even be more than that. And one issue with this system is if for some reason the signing key material gets compromised, you're going to have to revoke the certificate and this means that you now have to rely on certificate revocation lists. And we didn't want to have to rely on those for SigStore certificates, so we issue very short-lived certificates in this example just for 10 minutes. And when verifying, it's less critical that we check that the certificate is still valid, but that the artifact was signed when the certificate was valid. And for that we effectively need a witness, some other entity to say I saw this signed artifact when the certificate was valid. And typically we rely on the transparency log on Recor for that. Whenever you upload an entry to Recor, it includes an integrated time-stamp. And when verifying, it's expected that you check that that integrated time falls in the window of validity for the certificate between the not valid before and not valid after period. Well, one question is what if I don't want to operate a private instance of Recor for some reason? Well, another option is to rely on time-stamp authorities. And this is actually quite an older concept within this space. Time-stamp authorities had been around for quite a while. But roughly the way it works is you have some trusted entity, a time-stamp authority. And it signs over a current time along with something associated with what you're verifying. Typically it's an artifact signature and this is called counter-signing. And the time-stamp itself is signed so it can be verified. And it's the same as with Recor that when you wanna go verify a short-lived certificate, you take the time-stamp from the signed time-stamp and you check that it falls in the window of validity for the certificate. We can go one step further and use both a time-stamp authority and Recor to have distributed trust. Once again, when I talk about trust boundaries, you can take a time-stamp authority that you're operating yourself and put it in a different trust boundary than your transparency log, a different trust boundary than your certificate authority to reduce the impact of a compromise. You could also rely on a third-party time-stamp authority. For example, did you serve or Apple operate some? And maybe you choose to trust them to be the witness and to provide the trust at time. So in this example is a verifier what I would do is I would get the signed time-stamp and the Recor entry and I can simply ignore the integrated time from the Recor time-stamp, sorry, from the Recor entry if I want and instead rely on the time-stamp that comes from the time-stamp authority when verifying the certificate. So the last consideration is routes of trust and in particular how to ship these and how to securely rotate these. And like I mentioned earlier, these routes of trust within SigStore include the certificate, the CA certificate chain and verification keys for the Recor transparency log. Once again, the problems of key management come up here. With routes of trust, how do you securely distribute them in such a way that an attacker isn't able to inject something into the trust route? Storage is another consideration. How do you lock down wherever these routes of trust are stored once again so an attacker can't inject other ones? And compromise is also something to consider. And I think it's worth noting once again, expiration too. If a compromise occurs and I need to rotate out some route of trust, how do I do so in such a way that does not allow an attacker to inject their own routes of trust? Within the production instance of SigStore, we use the update framework or TUF. And this is a framework that aims to secure software updates. And it does so by creating this hierarchy of roles that are responsible for signing off on other roles and they can continue signing off on other roles and eventually will terminate and sign off on some metadata. So in this example here, this is what a typical TUF repository looks like. We have a target role which is comprised of some signing key. I'll note that the signing key could either be offline or online. So for the target role, it's I believe recommended that it be offline. For the root role, it's very strongly recommended that it be offline. And the target role is going to then sign the metadata or a set of files, right? And this could include the certificate, this could include the public key. And the idea is that because it's signed, it doesn't give attackers the opportunity to inject in any of their own metadata. The snapshot role is responsible for specifying all of the roles that should be a part of the system, once again as a mechanism to prevent injecting other roles. And the timestamp role is responsible for signing a timestamp that's meant to frequently expire and this forces clients to go online periodically and fetch the latest TUF repository. And this is very beneficial to give us a mechanism for rotation. So let's say we have some certificate that expired or got compromised. All we'd have to do is have the target role sign a new set of targets, sign some new target metadata. And as clients timestamps expire locally, they would go online and fetch this latest updates. Another thing to consider here is that what happens if the signing keys for any of the intermediate roles here, the target timestamp or snapshot roles get compromised? Well, this is the purpose of the root role, which is going to sign off on all the other roles. Within the production instance of Sigstore, the root role keys are held by a number of trusted community members and you need a majority of the key holders to do a new root signing events, signing off on new keys for other roles. And so this prevents the compromise of the single root key holder and I think it's important that if an organization wants to use tough, you have a similar setup where you have some keys that live offline that are very rarely used to update the roles that they delegate to. But you rely on this mechanism so that you have some sort of secure update mechanism for rotating in new target metadata. So we're about at the end of the talk. There's a couple takeaways from this. First, key management is hard. And this is whether you're referring to artifact signing keys or for managing routes of trust or managing the keys behind your service. So I think it's really important to when you're building the system to consider this and this is why we recommend things like the update framework. Next, auditability is critical whether you're using the public instance of Sigstore or whether you're operating a private one. I think it's really critical to have an issuance log of every certificate that gets issued. A log of every artifact that gets signed because it gives you this permanent immutable record of every interaction that's happened. So you can go back in the event of a compromise, see what happens, you can see how your identities are being used. Like I mentioned at the beginning, I didn't talk about how to actually deploy Sigstore. If you're interested, there is a channel on the Sigstore Slack called private Sigstore users where some folks have been working on how to deploy a system that looks like this with all of these components. So feel free to go check that out. Thank you very much. Are there any questions? Yeah, so the question was how often are we seeing organizations interested in deploying private Sigstores? This is a very new area, so I think we're just starting to kind of see organizations take an interest in this. I know we have a few represented in the Slack channel that I mentioned that have had an interest in this. So right now I think it's kind of a handful. I expect it to continue to grow over time, though, as more become interest in this. So is the question, do you need the update framework if you're relying on? Yeah, so I think it kind of depends, right? It's where, what do you trust, right? So if you're trusting the cloud provider entirely, maybe you can rely on their mechanism for something like KMS to rotate out keys. I think it kind of depends on what your environment looks like. It's kind of like tied to that one cloud provider. Maybe everything's integrated and you don't need to rely on something like the update framework, but I think once you start distributing trust, you start running your services within different cloud environments, being able to bring together all the roots of trust in a single repository becomes important, and that's where I think a tough could be valuable. Awesome, well, if anybody wants to chat more, I'll be around after the talk. Thank you very much.