 Am I on? Yeah, I'm on. All right, let's get started. So first of all, thanks everybody. Thanks for showing up. It's the end of the day. It's the end of the conference. You could have been having a beer or food or something, but you're here. So I'll try to make it worth it. We're gonna talk about keyless signing without Fulcio. So keyless signing is this signing pattern that was popularized by Sigstore where you don't have to think, the end user doesn't have to think about managing their keys. And I wanna sort of demystify what keyless signing is and show that you can also create the same flow without having to use Sigstore. You can use off the shelf components and get the same kind of experience. Okay, so if you love this talk or if you hate this talk, here's how to get a hold of me afterwards. I have the kind of pleasure of working with ChainGuard. So I get to think about this kind of thing throughout the day for work, which is wonderful. And then you can find my GitHub or Mastodon handles there to say hello or whatever you'd like. And so here's how this is gonna go down. First we're gonna take a look at what the experience of keyless signing is. What does signing look like for an end user and just show that user experience and what the value is, first of all. And if you haven't managed keys before, we're gonna take a look at what that feels like a little bit. We're gonna sign something with a key, a key pair and then see some of the pain points and feel that what can go wrong, essentially. And then if we're in the position to wanna build something similar, but we don't wanna use SigStore for whatever reason, we're gonna take a look at keyless signing step by step and understand the components that we need to run ourselves to get the same experience for our users if we don't wanna use the public good instance or for whatever reason, we don't wanna use all the components of SigStore's open source services. Okay, so the first bit about keyless signing being fun. It is, it's fantastically easy, especially if you've gone through the alternatives of signing any other way in the past. We're gonna see some examples with Cosign. This is SigStore's CLI client that just makes it a lot easier to use these services to sign things. And really all you need to do to sign a container with Cosign is to just type Cosign Sign and the digest of the container that you wanna sign, like that's it. The user experience that happens right after this, if you haven't done this before is, if you're a person, you're gonna get, a browser's gonna pop open and prompt you to log in with an identity provider. Google, GitHub, Microsoft, that list might grow in the future, but basically how do you wanna sign this image, which identity you wanna use and just prompt you to log in and that's the identity that's gonna be attached to the signature on this container. There are some variations on this. If you're running the same command inside of some kind of workload, Cosign's gonna kinda look around and try to figure out who you are. Maybe if it looks at environment variable and sees GitHub ID token environment variable. Okay, we're in GitHub actions and it's gonna use that identity to sign. If it's in Kubernetes, it might find the service account token and use that identity. So if you're a workload, it's not gonna prompt you to log in, it's just gonna try to figure out who are we essentially. And there's some other, yeah, okay, so this is it. And then a smaller side, soon you'll have to use a digest because signing a tag doesn't really mean anything. The content can change and then what are we even talking about on the signature? So that's gonna be a required thing soon. And I think the magic of this keyless signing is really this part when you verify a signature. The aspect that you wanna sort of look at here is we're gonna say Cosign Verify and we'll point to the same image digest. And then we talk about the signer. We don't talk about keys. In this case, if I was signing with my Google login, we talk about my email address, that's the alias for me, oh, sorry. And we talk about according to who, according to accounts.google.com. There's variations on this obviously. You could say I want it to be according to the GitHub actions identity provider and I care about this specific workflow to have signed the image. Soon they'll be support for a couple other providers like GitLab and BuildKite in the works. So hopefully this will be ubiquitous across CI providers and other workload identities soon. There's some variations on this verification process. You can run this as a Kubernetes admission web book. So there's Giverno support. I believe Gatekeeper might have support now, I'm not sure, but there's also a controller called a policy controller from SigStore itself that can help you check signatures and reject images that don't match your policy, essentially, right? This is really awesome. As an end user, if you don't know anything about public key cryptography, you still didn't need to do anything and know anything about public key cryptography and you can hold it right and not mess up, not switch the private key and the public key or store something incorrectly. You don't need to know that the keys even exist in this case. So there's nothing to mismanage, which is really wonderful. But if you've never had to manage keys, you might not know how bad that can be. So let's take a look at managing keys and see what that experience is like. So cosine can also sign things with a key pair. And the first thing you need to do, whether you use cosine or not, is to generate a key pair, private and public. If you use cosine to do it, it'll look kind of like this. I'll ask you for a little password to encrypt the private key and then it'll write out two files, the public key. It's cosine.pub here and the private key, cosine.key is the name of the file here. The private key is the kind of sensitive bit. There's some variations on this too. The private key can be managed in like a cloud KMS system where you can only access to sign over an API instead of having the actual key content on disk. But some of the pain points end up being kind of similar even in that use case when you're not managing that. And there's different tools you can use, like OpenSSL or the step CLI to generate these kind of key pairs and use them for signing. And the experience of signing always kind of looks a little bit like this. So here using cosine, we point to the same kind of image and then we pass in the private key to sign with. You need to know to use the private key for signing. And then when you're verifying you pass in the public key. Again, you could use the Kubernetes admission control and stick the public key inside of your little CRD to say it needs to be signed by this thing. And this is gonna check in, in the case of keyless and in the case of keyed signing, it's gonna check that the contents of the container haven't been shifted, kind of like a hash. But in the case of this check with the public key, it's gonna check that it was signed by the private key. That's the verification that's happening. And like, doesn't look like that bad really. Like it's just, make sure that you handle the private key and stick it where you wanna sign and then take the public key wherever you wanna verify, right? It's not bad when it's one key. One key's actually, this is like the temptation that leads you down the path of like, despair. You only really need to know a little bit about the keys. Always verify with public, always sign with private and don't leak the private one out there. Otherwise people could sign with it, right? So let's ask a couple of questions to kind of move beyond just like one key pair. This is maybe a situation that you have. I wanna verify that the things I run in my production systems were built in my built system. Like we should all be able to answer that question. That's like entry points to software security, software supply chain security, right? And the way that you would solve this to kind of fix this with key management is, okay, well we'll generate a key pair, stick the private key over in your build system, sign stuff in your build system by accessing it maybe from some secret store or something, and then you tick your public key and wherever you're about to run your containers or your software verify before running and then reject stuff that isn't signed. So in this case you start to see the pattern that emerges with these key pairs is that the public key becomes this kind of alias for the build system. And we know that it's the build system because we only gave the private key to the build system. But if you wanna have like slightly more nuanced language around verification like in this Kubernetes namespace, I would like to make sure that that it was built and signed by a particular CI workflow. The only way that you can discriminate between the different signers in this case, the different CI workflows is to give them all different private keys to sign. Otherwise, your statement can't be verified because you can't distinguish between the different signers, right? So again, the keys have become sort of an alias for this particular signer and they hold onto it for a long time. At this point, you better start kind of like track them somehow because just by looking at the key material, you don't know where you stuck it in which pipeline and it gets to be a little bit chaotic tracking where you put each one, right? You might have a lot of pipelines too. Okay, like your CI provider also probably is gonna get jacked at some point and people will take your private keys, right? Like no one is immune, if you create a secret store, people will attack it, right? The tough bit here is you need to make sure you create a new key pair, put the new public key in place to verify against both public keys, sign with the new private key, distribute a build for everything and then get the new builds everywhere and then remove the old public key from the verification policy. It's work but it happens, right? But you were maybe signing things for customers and they were verifying with those keys. It starts to get rough now, like talk to PR, right? You have to do the same thing but you also have to deal with the fact that you just kind of busted your customer's trust a little bit and they're just kind of thinking like how long until it happens again? I mean, the new cycle's rough, right? It just kind of keeps on going on like this. You know, if you have, if you're a big company, it's too much. If you have a public key, where's the private key? Like, no, believe me, it's painful and like the literature is riddled with like public key management or just key management just being a brutally hard problem. So if we can get rid of it, it's a good idea. So exactly what is it about the keyless pattern that is actually better? The thing that is actually better is the verification process, really. You only talk about who the signer's identity is and that's it and that's what verification should look like. You're trying to assert who signed this thing. So let's look at a different example, like verifying in Kubernetes admission web books. So if you were using a six-door policy controller, this is a little bit what the CRD would look like and the thing that I really wanna draw attention to is there's a pattern for what the images look like, gcr.io foo something. There's some details there about CT log and keyless but the thing that's very important is what's highlighted in color and again, like all keyless verification should have this property where you just say, I demand that the signer was this, this identity provider, this particular subject and maybe even a bit of a pattern matching thing where you say, I trusted this subject most of the time but November was a bad month, they were compromised and then I care about the subjects looking like this pattern or something like that but so it can be flexible, doesn't have to be one exact workflow. So this is what it might look like for someone to say trust a particular GitHub workflow. That makes sense. Okay, so can we build our own? There's a lot of people right now building signing systems inside their companies. They have a lot of different constraints and I guess what I'm trying to say with this whole kinda presentation is that it should be keyless. You should really, really question yourself if when people are verifying the signatures on your signing system, if they need to reference a key ever. It's just not a position you wanna put your developers in because then they have to manage them and understand what that key means and what is this referring to, right? It's gonna get bad eventually and so how can we get the same kind of keyless design even if we decide that like six or as actual open source systems don't fit whatever your requirements are. Okay, so there's kinda some analogies here about why you might wanna build your own, right? If you're building like WebPKI, right? There's a big one that is open source, Let's Encrypt. And it's designed in a particular way to deal with a large publicly trusted, being a large publicly trusted CA. It has a very large target on its back so it's designed a particular way, right? But you can use like Boulders, Let's Encrypt's Boulder CA if you wanted to. But when people have private instances of a WebPKI, they often don't, they often choose different technology that's built for smaller private certificate authorities. If you're running a Kubernetes cluster, you have a CA, you know, it's baked into the API server. If you're running a service mesh, you also have a CA, right? That becomes that root of trust for all the communication inside of your service mesh. And people also will kind of build them out of small step or vault, open SSL maybe is the wrong example to use here, but you could use that too if you wanted to to build these sort of smaller systems with a different set of constraints. Maybe certificate transparency, which, you know, Let's Encrypt has to have, isn't something you need for your service mesh. So I guess the question is, if Sigstore is kind of the big public code signing, Let's Encrypt, if you will, what are the options if we are trying to make something with different requirements that's private? Does it need to be the same? And I don't mean to say by any means here that Sigstore itself shouldn't be your first option. It should, it's fantastic. And it's built to be extremely secure. So it's a great option. But if you discover you have already an existing X509 like certificate authority that you need to use for whatever reason from some other team has put this constraint on you, let's see, like, can we still get the same behavior, basically? Okay, so to figure out how you can do this yourself, we're gonna look at the keyless signing process in detail when you use Sigstore, and then we're gonna keep like a little grocery list on the way for the stuff that we need to run ourselves if we're trying to get the same properties, okay? So that's the way it's gonna go. So when you're using Cosign, like the first example there at the Cosign container, the first step for Cosign is to figure out who is using Cosign. So who are you? Who's calling me? And that, you know, we talked a little bit around like looking around in the environment if it's a workload. If it's a person and it can't find anything, it's just gonna pop open a browser and say log in, tell me who you are. And the response from that is gonna be specifically for Sigstore, it's gonna be an OIDC ID token that describes who the user is. Along with according to who, like according to Google, according to Microsoft, that kind of thing. And that brings us to basically grocery list item number one is to think about the identity provider you will use for your signers. If these are like CI workloads, they need to have a strong sense of identity and the identity is gonna be tied through the whole system all the way to the end so that the verifiers can use that language the same language as the identity provider to describe who they wanted to sign their systems. But in your system, it doesn't have to be an OIDC identity provider. Thankfully, that's actually being baked into a lot of CI providers these days. Circle CI has one, GitHub Actions, GitLab. It's becoming more ubiquitous, but if your system for workload identity is spiffy and they're, you know, X509 Svid docs, that could work too. If you're building things in AWS and your identity provider in that case is AWS AIM, that's a perfectly fine identity provider. So long as it works with the rest of the system and we're gonna talk about that. But that's one thing to think up upfront is who's signing? Is it people? Is it workloads? And how am I gonna identify them, right? All right, next step. Cosign knows who we are now. All right, so the next step is that Cosign is gonna create a private key and this is a keyless flow, but don't worry. We're gonna throw it in the garbage really soon, I swear. Okay, so it's gonna create a private key in memory and then it's gonna take the public key conjugate to that and that ID token about who the signer is and it's going to pass that up to Fulcio which is SigStor's certificate authority and what it's asking for is a short-lived code signing cert. So Fulcio's gonna take a look at that identity and remember we have to wire the identity all the way through to the end so that someone can verify just with that identity like me at example.com. It needs to make it all the way to the end process. So Fulcio's gonna stick that metadata inside of the certificate along with the public key because certificates always have the public key inside them as well and pass that back to Cosign and now the key property here is that the certificate is going to expire very soon. I think the default on Cosign for Fulcio for the public instance is 10 minutes. You could and should maybe make yours short. The idea is to get a certificate for every signature. Yeah, so the shorter the time window, the better and what's really great about this is when you're when you have certificates that you're issuing for a minute or two, certificate revocation is just out of the question. Like there's no reason to have a certificate revocation list with windows that short, they're about to expire anyways. Okay, so that's item number two is that certificate authority is gonna have to be in your system somewhere because if you're just using public keys by themselves there's no additional metadata about the validity period and the certificate authority is giving us two things. It's pushing the identity information into the cert to pair it up with the public key and it's also giving us that window of it's only valid for five minutes or something like that which is the magic of certificate authorities. Metadata on top of public keys, right? So there's lots of different open source certificate authorities. I think in spirit you could probably do this with like an SSH certificate authority because you can do signatures with those two but we'll keep our discussions to X509. So the same kind of certificate authorities you see for TLS just with different data inside them, inside the metadata. And the key bit here when you're choosing your certificate authority is that it needs to understand your identity provider. So when you're choosing that and you know that your identity provider was for instance, it was going to be AWS IM, the certificate authority needs to understand that and be able to exchange those credentials for certs. That's what we have to keep in our head or you need to glue it together somehow to make that possible. Okay, next bit. We have our private key and we sign our artifact with it and that gives us back a little bit of data, the signature and then we chuck the private key in the garbage. Nothing more for our list now. That's just something we'll be able to do. So no more requirements. I think the only thing to remember here is we're kind of on the clock at this point. The certificate's gonna expire soon so we better sign quickly. That's kind of the only design requirement. Okay, next. So what happens next in cosine is that it tells, it needs to go tell recore which is the signature transparency log that it made a signature. So this is one of these really interesting and excellent security features of SigStore is that there's signature transparency so each signature that happens ends up in a signature transparency log and cosine is gonna tell recore this signature transparency log. I've signed something next. So it's gonna pass the signature it just created and the certificate that it signed with up to recore and recore's gonna give basically all that stuff back but it's gonna add a timestamp and sign the whole bundle. So the certificate, the timestamp and the artifact signature are all gonna be signed by recore. Now it's not signature transparency. That's what gives SigStore this keyless feature. Signature transparency is great but it's not the keyless bit. What's happening here to make keyless possible is that this is a verifiable timestamp of when the signature happened and recore doesn't need to necessarily be the thing that does this but you must have something do it. So the key element here, the last thing that we're gonna put on our grocery list is a timestamping authority. Now if you decide that you want signature transparency because it has a lot of great properties in its own right you could run recore if you want as the timestamping authority but you can also run timestamping authorities that only do these timestamps and don't actually have this big signature append-only log if that's what you don't wanna have. They're a little bit simpler to operate as well. Okay, so that's it. We have the three things on our list. We have an identity provider, certificate authority and a timestamping authority. And then the final question a little bit here is that's a lot of stuff. Like a lot of bundle came out of this. Like how did that turn into someone just saying, you know, cosine verify this email address and this identity provider, right? There's, this is what's in the hands of a verifier, okay? And here's how verification works and what you need to implement if you wanna make this keyless magic kind of work, right? You have the signature of the bundle of the big timestamped bundle and you have, we have our certificate, we have our signature, we have the artifact in hand. You have to ship all of this stuff to the end user. What's happening in cosine if you sign the container is that this is all attached onto an OCI image right beside your image you care about. So it's all stuck there inside your OCI registry usually. So here's the process. We kinda have to appeal this thing like an onion. So outside layers first and we're gonna make our way inside. The first bit is to just verify the signature of the bundle. So recore in this case has a public key. We verify that this signature was signed with recores public key which kind of removes the bundle which means that we trust that the signature happened at a particular time and we know that what the timestamp is essentially. The next part is to check that the certificate is valid. Now it's really important right here that we don't use the normal certificate validation flow that you would see like in TLS because this certificate is very likely already expired in that sense. What we need to do is do most of that aspect make sure that it chains up to our certificate authority but then when it comes to the time of validity we just need to check that the timestamp is inside of the validity window for the certificate. So that's a modification you need to make from the normal TLS kind of verification that kind of pops off the certificate from the onion and then second last thing, yeah go ahead. Yeah the timestamp was signed in the original bundle so if I go back here the signature is over the entire bundle which includes the time, the signature and the certificate. So we kind of verify that the time was accurate insofar as we trust recore in the first step there. So then we check that we trust the certificate authority and then second last is we check the signature actually is valid. So we take the artifact, the public key and check the signature and after that's done we're remaining with the thing the user actually asks and they put that input in and say do I trust me at example.com? It's just the identity metadata left here. Do I trust these details? And those could be very arbitrary for your situation. It could be AWS, you know, roll ARN, it could be all kinds of you know you can stick whatever you want in an OIDC token. So if it matters to you, this is business logic, right? It could be your own thing. Okay so let's do a couple of examples. I talked a little bit about AWS so here's maybe you're building things in AWS cloud code build. So we just need to choose these three things. We get an identity provider for free, right? We use AWS as one if we want it to. We don't have to but we can. We could choose for instance, Hashicord Vault as the certificate authority. If you all have ever used Hashicord Vault, you know Swiss Army Knife of many security things. You can use AWS as an off method. So you can authenticate using AWS identities and then you can exchange that for a certificate secret you'll call them. So you can issue short-lived certificates that would totally fit this need. And then the last bit is you do need to choose a time-standing authority. So Sigstore has an open source one that you could just grab off the shelf and use for this purpose, basically. Or you could choose Rekor here or you could try to find another one that is sort of, I think the RFC is 3161 that's time-stamping spec that you could use. But there's options. Let's think about maybe signing with a corporate identity provider. Maybe you want to sign Git commits this way and you want to use like your Octa or something. Okay, so if you have Octa, you can create most of these corporate identity providers have the capacity to make an OIDC compatible app. You know, that'll speak OIDC after people log in. And a great option here too, another open source one is to use StepCA. So StepCA has the notion of a provisioner. Which credentials can I exchange for which certificates? And certificate templating. So you can give StepCA an OIDC token from your Octa app or from any other place and template in the values from the token into extensions inside of your certificate. So that'll give you that mapping into your certificate details. So they're called provisioners with StepCA. And then the templating is attached to a provisioner. And then again, you just kind of choose a time-stamping authority. Maybe you care about signature transparency in this case. So we'll choose Recor. So this is the kind of feeling to get the magical three things together and get this keyless kind of flow. Okay, I'm gonna give good news first which means you know that the bad news is coming. You can keep signing with Cosign. It's already got flags for a situation just like this. If you're using StepCA for instance, you can get a certificate with StepCA certificate. Here's grabbing one for me at example.com. Cosign needs to have a particular format for the private keys. So you use this import private key command to structure it correctly. And then in signing, you just do Cosign sign. You know, you point to the private key which you can throw out right after signing. So that's the keyless aspect. You point to the certificate you have and then you have to have a root bundle that chains up to your root and then point to your image. Yeah. The bad news is that we can't verify very easily in Cosign right now. Cosign's verification really understands specifically Fulcio at this moment. So you can imagine if you stuffed all kinds of arbitrary extensions inside of those certificates, the very last unpeeling of the onion, it would be like, I'm not really sure what to do here. So you need to write your own verification logic and that's the pain point right now. Unless you make your certificates very similarly structured to what Fulcio is doing. And you can almost pull that off, but if you have arbitrary metadata, you can't really get there, right? And that's probably why you're doing this in the first place. But the future's brighter. There's a couple of rough plans to make this just extensible. So I think the dream would be for everyone to have the first parts of the keyless bit, the time-stamming bundle being verified, the certificate being verified, and then all of that just be shared inside of Cosign's verification logic. And then I think what some of the hope is here is just to have an arbitrary rego policy or queue or whatever you love for your verification language at the end to verify the really particular details of your certificate structure. So that people can bring their own PKI and get the same really unique flow, right? So come join the effort. We're obviously looking for contributors at all times. And if we write the logic to verify once, then we all share it and we can all object together, right? And then just finally, a couple of key takeaway thoughts. Please stop managing keys, especially if you're writing a new code signing infrastructure right now, or if you're consulting with people, it's gonna cause you pain in the future. And you can take off-the-shelf components and do better already. And not giving this pattern of trying to worry about where you put all your private keys. And then if you're in this pattern or you're even gonna use SigStore, start thinking now about the identity provider that you're going to use and to sign with and hence to verify with. So is there actually a strong sense of identity inside your CI system? That's the biggest one, because it's not true everywhere, right? And then I guess the last one is come get involved with making verification for just bring your own PKI shared upstream, essentially, so that it's easier. We don't have to write the same verification logic. Thank you. That's it. A couple of minutes if anyone has questions. Yeah? Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yep. Yep. Yep. And in the other direction, SigStore, SigStore has a test server for fire, so that when that number is running, you build the software, and it needs to be in SigStore, I guess. Yeah. So that it's all functional right now, it is really hard to set up for me, and I don't know for a year or so. Yeah, yeah. I hear you. How long does it take to go before a card set? I hear you. I think that's part of the reason some people look for something else, is that Falsio in particular can be a little rigid, you know? It's not meant to be this extensible system that can take an arbitrary identity and exchange it. Step CA was one that I found kind of just would take whatever, and it worked really well, and I could test that way. But I think that there's something that we can all do better here to take hopefully with more of these implementations. I think we need frankly some standards on that structure, so that when any kind of verifier sees a keyless looking bundle, you know, the verification is clear. And then I think the other one is, I don't know if we have, and I think that we should kind of have a bit more of a, what is the shape of exchanging something that is not OIDC for a certificate? What's that look like essentially? Yeah, yeah. I don't know if I have great advice for you other than to try different certificate authorities to make it at least easier to prototype, I think, because it is tricky. Yeah, it is possible. Yeah. What is that? Yeah. Is that using the JWTSFIDS or X509 that you're trying to do? Yeah, exactly. And that's one that's like even further away from easy right now, I think in Falsio. Right, right. It's not ready for prototype. Cool, cool. Any other questions? Thank you. Well, I'll be around and you know where to find me. You'll see these slides up. So thank you so much for coming. Cheers.