 Hello and welcome. We're going to be talking about Zero Trust supply chains with Project SixTor and Spiffy. My name is Andres Vega, and you are? Jake Sanders. If you're new to the Spiffy project, the community was involved in the authoring of a book with everything from planning, architecting, implementing, operating, Spiffy-inspired deployments at scale, borrowing from the experience of practitioners who've deployed Spire ranging from highly regulated industries to web-scale companies. The book is available online at spiffy.io slash book. It's free. We also have print copies available upon request, starting an excellent resource if you're new to the project, or if you have been a practitioner for a while to get a refresher also. Jake, you want to tell us about yourself, please? My name is Jake Sanders. I'm one of the maintainers on SixTor. I've been working on it since mid last year or so, around mid last year. Previously, I was at Google and Amazon. And then my co-host. You worked on some notable projects at Google? Oh, yeah. I worked on it. I was on the container registry, and GKE at Google spent my last year and a half at the company on the Google open-source security team. When they spun out SixTor, co-founded it with a couple of others from Red Hat. Awesome. I am Andres. I work at Control Plane. Control Plane is a offensive security company specialized on cloud-native security. We do all sorts of engagements ranging from threat modeling, pentesting, security architecture, and automation. We're here to help with any of your security needs in attacking and defending Kubernetes. Also, I am on the steering committee of the SPIFI project. I've been involved in the project approximately for five years in different aspects of product, program, and project management. I authored the SPIFI security assessment in preparation of the project moving from sandbox to incubation. I also led the preparation of due diligence from moving from incubation to graduation. I've also participated in a few other efforts through CMCF tax security. It's nice to see fellow tax security members here. We recently published a supply chain security best practices guide. It is available in the tax security repo. And as an addendum to that as a sequel, we also wrote down a reference architecture of a secure software factory. All of these resources are open source. I invite you to check that out. Give us feedback if it's something useful. If there are gaps, we'd love to hear from you. So let's get into the subject. What is a software supply chain, Jake? Well, a supply chain is everything from creating the code to checking out the code to building the code to deploying the code and everything else you might want to do with it. And why would it be important to secure it? Because mean people like to mess with your software at every step of the way. I sure I probably don't have to tell anyone in the room that software supply chain is a big issue these days. Basically, what the industry has had to deal with is not really having a good foundation to build upon in order to secure every piece of the software supply chain that they might need to. But now we're getting things like the executive order that is mandating that people working with the government have to have certain levels of supply chain readiness. We're seeing a whole bunch of initiatives from all sorts of companies, including Google and things like Stor, just in order to deal with it. It's a very good time to be in the space and it's a very good time to be looking how to actually solve this problem for yourselves. It is a full time job to keep up with the threats. The community led an effort pioneered by Santiago Torres Arias from Purdue to catalog cloud native supply chain compromises. This is available in the tax security repo, links on the screen. Here's a snippet. The list is much more extensive, is a good breakdown to develop understanding of the threats we're faced with and how adversaries may attempt to exploit vulnerabilities. So let's elevate the discussion for a second, looking holistically at software and where we're at today, where we come from. With the advent of cloud of applications evolved to be larger distributed systems composed by smaller pieces, if we look at high profile compromises that have occurred since, there is the common denominator of man in the middle attack, exfiltrated credentials as the entry vectors of how attackers gain a foothold. And why is that? How did that precipitate? Well, with the cloud, the perimeters evaporated. As we have a larger number of endpoints communicating across security boundaries, communicating across platform and cloud boundaries, all of this traffic is exposed. The perimeter, as I said, is non-existent. It's a really brittle barrier for anyone to gain a foothold onto it. So there is a need to drive down trust to zero, reduce the number of assumptions we have. Typically, traditionally, access control has been implemented by the use of API keys and shared secrets to grant access to a particular resource. These secrets, this key material must be secured, so you encrypt that. And encrypting it, you need a decryption key. And now you have yet another secret you need to protect. And this is a problem of infinite regression, turtles all the way down. You end up ultimately having a collection of keys. Some end up in paper, you end up putting these on a physical secret store, probably even a bank, to ensure the confidentiality of that in order to protect it. Now, it's not a great model. It presents a lot of risk. And what we post it is, how can we move away from proof of possession onto what be the equivalent of fingerprinting and readiness scanning code? And the event that this material should ever be exfiltrated, it has very little utility if the lifecycle is as close down to zero as possible and it's dynamically rotated. Based upon that, the community gathered and evaluated best practices of large scale organizations that were further down the road of solving this problem. We externalized these best practices and codified them on the SPIFI specification. SPIFI stands for secure production identity framework for everyone. It defines a set of interfaces and documents for managing the lifecycle of identity and how to distribute the scheme material and for services to cross authenticate to one another. Then we have ASPIRE, which is the software implementation. It applies the SPIFI specification so you can consume it within your platforms, within your software infrastructure. There are several reasons why you'd be compelled to use SPIFI and SPIRE. One is you can drop MTLS in one fell swoop. You can also reduce once again the likelihood of breaches by driving down the utility time of that credential in the event that it would be exfiltrated. It does create an identity control plane. It creates an identity abstraction that you can rely to do high velocity PKI and not have to worry about how do I teach my app to do TLS? How do I teach my app to talk to this other service securely? If I have a Kubernetes spot that needs to talk to a Lambda, how do I exchange this identity document for an SDS and have an IAM role binding and move away from managing that material manually? There's a range of specifications codified within the standard. There's the notion of identities that represent the name of the object that's being issued to. That gets embedded within an X509 or a JSON web token. There is a specification for the API or our applications to retrieve and discredentials. There's the trust bundle that is the collection of public keys to authenticate the far end of a call. Federation, which is how to exchange these sets of keys across different platforms over OIDC Federation. In the interest of time, since we start a little late, I'm not going to dwell too much on trust domains. The slides will be available online. You can reference the trust main section. Moving on to the software implementation. This is what a typical aspire deployment looks like. You have a spire server that manages the registry of all of these identities. It's concerned with the issuance. It's concerned with rotation of root certificates also. The second key component in the architecture is the aspire agent, which serves the workload API to containers, to any application that you're attempting to spiffy enable. The API that the agent exposes when a workload gets instantiated and has no notion of who it is, who is it supposed to talk to, comes up confused, who am I, goes from something that's never been seen before the infrastructure. How does this transition to something that we understand well what it is, have determinants identity with certainty, therefore we confer access control. That process of introspecting the application gets initiated by the workload API. You see here Kernel, Kubernetes, and Docker. The workload API ensures that the calling workload, what's the sdid, what is the user group, what do we know from Kupelet, what's the namespace, what's the service account. If it's running on a Docker container, what are the environmental variables, what are the labels, what's the image ID. If you're running on a cloud platform, it can interrogate say if it's AWS, the instance metadata API to determine what VPC, what availability zone, and if all these attributes match policy, then the identity gets issued, it's credential and passed on through the workload API. Now, having given you that overview of Spitfie Inspire, I'm going to pass it on to Jake to talk about Project Six Store and problem-solves and where does it intersect with Spitfie Inspire. All right, so as I mentioned, Six Store's Raise on Detra is helping you solve the most fundamental issues in creating a cryptographic root of trust and actually using it. The actual root of trust itself, the services that are built on top of that, and the tooling which relies on those services in order to actually make attestations about how an artifact was built, where it came from, who contributed to it, everything you might need in order to avoid many classes of attacks that people are most worried about nowadays because they keep on popping up in the news. It's very nice for us. Basically, the core projects of Six Store are Fulcio, which is our general purpose certificate authority. Think of it a bit like let's encrypts, but for binaries. There's Recor, which is our transparency log. The transparency log allows us to do something very interesting very magical, which is keyless signing, which is to say you create a key pair, you sign an artifact with it, you send the signature to Fulcio, it gives you a certificate, and then that certificate goes in the transparency log and that log will allow you to determine that yes, this identity had this key at this time. There's no worries about it getting lost somewhere, getting used somewhere else, because that key only needed to be around for the split second it took in order to create that certificate. This is the core value proposition of Six Store and upon which we are building from there. Like I said, we're solving the bottom turtles for you, so to speak, in terms of what we're putting forward is the value proposition. Those turtles are, like I said, the root of trust, Fulcio and Recor, which are built on top of it, and then finally the uses that you put them to. A couple of the most popular utilities that we have put out currently are cosine and getSign. One, cosine is to sign container images and also add attestations, S-bombs, etc., to your container, like to your container images so that you can make attestations about them at runtime. This is where, oh, and there's also getSign, which is pretty self-explanatory. You can sign your get commits, good stuff. It's always been really difficult. I don't know if any of you have managed PGP keys in order to get old style getSigning work. Yeah, it's awful. Use getSign instead and do it. You should do it. It's super easy. Try it out. If you don't like it, money back. This is where Spiffy comes in. If you really, really, really want to make it easy and already have a Spiffy Spire-based architecture, well, Sigstore loves you too. We support Spiffy identities through OIDC. If you happen to be using Spire, they have a reference implementation of an OIDC server that allows you to translate your verifiable identity documents into actual identity tokens and attest identities in the actual Spiffy parlance. You can see you have your domain, your organization. This is an example of a Sigstore policy controller configuration. It's attesting the identity of someone that assigned a container image. In this case, cluster this domain in this namespace with this service account, that kind of thing. Everything you would expect from Spiffy ID. It just works. You know how I said, know your customers. We say know your robots. We want to make sure that every single component, carrying out a task in your CI CD, has been strongly attested, has gone through that workload attestation process, has a granular canonical identity that we can reason about, understand every single subject and object that encompasses the build pipeline. So this is where a demo of that would actually go. But the demo gods, apparently my sacrifice to them was found wanting and I was smote. So here is basically an architecture of what was supposed to go on. I'm turning the demo into a tutorial. So if any of you guys would actually like to put this into practice with existing Spiffy architecture, I will be sure to let you know how to do that later on. But basically you can see the CI CD system, which is creating the artifacts. It's using Spire-based identity. It is using this verifiable identity document within the workload. It has access to the Spire agent. The workload has access to the Spire agent is using that identity to like ask for a signing certificate from Fulcio. Fulcio then goes back and asks the OIDC server that is the authority for these identities. Did this actually come from you? Yes. All right. Fulcio will mend the certificate. The CI CD server sends it to Recor. Recor gives it its own transparency long entry. Then finally, the CI CD system will basically all of this operations with Fulcio Recor and the OIDC server and the container registry all happen within the context of Cosign. So like a single command within the context of your CI CD system will handle all of that for you. But the signature is going to container registry. That part is done once the workload actually wants to be deployed to a cluster that is being managed by this policy controller. You do a kubectl apply dash f, some pod. The policy controller will resolve that image to its digest. It will look for the signatures on that digest. It will look at the policies like the one described in the previous page. It will look at the signatures for that image. You'll say, okay, this image was signed by the workload associated with this spiffy identity and that was attested by this identity authority, the OIDC server. All good. We'll let it run. That's a lot of work for all of two commands that are the core of the demo. But unfortunately not. Anyway. Here's some further reading. The first link is a link to what would have been the demo, but it is now turning into a tutorial. If you follow that repo, I promise I will create a release. Once it is fully ready, I'll actually take the time to explain all of the pieces and how they fit together. So if you already have a preexisting infrastructure that you would just like to put six store into, hopefully that will be a good resource for you. Furthermore, there's some other reading related to the iterations between cosine and spiffy. That should be interesting. Thank you for coming. You have a question? If you can go back to your demo slide, I want to understand which component is actually doing the digital signing and which is actually attaching, generating and attaching, what do you call them? Attestations here. I understand that policy controller is actually validating that. But in the process where my code is getting pushed, starting from that point to the repo, how is it getting signed? How is it getting attestation attached to that and all that stuff in this diagram? Right. So you could just imagine that somewhere within the CICD system, just pretend that some workload has access to the cosine binary. It builds a container image, it pushes the container image, then it says, cosine sign, that image. And that's when all of the rest of this stuff happens. That's when it goes out and gets the signing certificate from Fulcio. Fulcio goes ahead and asks the OIDC server if this thing is real. Okay. Fulcio gives you your signing certificate. Cosign will automatically upload that to recore. Then you get your transparency log entry. Finally, once all of that is done, cosine will bundle that all up into this signature manifest. Upload is the container registry in a well-known area in the same repository as the image that you pushed. Then it's all done. That could be the end of the CICD system, but usually the CD part is after cosine sign, it would do kubectl apply and tell the kube API server to then pull that image. At a station implying stuff like you have sent it through a vulnerability analyzer, for instance. What you can do at the same time is you can also use cosine to, quote, unquote, attach at a station like that, which are just a special flavor of certificate. It's just a certificate with a more interesting signature kind of. The only difference between the two is what commands you use in order to create them. You can attach arbitrary binaries, like arbitrary data to an image. Every single one of these components has been pre-attested by Spire. The moment you bring up the setup you see on screen, each of these would have called the local workload API. Gone through that selection of does the policy match in order for this workload to be issued an identity. That's all performed by Spire agent. That generates the CSR that gets sent to Spire servers, Spire servers, ScienceStat, the key materials propagated back. Does that answer your question? Awesome. A couple more questions. Justin Kapo is in the back. Sorry, here. Clearly your talk here was about the code and the progression of that through, but I'm just curious, assuming you're using an OIDC system for authentication, what do you do then for, say, the CI system, which is actually building the images that are being loaded into the registries, and say, yes, you need to authenticate that the code going into those is good, but what does the CI system itself do? It's obviously not going to be communicating back to it with an OIDC chain, but maybe it has some kind of refresh token or maybe it's got a TLS or something. What is the level of that? That's up to implementation. There's sort of ambient tokens like you have on GitHub and all the cloud providers. That's one form of identity token. There's what Spire does, and there's just an agent. You talk to it through a socket. As long as there's an OIDC identity token that is created somehow, that's all we need to know and also be able to chain is part of what goes into, clearly it would go into the log, but is that also part of the signing process? In other words, if you go into the container and look at the image, is there some kind of signatory of the CI and that chain entirely back without having to go to the transparency log? So, Fulcio actually has to trust the OIDC server. The public instance that we have put out, it trusts Microsoft, GitHub, Google, and I think exactly one SPIFI-based public OIDC endpoint. If you are implementing a public OIDC identity authority, there is a utility in the Fulcio repo that will allow you to create a PR to add yourself to it, but the alternative is just standing up Fulcio on your own, and that's what I was doing for the demo, and just trusting it in the config. Obviously, the SSL route of trust is its own separate thing, but you also add a configuration for this is the specific end, a list of end points that I trust, and as long as the endpoint is the issuing authority in the identity token. No, we don't specifically bundle the entire cert chain for your specific OIDC provider in the signing cert for the artifact. I have, I think, a quick last question. So, first of all, thanks for the tough shout out, and secondly, I'm a little curious about how some of the namespacing, you handle a lot of namespacing issues, which is always kind of a problem here. Are all the OIDC servers equally trusted, and how do you, because that seems problematic? That's an excellent question, and no, they are not. So, like I said, you have to, OIDC providers are explicitly trusted in the config for Fulcio. There is one like a trusted OIDC providers of record that, you know, our public good instance uses, like I said, those are like Google, Microsoft, GitHub, etc., but you must explicitly trust it. There's no, you can't like have another OIDC provider that's just like, oh yes, I am Google. Here is sirgayatgoogle.com. It was signed by him. But your users do end up trusting all of those. In other words, like, GitHub could claim, could provide certificates or things like this, even if they're for other types of, yeah. Sorry, one thing that I, one thing I did not include in this, oh actually, yes I did. Okay, so you can see that you have, you may and you absolutely should explicitly trust an issuer in your like client side policy. Like the specific, the specific providers that are allowed for, you know, in Sigstore's own, you know, representative Fulcio instance are just things that make sense and are not problematic. But like I said, you can stand up your own and you can do anything with it. Users always have the option of not, you know, setting the provider that they want to trust, but you know, that is its own issue. So in pretty much all of the examples we give, we encourage people to set the exact identity provider that should be issuing this identity. So just to make sure I understand this. So let's say that I'm running something like PyPI slash warehouse for the Python community where anybody can go and, you know, get an account and do things like this. So you're saying that at the moment that as a developer, I decide that I'm going to use OIDC with this, then how does, how does PyPI slash warehouse indicate that then that identity has to come from Google and couldn't come from GitHub, for instance? Or is that something, is that level of isolation possible? Do you understand what I'm getting at? Yeah, no, that level of isolation is totally possible. It's in the issuer. So it's like, you know, when you get an identity token, the issuer is one of the fields in, you know, this gigantic ugly jot. So it's, it's, there is an ambiguity unless you really want it. And it is totally, totally possible to, you know, narrow down the, like, explicitly say, no, only, only these identities come from here. Thank you, everyone.