 I welcome folks to this talk on Salsa and Fresca. I subtitled it The Road to Unfalsifiable Providence, because that's what I kind of believe it is. And this talk was really born out of conversations that I had with folks at previous conferences where there wasn't a lot of awareness of Salsa and Fresca. So I wanted to make sure that I submitted a talk that brought more awareness to both of these topics. So that's really kind of what this is about. My name is Christopher. I go by Chris. I shorten everybody else's name, so you're welcome to shorten mine. Consultants, I'm an instructor, and I help folks build platforms and adopt new technologies. And it's been a number of years that I've been doing that. So I helped with some of the certifications in the space. So for example, I helped write some of the questions and help author the CKS test. So if you're going for that certification and it's really, really hard and you want to curse my name, you're welcome to do that. All right, so what is Salsa? It stands for supply chain level for software artifacts. It is a framework for assessing and communicating the trustworthiness of software artifacts that are produced by the software supply chain. So think of it as like a checklist. You check off a number of items from Salsa, and you can communicate that your artifacts are compliant with Salsa levels in the supply chain. We're going to go over those as we go through this talk. The framework is aimed at software producers. So there's another framework that's about ingestion consumption on the consumer side. So this is all about you producing artifacts for consumers downstream. They want to be able to verify that those artifacts are indeed what you told them you were building, right? And the idea is to provide that transparency to consumers downstream so that they can do that. And when we build artifacts, we include verifiable trust metadata with those artifacts so that they can do exactly that. There's details about how the artifacts were built, the platform where it was built, and then we cryptographically sign those that made a data so that we can ensure that there's a guarantee about how that artifact was built, and then the consumer can trust it. So the Salsa 1.0 standard is split into tracks and levels. The previous, before 1.0, we didn't have tracks or levels, but when they released the 1.0, they broke it up into tracks. I misspoke, actually. There were levels before, but there weren't tracks. The idea behind having tracks is that as an organization, you may be really good at one part of this and not so good at something else. And so that could block you from being higher on the scale, being a higher level than you normally would have been if it's going across disciplines. So right now, in the 1.0 specification, there is just the build track. This was an important one to release first because of the fact that software prominence is such a challenge, right? There's going to be additional tracks that come along, so there's going to be a source track that's planned, build a platform operations track, et cetera. So you're going to be able to assess your Salsa level for these other tracks when they come out. And then, of course, saying we're Salsa level 3 build, then recognizes your progress, even if you're source level 2 or something. That way, we're not boiling the ocean because that would be really difficult. And oftentimes, when we try to do that, we fail. So each track has levels indicating the supply chain security level guarantees. So of course, higher levels indicate that you're providing more guarantees to your consumers. And having these levels also encourage iterative improvement, right? So we don't have to hit the highest level at the beginning. We can just start with something. And a lot of the folks that are working with Salsa and as well as Fresca, they'll tell you, right? Something is better than nothing. And so even if you're at level 1, you're doing something. And it's better than just not having any sort of provenance on your artifacts. So why Salsa? We want to be able to prevent any sort of unauthorized changes from unauthorized actors. The background of Salsa and the open source security foundation comes from partly from Google and then partly from the NSA and the CISA. And so the SolarWinds attack was a really bad one. I'm sure everybody here is sort of that. And so we want to prevent that type of attack from happening again. And so because the governmental agencies rely a lot on open source, they want it to be able to verify the provenance of those artifacts that they're using and as consumers have those guarantees. So other reasons, we want to prevent pulling materials, build materials from code repositories. Let's say an attacker switches out the Git repo that is being cloned in the build and swaps it out for their own nefarious loaded Git repo. If I have made a data that was signed related to that build, and this comes from concepts of Intoto, which is one of the things that Salsa relies upon, I can detect that sort of change and prevent it. Again, giving confidence to the consumers that the software that they're using is not compromised. So let's talk about the build track. Right now, 1.0 has three levels. There is a fourth level plan. So this is also changing over time. Again, you don't have to hit four right away. Start with one. It's a long road. That's kind of the reason why I titled this talk, the road to falsifiable provenance, because it's going to take a bit to get there. So the focus here in the build track is on provenance. We want to track from the source all the way to the artifact, the build ingest, the code sources, all the dependency sources that we're bringing in, all the build parameters that are used for that build, and any sort of outputs. That will give us a full transparency log of everything that was done during that build. The provenance statement, as I mentioned before, is currently in TOTO, is the standard. I kind of remember the first time I heard the word in TOTO dating back all the way to DockerCon 2016, where Justin Capos was on stage talking about Content Trust, which is a Docker product, and then immediately identified the problem, which was, well, upstream, if your build infrastructure was compromised, you just signed an artifact with your key that says, yeah, it came from me. And then you let someone use it. So it kind of opened my eyes to this large problem, and we're a lot of years down the road, and we're gaining maturity. It's nice to see. So shout out to that talk. It kind of changed my perspective on things. If you get a chance, go back and take a look at that talk on YouTube, it's good. So the in TOTO attestation is actually not required. The reason they're using it is it is easy to generate. It's easy to consume. It's easy to verify. But if you're just doing this internally in your own organization, it's OK to use another format if you want to use it. So why would I say that? Well, not all the tools that are available right now can generate an in TOTO formatted attestation, or they can't generate the version that is recommended. So that's OK. Again, something is better than nothing. So I'm actually going to show you an example of that because Docker has a way to do this, but it doesn't follow the salsa v1 standard. But that's OK. They make it often easy to do things. And so we're going to see some commands that it's fairly easy to do it. And that way you have some prominence that you can generate even if it isn't an official in TOTO attestation. OK, so level one, a couple of different requirements. The producer follows a consistent build process. You have a build script of some kind so that we know that it's a correct build or we can reproduce that build from that script. The only command, maybe the command that's used to kick off the script is imperative. That's OK. But we want to use a build script. We want to make sure that prominence exists, which means that this is going to describe how the artifact was built, build platform, the process, top level inputs to that artifact. So really any builder that generates provenance at all qualifies for salsa level one. So if you're generating provenance right now with some tool, even if you're not signing it, congratulations, you're salsa level one. So easy, right? We're already making progress. And then the producer distributes the provenance to consumers. The OCI registry, the registry specification was modified to allow for this type of metadata to be pushed with the image. Again, you don't have to use that. There are other ways to distribute the provenance as long as the producer and the consumer agree upon those things, then that's OK. But I'm a strong believer in if it's hard to do, we're not going to do it. One of the reasons why Docker gained so much popularity was they made things really easy to do. I remember listening to Solomon Hikes say this on the stage, right, their former CTO. The tools should get out of the way. So as long as it's easy for you to do, we make it approachable, then maybe you'll do it and you won't avoid it. So we'll try and do that. So we're going to do a build level one example here. I'm just going to use Docker desktop, right? You can use Minicube. You can use Cubadmin. Anything that will throw up a Kubernetes cluster doesn't have to be hosted, but it could be. And then we're going to use Tecton pipelines. This is going to run tasks to build our artifacts. And then TectonChains is our pipeline observability piece. And that will give us the prominence generation. So I'm going to switch to the terminal here and we'll run that. So there's a number of different commands. This Docker desktop is just empty, right? Let's just say you're coming at this from scratch and you're just using Minicube or some Kubernetes cluster doesn't have anything set up. The idea here is that this is fairly easy to set up. So what are the commands we're going to run? We're just going to apply the latest release of Tecton. We're going to wait until all the pieces are ready so we can move on. We're going to apply then TectonChains. And then we're going to patch chains configuration to use OCI storage, right? So you have a choice in which storage to use. I want to deploy my attestation with my registry. So I'm going to just patch it to use OCI and Tecton for storage. We'll then just restart our Tecton's controller to pick up that change and then apply a few tasks. These are just example tasks that come from the catalog, right? So Canico and Git clone are just standard tasks. This SIFT one, I modified a little bit because I have a local registry here that I ran that I'm just running through kubectl proxy. And it's not secure. So basically what I did is just, for the purposes of the demo, I modified the SIFT tasks to just allow me to do something insecurely, right? So I don't have to worry about logging into a registry somewhere. I didn't want to rely upon conference Wi-Fi to do all that. So this is my way of avoiding that, pushing up to the cloud or something. And then it's going to apply the demo pipeline. I was just going to invoke those tasks, right, one after the other. And then it'll cat the pipeline just as a reminder for me to show it to you. And then another note about starting the pipeline. So let me run that real quick. So that's going to deploy everything. We'll wait there for a second while it deploys. Fairly straightforward to get this up and going on just on my laptop. And at the end here, we'll look at the pipeline, and I'll show you the different tasks that were created. All right, so these are the things I was talking about. I'm telling SIFT to skip the TLS and using HTTP again, just so I can use a local registry. So next thing I'm going to do is I'm going to create a signing key. Now, this is not required for Salsa Level 1, but it's easy to do, so I wanted to show it. So I'm just using cosine, generate key pair. And I'm just going to upload it to my local Kubernetes cluster. And that'll give me a key that I can use to verify the secret when I'm done. So I'm going to kick this off. And we're going to watch for the tasks and the pods that get created by Tecton. And it'll go through, and it's just three stages or three tasks. I didn't include something like a scan of the image before we signed it just because it's not necessarily pertinent to this particular conversation that we're having, but you would want to make something more sophisticated that made sure you didn't have CVEs in your image before you ran it. So yeah, so it threw an error, I think, because it's probably trying to pull an image from the cloud. And it's not going to do that because of Wi-Fi. So I did a thing where I recorded a video so that we can watch the local demo here, because I assumed that that was going to happen. So we'll jump ahead here a little bit. OK, so we're back to the point where we were creating the demo pipeline. We're going to watch that go a little bit bigger here. So I had more reliable Wi-Fi, and I was able to download the actual build images. So give that a moment. We've got the cloning of the repo, which is just a demo application. It's a Python sort of hello world thing that we use for our classes. And then the build and push uses Canico, just out of the box. And then the generate bomb is, again, the customized sift. But it's mostly just the out of the box sift that comes from the tecton catalog. All right, so at this point, what I'm going to do is I'm going to use a crane. And I'm just going to query for that metadata. So I'm going to pause it there. So you can see the artifact. It's the tag is salsa1. And then you can see the artifacts that are using the shahash. So I've got the software bill of materials. I've got the attestation. And then I've got the signature. Now, again, the signature is not really required, but I wanted to show you how easy that was to do, to include a signature for that. And so what I'm going to do is download the attestation and just decode it so we can look at it. So we do that. And then we'll cap that attestation. We'll see that it's, of course, machine JSON. So I'll redo it again. And we'll look at it through JQ, just so we can read it. And so there's my attestation. And now I don't have to do anything extra to distribute this because it's already in the OCI registry. And people can rely upon my attestation. And it's in the salsa v1 format. So then I restarted the video there. OK, so fairly straightforward. I just did everything on my laptop. But I still had to fire up a Kubernetes cluster. I had to install chains and all that. I added all the commands here. So that it's not magic. You can repeat all of these commands. And they should work for you, except for the one special SIF task. But everything else is kind of out of the box. So I included that in the slides. Let's look at an alternative. As I mentioned, Docker has the ability to generate provenance and S-bombs. So we can use the buildX command with building. There's a couple caveats to this. The default Docker image store doesn't support attestations. So we have to push it. So the workaround is to use a push flag and push it directly to a registry. Or use container D for pulling images. And that works around it. So let's take a look at that. So this is the first command. BuildXBuild, we're just going to tag it. I'm going to tag it salsa0.2. Why am I tagging it salsa0.2? Because it doesn't, right now, follow the salsa version 1 specification. It follows salsa0.2. So I just wanted to make that clear. We're going to create an S-bomb. We're going to generate provenance. So hoping again that this builds, pulling images down. And yeah, it looks like we've got a successful build. So now what I'll do is use image tools inspect. I'm just going to inspect the document. And there I have my annotations. I've got the manifest associated with it. And if I go and add the extra provenance.salsa, I can inspect the salsa part of the attestation. So we can go through here and see all the different pieces of that attestation. Now again, it's not signed, but that's not required for salsa level 1. So again, fairly easy, right? If you're already using Docker in your workflows, you can use that right away. So again, I included that. So again, it's not magic. I want to make sure that we're transparent about all the commands here. OK, so let's move on to the salsa build level 2. This is all of build level 1, right? You have to have provenance. And then you have to do it hosted, right? So if you're already hosting your build platform, you're getting even closer to build level 2. Provenance is then tied to that build infrastructure through a cryptographic key, right? And that signed key should only be accessible to that build platform. So the factory for repeatable, secure creation of artifacts, say that 10 times fast, Fresca is much easier. That is capable of build level 2. Part of level 2 is, of course, then downstream when a consumer goes to consume the artifact that they validate the signature. That's the goal. So what is Fresca? Fresca is a reference implementation of the Secure Software Factory white paper, which itself was a reference architecture for the secure software supply chain white paper that was before that, right? So we went from things that were more abstract to things that are more concrete, right? So we have a reference implementation of open source tools that will help us create a hosted build cluster, right? And let's take a look at its architecture. There's a lot of pieces here. We've got, again, tecton for the pipeline build. And when I do a demo, I'm just going to run the exact same pipeline. We've got chains, again, for pipeline observability. But we're adding to that spire, and vault, and some other pieces here that we're going to go through. Why? Because the goal is what? It is unfalsifiable provenance. So I could falsify the provenance in level one, right? We want to get to the point where we can't do that. And so through Fresca tooling, we're going to try to get as close as we can to that. So how does that work? Spire gives us the ability to attest to these build workloads, right? So when I deploy spire on the cluster, the agents will go through a node attestation where they can prove that they are an actual agent running on a node in the cluster, right? So how do they do that? It's through a third party. It depends on what sort of platform you're talking about. You've got AWS, Azure, GCP. You can even do it through Kubernetes projected service account tokens, right? Once we prove that that node is indeed a process running in our cluster, now when we go to build, we want to make sure that that process too is also a process running on the cluster. So we use the workload API and workload attestors so that the workload has attestations associated with it, right? So we use things like the fact that it's a UNIX process, the fact that it's a pod or a container running on the platform. This is all important because we have to validate that these things are real, right? That they can't be spoofed, that it's not another process that generates the artifact, but this process that did it, so it can't be spoofed. So once we have Spire up and running, we have, by the way, authored a blog on a Spire deep dive. I'm sort of taking a high approach to this, but we went through Spire concepts, agent deployments, agent configs, workload registrations in depth. So you can understand all the different pieces and all the registrations that you make. And then we even test a spiffy ID, the idea being that once we have that spiffy ID, we can use that to integrate with other systems. Vault specifically is used to keep our secret material safe. We do that through putting our keys rather than on my file system as we did locally. We put it into Vault and then we use Vault's transit secrets engine so that that key never leaves Vault, right? We have that workload who gets assigned, right? The agent has attested that the workload, the builder, is a process on a node, and it will assign it a spiffy ID. We then tell Vault to trust Spire and trust spiffy IDs. We do that through an OIDC endpoint on spiffy so that Vault can then call, sorry, through Spire. Vault then calls Spire and can validate. Again, we're doing what? Third-party validation again, right? So we're presenting a token. And then the system that's holding the key is saying verify this for me, right? Trust but verify. And that way the key material is safe and we can sign the attestations so that, again, we're trying to get closer to unfalsifiable provenance. We're about to publish a blog on Vault, so that's coming soon. So if you follow our blog, you'll see that. We talk about the OIDC configuration. This is a lot of hard-won information. We learned a lot through this process. And then we even test secret access. To tie it all together, again, we go back to Tecton and we use Tecton chains. So what happens? The Spire node attests that the Chains Controller is a valid process running in our cluster, right? It will then grant the Chains Controller a SPIFI ID. Vault can then validate the Chains Controller SPIFI ID via that OIDC endpoint. And then Chains Controller is then granted access to that Vault signing secret and uses that secret material in Vault to sign the artifacts and the provenance. So now we have signed provenance. And we can attest that all of these processes were indeed in our cluster, and they weren't done by someone who injected some sort of malicious payload into it. So we're going to look at the Solstice Level 2 example. And so typing with one hand is harder than you might think. I'm going to actually SSH somewhere, because this can't be on my laptop, right? So we've built a hardened Kubernetes cluster using Kubernetes operations or cops, if you've heard of that. And so this is a Bastion server that is in front of a cluster that has been hardened and doesn't have public access. So we'll just back through these commands. Again, same command. I'm going to create that demo pipeline. I already created all that stuff. I don't need to create it again. And then we'll watch. There's a previous run here, so we're going to watch this next run happen. Again, same steps. Clone the repo. Build the artifact. Create the SBOM. And then change is going to sign it. And we should be able to verify this provenance if the cluster behaves. So we'll give that a second. So as it's doing that, in the background, I described a lot of things happening between Spire and Vault. The idea is that this is transparent to developers. This should all happen in the background and happen automatically through automation. So we have a successful completion here. So we will then use Crane again. OK. So in this case, I'm hitting a registry. Again, that's running in this cluster for the demo purposes. I've got the SBOM. I've got the agitations. And I've got the signature just like before. Except for this time, again, the signature we know was done in a safe way rather than just on my file system. And so I can validate that signature. Whoops. I'm typing validate and it's verify. There we go. Cosign verify helps if I use two hands. There we go. The cosine claims are validated. The signatures are verified against the secret material. And I use the secret material that's coming from Kubernetes. I'm not using the local key. I'm using the key that is stored in Kubernetes for validating these cosine claims. So before, I was using the local key. And so there you have it. Now we have a much more robust way of building artifacts. And I can have more security guarantees that my secret material wasn't compromised. And the build process was done through attestations that we can link directly back to the processes on my build cluster. We already looked at the attestation file. So we won't look at that again. But because it's going to look the same. Now we're not at salsa build level three. Build level three has much higher requirements. We have to prevent runs from influencing one another. And we have to prevent secret material used to sign the provenance from being accessible to any of the user-defined build steps. We're pretty close. But Fresca doesn't qualify for this because tecton chains right now will essentially just observe tecton tasks and then just sign things. So I could inject in between that maliciously and then just have change sign something. So it's not right now non-falsifiable. However, there is work being done to improve this so that the integration between tecton and spire is tighter. And that would guarantee our non-falsifiable provenance. So it's a work in progress, but we're getting there. It's a road. It's a road for organizations. It's a road for the open source community. But the idea is to get there. So I mean, if you're using GitHub Actions or Google Cloud Build, you're already there. But if you're sticking to open source, like we've been going through the last few slides and demos, then we're getting there. We will get there eventually. OK. So thank you. Hopefully, you learned a lot about the Salsa standard and the Fresca architecture. Again, we've got deep dives in our blogs for each one of the individual components. So we're going to keep going with that. Because I almost subtitled this Fresca the hard way, because that's the idea was to deploy everything piece by piece and really understand what was going on. Because I found that when we did the research for this stuff, it wasn't always clear what was going on. And we want to make sure that we're clear about that. And I do owe some thanks to a couple of my colleagues in the audience. My colleague, Valentin, Sigstor, is one of his favorite topics. So if you see him hanging out with me after asking him about Sigstor, Ilian did a lot of research on Spire. I'm just the guy that got to get up here and talk about it. So thank you, Ilian. And then my other colleague, Christian, he's not here. He's back in the States. But he had a lot of input on Vault. So I just wanted to make sure that they got proper credit. Thanks, everybody. I appreciate it. I guess we're taking questions. I wasn't keeping track of the time real well. Did I go over? Go ahead. Why did you add Vault to the whole thing? I mean, the Sigstor supports keyless signing. Is Vault really necessary in this scenario? So in the Fresca architecture, the important part about Vault is that the transit secrets engine, because it keeps the key material in Vault so that we trust Vault. We use the spids to cryptographically sign the annotations essentially in flight so that the key material stays there. And that's the most important reason to use Vault. So your key is secure in Vault. Thank you. Yeah, I'm a bit sidetracking maybe. Outside of this, for the actual build and artifacts, is there anything going on for tracing provenance of configuration? Because with stuff like him, for example, I'm loading configuration from a bunch of random places on the repository. That I don't think I can answer. We were talking about when you deploy workloads, like deploy actual applications on clusters, is there efforts for tracking the actual configuration there? Is that the question? Yeah, exactly. Because more and more stuff is going into the OCI registry and stuff like that. So I was thinking, is this framework maybe in the future going to be adapted to also the other parts? That's a good question. The specification allows really for any sort of arbitrary metadata to be stored in an OCI registry. So I would imagine that it wouldn't necessarily be something you couldn't do. I'm not aware of any particular efforts to do that. But the registry will support that type of metadata and the new specification. OK, thank you. Yeah. Over here? Yeah. Thanks for the great walkthrough. So I have kind of a basic question. So you went through all this effort in making sure that the node assigned and so on, configuring spliff here and so on. As a consumer, can I verify that you actually did this correctly or do I have to trust you that you set up everything? So that's what the Providence is about. In that Providence document, we didn't really deep dive it too much. But yeah, you know the identities that were associated with my build infrastructure. And those can be verified. So it's full transparency of not just the materials, because the SBOM gives you the materials. The attestations give you the additional information about what the process was, how the thing was built, and all of those extra details. There are a lot of details that we skipped over because we only have so much time. But yeah, absolutely. You can validate all of that. And I would even add you want to do it automated in an automated way. So I don't know the state of the support for that type of automation downstream yet. That's really what that other group I was talking about at the beginning was for consumers, for ingest. They're really working on that sort of thing. So that once we have the artifact signed and we have the attestations, then there's an easy way for you as a consumer to then consume it and validate it. So I can't say I'm paying attention to that or part of that effort. But they are trying to do that. Yeah. Thanks. Yeah. All right. I think they're serving beer somewhere in the other room. So probably a good idea, right? Congratulations. You made it to the end of the first day through KubeCon, right? That's always an achievement.