 Hi everyone, welcome to securing your container native supply chain with Salsa at GitHub and Tecton. Hi everyone, I'm Laurent. I'm a security engineer at Google. I work on the OpenSSF scorecard project and supply chain security. And I'm Priya. I'm a software engineer at ChainGuard. I work on open source projects like Sixdoor and Tecton, which we'll be talking about today, and a bunch of other open source projects as well. So you've probably heard a lot about software supply chain security in the news recently. As the graph shows, attacks are on the rise, and it's becoming really important to start to understand how you can secure your supply chain. So let's take a recent attack as an example. The NPM caller package attack. The attack happened just a few months ago, and one of the maintainers decided to add an infinite loop in the package. The code was never changed in the repository itself. The maintainer pushed directly to the NPM registry. Around 9 million projects depend on this package, either directly or as a transitive dependency, which is really scary. As a developer, it might be intimidating to know where to start when it comes to securing your supply chain. So in this presentation, we'll discuss practical steps you can take to start improving the security of your supply chain. So the good news is that if you already used Tecton or GitHub workflows today, there are a few easy steps that you can take to start securing your pipelines today. So that's what we're going to be talking about in this talk. Let's jump right into it. So a quick agenda for today. When discussing supply chain security, it's important to have some metrics or a framework to actually quantify how secure your supply chain is. So that's why we're going to start by talking about Salsa, which is the framework we're going to be using to determine how secure our supply chains actually are. Then we'll jump into talking about Sigstore, which is a project we use for signing and verification. And at the end of the talk, we'll talk about how we bring Salsa and Sigstore together to achieve higher security levels in both Tecton and GitHub workflows with demos for each of those platforms. So what is Salsa? Salsa is an acronym that stands for supply chain levels for software artifacts. You can think of Salsa as a framework or a checklist, basically a list of things that you want in your supply chain depending on how secure you want it to be. Salsa is not a tool or a service. It's more a set of guiding principles. Some key principles about Salsa include the following. Artifacts should be signed. And artifacts should also include metadata or so-called Salsa provenance about who and how the artifacts are created. Now, you might be wondering why we don't just use signatures. And the reason is, signatures do not protect against all sorts of attack that happen on the supply chain. Let's take again the example of the NPM caller attack. As we said earlier, the NPM maintainer pushed directly to the NPM registry. So the maintainer had access to the credentials. The maintainer would probably have access to the signing key as well if signatures had been used. Salsa provenance solved this problem by including who created an artifact and what source code was used to create it. So in a nutshell, Salsa provenance provides the following benefits. It allows tracing an artifact back to its origin, including the exact source code, repository name, hash commit, and branch. So the Salsa framework has four levels ranging from level one to level four. It's meant to be incrementally adoptable and it's a set of security guidelines established by industry consensus. Level one is supposed to be the most achievable level while level four represents the ideal end state. Level four can be difficult to achieve, so levels two and three are meant to be stepping stones on your path to achieving Salsa level four in your secure supply chain. Depending on your build system, you can start making progress towards Salsa level two and Salsa level three today, which is what we're going to be talking about. So a quick overview of Salsa level one. Salsa level one is the most achievable level with only two requirements. The first is that your build process is scripted, and the second is that you have some kind of provenance available, some sort of metadata about how your artifact was generated and maybe what also went into it. Salsa level two builds on top of Salsa level one and has some additional requirements. So this is where we start getting a little bit more secure. Salsa level two requires that your source code is an aversion control system like GitHub. Hopefully you're already doing that. It also requires that your builds are built in a service and that provenance is service generated. Provenance should also be authenticated, which means it needs to be signed so that end users can verify that the provenance for the artifact they're looking at is coming from where they expected to come from. So now that we've talked about the framework we're going to be using to evaluate how secure our supply chain pipelines are, how are we actually going to start implementing these Salsa requirements? So this is where Sixter comes into play and both of our demos involving GitHub and Tecton later in this talk are going to be using Sixter under the hood. So Sixter is an open source project aimed at making signing and verifying software free, easy and accessible. And how does Sixter accomplish this? It provides a few tools and services which are available for public use. And the first tool that it provides is called Cosign, which is a CLI tool and library used for signing and verifying software artifacts. This could be things like OCI images, espoms or provenance. One of the services that Sixter provides is called Fulcio, which is a certificate authority service that issues short-lived code signing certificates. It also provides a service called Recort, which is an append-only transparency log for storing signatures, certificates and provenance. But why do we use Sixter instead of traditional signing methods? What kind of problems do the Sixter actually solve? So the main benefit of Sixter is that it eliminates the key management problem. So we have a diagram of a traditional signing situation here, where we have a maintainer who holds on to a public-private key pair, signs their piece of software and then distributes it to an end user. The end user is responsible for verifying that software before they actually use it. But there's a few issues in this scenario. First, it requires that the maintainer holds onto the private key and keeps it safe for long periods of time, potentially years on end, which can be really difficult. What if that private key gets lost or ends up in the wrong hands? It's really hard to then revoke old signatures. It also means that the maintainer is responsible for distributing the public key to end users, figuring out how end users can get their hands on it. A lot of times, instead of verifying signatures, end users just choose to use the software without doing the verification because it's not easy to do the verification. So this is where Sixter comes in. It solves this key management problem so that maintainers no longer have to keep track of private or public keys. But how does this work? So you can see we have this updated diagram now, which includes the Sixter tools and services that I had mentioned earlier. So instead of holding onto a private key pair, the maintainer actually just has to request a certificate from the full certificate authority. This certificate is a code signing certificate and contains and is bound to the identity of the maintainer. In this case, it's the maintainer's email address. The certificate also embeds the public key that was used to sign, that is associated with the private key that was used to sign the artifact. And the signature is bundled along with the certificate and stored in the Recor transparency log. This is really convenient because the log is public. So when the end user attempts to verify the software artifact, they can use the Cosign CLI tool, which under the hood will retrieve the correct entry from Recor and do all the verification for you. All the end user has to do is decide if they trust the identity that is embedded in the certificate. So the nice thing about full CO is that both people and systems can request full CO certificates. People can prove their identity by signing in with an email address. So this would be like a sign in with GitHub or a sign in with Google type flow that you might have seen on different websites yourself. Systems can also authenticate and request full CO certificates. So workloads can use the Spiffy S with specification. And there's also support for using Kubernetes service accounts and GitHub actions invocations. So we're going to be using these last two in our demos later on in this talk. So how do we bring Salsa and Six Store together to actually start securing some build systems? So we'll take Tecton first and we'll talk about GitHub after. So what is Tecton? This is a big question and there's a lot to Tecton. So in the interest of time, I'm just going to cover the important bits for today's talk, but I would highly encourage you to go learn more about Tecton if you're interested. For now, all you really need to know is that Tecton is a continuous delivery system built on Kubernetes. It leverages custom resource definitions to run tasks and pipelines on your cluster. You can kind of think of Tecton just as a CICD system where you would do anything you would normally want to do in CICD, including running unit tests, running integration tests, building some sort of software artifact and then storing it somewhere for your end users to be able to use. The basic unit of Tecton is the Tecton task, which is you can deploy your Tecton task to your cluster. If you look on the right side, we have some sample YAML for a Tecton task. If it looks like a pod or it reminds you of a pod specification, that's because the Tecton controller is responsible for turning this task into a pod so that it can actually execute in the cluster. You can instantiate this task with the TKN CLI tool to turn this task into a task run. Task run is actually executed in the cluster and we'll be doing this in the demo later on in this talk. So what do we know once the task run has completed? Well, there are a few key pieces of information. First, we know whether the task run was successful. We also know what steps were run and exactly what arguments were passed in. And then finally, we know what artifacts may have been built in this task run. This is really useful when trying to achieve certain salsa levels. We know exactly what was run so we can put that into our salsa provenance and we also know what artifacts were built so we can generate provenance for those artifacts and also try to sign them. So how do we do this? This is where Tecton chains comes in. You can think of Tecton chains as a supply chain security manager for Tecton. It leverages six store under the hood to sign artifacts that are generated by Tecton and also generate signed provenance. Here's a little diagram, I drew of how this works. Basically, you have your cluster running and you have Tecton pipelines, which is the core Tecton project running in your cluster. When you start a task run, the Tecton pipeline project is responsible for executing that task run. Meanwhile, Tecton chains is observing all the task runs in your cluster and once it sees that the task run has successfully completed, it'll try and extract any artifacts that it recognizes from that task run. It'll generate signatures for that artifact, generate provenance, and then store those two things alongside the artifact in whatever storage backend you have configured. This could be an OCI registry or a GCS bucket. You can also configure various types of signing keys to actually sign these things with Tecton chains, including KMS keys, public private key pairs that you might have set up yourself, and full CO, which we had talked about earlier. So I'll do a quick demo of everything we just discussed with Tecton. So let me share my terminal. So in the interest of time, I've already created the cluster that I will be doing this demo on. It's an Amazon EKS cluster. And the most interesting thing about how I created this was I passed in this with OIDC flag. This is basically going to allow my Kubernetes service account to authenticate to full CO and request code signing certificate for the piece of software I'm about to build. So I already have Tecton pipelines installed in this cluster. Installing chains to add in supply chain security, the supply chain security manager is just a simple COOP control apply command. I've already run this, so I won't run it again. But the last piece of configuration we need to do is tell Tecton chains exactly how we want to generate our provenance and sign it. So there are four commands here. The first one is telling chains that we want the format of our provenance to be the in-todo format, which is what's recommended by the salsa framework. We want to store our signatures in an OCI registry, and we've also enabled the transparency log, which means we want to store our signature and certificate bundle in the record transparency log. We've enabled signing with full CO, so we don't actually have to set up our own public private key pair or point to KMS. We'll just request a certificate, and that's how we're planning on signing our artifact. So let's take a look at the artifact we're actually going to build. We're going to be using Kanako to build an OCI image and push it to a registry. So this is the task that we plan on running. There's just a few parameters passed in, including the name of the image we want to build, a few results here, which is how we're going to tell Tecton chains that an artifact was built. The first result is the digest of the image we're going to be building, and then we have the URL or the name of the image that we're going to be building. There are three steps in this task. So the first one basically just creates a simple little docker file, which we want to build. The second one actually does the build and push of the container image, and the last one writes the name of the image to our result so that we know what image was built. So we can apply this task to our cluster, taking a second. Amazing. And we can kick it off with the TKN CLI tool. So while this runs, I basically passed in and I've already stored this image name as an environment variable, or I have not. Let's show this again. I'm going to delete this task run that I just created. Let's start over. This is the image that we want to build, and let's kick off another task run. So now we're passing in the image name that we want to build, and we're running it under the service account I created earlier called ECR pusher, which has permission to push to my ECR registry. We take a look at the logs for this image. We can see that we're running through the steps, adding the docker file, building and pushing the image, and we have written the results. So this task run has completed successfully. If we take a look at the task run using that same CLI tool that Tecton provides, you can get a little bit more information about the image that was built. So we have the image URL, which I'll just store into an environment variable, and the digest. So we should now be able to run this image. It'll pull it down, and we can see that it has printed out Hello World, which is exactly what we expected. But now we want to make sure that the image that we built was signed and that provenance was correctly generated. How do we do this? The nice thing is that under the hood, Tecton chains has already realized that an artifact was built, attempted to sign it, generated signed provenance, and stored it in our OCI registry. As an end user, it's really easy to verify that this happened using the cosine tool. So at the moment, transparency, log, and full state of support are experimental in cosine. So I have to set this experimental flag. But all we have to do is a simple cosine verify, and then the name of the image, and we should see that verification is successful. If we pipe this into JQ, we can get a little bit more information around the certificate and the signature. And you can see that the subject in the certificate is the service account that we were running under, and that the issuer points to my EKS cluster that we ran in. So it's really cool. We didn't have to worry about keys. We didn't have to do anything. Everything just happened under the hood, because we already installed Tecton chains. We can also take a look at any provenance that was generated with the cosine verify attestation command. And looks like it was successful. Again, we see our certificate subject, our certificate issuer URL, and we can take a look at the provenance that was generated by decoding it and piping it into JQ. And you can see that we have our provenance now in the salsa provenance format. The subject is the image that we just built, and you can see that all the steps that we ran exist in this provenance. So we know exactly how it was built, and also when it was built. Let's transition back to the slides. Cool. So you can see that by just by adding Tecton chains into our existing Tecton deployment, we were able to achieve salsa too. Assuming your code is already in version control, our builds are scripted in Tecton. This build service is Tecton. Provenance is available, authenticated, and service generated, since it's been generated by Tecton chains. So Tecton plus Tecton chains equals salsa level two. Yes. So now let's talk about GitHub workflows and GitHub actions. So many projects are natively developed on GitHub. So it would be great if we could also generate salsa provenance on GitHub. And this is the motivation behind this presentation. So on GitHub, the standard technology that you use to run CI are GitHub workflows and GitHub actions. Whether you want to run simple unit tests or end-to-end integration tests, or maybe release pipeline, you can do all this using GitHub workflows and actions. Typically, you define GitHub workflows by creating a file under .github slash workflows on your repository. GitHub workflows are very powerful, and you can run arbitrary commands. You can also define so-called triggers to start your run. For example, you can start to run on an incoming pull request, or when there's a new push to a branch, or when a new tag is created on your repository. And here's a simple example of a hello world GitHub workflow. All it does is print hello world every time there's a push to the main branch. One powerful feature of GitHub workflows is the ability to support isolation between jobs. So you can define different jobs in the same workflow. Here we have two jobs, one in red and one in green called PR and secrets. Even though they're defining the same workflow, those jobs run in isolation. In red, we have a job that runs on incoming pull requests. The pull request may contain malicious code, might not have been reviewed, so we want it to be isolated from other jobs in the workflow. And in green, we have a sensitive job that accesses credentials or other sensitive information like a secret. This could be the job that publishes container images to your registry and needs a secret to log in the registry. So what I'm presenting today, using GitHub actions and GitHub workflows meets the salsa level three requirements. So let me tell you a little bit more about what those requirements are. The first requirement is that the build must use an ephemeral environment. The motivation behind this is that it makes it harder for an attacker to gain consistency to your release pipeline if they were to gain, if they were to attack it. On GitHub, each job in a workflow runs in a fresh VM, so we get that requirement for free. A second requirement is isolation, meaning that the provenance generation must be isolated from the maintainers. So what this means is that the maintainers that are building an artifact shouldn't be able to interfere with the provenance generation. And this leads us to the last requirement, which is that the provenance should be non-forgeable, meaning that maintainers shouldn't be able to lie or temper with the provenance information. They shouldn't be able to lie about what source code was used, for example, to build the artifact. So let's see how we can generate salsa three provenance on GitHub using an example. All you have to do as a user is define a workflow and pull the trusted builder. Once the run is triggered, the execution is handed over to the trusted builder here on the right. The trusted builder is hosted in a different repository. In our example, it's called github.com slash trusted slash builder. And the trusted builder is not under the control of the original workflow on the left, and therefore the workflow on the left cannot temper with the execution of the trusted builder. So we can do anything we want in the trusted builder. And the first step we take is building and publishing a container using code. And then in a second step, we can generate the provenance, sign it, and upload it to a registry. And here, we can just use cosine API and the six-store tooling to do that. So something I want to point out is how signature generation happens. I think Pri already talked about this, but I want to mention it again. In the example that you saw, we didn't have to provide any sign-in keys. And this is a key feature of generating source of provenance using six-store. Key generation and management happens transparently through workflow identity. So when the trusted builder runs, it is provisioned with a unique X509 certificate that uniquely identifies it. And for this, we use OIDC, as Priya mentioned earlier. This certificate is then used to sign the provenance. So during provenance signature verification, we can inspect the certificate and learn who produced the build. Just to make things even clearer, I want to compare the WebPKI and the SalsaPKI that we've built using six-store. On the web, a CA may be verisign. And when you visit a Google website, the X509 certificate that you get during the TLS handshake will have the Google identity, such as 3w.google.com. In the SalsaPKI with six-store, the CA is six-store. And when you verify the provenance, the certificate contains the identity of the builder. So during verification, we can inspect the certificate and identify the builder by name. And if we trust the builder, then we can trust the sun provenance and its content. So it's pretty easy to verify provenance. We've built a tool called SalsaVerifier that can help you do that. And under the hood, it's using the cosine API. All you have to do is pass it an artifact OCI of your container and tag. And one option that is mandatory for you to pass is the source code that you expect this container to have been built from. In our example here, it should come from github.com slash origin slash repo. Optionally, you can also pass a branch, which by default is set to main. And then you can also pass a tag if you expect the container to be built from a particular version or tag of the repository. So as Priya mentioned before, thanks to six-store, we don't have to handle public keys. And you can just pass the identity of the repository you believe the container should come from. And the verification happens automatically for you. Okay, so let's have a quick demo. Let me share my screen. So the example we're going to take is hosted on github. It's called KubeCon minus EU22. It's a simple hello world, which will print a random UID. So we're going to take this go file and package it up as a container using code. Doing this is pretty easy. As a user, what you have to do is define a workflow, a github workflow. Here it's called code minus publish. And then here we define different jobs. You can ignore this part. You can look at it if you're interested, but the most interesting part of this workflow is this part here, the build part. All we do here is we basically call a trusted builder, which is hosted under this repository. And then we pass a bunch of arguments to code. So here we're saying build a multi-architecture image and tag the container with tag five and tag six. We also pass environment variable that coexpects such as code docker repo, which is just the docker account where we want to publish the resulting container image. And we pass a password to be able to publish to the registry. Then all we have to do is run this workflow. So you can go to the action tab here on github and here on the right, run workflow, and we run the workflow. So it's going to take a few minutes for this to complete. So I've already run the release a few times. To save up sometimes. And here on the left, you can see all the different steps that were taken by the builder to build this image and publish it and sign the provenance. And this is the resulting container image that we've got. So let's try to verify the provenance. Okay, so let's first verify that we can run our container image. Okay, so this seems to be working. We have our hello world. We can also run it by using the hash, which is the same thing. Okay, so at least our container image is working. Now let's verify the provenance. Okay, so sasa provenance was verified and docker, the docker container is printed. That's what it resolved to these tactics. Now let's take this command and change the expected source repository. Let's say 23. And as expected, it doesn't pass the verification because the source repository doesn't match. As we said earlier, we can also pass it a main branch which verifies. And if we were to pass a different branch, it would fail. We can also print the provenance. And as expected here, it looks very similar to what Priya presented with tecton. Here we get the source origin repository and the commit hash that was built from. We also see a list of environment variables that were passed to Co to build. And also the command that was used by Co to build the container. There's additional information about the environment used by the trusted builder. Architecture was AMD64 running on an Ubuntu machine. And we see that the trigger event was a workflow dispatch, which is just saying that I clicked on the button to run the workflow. We also see up here that the reference is the main branch of the original repository. So we have all the information that we need. So let's quickly go back to the action run. It just completed. And we can see again that this build is reproducible. And we get the same container image that we verified. Awesome. That was super cool. Let me present it again. Yeah. So to conclude, GitHub Actions and GitHub Workflows gives you Salsa Level 3 if you use a trusted builder. Thank you, everyone, for attending. If you have any questions about this talk, feel free to reach out to either of us. I'll be there in person. So look out for me. And yeah, thank you so much for attending. Thanks, everyone.