 Hi, I'm Michael Lieberman and I'm a Supply Chain Security Engineer. This presentation and demo is on putting the supply chain pieces together, a deep dive into the Secure Software Factory. I will talk through what the Secure Software Factory reference architecture is, what SSF and implementation of that architecture is, and finally there will be a short demo showing off some of what SSF can do. So, what is the Secure Software Factory? Well, at a high level, the Secure Software Factory is a cloud-native but technology-agnostic reference architecture for building software while tracking provenance. In other words, it's a reference architecture that takes your continuous integration and build systems and helps fill in the supply chain security gaps like helping provide a chain of custody for your project from source code to package software. This chain of custody can then be used to help prove that your artifact went through all the appropriate built pipeline steps with appropriate controls applied to them, as well as be used for analysis later in the event of a discovered supply chain vulnerability or attack. Let's take a look at what the Secure Software Factory is within the context of your broader software delivery environment. As you can see here in blue, the Secure Software Factory more or less takes the place of your continuous integration or build system. It pulls in source code from your source code control, pulls in dependencies from artifact storage, performs various actions on them like lint scans, and eventually the build. It then packages the software up and pushes it out to artifact storage. The areas where things might begin to differ from a normal CI system is the inclusion of IAM at the top and the emission controller at the bottom. IAM here provides identities to the components within the Secure Software Factory, and you can use those identities to sign attestations on what is happening within the pipelines that run inside of the Secure Software Factory. The production emission controller can then be used to validate that only artifacts that have valid attestations signed by trusted identities from the Secure Software Factory are authorized to run in production. Now, let's dive in a layer deeper into what the Secure Software Factory actually is. This might be a bit hard to read, but you can follow along in the CNCF Secure Software Factory reference architecture document. The core of the Secure Software Factory is the build environment, which is here in the center. And then from the top and go encounter clockwise, first up, you can see that there's a pipeline framework which orchestrates the builds happening within the build environments. These are build pipelines defined as code, for example, Tecton, Jenkins, or GitHub Actions. Unlike a CI system with no or minimal controls, what types of builds and steps are allowed to operate in the build environment are constrained by an emission controller like Kiverno or Opa. This emission controller would have policy defined as code intended to constrain what is allowed to run within the pipelines. This allows you to have greater confidence that if you support a specific signed and attested version of a build tool, only that version is allowed to run in your builds. If you require all pipelines to include the step that generates a software bill of materials, you can enforce it through an emission controller with appropriate policy. Next up are the attesters and pipeline observer. A common attack against build environments is having those builds misreport their own output. You can imagine a compromised build could generate standard output and report its runtime parameters to look identical to a real build. A pipeline observer helps in this case by being a separate component that has access to record what is happening in the build environment instead of the build environment itself reporting its own output. It doesn't prevent every case of this sort of attack, but it does help. An example of a pipeline observer would be something like TectonChains. The pipeline observer gets much better, however, when paired with node and workload attesters. The node attester will validate that the build is running on approved hardware, for example, a Kubernetes node you run versus one spun up by an attacker. The workload attester, on the other hand, will validate that a build workload pod or container has not been tampered with after initial orchestration. This means if someone attempts to modify the Kubernetes pod while a build is running, a workload attester will be able to detect that and not sign off on the build. The most common set of attesters is Spire, which generates spiffy identities, which are short-lived identities that essentially live only for the lifetime of an individual build step and are linked to the parameters of the build step's pod. These short-lived identities help by both reducing attack surface and limiting blast radius. The attack surface is limited by the fact that the identity is bound by both its short lifetime and by only being valid against the initial configuration of the pod. The blast radius is also limited because if an attack somehow does compromise the build step and get access to the signing secret, that signing secret is valid only for a very short period of time and also only valid against signing the output of the compromised pod. The previous components mentioned help you enforce what build workloads are allowed to run as well as validating that those build environment workloads were spun up on appropriate infrastructure with allowed build parameters. But what happens if the build tool itself, for example, GoBuild or Maven turns out to be doing something bad? How can we detect that? That's where runtime visibility comes in. By using EVPF and other tracing tools like Tracy or Falco, you can monitor what is happening within the build itself. So if a build is attempting to reach out to malware.com or something, your runtime visibility should be able to detect it. You can also use runtime visibility in helping provide profiles for what your builds normally do. You can run builds in air-gapped and other isolated environments while tracing the runtime characteristics. You can then use these recorded profiles to detect anomalous behavior in builds in the future. If a build's memory syscalls follow a particular pattern and then later on you start noticing they're doing something quite different like writing unknown data to executable memory, this should help you be able to detect that. These components all together help fill in the supply chain gaps in your CI and build systems. Now, let's take a look at what an example build looks like. When using the secure software factory architecture, you shouldn't change your pipelines to no longer do things like security linting, SAS and DAS scans are similar. Keep running security-focused build steps. The key thing that the secure software factory does for us is help you generate provenance around your supply chain that you can have some confidence around. As this diagram shows, each step is recorded by the pipeline observer. The pipeline observer signs attestations regarding each step and forwards that metadata over to some storage. This can often be a transparency log like Recor, a document database like MongoDB, or even be packaged up alongside your artifact itself. For example, it is common to push attestations to live alongside your container images in an OCI registry. By recording each of these steps and pushing them out, you can begin to generate a chain of custody around what is happening in your build. If your build artifact step reports the hash of its output as something different than what the publish artifact step does, then you know something has gone wrong. This also helps you enforce that if a piece of metadata is missing like an S-bomb, you can deny that artifact from being authorized for production. Now we've gone through what the secure software factory is, but what is SSF? SSF is an implementation of the secure software factory reference architecture. The name might still change, but for now, it's SSF. SSF is a tool contributed by Citi to the open-source security foundation, a sibling to the CNCF. Its purpose is to act as both a system for people to use to help secure their internal builds and software supply chain, as well as being a project with a secure supply chain. We plan to do this by following the various best practices and standards defined in open-source groups like the CNCF and open SSF. Some of the best practices for supply chains to security today can be enforced in various, you know, SAS build tools. However, that often doesn't necessarily fit the needs of some organizations and projects that might not be able to be allowed to use SAS. This is also an issue for folks who are required to enforce different sorts of policy that are not supported by some of these other tools. SSF tries to help out here by enforcing policy at multiple different levels. There are the SSF defaults, which are supply chain best practices that should be enforced regardless of an organization's individual requirements. These are things like pinning to a hash for a container image instead of using the tag-latest, or that all build pipelines must contain an SBOM generation step. An individual organization, whether it's a company or organization like CNCF, might try to enforce high-level governance rules on all teams and projects that fall underneath them. These are things like naming schemes, allowed programming languages, and versions of build tools. Underneath that, there might be further restrictions based on an individual's team's requirements. An organization might say Python, Java, and Go have all been approved, but a specific team might say that only Go is allowed for their project. You can then further constrain configuration and policy to individual projects, so a new project under a team that uses Go might be required to use version 1.18 or above. These constraints help organizations, teams, and projects better enforce rules and controls, and thus help increase the confidence in what the provenance attestations are telling you is in fact true. Now, how does SSF handle and use this configuration? It does so by using a configuration language called Q. Q, if you're not aware, is a language that has been seen as a successor to Google's internal configuration language, similar languages like JSONit. It helps us by instead of having to write individual YAML for both our tecton pipelines as well as Qverno policy as code to enforce the rules, we can write specifications in Q, which lets us easily generate interfaces, so a lot of the cross-management of Qverno config maps, tecton tasks, and pipelines can all be managed from a single set of specifications. As mentioned in the previous slide, those specifications can be further refined as we need to for individual teams or projects. All right, now let's take a quick look at what SSF looks like in action. If you want to, you can follow along at the Git repo listed here. All right, so I already have SSF running, which is running on Minicube, so if I just quickly show off Minicube status here, you can see there's a Minicube running, and if I run, whoops, you can see there's a bunch of different namespaces with a bunch of different things in them. All right, so I already have the initial configuration set up, so it already has a key for this particular build setup, and I'm just going to run an example of build packs build here, and to be clear, this is mostly just running tecton, but it's running it within the context of a secure software factory configuration and setup, so it's running with all the various best practices you would want to have set up in this, and it also has all the other associated tools you might want to have running here. And then there's certain things that haven't actually been deployed out to SSF yet, like runtime visibility, but it is one of the things that are currently being worked on. So while that's running, I can show you the logs here. Oh, actually ran and completed already. This is just sort of the build logs here. You can see it generated this image, which it's using TTL.sh, which if you're not aware of, it's just a free sort of test registry for pushing out images that you want to expire. So for the sake of this demo, very straightforward. And I can go and validate that tecton chains here, which is one of the components of SSF, has come through and signed. Yes, it has. So if I run a couple of these other commands here, and all I'm just doing is just getting the image URL and task run to make some of these other commands a little easier. So if I run crane, which is a tool that allows me to manipulate and see what's inside of an OCI registry, I can see that, hey, I have an image here. And this image contains the latest, and it contains a .att, which if you're not familiar with, is just sort of a cosine, sorry, a six store based attestation. And it also contains a signature here. And now if I want to go and validate that this image has been signed with the appropriate signature, I can go and validate that, right? I can go and check that. Yep, it has been signed. Cosine is saying, yes, you know, based on this key. Yes, sorry, based on this public key. Yes, it has been signed. That's a valid signature. And then what I can do here is I can also validate that there is a valid attestation with a valid signature. Yep, and so it's able to say that as well. But now let's go and actually take a look at what that attestation is telling us. So I can download that attestation and essentially base 64 decoded and fetch out the actual data that's in there. So as you can see in here, you know, I have the name of the image. I have a digest of the image. And this is, for those who aren't familiar, this is a salsa attestation. And what this tells us is it tells us all sorts of nice little bits of information. This information is coming out of stuff like tecton chains. But in here, I'm able to do things like, I'm able to see what are the environment variables, right? I'm able to go and check stuff like what was the entry point for the build? What was, I can also check in here, what was the builder for the build, right? Like in this case, it's using, you know, this sample builder image. And I can do all sorts of other things. You can see when the build started when it finished. And all of this metadata is super useful to me when sort of figuring out, you know, did something happen to my build, right? If I expected, you know, a particular builder to be, you know, if I expected to use a different kind of builder and it didn't match this, well, then I know something went wrong, right? And I can use all of this metadata coming out of here. And you can imagine if I generated an S-bomb, I would have a similar sort of attestation that looks like this that has all the metadata associated with that S-bomb. And I'd be able to go and validate that, you know, was that S-bomb generated by a build tool that I, you know, that I trust? If yes, then, hey, great, you know, we can approve this sort of thing to go to the next level to be promoted to production, whatever. If I notice that, you know, it's using a tool that hasn't been approved, we can control that through stuff like policy as code. All right. So just to kind of quickly show a little bit of what this configuration looks like, well, the configuration here is all in queue and it looks very similar to JSON. But what this actually does is using the rest of our tooling, it allows us to generate the YAML configuration for Tecton, as well as other things like, you know, if we wanted to generate an associated policy with the build, we can do that all by sort of generating a single sort of specification here. And so all of this sort of stuff allows us to then sort of have some increased confidence around that, you know, a build happened in the right order with the right build steps, with the right controls, and that all of those different things happened with the right identities signing off on those things. And then, you know, those sorts of things help provide increased confidence. And so when you go to production, you can start to, you know, ask via also policy as code using stuff like emission controllers, you can say, does it have a valid, you know, salsa attestation? Does it have a valid signature? Does it have a valid S-bomb? Does it have a valid SAST and DAS scan with appropriate attestations? Do those SAST and DAS scans, you know, meet our policy, you know, for no vulnerabilities discovered or whatever, right? Assuming the answer is yes for all of those, then you can have increased confidence that everything looks good, that your supply chain is secure, and you can then approve it and, you know, do something akin to, you know, what Google calls binary authorization, but the basic idea just sort of being, you know, you can sort of stamp your artifact for use. Anyway, that's about it all I have right now, and I wanted to then open it up for questions.