 One of the first tactics mentioned when the executive order first came out was the need to have a blueprint for everything that makes up your software application. This blueprint is called a software bill of materials, often shortened to S-bomb. It shows all of the third-party code and dependencies that together make up the application. GitLab provides a dependency list, which is much the same thing, but the harder part is certifying it. One of GitLab's key channel partners, BoxBoat, has solved the challenge. Cole Kennedy from BoxBoat and Nicole Schwartz from GitLab will together explain more about the ins and outs of S-bombs and how they can help you secure the integrity of your software supply chain. Hello everyone, welcome to my talk, securing the software supply chain with S-bomb and attestation. Let me introduce you to myself. My name is Cole Kennedy. I'm a director of defense initiatives at BoxBoat Technologies. You know, I've been working a lot in supply chain security over the past couple years. Recently, I contributed to the CNCF Secure Supply Chain Best Practices paper. Highly recommend you give that a look. I'm a member of the CNCF Security Tag and the Supply Chain Working Group. I have customers in defense, financing, and critical infrastructure. So I'm here today to talk to you about a problem, right? Right now, software consumers have really no way to really understand the risk level of the software that's running on their systems. And this is because software producers have no real incentive until recently to produce a software bill of materials that gives those consumers of that software understanding of the risk level of that software. And this is a problem because zero trust architecture really requires a strong identity system. And this identity system must really be based upon the immutable attributes of the workloads and the users running on the system. You know, in the recent executive order that I said, the trust we place in our digital infrastructure should be proportional to how trustworthy and transparent infrastructure is. So this is why they are requiring SBOM and software producers must produce the SBOM along with their software artifacts. This gives the transparency software consumers need to understand the risk level of the software on their systems. So the solution, right, is that executive order, right? We have to have some sort of regulation because we are federal, our IT systems and our federal government and our defense systems are at risk if we really don't understand what's in them, right? So the NTIA in this recently defined what critical software is. And the list is very extensive. Anything that deals with authentication access to network resources or network controllers, right, discuss this. I encourage you to go look at the link that I have right here and actually look at all the different requirements or all the different definitions of critical software. And additionally, very recently, the NTIA released what the minimum SBOM elements are, right? And it's important to look at that to make sure that your software is in compliance and your generation of your SBOM. So again, on the executive order on section 4, the timeline is very, very, very fast, right? You know, especially for how fast the government generally moves. By May 13, 2022, there is going to be a report due for pilot programs implementing everything that was in that executive order. So that's within one year of that order coming out. So you go back to the CNCF paper that we contributed to at Box Boat, right? You know, we really pushed for the idea of this out of band verification, understanding that each piece of the software supply chain is actually trustworthy. And once we put all those pieces together, right, we're able to test that the artifact that came out is also trustworthy, right? But we need to explicitly define those trust relationships. And I'm going to show you a little bit about what that looks like in the demo that we have going. But really to understand this demo and why it's so important, I want to step back and explain, you know, zero trust architecture and how to decompose that in a way that really makes sense, right? Zero trust architecture is not a product that you can buy. It's a design decision that you do, that you place upon your networks. At the core of zero trust architecture, I feel is identity. Now, identity is the immutable attributes of either your workloads or users. And those should be cryptographically, they should be identified in either cryptographic tokens such as JWTs or X509 certificates. So everything in your system can be cryptographically verified. The next is policy. Generally, when we talk about policy, we talk about change control boards. And hey, we want to know if a change going into a system meets organizational policy. So we have a group of people that go together review it and see if that works out. But with zero trust architecture, we change how that works. And we actually encode those policies and pieces of code or documents that we can use to make decisions in real time. And last is control. And control will take these identity documents from either workloads or users along with those organizational policy documents and put those together to make a decision when a user or resource on the network requests access to compute or network resources. And when you have all these working together, you really do have zero trust architecture, you don't need Kubernetes, you don't need Docker, you don't need any of that. But what you do need is a foundation of identity policy and control. So what happens when we apply these these concepts to a build system? Well, traditionally, right, we have our build steps, and there are chain of steps, and each one relies on the other, right? And if those break down, right, or if there's a compromise somewhere in there, we may never know. And the key to understanding if there is a compromise within that build system is really looking at the metadata inputs and the outputs of each one of those steps, analyzing them to see if they're trustworthy. And if they're actually correct. So without that actual metadata verification, what we're doing is actually placing trust in its producer of the software, rather than the actual software artifact. And that's not good. We saw with some recent attacks from some big companies, right, you know, everyone's really vulnerable to this. If they place all their trust in their software producers, because everyone's going to get hacked. But we need a way to verify that, hey, these build systems are secure. And the artifacts that came out of it, pass this test and the rigor that we expect it to. So MITRE has actually been working on this a little bit in this concept of evidence based trust. This is a graphic from a paper that was released earlier this year, called Deliver Uncompromised. Now, if you go look it up, there's two of these papers, right? There's an older one or the newer one. I recommend that you look at the newer one. Because it really builds on this SBOM and this evidence using the evidence actually make decision execution time, right? So the concept here is that we bubble up all that metadata from the different stages of your CI pipeline, you sign that metadata. And then you can use that metadata downstream to actually make decisions when that resource requests access to network resources in the form of certificates with an identity or it requests access to compute resources, right? Add admission control. So let's decompose that, right? Each of those CI steps has different inputs and outputs. And when we look at these inputs and outputs and we can build policy against them, we can have a lot more trust in the system. So different things that go into a CI step, right? You get your runtime environment, it might be what's your Docker image that you're using it. Then you have your source package, right? You have your internal code that you're compiling. And then you also have packages that you may be bringing in externally. You have build tooling, you have compilers, linters, packages, right? You have all these different things. And you also have machine identity, right? Where is that actually being built on? Is that being built on a server in a basement somewhere in Russia or is it being built on your hardened infrastructure in your AWS security group, right? Those are things that we really want to know about. And then, you know, the outputs of that CI step, right, are of course, you know, your artifacts, you're going to have your S-bomb, you'll have different test results, you have some logs, and then maybe you have some traces or sys calls so you can really understand, you know, what happened at the kernel level during that CI step. So when you can take that CI step metadata, you push it to an external store, you make that available, right, to a metadata verifier or some sort of a mission control, that allows you to have a really strong identity system over your software artifacts, right? And this is really what the NTIA is looking at with the S-bomb, right? With that S-bomb, we push the S-bomb to an external store, and we're able to get insight into exactly what is in that software artifact and make real-time decisions of whether we're going to allow that artifact to have access to network resources and compute resources. So, you know, we've been putting together, we've been doing some work on Intoto to actually make this happen. Intoto is a great open-source piece of software. What it does is it allows you to sign metadata from each of the build stages in your CI process, and they compare those pieces of metadata to a build policy that you create. And so that build policy then verifies that those steps actually took place as you expected them to. You'll see a little more of this when we get to the demo. I'll make a little bit of sense. But the key part of this is that it really decouples verification from the build steps. And almost every single CI system that I've seen, right, that verification actually happens as part of the CI process, right? So, you're almost forced to place that trust into the software producer, right? Now, within Intoto, in this pattern, the software consumer is actually able to do that. The other piece of software that we used was Spire. Spire is a system for distributing certificates across infrastructure. It's got a robust plug-in system, right? So, you know, we actually wrote a plug-in that ties into a hardware road trust using a TPM 2.0. Works absolutely great. I really love the plug-in API. They have a great community. And, you know, it's a standard-based, it's standard-based, which means there's a government body to it. So, it's not just going to change whenever, right? They actually want multiple implementations of the Spiffy, of Spiffy. And Spire is just a reference implementation. So, when we take those two projects and we put them together, right, we have this automatic way that we can deliver certificates to our workloads using Spire. And then we also have Intoto, which takes the inputs and the outputs of a build step. And it allows us to verify that that happened. That allows us to actually build an automated system that satisfies those goals that that MITRE had. So, I'm going to switch over to a demo here and kind of show you how everything works. So, the first thing I want to do is kind of bring you through, you know, what a layout is. So, this is our build policy, right? And this is completely decoupled from our actual build steps, right? So, what we do is we sign our layout that validates that our build step happened as we were expected. And we do this with an offline key. In this demo, we're kind of faking it a little bit. And we have our key just sitting in the repository. As you can see right here, it's his owner key. And I have a little script that actually signs that layout file, takes that unsigned layout and then signs it, right? And then we verify that that signature is actually valid at the verification time. So, then let me go into the GitLab CI and kind of show you exactly how we're wrapping our commands with Intoto. So, right here you see we have the Intoto command. Now, this Intoto is from our fork. This is not the upstream Intoto. We're really working hard to get this upstream. There's a few changes that we need to make. Hopefully, we'll have a PR in there by the time you all see this. So, we have Intoto, then we run and then we're cloning where the name of the step is cloned and we're specifying some products, right? So, the output of this is going to be a directory named go hello world, right? And that's expected. And then we have our Spiffy workload API path. Now, what we did is we exposed, we deployed Spiffy on the cluster that GitLab is running on, and then it mounted this socket into each one of the build containers. And then we actually just have our command right there, right? So, we're actually doing our clone command and then making sure that we're at the right SHA hash, right? And so, the artifacts for this are going to be something called a link metadata file, right? And this will contain all the information about what happened, the products that are the materials that went into this, and the products that came out. So, and then let's go down to the next one, right? So, the next command, right? We're doing our Intoto run again and this is our build step, right? So, our materials are going to be everything that's in that directory. That's everything that came from that clone step. And then our products, right? We're building it, so it's going to be a binary that comes out. And then what's cool here is we're actually creating an spdx, which is that sbomb that we talked about a little bit earlier than talk. So, we have our Spiffy workload path again. And then we have our go build command. And then we're using this bomb tool from the Kubernetes distribution. So, there's a recent PR where Kubernetes is actually creating an spdx sbomb. So, I pulled that tool and added it to that base container and generating that sbomb right there. So, that should work, all right? So, then let's look at the next line here. So, now we're building an image, right? And we're creating a tar file docker image. And then we're doing a scan. We're using Trivy. We're using the latest image there. And then we have our script right here that, hey, downloads Trivy, gets everything ready to go. And that, hey, again, you see here where our products are the Trivy scanning report and our materials are that gohelloworld.tar. So, we're telling them we tell Trivy, hey, go take that helloworld.tar and create a scanning report and that JSON. And now we have all these different artifacts. So, we see we bring those out and those are going to be available at the end. And then the last step, this is a verify step. Now, in this pipeline, in this example, we actually have that verify right before we push it to our registry. Ideally, in a zero trust architecture, right, we want to actually verify it before we give access to network resources or compute resources, right? So, this might be at admission that we'd start running some of this stuff. And, you know, that's some work that we still have to do there. But you can see we have this in total verify. We give it that sign layout file that I was talking about. And then we give it the public key for that layout. We can also give it X509 certificates too. So, you know, really interesting you could sign it with a smart card if you're a DoD user or maybe a UB key. And then you really have strong control over what goes into that system. So, you can see, and then we also connect to the specifically workload API so we can grab that trust bundle, right? So, then if that gives us a good exit status, then we push it to our registry. So, let's go ahead and kind of actually see what this looks like. So, this is a recorded talk. So, kind of cheated a little bit. I know this is going to work. If my internet will get fast. All right. So, let's go look at that one. Don't look at that failed one. We'll talk about that one in just one second. So, right, we have this clone step. And you'll see artifact. We have this link metadata file. All right. So, now we have, you can zoom in here. It might be a little bit tough for you something to watch. So, you can see in this link metadata file, we have the products, right? Hashes of everything in there. And then we have byproducts, right? What was our return value? What happened on the command line? What all that happened? You know, and then it has our command. So, everything here and then you can see we actually have a signature and then we have our certificate here, right? This is some of the work that we did actually with the certificate constraint system to make that happen. All right. Let's close that out. And let's go into the next job. So, you see we have that link metadata. I want to actually open that up. But we also generated an spdx file. So, let's go ahead and download that and actually see what that looks like. There we go. All right. So, look at this. We have a compliant spdx file, right? And then if we actually go and we look into that link metadata file, let's download this guy. So, you have all our material. So, we know what went into that sbomb. And then our products, right? There we go. We have our hash right there for our spdx file. And we have our signature over this whole file. So, we can actually validate that that's correct. And we do that. So, let's go back into our verification step, right? And we can see right here that, right? There we go. In total verify. And we have some mornings, right? I think we can we can lock this down a little more if we want. But, you know, all the files, all the inputs and all the outputs actually pass. And we'll go actually take a look at this layout file one more time to show you actually what we're doing there. So, we're doing actually a verify and we're doing an inspection. We're making sure that our go-hell world is made with products from the build image stage. We're matching the image ID, right? That's a hash of the image and we're making sure that's made with products from the build image. Same with the trivia scanning report from the scan image and then that go-hell world spdx with products from build. So, right here, we're actually doing cryptographic verification that everything that came out was from those materials that we put in that was from our Git repository. We can extend this even a step further, you know, and verify that all those commits were signed by developers if that's within our organizational policy. And then we're checking here, right? So, our cert constraints. This is the work that, you know, we really spent a lot of time at Boxboot was because traditionally in Toto, it requires, you know, public key, private key pair, right? That doesn't really work with enterprise organizations or using short-lived keys. So, we added a certificate constraint system that allows us to specify fields on that certificate to see if that step was valid. So, during this step, we used a container that we gave the identity of in Toto builder to and we're making sure that, hey, yes, that container was the one that we expected to. So, if a malicious actor replaced that container with something that they put in there that might have some of their malicious tooling in it, right? It would never be issued this identity and we would not pass this step. So, you know, and let's go take a look at what that looks like to what happens if we do do something. So, we can see we had a hacker come in here and try to do an MR on our repository. So, let's go look at the changes they did. Look at this, right? So, they did a said in our GitLab CI.yaml, right? What developer looks at their GitLab CI.yaml after they already got it set up? So, a malicious actor could have put some sort of trojan horse in here, but because we're using in Toto that verify actually failed. Go look the logs for that, right? Inspection failed. Air artifact verification failed. Recept failed. Materials main.go disallowed by real disallowed. So, right there. We caught the hacker and now I think that's the end of the demo. So, I'm going to hand it over to Nicole and she's going to go over how this applies to GitLab. Thank you, Cole. So, what does GitLab think about this? We love that Boxbo did this proof of concept, leveraging the open source tools in Toto Inspire to incorporate attestation documentation into a pipeline. It weaves from multiple GitLab stages, manage compliance, verify, package, release, and secure. They're showing off the power of all in one DevSecOps. We at GitLab have also had an increase in customer interest in software bill of materials, signed build documentation, and many more complex verifications within the build process. As Cole mentioned, we believe this mostly has to do with the executive order here in the states. However, there are many regulated industries who want these capabilities. GitLab is watching the Linux foundation, digital bill of materials, and other projects with the desire to integrate them into our pipelines. Now, here are some roadmap items. If his demo peaked your interest, you may want to keep an eye on these. Verify build artifacts, package release evidence, release compliance tooling, and secure composition analysis software bill of materials. And here are some of the relevant GitLab backlog issues. The best way to make sure that the relevant product manager is alerted is to upvote and comment with any additional context on why or how you use the proposed feature in the issue so that they can be sure that what they've designed is really what it is that you want. You, of course, are not limited to just showing your interest on existing items in the GitLab backlog. You can propose new features if those don't cover your needs. If you're a developer, you could actually go ahead, take the proof of concept from the BoxBoat blog, and start incorporating that into different pieces of GitLab and move to get those relevant pieces into GitLab itself. Remember, everyone can contribute. Or if you would prefer and you're a BoxBoat customer, you should chat with your account team and let them know that this is something you would like to invest in maturing. Thank you for coming to listen to us talk about securing the software supply chain with S Balm and attestation. We look forward to your questions. Please enjoy the rest of commit.