 Hello everyone, welcome to our talk securing the supply chain zero trust builds with nteto-inspire Let me start by introducing myself myself there. I am right there I'm a solutions architect at box boat technology I also lead a lot of our technical initiatives are at box boat dealing with the federal and high compliance Mysteries, I'm a contributor to the CNCF SIG security supply chain working group right now We are working on a supply chain security white paper Hopefully by the time you see this video, it'll be released or soon to be released. So so be on the lookout for that I'll be posting it on my LinkedIn and Twitter Once that becomes available. I work in defense banking and utilities You know generally those high compliance environments or high compliance verticals All right, they they all want to move to dev sec ops, but they have all this bag is in regulation involved with it It makes it very very difficult To do that. So so we help them out there and then with me. I have Mikhail Swift. He's an amazing engineer He's a dev ops engineer with us at box boat as well. He contributes with me to that white paper I talked about And then he works on the US Air Force Platform one so he helps bring some of their mission partners on board So if you think like these large traditional slow-moving defense integrators, right? They need them as the US Air Force moves into a dev sec ops environment. They need to bring these large So Mikhail's working on a team that is really helping enable one of those large integrators to work in that US Air Force platform Really proud of the work And then I'll talk about a little bit about box boat. We are a professional services Company I like to call us a cloud native integrators So we can take you know your business requirements, and then we take the cloud native landscape and come up with a plan to really accelerate your Devops journey But let's get to the problem at hand, right? There was some stuff that happened This past winter and it really kind of shook the entire industry Security administrators are finally trying to figure out that they have no way To assess the risk level of software running on their systems, right? It's a system based upon trust And that's a problem, right? Because a lot of organizations don't even sign their artifacts or any other bet it out It would look at the open source community, right? It's even a larger problem And it all goes back to right how do we sign those artifacts, right? It becomes very difficult, especially when you're a large distributed organization to Distributing those prep that private key material and send If you've ever worked in a large corporation and have tried to do some sort of Request for certificates you understand that the process is different everywhere, and it's generally very streamlined, right? It's something that that takes, you know, multiple levels of approval It can take multiple days. So it just is a really difficult thing to do And all this really boils down to is that sometimes when we deliver software faster, we're really delivering vulnerabilities And in these high compliance environments, the fence banking utilities, right? That's just not acceptable We'd rather be slow in decades behind than have a system that's vulnerable Now we all know that, you know existing systems that haven't been passed are vulnerable But we can't introduce new to vulnerabilities into a system And we just can't move fast enough to have the assurance that we know that that's So When I open this talk, I talked about zero trust That's a big market, and I kind of want to break that down a little and There's this idea, right that That zero trust architecture is a product that you can buy but that that's just not right It's an architecture. It's a design decision that that your company or your organization needs to make And I like to distill a zero trust architecture three So you have identity that is your workload's identity based upon the attributes of that the immutable attributes of that work such as a container hash the Process binary hash, right? There has to be something about that workload that I uniquely identify as it or identify that as a part of a group The same goes for users, right? Identity is not just workload identity, right? Most of us understand identity as a user So we need to have strong identity systems for both users and work The next part is policy, right? So traditionally when we talk about policy, we're talking about organizational policy, right? What can users do? What can administrators do? What are our left and right learns? Generally in a traditional architecture, you'll have your policy that's managed by a change control board So you'll bring a change to that change control board and they'll examine the policy to know Hey, is this going to go can this go into production based upon? The very slow process and it really doesn't work with a DevSec ops methodology So what we do in a zero trust architecture, so we take that policy and we We enable it as policy as code, right? So we have our left and right limits defined as some piece of automation You know open policy agent is a great tool to do this and then finally we have control, right? What is actually taking these identity documents and policy documents to make decisions in the system? So an example here may be that you know it if you have a network policy that says two workloads can talk to each other Right? You need some sort of control of that that actuates those IP tables to make that so When you have these three things working together, you do have a zero trust architecture You take these three ideas and you can apply them to just about any problem in your space Design it securely. So when we apply it to the CI CD pipeline, we see that there's a traditional CI CD pipeline all of the data That is collected or generated during the build process and stoned away and not distributed along with the Artific that that metadata is extremely important to attesting the validity of Of that software that it was built on the machines. You expected it to be built on It was built by the compilers. You expected it to be built on There was no man in the middle of taps that that modified the code in between these different costs You want to have that assurance that your vendors did the right thing there before you deploy This software into mission Chris critical system and right now We're placing our trust in the problem. It's like, okay, you sign that artifact so we're gonna trust it in our system and You know the events of this past winter or past year 2020 with with some of the supply chain attacks We've seen we just see that that's not the case, right? We cannot trust External processes in our mission critical system. We need to place trust and data not the process. So this Means we need to move to a system of evidence-based trust rather than a signature base And this is this is somewhat of a new idea. I know the NTIA and MITRE have been working on this the past couple of And there's a paper called deliver on compromise that that might have released It really kind of explains this concept and expands upon And you have this idea of an S. This is a piece of metadata that is emitted From each step of the build and we take those s bombs assigned s bombs and you know Put them all together we and use it and deliver them to the execution phase, right? We can actually use a controller at the execution phase to determine whether we trust that software And that allows our system administrators to examine that software to make sure that it does meet that organizational policy We're placing our trust in signed data. We're placing our trust in public key infrastructure Not the process when we move to an evidence-based trust model for The problem is that MITRE actually didn't release an implementation or any implementation guidance So at box, but we kind of took it upon ourselves, you know We've been working in supply chain and supply chain security for you know the entire history of our company It's what we do traditionally we've been focused on delivering software faster But as we moved in these high compliance environments, they have new requirements, right? They want to actually be able to trust that that software. So we started looking at the in-toto Project in total is a great project. It does what we need to do, right? It emits those s bombs that metadata that signed metadata for each stage of the build It allows us to verify that by an out-of-band Policy enforcement they call that a layout file, right? So that when you put these together it allows you to cryptographically verify the build process Wherever you are, right? So we can we can verify that build process at the execution phase not having to rely on Trust of that that business process from that vendor that so how does this actually work so Didn't total effectively decouples the build policy from the build process itself. So in a traditional build process you may have GitLab CI pipeline That that is enforced at some point, right? But if you're using But once you leave that GitLab environment or that CI environment, you have no way to validate that Right? So what in total does is it gives us an out-of-band validation In this in this diagram you see there's a project owner Diana She's going to define what that build policy is she can say hey all my builds need to meet certain criteria And then each of the functionaries here are responsible for fulfilling or satisfying those requirements Each function area Which could also be it could be a person or an actual build stuff Does require a private key to sign that metadata? Which kind of causes some issues when we bring this into enterprise environment? So enter spiffy spire, you know, we're not going to do a real big deep dive into what spiffy spire Is and how it works. There's a lot of really really great talks on that and a great active community but Spiffy is the spire is the implementation of the spiffy API. It's a reference representation is what we used in this project here and spire allows us to Issue identities in the form of x519 certificates to workloads based upon their immutable Right, there's there's a lot of different selectors that we can pick up including, you know What machine that they're being built in one of our engagement? We're actually able to tie issuing these identities down to a the public hash of that TPM key on that machine Right, so we can cryptographally prove that build to happen on a specific machine right and So this allows us to register these functionaries to the spire registration Once we register these these functionaries to the spire registration. I'm API Those functionaries are issued the keys that we specify and then they can sign that metadata and the whole process becomes automated You know, we are able to again Attest to the machine is running on or the cloud environment was running on as well as a specific workload That is running on the When we take those two projects and put them together It allows us to create an automated architecture that that implements that reference design that that we talked about earlier that Might release and that deliver uncompromised paper Um Allows us to tell allows us to attest for the workload identity, right? What who built the software and where it was built? That allows us to pass that information down stream to consumers of the software so they can actually So without further ado, I'm going to pass it over to Bickel and he's gonna actually go over a demonstration of this working Thanks, Cole. I'm gonna give a quick demo of the work that we've done with in toto inspire We have a sample hello world project here As part of this hello world project we define the simple build pipeline The clones build both the binary and a docker image Stands a docker image and then finally uses in toto to verify the image before it publishes it to a docker registry The only difference between this and any other pipeline that you've probably seen is that we wrap every command in an in toto run This allows in toto to capture any materials which are things that we use in our built on our stage and Products which are things that we create in our state We are also connecting to the spiffy spire Docket which allows us to get ease and do some workload Attestation to make sure that we're actually verified to do this and we're a trusted workload and Then finally it just wraps our get clone or whatever else we're going to run here and We can see this reflected in our in toto layout The layout is what's going to tell in toto what each step could do who's allowed to do that step as a Functionary and then what materials and what products are expected for each step Everything here has to match what happens in our pipeline For example, if our command doesn't match in toto will throw a warning We won't outright fail because sometimes commands not matching is expected There are some things you may your variables that may vary that you just won't know ahead of time But everything else such as our materials will fail and cause a hard failure on verification So for example in our in our clone stage, we are saying we don't need any materials We should not have any materials because we're cloning We're going to create everything in our folder saying that we basically clear it created the world when we clone And as part of our build stage, we're gonna say our build command should be go build And then our expected materials must match our products from our clone state So this prevents any files from changing any files from being added Things like that in between stages and making sure we only use files that we created in the clone step and our build step And then the final part of the in toto stage is the inspection and verification So by default in toto will look at every single meta data Vented by each of these steps and it'll make sure that the materials and products match you to find rules You can also define this inspection stage, which may do extra stuff that you define So in our case in our previous build image stage, we tarred up our file or our docker image and Now we're going to say that this docker image are that we have now Let's match the one that was created from the build image stage So if someone were to download a tarred copy of our image They'd be able to verify with the signed meta data that the image they have was the same one that we created as part of our build process The final part that we added is this root CAs By diva are currently today in toto allows you to sign meta data and verify functionaries with the EPG and Public private key pairs that are embedded in this layout With our integration with spiffy we wanted to make sure that we could sign meta data with any key and certificate pair belonging to a trusted root So as part of that we added in the root Certificate as part of the layout that will then be signed by another trusted key one that could be pre-shared between bill agents and Anyone who needs to verify in a separate out of band process From there each step we added to a certain constraint field to This is saying that certain attributes of our certificate must match these things So for example the clone any common name is okay We want this specific uri in our certificate and this is a spiffy ID that will be in any key made by our Biffy agent and provided to our builders To show how in toto can catch unexpected things from happening in your build process You can take a look at this murder quest. I've opened In here. I'm just adding a line after our main function This is printing. This shouldn't be here as our opening statement in our main function Doing this outside of in toto run so in toto should have no idea this happened But take a look at our build probably buying for this murder quest. I can see it's failing image verification if we take a look at so why it's in failing the inspection step for Artifact verification failed for step build materials main.go were disallowed by the rule disallow star As part of our rules for verification for our build stage We said that everything uses materials had to match the products from our clone Since that didn't it got caught by the disallow star stage You can distribute your in toto metadata from these build stages So that anyone who gets your software and the metadata can verify that you are the one who built it and that they can trust it When you start to combine in toto with things such as reproducible builds and distributed builds It really gives you a strong idea of maybe when a build environment has just diverged and is producing different builds Or maybe something malicious is going on in that build environment and you need to dig in deeper If you want to play with our code, we've published it here on github on the box vote in toto go laying repo Thank you, and I'll hand it back to Cole As you can see we built a system with both in toto inspire Decouple will be sticking around for any question