 All right, thank you everyone for coming. Welcome to the open source summit. My name is Dan Lawrence, and I'm gonna be talking about zero trust supply chain security. If you've heard of supply chain security this year, then you're not alone. It's a pretty hot topic in the news lately. I'm gonna be covering some of the history of supply chain security, a bunch of the attacks that have happened lately, and then how to combine some buzzwords. Supply chain security has become a buzzword, zero trust is a buzzword. We're gonna put them together and make a double buzzword, and then have some real examples though, showing how it can be used to actually help in the real world. So yeah, thanks again for coming in person to everyone here in person. I might know the pandemic has been long. This is one of my first time speaking to real people, and we're sick of the prerecorded talks and the videos being played at us all the time. So we'll try to make this one a little bit more fun. We'll start with this one. If you've seen my bio photo or anything on the internet, yeah, this is what I looked like before the pandemic, how it started, how's it going? Even this one's a little old now, so my hair kind of collapsed under its own weight, and now it goes down instead of out. This was about the peak of still going up. Yeah, so this has been a long year, not just in pandemics and in hair, but also in supply chain security and supply chain attacks. So here's what we're gonna cover in the first couple of topics, why does this matter? Why are supply chain attacks important now? Why do we need to protect ourselves against them? I'll try to explain the zero trust buzzword a little bit too, and then jump into some zero trust supply chain architecture and some demos showing how we can actually use some of this today using projects like Tecton, Sigstore, Spiffy, Inspire, and in Toto. Cool, so supply chain security, why now? I've got a couple of memes here. I put these together. There are a few different versions of this, but this kind of sums it up. We've spent kind of decades in software building things and kind of neglecting the build environments and the build systems and the people that operate them. We've spent all this time securing our production infrastructure, but then build things on Jenkins boxes sitting under somebody's desk that nobody is really looking after. I'm guilty of this. I've done this. This is how I got into this space. Everybody is kind of today, and this is what led to a whole bunch of high profile attacks this year, things like the SolarWinds attack, the CodeCov attack, the recent Travis CI secrets leak. This happens constantly. This is the big recurring theme in software supply chains. We spend all this time and infrastructure and dollars buying these expensive solutions to protect our runtimes and then deploy to them kind of using amateur tooling, not to use that word. Apparently it's just we all know we're doing it and we haven't fixed it yet. So yeah, the world's software supply chain security and a bunch of unpatched Jenkins servers and closets holding all of that up. There's another version of this one too. I thought of yesterday, so I added it here. This one. Yeah, this is the ship getting stuck in the canal, the other kind of supply chain that's had huge issues in 2021. Software supply chains aren't the only ones under attack this year, but this one summarizes it pretty well too. The big theme here and the big advice I give to people in talking about how to secure their supply chains is simple. Treat your build system like a production environment because it is one. It sounds kind of silly, but it's like those movies where they build this huge jail cell or a bank vault or something like that that's super secure and then somebody sneaks in in the food cart or sneaks out in the food cart or delivery van or something like that. We've gotten so good at protecting our infrastructure that people of attackers have looked for the next easiest way in, which happens to be through the supply chain. And just like physical supply chain attacks happen using vendors or the doorways that we've left open, software supply chain attacks work the same way. Instead of attacking one company, if they've done a good job protecting themselves, you can find a vendor that they use or an open source dependency or a library or another system and attack that and then pivot to all of their customers. So here's a little timeline of a bunch of attacks that have just happened in 2021. There's an unfortunate cutoff here, which was the one that kind of started this big rising coverage and the attention and even an executive order this year was December 2020, which was the SolarWinds attack. That was cut off here a little bit, but there are a bit of a bunch of scary ones affecting pretty much anybody working on open source software across the internet. And I've left a lot out here actually because there have been too many to really fit in and there's a couple of graphs I'll show a little bit later. January started off with a really, really scary one that thankfully was caused by a researcher and not an attacker. This is one of the ones where the what ifs still worry me. VS Code, which is an open source text editor from Microsoft that was developed on GitHub. There's a misconfiguration and some GitHub actions on release branches where an attacker was able, a researcher was able to get push permissions to the release branches because the branch protection was set up incorrectly. Thankfully this person got a huge reward and everything and nothing bad happened, but he showed that he had permission to push releases on VS Code just because of some misconfigurations on GitHub. The really scary thing here is that this supply chain attack could have pivoted to other supply chains. This is an IDE, this is a development environment that's used by developers all over the world. So if that got hacked, if that got tampered with, now we couldn't trust any code that came out of that. So this is one of the reasons this one scares me so much. There have been a bunch of other ones. Dependency confusion was one that took off because this was kind of a novel attack. Nobody had done this before and it was another researcher again, thankfully, by confusing people about the name and the name space and the location packages would be downloaded from. The researcher was able to upload packages with internal names to public repositories and the package managers try so hard to download something for you that they'll fall back or fall forward and retrieve things from public managers first. And so the attacker got a whole bunch of, the researcher got a whole bunch of prize money for this one too, which is great. PHP was another near-miss where this was an actual attacker but the PHP development team caught it pretty quickly. But somebody pushed Forge commits directly to the upstream PHP Git repository because it wasn't secured correctly. They quickly corrected that, they moved it to GitHub instead of hosting it themselves and noticed it fast before a release got cut. But this is another one with huge downrange implications because this is a language interpreter, a language runtime that's then pulled into websites all over the world. I'll go through all of them and this is just a summary anyway. Jump ahead to some of the more recent ones. Just maybe a month ago, anybody doing work on GitHub probably had to deal with this one, did anybody have to deal with the Travis CI secrets rotation issue? Anybody have to deal with that one? Couple people here. If you didn't raise your hand, you should probably check to see if you're using Travis. You might not have seen it. But there's a misconfiguration in the Travis CI infrastructure where all secrets were made available to pull requests and pull requests are a form of remote code execution if you have tests. Anybody can send you on and your tests get to run. And if you have secrets in those tests, the Travis CI issue made it so anybody could get those secrets. And if you had stuff like package manager upload secrets or signing keys or anything like that in there, that's another way supply chain attacks can pivot downstream. So a bunch of these are both open source and closed source. Open source isn't worse at this, it's not better. It's all source code and all source code is developed and all source code is uninvested in from the build perspective like those memes showed before. Been a bunch of other scary charts here. So Sona type, they're here at the event. They just put out a report last week, maybe two weeks ago, their 2021 state of the software supply chain report. There's a 650% increase in supply chain related attacks in 2021. This number is much higher than the one last year. I think the EU put out a report too, predicting a 400% rise next year. I don't know how they're coming up with these numbers, but they're all big and they're all similar and they're all scary. So it's a big problem. One of the best reasons I've heard for that problem and why this is only finally a thing is because we've gotten so good at securing the rest of our infrastructure. So attackers are pivoting to the next easiest thing. It's not that this is easier than it was before, it's just easier relative. People are finally turning on things like two-factor authentication and strong password managers and HTTPS everywhere to do efforts like Let's Encrypt. So the rest of our infrastructure's gotten harder to attack making supply chains the easiest relative target. This is what it culminated in. This was May, I think of this year was the Biden administration issued an executive order to improve the nation's cybersecurity and a huge portion of this was around supply chains, particularly software supply chains. If you've heard the word SBOM or Software Ability Materials lately, it's probably because of this executive order unless you've been working on SBOMs for years since before then. There are a whole bunch of recommendations in here for government agencies to work with industry to figure out how to protect this because it's seen as an existential threat to the software industry and to our national security which is a good thing because it really is one. And so this is where we get into all of the solutions that we're working on today, things like SBOMs for Software Ability Materials which help us understand what's in the software we're using and then security frameworks to help describe that. And this is where a bunch of the other buzzwords like zero trust come in. So we'll start with the first buzzword here on zero trust. I don't know if there's a great definition that I like and it's become one of these things that if you walk around the RSA floor or any of the other big security conferences or hundreds of vendors trying to sell this for you. So I have to think about it the other way in what is not zero trust and work backwards to what zero trust might actually be. So the big theme here is perimeter based defenses. That is not zero trust. If you're relying on a fence like this or a firewall or some allow list of repositories that you want to put your packages in from and just assume that everything inside of that boundary is safe and everything outside of that boundary is dangerous. That is not zero trust. That is the opposite. So things like secured networks if you're using bastion servers to get into a production environment. VPNs, this is all in the network security angle. So I'll explain how this translates to supply chain security in a few minutes. But these are all the things that represent not zero trust. And it's obvious, right? As soon as you find one hole in one of these barriers and get in, now you have full privileges and you can pivot around and attack everything behind that infrastructure. So this is why these older techniques don't work. So what is zero trust? This is when you start making things a lot finer grained and based trust on individual entities. So instead of saying everything inside of this room is safe. Everything outside is dangerous. We actually know and base all of our trust decisions on who everyone is. I know some people in this room. I don't know some people in this room. So instead of just setting up a fence or a perimeter or actually basing it on every single person and they carry around the credentials with them everywhere they go. So this has been particularly great during the pandemic when all the work from home started and everybody stopped going into an office every day. So if you had a setup before where your office had permissions that you couldn't do from home, you likely switched to something like this where you got permissions everywhere you are. So if you have the same permissions inside and outside the firewall, that's what we're going for here. And this is a zero trust network architecture. So how do we translate that to supply chains? I talked about it a little bit before, but when we're talking about people inside and outside of a firewall, we're pivoting this over to artifacts, right? And supply chains, we're dealing with software code, build systems or artifacts that get built. These are binaries, these are container images, these are Python or Java packages. And so if we want to get rid of those firewalls and get rid of those allow lists and trusted registries, then we have to actually be able to trace back artifacts back to where they came from. Artifacts don't have names, they're not people, they can't tell you who they are. We want to base these policies on where the artifacts came from, how they were built, who reviewed the source code, those kind of things. So if we want to combine these two buzzwords and come up with what a zero trust supply chain is, it's one where every artifact can be verifiably traced back to the source code and the hardware it was built on. And these are roots of trust, right? People have roots of trust today, we have two factor auth, we have identification systems, of all these things to vouch for a person's identity. Hardware has the same thing, we have TPNs, we have trusted hardware, we can prove that hardware is safe and hasn't been tampered with when it boots up. But we don't really have that for source code, we don't really have that for artifacts today. So if we can kind of connect that bridge of artifacts all the way back to those trust roots for people and the machines that they came from, then we can start to apply these finer grained policies to our production environments. And right now it's a mess, right? This is just a little example I have and we'll kind of play around with something that's in the demo at the end. But this is not to pick on this example, this is just the first one when you go to the CNCF Artifact Hub. So if you go to artifacthub.io, this Prometheus chart is one of the top ones that shows up. If you install this chart, you do a helm install into a cluster and then look at all of the images that get pulled and start running in your cluster. These are the artifacts that get deployed. So these are container images that end up running just from a simple installation here. This is not zero trust, right? We're trusting a whole bunch of different things, a whole bunch of different people, a whole bunch of different systems. And every single one of these can be tampered with or attacked or could be malicious. So this is kind of the opposite of zero trust. We're trusting dozens of systems that we don't know anything about. Here alone, there's eight images, something like that. And so we're trusting the authors to not have injected malicious code. We don't even know who some of these authors are. These images also have their own artifacts because supply chains are recursive. So we're trusting the people that authored the dependencies of these artifacts. We're trusting the people that operate the repositories. These are just strings, there's no digest here. So if the repository operators get compromised, now they're part of your trust circle. Anybody operating one of these repositories can change something and attack you. In this here, there are three different ones, at least, trusted in this one Helm chart. So we've got Docker, we've got Quay. I guess it's just two, yeah, because the bottom one is also on Docker Hub. So we're trusting both Docker Hub and Quay in this example, and they might not be part of your trust circle. You might not trust those operators. Then there's the build systems that actually built these things. And I'll show some examples here. If the build system gets attacked, then that can insert malicious code. That's exactly what happened with SolarWinds. So even if you know the authors of these images, you know the people operating the systems that host the images, if they were built wrong, then some malware can get inserted. Then again, recursive all the way down. So we've got the dependency maintainers, the build systems they use, the artifact managers that they're all hosted on. So there's a lot of things that you trust here to run one command implicitly, whether or not you meant to trust them. So these are some kind of top level principles on how we can get to zero trust supply chain security, some don'ts and then some do's if we wanna work backwards like we did in the beginning. The first don't is obviously nothing, which a lot of people are probably doing today, just allowing anything to run in a production environment. And if you have no security around what artifacts can run, you have no idea where they came from. Any one person could have pushed something by themselves. This is kind of the worst of the worst. You're not paying attention at all. Allowsing specific tags is kind of the next step up. If you go through some of the tutorials around best practices with Kubernetes, this is a pretty easy thing to set up. If you push all of your images to a single location, you can at least restrict things to that. That's way better than doing nothing, but it's still not great because then anything that gets pushed there is now inside of your trust circle. It's inside of that barbed wire fence from the first picture and it's not really zero trust. Specific repositories, same thing. And even a whole bunch of the basic signing schemes in all of this are also kind of just papering over the same problem. Anything that got signed now gets to run regardless of where it came from. There are tons of attacks involving tricking somebody into signing the wrong thing or being unable to revoke a certificate. So how do we flip this around to things that we should do? And this is where we start to get to much more complicated policy decisions, much more complicated policy definitions. But in these systems, you have to base the policy on exactly where something came from. That's where the term provenance comes in. If you haven't heard that term, it's the same in physical supply chains and software supply chains. It represents where something came from. And we wanna base our policy on that. And it should be tamper proof, so people can't just trick us by sending us the wrong provenance. So instead of basing policy on where an image is or where an artifact is, we wanna base our policy on where it came from and how it was produced. And we wanna capture this at every step of a build. Builds are recursive, builds have multiple steps. Anybody's looked at a make file, there's probably hundreds of lines in there just for the simplest things. This applies to dependencies as well, so we have to do it recursively. We wanna capture this provenance at every single step. We can't do it once at the very end. This applies to every single step, every single piece here. So the source code, the build process, the publication process, everything. When we sum all this up, we get a system where we can trust what the artifact is and how it came to be, not where it is at any given moment in time. That's the big distinguishing characteristic between zero trust and non-zero trust. This is just another map showing all the different attack points in something. And this is kinda stolen from Salsa or S-L-S-A dot dev. This is a very simple single step, software factory or malware factory, depending on how you look at it. These are all the different steps that happen just to build one package and get that from source code to one consumer without any of the recursive elements. And all these letters here, A, B, C, D, E, F, G, H, are attack points. And if you go to the repo there, you can see a whole bunch of sample attacks that have actually happened. So this isn't theoretical. These are places that actually do get attacked in a single artifact. And so if you're relying on no trust or not caring at all, then you've got a whole bunch of different attack points here. I mean, you're trusting nothing bad to happen at any one of them. Or if you flip this around and you get signed provenance captured at every one of these steps, stores somewhere that we can then apply policy on, we can skip everything. So we can skip everything from A, B, C, D, E, F, G, H, and the consumer can apply trust policies directly all the way back to the developer. I mean, we're taking everything inside out of that trust loop and out of those trust circles. So, yeah, another meme here. This is kind of how we have to reason about this stuff today. I'll play around with a couple of examples here, particularly with that Prometheus chart that I started on before. But these are the type of questions we want to be able to answer so that we can write and enforce policy. If you find a binary on the ground, if you just find something on the internet that you want to curl, pipe to bash for some reason, you probably want to know who built that, who published it. You might find it at a URL, but you don't know who owns that URL. It could just be some random hosting system. So who published that binary? The next step is who, how was it built? Was it built with the version of the Go compiler that was full of CVEs because it was old? Was it built with the toolchain that we trust? Was it built on a system we trust? If you just have a random executable or a container image, you can't really answer any of these questions today. Then all the way back to the more important questions, what source was it built from? If you're using open source, it's great because you can audit the source code, you can review the source code and look at it for malicious code or vulnerabilities. But that's only true if you know the actual version and the actual source code something came from. A lot of the package managers today show you a repository URL and tell you where they think something came from, but in most cases, that's just a random text field that anybody can edit and they can put any repository they want to there. So it's just a hint, which is almost worse than not having it there in some cases. Once you've found the source code, a lot of these are multiple 100,000 line repositories that you're not gonna go look through every single line of. But if you know that it's maintained by trustworthy people, if you know it's maintained by a company you trust or a community you trust, then you can implicitly trust that they're not doing bad things and inserting malware into the code. But again, if we don't know who built it, who reviewed it, who authored the code, then we can't make those decisions. And again, it was anything tampered with. Even if we think we know all of this, we might not know, because an attacker sees each one of these steps as a potential place where they can jump in and do bad things. So this is a little reference architecture that we've put together, showing a bunch of open source projects today can be combined to get something similar to one of these architectures. We can capture these provenance statements inside of a build system. We can operate the build system securely. We can publish things to repositories along the signatures. So even if the repository gets hacked, then we would know. And we can combine all this to get all the way from code to a production environment and know what happened in every step along the way. What we're trying to do here is cryptographically verify every step in a simple supply chain. So from an artifact all the way to a root of trust, whether that's a person touching a two-factor auth token or hardware with the TPM booting up an unknown good state. And since this is open source, we want to build this inside and outside of organizations. So you might be able to set up something like this in your own company where you don't build any source code from external dependencies. You're at everything from scratch yourself, but this is an open source conference and with anybody in the world really is building code without using open source dependencies. So a recent survey that said 99% of companies have open source in their supply chain somewhere and 1% was probably lying or didn't understand the question or something like that. It's virtually impossible to not use open source whether you know it or not. So we want to be able to carry this inside and across organizations. Nobody's living on an island. Nobody's living in a bubble. If you build something that only works in your internal system, then that's not solving the full problem. Yeah, so a whole bunch of different projects we can combine to get these five pieces. So the first one is identity, both for people and systems. We want to be able to address machines. We want to be able to address services, whether they're ours or someone else's. Naming is one of the hard problems that every engineer's had to deal with. And this is really a naming problem, a secure naming problem, which makes it even worse. Thankfully, there's some awesome projects that helped deal with this. One of the first ones here is Spiffy. The Spiffy and the Spire sibling projects help uniquely identify and secure the authenticate systems in a federated manner. So across environments, across organizations, those kinds of things. They can use TPM attestations or other types of hardware attestations to attest to machines being in a known good state before they issue credentials. You can take those credentials, share them with other people, and they can validate that those credentials actually came from a healthy system. Then we combine those with build systems. We want to operate our build system securely. The build system has to be designed with security in mind. For these demos and examples, I'm actually going to show some GitHub actions stuff as well as Tecton, which is an open source project, designed with supply chain security in mind. Finally, we need a place to store and access all this information, all the signed providence statements in a place that can't be tampered with. For this, we're going to show the SIG Store project, which is designed with a whole bunch of cool transparency logs and Merkle trees to make it so that we can insert all of this query in and access it anywhere without having to worry about the SIG Store infrastructure being tampered with. And then finally, the Salsa framework shows how to put all of this together and evaluate how far along you are while building your zero trust supply chain architecture. This is a super complicated graph is diagram showing how it's all going to work. I'm going to zoom in on a bunch of these though and then show some demos so we don't have to understand it altogether. But we have Tecton running inside the Kubernetes cluster, building things. Those build jobs get scheduled on machines. Those machines are VMs in this case. They are monitored when they spin up to get TPM out of stations and prove that they're healthy before credentials get issued. This is a multi tenant system. So we know that we're issuing credentials as finely scoped as we can. If you've got hundreds of builds going on, each build gets its own set of credentials. It's not a global namespace or anything like that. Those are coming from the Spiffy Spire system. All of that gets published into source repositories or artifact repositories and into the sig store transparency logs which is where we're going to pull them out as part of these demos. So to zoom in a little bit more and talk about the individual components, this is the role that Spiffy inspire play. You have agents installed on the virtual machines. The agents do a bunch of health checks when they first come up and prove that the machine is in a good state back to the central server before the server authenticates that node. These nodes run a whole bunch of different workloads. There are VMs in this case in a Kubernetes cluster so they run a whole bunch of different pods. When we know that the Spire agent on each node is healthy and it's running as root, it can then authenticate and attest to the individual processes running on that machine and issue credentials for the specific build jobs. And you can tie these out of stations to a whole bunch of different things. If you're running a physical hardware, you can do TPMs. If you're running on cloud providers, a bunch of cloud providers provide great technology for this out of the box that are a little bit easier to work with. So we can get workload out of stations from each node, proving everything was healthy and nothing was backdoor when everything started up. And then the really useful part, which is what we're gonna use here in these demos, is the extensible system for defining this in a way that can be validated across organizations. So we get these credentials and they're actually issued in meaningful identifiers that I can use in my internal system that I can hand to somebody else and they know what to do with it. They know that it's tied back to my internal organization and then I can look at that and tie it all the way back down to the individual machine. If you're at a different company, the machine name might not make sense to you, but you can tie that back to me and then I can trace it back to the machine if something bad ever did happen. The tecton here has this really cool cat logo, but it is a Kubernetes native CICD system. It was designed with supply chain security in mind, like I said, so it has very predictable, very declarative definitions for build pipelines and build tasks. This means that there's a lot of yaml because we have to actually specify what everything's going to do. There are no surprises, but that's great because when we look at that later, we know exactly what happened and if we look at it before it runs, we know exactly what is going to happen. And then it's got some other cool properties here, like automatic provenance capture and signatures and other fancier stuff like hermetic builds. So you can instruct it to run builds without network access, which prevents a whole other set of attacks. Then the final one here is the Sigstore project. So this issues a bunch of code signing certificates for free, just like Let's Encrypt does for HTTPS in your browser. The goal here is to make it easy to get code signing certificates for free that can be used to sign open source artifacts. This is stored with some pretty cool transparency log technology, so you don't have to trust the operators of the project. You can audit them to make sure that nothing is deleted, nothing is tampered with. So we're trusting the operators to keep this up or not trusting the data, and we're going to use them to not tamper with the data, which is a nice little hack here to make sure that we have a central place. All of this can be found and queried without having to add another trust element to everything. So those are all the pieces broken down. This is all of it again. Now we're going to jump into the actual demos. We're going to take some containers and try to look everything up and prove where it came from. So we're going to first start with the anti-demo. We're going to take this chart and try to trace everything back and see all the places where we have to do guesswork and where bad things could have happened. So we're going to tab out of here. The first one I'm going to pick on is this middle one here, the Jimmy Dyson config map reload. I know Jimmy, I've worked with him before, but this is a cool utility he wrote a long time ago to reload a Kubernetes pod when it config map changes because it doesn't do that by default. So if we grab this one here, we got v0.5. Type that here, we find a GitHub repository. There's some source code. It's pretty simple. It looks like it's Go, sorry, not Python. And this is, here it's got some great scores and everything, but we really have no way of knowing that the container image that's running in our cluster actually came from this GitHub repository. So if we wanted to, we could look at all this code, we could audit. You can trust the contributors, but we don't really know that the Docker image came from here. So we'll go back. We can find the image here on Docker. This is v0.3. I think we had the 0.5 tag. If we go here, we'll come back up. Latest. Yeah, so we've got some digest here. There's even a link in here somewhere to the GitHub repository if we go to overview. Yeah, so this is all getting strapped from the GitHub repository. This is the readme here. But again, the owner of an image on Docker Hub can put anything they want here unless you're using Docker Hub's automated, verified build feature where Docker Hub actually pulls the code and does the build for you. Then this is just a best guess. So this isn't really something safe that we can rely on. If we go back to the GitHub repository, let's pretend that we trusted that. We can recurse down a couple more steps. Let's find the Docker file. So this isn't even from scratch here, right? This is from busybox, which is another Docker. Sorry, yes. Awesome, yeah, thank you. Yeah, so if we read this Docker file here, it's parameterized a little bit, so it's hard to tell exactly what this would build if we built it because these are all arguments. But this base image here from base image comes from right here. So it can be overridden when it's built. If you build this, you could build an Anubuntu or some malicious image. But let's just trust that Jimmy did the right thing and built it from the busybox image. If you wanna follow that one back now, we can go to Docker Hub. And now we've got busybox. There's a whole bunch of tags. We don't know which one this happened to grab at the moment. It was built. This stuff changes constantly. This is part of the actual official Docker Hub images program, but these change constantly. So we don't know which CVEs were in the version of this that was built because we don't know which version of it was built. Any one of these steps could have had an accident happen or an attacker could have tricked them into using something old or they could have just forgotten to update it. That image was already six months old when we looked at it on Docker Hub. And that was just one in the six or seven images here. So basically from the start, we can't actually figure out where that image came from. We can just kind of do our best guess about it. And if we trust it, then we also can't figure out where the dependencies in that image came from because the problem just keeps applying down the chain. So I've got another image though that I built using GitHub Actions and Tecton with signed provenance at each step stored inside of Sigstore. So I'll show what this looks like a little bit. Generically, you've got two images. This could be three images in a chain. This could be 10 images in a chain. But we're starting from a scratch. We have a base image that's got some code in it contributed by one person. I'll full screen again real quick before we jump back out. So there's the base image with some code in it. The commit is signed by one person. That is built with Tecton. This happens to be a distrelous container image which is built from scratch. The commits were signed. This was built in a cluster with all of those features turned on in Kubernetes and it was published. And then I have a sample GitHub repo with another image. It's a Hello World Go app that was built with the tool called Co. We'll look at that code in a minute. It's got some code in it as well that is layered on top of this base image that was signed by another person and that was built in GitHub Actions using some of the OIDC features to sign the commit and sign the final image. So we can actually start and trace all this back from the image itself. I'll start over here. Here is the repo itself. Where all the code came from. It's a very simple, I made this too big. Hello World Go line application. About as simple as it can get. If you run it, it just prints one string. And it was published in the GitHub Container Registry and was built using GitHub Actions. So we can see the run here that actually worked and a whole bunch of failures. But we're gonna start with just the image and we're gonna trace all of that back using the City Store transparency logs. Let's switch to my terminal. See if I can type live. The cool part with recorded talks is you get as many takes as you want when you're typing. Let's see if my bash history saves me. The first thing we're going to do is look for signatures on this image and it's published at, I'll just show the image itself first. I'll do the verify. It is here at ghcr.io slash dlorenc slash helloco which happens to be the GitHub repository as well. I'll just show what it looks like first. It's restored in manifests. So we can see the actual image manifest. If you've never dug inside of what a container image is before it's just JSON and tar balls. So here's the JSON that references a bunch of tar balls. So this is what came out of the GitHub action. And so we normally reference this overall thing by its own SHA-256 digest. So that's what we're going to look up signatures for. Using the cosine tool to look for and validate any signatures. So we're going to print those out too. I built this a couple different times. So we're going to see a couple different entries here. And this is the signed data as well as the verification information for those signatures. So zoomed out a bit looking for the part I'm looking for. Here we go. So this shows the Git SHA that this was actually built from in this repository. And the subject here is using the cool on new GitHub open ID connect flow. But this is a token that was issued to the specific GitHub action run. So this isn't using spiffy or spire but it's similar conceptually where GitHub issues tokens only to this action run. So we can jump over here. You can see the actual action run. That's here. That's the name of it. And this is the commit that was here. So now we have cryptographic proof that this image was built at that commit in this exact action run. And this is all stored in a transparency log. So we verified it. We can see the timestamp when it was entered and all of that cool information. But now let's look at the commit itself too because I signed that commit myself and also put that in the transparency log. So we're going to take this commit number and we're going to do another verify and we're going to do a search in the transparency log and we're going to type in that commit into bash. Good thing I have all of this saved. So we're going to look, this command uses the recor CLI to look up any signatures, any data in the transparency log tied to that shot to 256 entry. And we found one entry here. I could have signed it multiple times. Many people could have signed this because it's a global transparency log, but there's only one in here. So we're going to look at that. Get UID, change it to the right one. Same command, just a lot of stuff. Let's find it. All right, so we'll strip off all of this. We want this one. So we're going to see a bunch of data dumped out here. We can decode this now. Here we go. Yeah, I've got the whole command now. So we're going to do a whole bunch of JQ parsing, magic, pipe this through open SSL, and we're going to see the actual code signing certificate that was used to sign this. And you can see the subject here is me. That is my email address because I signed that commit. So I signed the code, it was built automatically and get hub actions and we found the actual action invocation that was used to build it. So we started backwards from an image and we found quite a bunch of information about it. But we can keep going now. We can look at that image again and find the base image that I mentioned before. So we're going to go one more level down. This is the manifest tool again, inspecting the image. In here is a base image annotation, which is part of the open containers initiative spec. Tools can indicate the base image that they used to build an image from. So we can see that this was built from distrolis slash static colon non-root. And here is the actual digest at that time. So we've got the tag and the specific version of that. And we can apply that same thing again, looking for signatures on that image. Actually, we will first, yeah, let's do that one and see where this one was built. There's a bunch here because the distrolis images happen to employ another cool technique called reproducible builds. This turns out to be pretty hard to do in Docker images so it doesn't happen often. But these ones are reproducible. So if they're built daily, we get hundreds of entries with the same digest unless something actually changes. So there's a lot of entries here so we'll just look at the bottom one. We can see the subject here. This is an actual service account that is tied to the machine. This one happened to be built on and we've published this. So this service account is only used for builds of the distrolis images. We can find a whole bunch of other information in here in the signatures. But the cool little hack I mentioned around reproducible builds means this is built in a couple different systems where we get the same digest each time. So we found one valid signature for the service account but what else is in the transparency log? Let's do a, let me find the digest first. Go back here, cool. So this is the digest. We're gonna look up any entries in the transparency log we can find with that digest. So we're going to do a search. Just grab this one, stick that in. And see we've got a bunch of entries in here because it's been built a whole bunch of different times and we've gotten the same digest. I think we can just grab anyone. Let's get, we don't want the open SSL one, here we go. Yeah, so this is a provenance statement explaining exactly how it was built and by what tool versions and all of that stuff. So we're going to decode that with a bunch of JQ, awesome. So this is the one that was built from Tecton using the workload credentials from the Spiffy-Spire system and we have a payload here showing exactly what happened. This is captured by the build system itself. So this is not something that was generated inside of the build script. The build script was compromised. This wouldn't be affected. This is the build system logging exactly what source was fetched before the build was kicked off and what artifacts came out at the end. So there's a bunch of images built at the same time in here. So there's a lot, but we can see the date it was built at four days ago, according to the build system. We can see the source code repository that was fetched from the exact commit that this came from. We can also see some other cool stuff like the exact tools and versions that were used as part of that build process too. So this was one step here. You can see the exact batch script that was kicked off. This happened to use the goal-lying official container images, one of the build steps. We have more than just the base images. We have the actual versions of all the tools that are used in here. We built a bunch of different things in this same build because we built a bunch of containers. So we can see all the other ones that came out at the same time. They're digests and jump around in that graph by searching for all of these things. If we come over here, we can look at that commit. Back in the GitHub UI, zoom in a bit. Maybe too much. We can grab that commit. Get there. I think that made everything too big. Yeah, so that's the latest one. I'll type that one in so we can see. Yep, that was this commit. Some W-11 builds got updated and this is the automated build that kicked off that run. Now I think I signed that commit too so we can do that same trick that we did before and we take that commit, look for signatures associated with it. No, okay, I didn't sign that one, I guess. But I could have signed that commit and that would have been reflected in here too. So we traced the one image back to the source code it was built from and then we actually traced it back another level from a completely different build system. So the first one was built in GitHub Actions. The second one was built in a Tecton installation running in a Kubernetes cluster maintained by a completely different team and then we actually traced that back to the source code and tool versions used here as well. So all this metadata is available. Anybody can go and query that if you happen to know the right incantations on the command line. So this is just one demo showing you you can hop around multiple times and do some cool stuff. So yeah, thanks. This is the end of the demos. This is the end of the Zero Trust Supply Chain talk. So do we have any questions for the last couple of minutes? Yes. In the definition of Zero Trust Supply Chain you mentioned hardware, how low into hardware can you actually get? Realistically though, how low into hardware can you actually get? Microkernel, go ahead and then check. Yeah, good question. I'll repeat that back for the people on there. Yeah, so the question was we mentioned we can get hardware out of stations. How low can we actually go? Because there's always a turtle below the one that you're thinking about at any given time. So the cool part of the Intodo and SpiffySpire project is that a lot of those mechanisms are pluggable. If you're running in a cloud environment then you just get whatever the VTPM or workload out of station injected automatically is. But if you want to configure your own TPM 2.0 hardware out of station with your own certificates and your own remote out of station system you can do that too. So you get the same style of credentials you can just kind of define and configure your own policy and write your own plugins for as deep as you want to go. I'm sure there are levels to that that I don't quite understand or grok though and some hardware experts can back it up a little bit more. But if you have something like one of the new 502 tokens which you've got something to give away if you want signing your own commits and then you've got that hardware stuff as deep as your cloud provider or whatever infrastructure you happen to be running has as well, then you can get pretty low. Yes. Do you need to make any changes to the tool changes that you used to build? Like you mentioned having trusted tool changes. Yeah, so that question was, do you need to make any changes to the tool changes you used to build? You don't need to. I think the biggest change that, the biggest change is actually in build systems themselves. And if the build system that you're using doesn't have a built in way to capture that provenance out of band from like the build script that it's executing, then you can only get so far. But if you have a build system that can log the commit it fetched and it can log the script that it ran and what artifacts came out, then you don't really need to make any changes. You can use make files, you can use whatever I guess these systems you want to. Yes. So it sounds like you've got whatever you were just got kind of an idea of how to build stuff and then you just sort of racks and build costs itself. How about comparing something that changed at a lower level like it's in pilot, but goes in there and builds out what you think. Yeah, so the question was, it sounds like you do have to make a change to the build system to run a step first to fetch some data or log some stuff that was grabbed or log the tool chain version. How would that change if you just interacted the tool chain itself? I think that's one way you could think about it. I like to think about it more in terms of like which components you trust and which components are dynamic as part of a build and instead, so you can kind of apply it to either level. If you're working on an open source project and somebody can send you a pull request that runs build.sh at the top level of the root repo, you probably wouldn't want to trust the output of that script because the person can change it when the pull request gets kicked off. But if you have your own build.sh that's checked in somewhere else that's audited by 10 people and that can do those things, then you might trust that version of the script. The same would apply to the tool chains. If you're gonna run a compiler that has instrumentation built in as long as that compiler is stored somewhere else and it's built itself securely and signed and does all this cool stuff, then you can trust that. So it's really about what part like one developer on your team or one person on the internet has access to modify as part of the build and you can draw a lot of different circles yourself around how you define those boundaries. But in general, the lower level you can make those integrations the better. Compilers know which files they fetch, they know what outputs come out, it's way easier to do it at that level. As long as you don't just curl, pipe to bash the compiler installation script. Anything else? Yeah, so the question was there's a lot of comparisons I had to do there, there were a lot of hoops I had to go through and look at and inspect. Could you automate a lot of that with having a trust policy? Yeah, so it's a whole bunch of JSON and it's a whole bunch of YAML. Nobody wants to look through every time they wanna do a deployment. I like to kinda keep those concepts orthogonal so we get as much data as we can. It was generated in a trustworthy way and put that places where you can access it when you wanna make those policy decisions. There's a whole bunch of different great policy engines that you can use to configure which things you trust, which repositories you trust, how many people have to sign off on each change. So yeah, you can hook it up with different trusted key management systems or different really rich policy engines that let you get down to even lower levels. Yes? Is the scale of work to hook up this recording if you're on zero trust policy? Yeah, so the question was what's the amount of overhead and to hook all of this up for a new project using something like GitHub Actions. Not terribly high, right, as the build system. The cool part of this is that the improvements have to come in from build systems. And if a build system rolls out improvements, then a lot of it can just apply to everyone using that build system without you having to do a whole bunch of manual work on your own. So it is a lot of work. If you're using the meme from before of a whole bunch of unpatched Jenkins servers, then you're gonna have to patch them to install different plugins, which is its own nightmare. But if you're using something managed like GitHub Actions when they roll out new features, theoretically you should just be able to take advantage of them without even thinking about it. So especially if you're starting with something new, it's really easy. But if you've got something super old that is already a whole bunch of versions behind, which a lot of us probably do, then it's gonna be more work. All right, so we are out of time now. Thanks for all the questions.