 All right, sorry for the wait everybody. So welcome to my talk and I guess welcome to supply chain security conference. My name is Adolfo Garcia. I am a software engineer with Jengard, a company trying to solve the supply chain security problem. And I am based out of Mexico City. I am also one of the tech leads with Kubernetes Sieg Release. And I work where I work on the release engineering team leading mostly the new improvements we've been doing to the Kubernetes release process and adding more supply chain security features. I also like to write back around the world and I'm very happy to see everybody here today. I usually don't do these kinds of slides where you present the program to everybody, but today I have like a really packed session because I wanted to give you a lot of information because I want to really understand what we're trying to solve, how we're trying to do it. And I hope that the tools that we're building make sense to you and you decide that they may be useful for you. If I have time, I have a short and a longer demo and I'll try to also cram in some questions, but it's gonna be tight. So let's try to rush through everything. So first I wanna talk a little bit about the challenges of releasing Kubernetes. For anybody that's not aware, Kubernetes is the largest go program in the open source world. It has a vibrant code-based and community-rounded around a thousand people actively pushing comments to the project. We, in secret release, are the guardians of the releases of Kubernetes. So we build all the guidance and the tooling that makes the releases possible every month. And we also manage not Kubernetes itself, but a wide range of other projects that live under the Kubernetes organization. So we have two kinds of releases. One monthly release that involves the batch releases of those branches that are under maintenance, which typically involve high critical security patches and bug fixes. And we also have the big release that happens every three months, four months now, because we switch from four to three, which is the very big release that we do every four months. This release is powered by a team of around 30, 35 people that sign up every cycle to be part of the release team. And they handle everything from bug triage to communications, writing the release notes, making sure the enhancements are well tracked of what goes and doesn't go into the release. And if you want to get involved with the project, it's one of the best areas that the project offers. Anybody can join, no matter their technical level. So as I said, we also manage the tooling that releases the artifacts for over 100 projects that go under the Kubernetes umbrella. Everything from storage, container runtimes, and lots of other pieces that go into the Kubernetes deployment. We release, we support three different operating systems in different architectures under those. We put out over 25 images now, and over 50 binaries ranging from everything from Kube CTL, Kube ADM, the single binaries for all of the Kubernetes components. We produce star bar bundles for the hyperscalers to build the distributions from. We produce also system packages, RPMs and devs, and package the release notes, everything through an automated project process that we've custom written for the project. And important also, whenever we got a new release, parts of the Kubernetes code get spun off into repositories, which are some of the libraries that are used in a lot of cloud native projects, like for example, Client Go. Whenever we tag the new release, those projects get tagged as well. So I hope that gives you an overview of how much stuff goes into a Kubernetes release. And this is the artifacts that we put out are consumed downstream about thousands of users. So we take our responsibility very serious. So we started adding some security features to the Kubernetes release process about, I started working on it about three years ago, adding some of the common things that you now find in every supply chain security problem. So I want to talk about a little bit about why supply chain is a big concern. So when you think about other areas of Infosec, you'll see that those areas have come up with solutions that have been tailored to the particular challenges. So for example, if you think about network security, we have come up with solutions such as making our networks a tiny impregnable fortress where nothing goes in and out without supervision. And we wrap traffic in cryptography so that people cannot snoop on what's going, what's traveling through our networks or forge the origin of that traffic. And supply chain security has its own set of particular challenges, which are, well, we'll see what those are. And I want you to hold on with me because I'm gonna try to rush through all of this. So let's start with this. So this is you and your code. And typically the way we work today is that someone, a developer, will push changes to a code base. And those changes need to go out to the world. They need to be released. The new features need to be used by users. So if you're doing anything meaningful, you're most likely pulling dependencies that can come from your language or form a third party that is some public library that you're pulling into your code. Most people will also host their code in a version control system, most likely Git. And in order to build or release your project, you'll most probably have a builder. Builders can take the different shapes and forms from a simple bar script or make file all the way to a sophisticated release process like Kubernetes has. And if you think about it, builders are also code that gets executed arbitrarily whenever you run and whenever you release or build the software. So that introduces a new set of dependencies into your project, which have to be tracked as well. Then typically your builder will be running inside of a build system, something like GitHub Actions or GitLab. And unless you're, of course, you're laying your project away and releasing from your laptop, which you probably shouldn't do. And whenever you run the builder, it'll put out some artifacts. They can be binaries, they can be system packages like RPMs or devs, they can be container images and lots of other things. And finally, adding to the complexity of this, that there's a human element. So most people don't work, especially if they had a setup like this. They don't work by themselves, they are part of a team. Oh yeah, container images, they also pull another set of dependencies, the operating system dependencies that make the container run. And as I was saying, the other part is the team. People are not alone. They typically are part of a team with more people. So of course, this is a super simplified view of the software release cycle or development loop, but it will serve as well to explore some of the challenges in where your project may be open for attack or become vulnerable simply for the things you're pulling into your code base. So the first one is you can, for example, pull in a dependency which maybe have a vulnerability in it. And this vulnerability may come from an attacker injecting malicious code into the dependency or a well-meaning bug that some programmer introduced on inadvertently upstream and you pull it in. This is like your log4j situation, for example. Then the next one, an attacker could go and breach your version control system. If you remember there was a hack, I think in the PHP project where an attacker was able to breach into the Git server of the project and start pushing malicious commits into the code. Luckily, it was caught really quickly and the Git server secured. The next one, you can attack the build system. If you've seen the wonderful demos of that people like Doffy Cooley, Ori and Coldwater, Roy McCoon, all of those folks do about attacking Kubernetes clusters, those techniques can also be used to own your text on deployment or your Git lab runner. And once you have control of the node, you can pretty much make it do whatever you want. You can attack your builder, especially sometimes builders are left in another repository forgotten and not very well taken care of. If you take control of the builder, you can also make it produce anything that you want. For example, binaries with malware in them. You can attack the system package builder. So for example, imagine an RPM build command modified to inject malware into RPMs or same thing with the container image builder. So you can potentially envision a Docker daemon modified to inject layers of fun stuff into every container it turns out. And finally, if everything else fails, you can always attack the artifact distribution network. I think there was a case with Sourceforge where one of the mirrors was compromised and that mirror, I think it was the Korean mirror, started distributing versions of PHP, MyAdmin with malware in them. And finally, there's a rogue element in your team. Everybody thinks that their team is super secure unless until you something that happens without this rental employee or someone that wishes to make a statement with your project. And that person happened to have push access to your code and decides to inject something unwanted into the code base. And when you consider all of this, it's kind of scary. And then if you're a software developer, you have to being aware of all of this, turn back to your client and say, hey, it's okay, you can run this new version in your nuclear reactor or whatever, it's perfectly fine, go. And it's a scary thought, but that's not the worst part of it all because if you consider this system, this thing that we just saw, is simply just a component of our giant network of projects, each feeding into their artifacts and libraries into the next one, into the next one. And if you think about it, it's a giant mesh that simply defies human comprehension. And the scary thought is if you think that the compromises that we just saw could happen in any of these projects, then that vulnerability can flow to any of them. So it's a scary thought to just be aware of what's happening. That's why we take the same supply chain really, really seriously and we're trying to put to work the latest techniques that we've been finding. And even when you see this, people are still going and saying, okay, sure, run this version in your financial institution. It's perfectly fine and people will just go and do it. So we are in a really delicate case right now in a really difficult situation because we still have the supply chain pretty much insecure and we're trying to come up with fixes and making sure that everybody's doing their part to secure it. So as I said, it is scary, but there's hope in the way. And so as I was saying in the beginning, we are trying to come up with techniques on the supply chain security space to really alleviate this a little bit. So what's the first one? Well, the first and most important thing that we're trying to do right now is to gain visibility into what's happening in the supply chain. And so what we do is that we take these built environments that we have come up with. And we start if you think about an artifact and you want to make sure that it's safe to run, you need to make sure that you can answer some questions about it and some questions like for example, if I take an artifact, what's inside my artifact? What is it made of? What are the components? What does it look like inside? Do those components have to be compromised? Did it come from the repository it's actually supposed to be coming from? Who committed the latest changes to the artifact I'm using? What was the version of the builder? Because if it's an unexpected version, it could be a sign of trouble. What's the build system that produced it properly secured? Who was, how was the builder invoked? Did I pass any parameters to it? Or better yet, tell me, you didn't pass any parameters to the builder. And are these the expected artifacts? So do they match that part of the build process where they were produced or were they modified in any way? And answering, if you want to do a good determination, whether it's safe or not to run one of these artifacts, you can answer at least some of these questions. The more you can answer, the better. But the problem is they start piling up. And all of a sudden, you end up with lots of questions that you need to answer. And if you think about rapidly shifting workload as we have, for example, in containerized workload in the cloud, they start to become a data problem. Because potentially every new version has a different set of answers. And it's too much. But yeah. But the good thing is that machines are here to help us. And so we have new tools at our disposal and new manifests that are machine readable, which can give us that information. So the first one is the manifest that answers what is inside a piece of software. And it's, of course, the software wheel of materials or the S-bomb that we so lovingly like to call it. A software wheel of materials is a list of components inside of a piece of software, which will give you lots of information about it. What are the components? How does the structure inside of it look like? What's the licensing? How do the components relate to each other in that software project? And even some information that isn't detectable directly in the artifact itself, like what were, for example, some of the tools used to build it. The second question, how was this software put together? And that's why we have proven and set the stations. This help you answer the rest of the questions. So just like a short break here, when I talk about proven and set the stations, I'm talking about internal salsa proven and set the stations because there's, I don't want to sort out the holy words about what an attestation or what provenance is. So when you have an attestation, it'll give you information. An attestation is composed of four key pieces of information, which is what were the materials used to build my piece of software? What was the transformation done to it to build it? Who did the actual build and using what? And what were the artifacts that came out? So if you complete those four pieces, then you can answer a lot of the questions that we just saw about the artifact flowing through the release process. So once we visited those two, I want to give you an overview of how we are trying to build a very salsa compliance in Kubernetes. So salsa, for those of you that are not aware, is a framework that helps you do gradual steps to harden your release processes. It'll give you information and assurances about the artifacts that you're putting out that they have been built in a secure environment. So this is the acronym supply chain levels for software artifacts. I actually wrote it in big letters because I always screwed up. And then let's see a little bit of what we're building here. So the discussions to build these newer software artifacts in the Kubernetes process started around 2020 at the end of 2020. And the first project we were to was building the Kubernetes system. As we saw at the beginning, Kubernetes huge project. And we put out a lot of artifacts which need to provide their users with some visibility into what went into them. So we started building the S1. Kubernetes not only is a big project in terms of the artifacts it puts out, but it also pulls like I think it's around 240 dependencies, which are all listed in the S1. And so if you use the S1, you can know which appendices, the versions, and the licensing information for all of them. And we started adding an S1 to the releases in June 2020, 2021. And then on October during KubeCon in Los Angeles, we announced that we were doing the last missing bit to take Kubernetes to Salsa 1, which was providing SAS attestations of how we were building the software. At that point, we added the SAS attestations inside of the release build, which is something that you should not do. That's why we built some of the projects that you're going to be seeing today. Then in March 2022, we finished setting up the signing infrastructure for the project, which meant we had to set up a bunch of cloud resources to get identity and hardware and other cloud resources to make signing possible, not only and also modifying our tools to support signing. We partnered with the Sixter project, which for those who are not aware of Sixter, it's the best way to sign the software, your software project out there. Highly recommended that you go and check the project if you want to start signing your artifacts. So when we finished modifying our tools to add the Sixter libraries and making them into our processes, we started signing the container images in last March. And but this also meant that we have in place the necessary libraries to start signing other kinds of artifacts. For example, we are about to start signing the binaries, and we are about to start signing the attestations that we produce. And planned for this October for the next KubeCon, we are trying to do some Salsa level three-ish. And by level three-ish, I mean we're going to do the best that we can to get it to Salsa level three. And this is because Salsa level three imposes some requirements on the build environment and the infrastructure that some of them we do not have control of because it's shared with some of one of the parent company supporting Kubernetes. And so we are going to do the best that we can, but we are still deciding. We're also shifting a lot of things inside the project, so we'll see. But we are going to definitely do sign attestations, for example. So as you can see in the Kubernetes case, the best way to try and start your Salsa journey is to do it gradually. So when we started in provided artifacts to comply with level one, it wasn't perfect. It is not perfect to this day. But at least we were able to start giving our users at least some visibility into what's going on into the release process. Yes, it's falsifiable. It's not the most secure document in the world. But if you choose to at least take a look, it gives you that visibility. And not only that to our users, but it also gave us some visibility because at some point we were at a point that we didn't even know what we were releasing. So we now have full documentation of everything. The second one is the anti-temporal artifacts. So when you have attestations for what it's happening in your release process, and you sign them, that means that as the artifacts flow from one stage of the build to the next one, they cannot be modified because you have an attestation which has all the hatches of those artifacts. And since it's signed, you cannot modify it. Salsa 3 provides assurances that the infrastructure has been hardened. And as I was saying, the project is trying to do its best with the access that we have. And we'll see up to what level we can get it. And level four is that mythical level which is very hard to comply with, which we are not even trying because in that case, since we're open source and some parts of the infrastructure are of control, we are not trying. So in order to do this, we built some tools. And the first one, that came out of the S-Bomb effort. So we built a lot of custom tooling to produce the Kubernetes S-Bomb. And the first one is BOM, which is a tool that lets you create a software build of materials in SPDX. It's now being updated to work with SPDX 2.3. And it has a few features that are specialized, mostly for containerized workloads. And those features have been tested really well until, because it's been battle tested inside of the Kubernetes project. It has already over 120 starts in GitHub, which is, I don't know, at least the biggest one, the highest number I've achieved. It has a base of 25 contributors and supports generating your full dependency for Go projects, which is what most of the cloud uses, the cloud native landscape uses. It's been, it's in use in several well-known projects already to generate their S-Bombs. And it has some cool features like visualizing S-Bombs. As you can see here, it'll give you, you feed it an S-Bomb and it'll show you how the information is structured inside. And it also has a tiny query language where you can feed it an S-Bomb and ask it for things like, give me all the Debian dependencies that you can find there. It'll do that for you. And it's also now in incubation in the automated compliance suiting attack of the Linux Foundation. But I've done lots of talks about BOMB. You can look them up on YouTube or ask me questions if you want. But the one I want to show you is this new one, which is called Tejolote. So for those who are not Mexican or not Mexican enough, the Tejolote is the handpiece of the Mexican mortar used to make salsa. So you take the mortar, which is called the molcajete. Tejolote, you grind your tomatoes and chilies and other delicious stuff in it and you get salsa out of it. And that's why you named it Tejolote. Also because I wanted to have fun with people trying to pronounce that. And the features is, of course, it's new, new, new because it's, right now it's under test on the Kubernetes release process, but it's already working. And right now it's in shift from my personal GitHub to the Kubernetes organization. Maybe by the time if you look it up, it will be under the Kubernetes organization already. So it works, it's API based. So it's an attestation. If you can make up the information, it's really not very trustworthy. So it works by querying your build system to extract the information it needs. Supports GitHub Actions. It supports Google Cloud Build and pretty soon it'll support Proud, which is the CSED system that the Kubernetes project uses. It has sleep and resume capabilities where you can store its state in a file and resume after the build is done. It also supports sending invoking messaging itself in pops of messages. It supports, of course, signing the attestations natively with Sixer and has a bunch of artifact collection back and switch I'm gonna show you in a little bit. So the way it works is like this. You have some source code that fits into a build system where it triggers a builder and it outputs some artifacts. I intentionally left the build system black because we're treating it as a black box. So we will only tell you what the build system told us it happened. We don't make any assurances of the quality of the build, but we'll at least record what went on in there. So the first thing that the whole thing will do is it will observe the source code as it flows into the build system. It will record things like, for example, the build point, the Git commit where your project is building because that one has to be set forth, has to be registered in the attestation because if you have an artifact, you wanna make sure that you can go back and know from which build point it was compiled, for example. So once it observes all of the inputs to the build system, it will watch the build run. So you can leave the whole of the running and it will query the API every few seconds to watch as the build runs or you can put it to sleep with the information it gathered and resume it at the end. Once the build is done, it will query the API of the build system and it will collect the artifacts and it will collect information it needs about the run. So which builder was used, from which repository, the version, the identity of whatever run the build process and so on. And finally, it will collect the artifacts. So to collect the artifacts, it uses a number of backends, which I'm gonna go through. The first one are the native artifacts of the build system. So cloud build and actions, both support storing artifacts natively. It can use those if you want. If you don't specify any, they will, those are the ones that will be used. You can tell it a directory to look for artifacts so everything it finds in there will be recorded in the attestation as artifacts. It can read from buckets currently cloud storage and S3 is almost ready. And it can also query OCI registry. So the way it works is you point it to a registry and before it runs, it will index everything inside of the registry and after the run, it will index it again and the deltas are gonna be reported as artifacts of the build. Finally, well, not finally. It also can use an attestation. Why? Because if you have as a multi-stage build, you can have, you should be attesting to every of those stages. And if you're producing attestations of each of them, you can feed the attestation to the Holote to build, to attest the artifacts that came out of that stage. And finally, and this is the one I'm most proud of, if your build produces an S-bomb like it should be doing, you can feed that S-bomb to the Holote and it will record the artifacts that came out of the build using the S-bomb to use as subjects in the provenance attestation. Finally, it will take all of that information, build the attestation and sign it with Sixer. I may have, since we started a little late, I'm not sure if I'm gonna be able to make it to the demo, but let me try. So, luckily, I have a short demo, which is this one. Yeah, so this is the short one. So this demo, what it does, it's just a Go program. And the only thing that it does is it prints out, I'm ready to be attested. It's just a binary. And the way it's set up, it has two workflows in GitHub Actions. The first one is a simple Go builder which will output a name binary called Hello and it will store it with a name binary in the GitHub artifacts. So let me trigger this one and I'll show you the other workflow. Okay, so it's running. And let me show you the other workflow that actually builds the attestation. Oh, sorry, that's the other one. So this is the attestation. We currently do not have an action that you can simply put it in, pull it, but this works, it basically pulls down the code, compiles it and runs it. The invocation is here at the very end. I don't know if it's big enough, probably not. So it's the simplest invocation of the demo. It's called the binary, the holota test and point it to the build, which is past the workflow run here and I'm telling it to sign it. It will use the OIDC identities from GitHub Actions to do the signing. And if everything worked, we should already have, yeah, it's building the attestation. So it builds already the project and it's doing the attestation and it's done. So this actually was triggered after the build was done. We didn't need the multi-stage because it's such a simple project. And right here, it actually took all the information it needed from the GitHub API. This is on the development, so it's had a lot of output here. And basically at the end here, it collects all of the artifacts here. Yeah, this is it. So it gets information from the GitHub API and detects the artifacts in the native artifact store of GitHub Actions. And well, it's signing the attestation here and you can see it, this is the actual signed attestation. It's basically for encoded, so it's difficult to understand but I have it copied here so that you can see some of the parts of it. This is almost the last slide. So this is, it has the subject and you can see here the binary that I specified in the build. It has the predicate with all the information and the repository where the builder was and well, there's a bunch more information below that, which I'm happy to show everyone after this if you want. And yeah, so this is basically it. This was the simplest configuration of the attestation but you can use it to attest to more complicated builds. And I had a, yeah, this one. So if you want to get involved in developing any of this, please come join us in C-Release. We're always happy to see new friends show up on our calls. You don't have to be an expert. You don't have to know anything. Just show up and say hello. We meet every Tuesday and come by, say hello. I can introduce you to the code base. I'm happy to do workovers and I don't have any more time for questions but I'm happy to answer any after we are out of the session. So thank you.