 Thanks for coming to our talk. We're going to talk about the security or project with a security supply chain kit. Yeah, hello. My name is Adolfo Garcia. And I am one of the tech leads with Kubernetes security. I work at a company called Changer. And we do supply chain security. I also do other stuff like, besides working with Carlos on security, I also sit on the K-native steering committee. I am one of the contributors to the SPDX S1 standard. And I have been working on the Kubernetes release tooling for a while now. And my name is Carlos. I also work at Chengward. I'm a member of the Kubernetes steering committee and also tech lead on the security side and contribute to other programs inside Kubernetes. Also, I'm a maintainer of SiegStore. And that's it. So before we start, I would like just a small order of business. So contrary to popular belief, we are not the same person. And I mean, none of us mind when Carlos, when I'm flattered by being called Carlos, or he gets insulted by called Puerto Rico, but not the same person, all right. So before we start, I would like to talk about the origins of the projects that we're going to be talking about. So we are Sieg Release. And we are the release engineering team is the outfit responsible for creating the tooling that releases Kubernetes after each cycle. So whenever you see one new batch release or one of the minor releases of Kubernetes getting cut, it's us behind it. And it's our tooling that does it. And the way we do it is we have this big monolith binary that we run to cut the releases. And it takes care of a bunch of stuff. It builds Kubernetes, the binaries, the images. It builds the system packages. Now it pushes the artifacts to the registries to GitHub. All of the GitHub objects like tags get taken care by that binary. And it also creates a release node. And then, well, now it's handling the generation of the supply chain security artifacts that we publish with every release. So a lot of work goes into this tooling. It's like several thousand probably lines of code by now. And for a while, we have been thinking that all of this work cannot go to waste by just applying it to Kubernetes. So the first thing that we started doing was considering how can we help other projects on the Kubernetes organization get the benefit of this? But now we're also thinking, well, this is like general purpose software that other projects can profit from. So the binary I was talking about is called CRL, which stands for Kubernetes Release Toolkit. Well, yeah, I forgot the name. But that basically combines all of our libraries that take care of all of the release steps. Like all of the release functions of that are contained in there. And just worth mentioning that before CRL, we had like another CRL, but it was written in Bash. And then we had like a lot of effort to convert all the Bash scripts to become like a goal binary that we can test as well. And yeah, so, and then when we started like thinking how to make better use of all of this code in not just libraries, the first thing that we did was we spun out the release notes program or maybe, well, in reality, release notes existed before, but it's integrated into CRL. So you can now have the Kubernetes release notes program and use it in your own tooling. And it's already used in several projects like Knative and others. And then when we started thinking about supply chain security and the way we were building things, we started thinking about, okay, so how should we should build an S-Bom? So we built S-Bom generating libraries into our tooling. And first we started generating the S-Bom inside of CRL. And then we spun out a standalone utility called BOM, which now has evolved into like the multi-tool of multi-tool to work with S-Boms. It can, it lets you not only generate them, but visualize them query and a bunch of other functions. We also created, we started generating SAS attestations inside of our of CRL. And then we started thinking, well, this is not the right place to do them. So we're gonna move them out. So we now have another program that does that for us. And we also have a tiny utility called publish release that handles all of the GitHub page in artifact uploading, which all of this is derived from inside of CRL. So, and this is what we chose to release in package together and calling it the SIG release toolkit. So as I was mentioning, we're gonna go and show you a little bit of some of the tools that we have. So the first one is the utility to generate S-Bom's. This one is called BOM. And for those not familiar with an S-Bom is, it's basically a structure list that describes both the components inside of your piece of software, but also the structure so that you can keep transparency, be more transparent in the software that you're releasing to your users. I'm not gonna go deeper into the S-Bom because I suppose most people here came from that and are already kind of familiar. So, BOM has a number of features. Kubernetes generates S-Bom's in SPDX, which is the S-Bom standard from the Linux Foundation. It's an international ISO recognized standard. And the way BOM works is it will go into strengths, but some of the strengths are like, it lets you analyze code bases in Go and capture the structure of container images. That's where its strengths are. It has a fair amount of adoption. It has now over 200 stars on GitHub, which is not a lot, but I'm proud. And so, its strengths are like, analyzing your source code and capturing it. It captures the container image in a really opinionated but detailed way. It will capture all of the Go dependencies. It analyzes, when you point it at an image, it will also extract all of the system dependencies. That means the operating system packages is starting to have a rematch. We support both Alpine and Debian. And of course, one of the things that are lacking in S-Bom tools today is honesty. So, what is BOM not good for? Like creating deploy S-Bom's, which are the S-Bom's that capture a piece of software that is already installed. That means the configuration files and the way it's running in an actual, like once it's deployed in a system. It doesn't support, it doesn't support and it's not supposed to ever support SCA type, the composition of software packages and pieces of software. That means if you have a binary or another already compiled piece of software, BOM will never try to guess what's in that because it's designed to work earlier in the supply chain. And finally, it doesn't currently work with other languages than Go. We have had some requests to add them, but we're partnering with another project that's been off of SPDX, which is called the Open S-Bom Generator. And we're trying to build their reusable code analyzers that we can plug into BOM. That's subject for another talk, maybe. And then the other big tool that we have is our Salsa tester, which is a Salsa, is a standard to express the provenance of software. That means basically a signed piece, a signed attestation that tells you who, how and when they build the software so that you can understand that what you're holding on your hand and about to install your system or run it inside of your cluster actually matches in what's built by and how it was supposed to be running. So Tejolote is, okay, so Tejolote comes from the Mexican Spanish word of the Mexican mortar, which is the handpiece that you use to grant salsa. That's why it's called Tejolote. And I also enjoyed watching people from other countries trying to pronounce it. So what it does is that the way Tejolote works is that it will observe both two things. One, your build system as it runs, and then you tell it where to expect the artifacts. So that build system's gonna output those artifacts and it's gonna record whatever it outputs. And this is, we are about to start replacing the current salsa, the station generation set of Crayol when we release Kubernetes by this externally running system. And the reason why we're doing this is because you should never try to generate the provenance inside of your builder because you should treat it as an untrust piece of software. So this is the way it works. It watches the build system, records the software as it goes in, watches the run using publicly, like externally APIs that you can ask the build system. And then once it publishes the artifacts, it will record them. So the current build systems that it supports are GitHub Actions, Google Cloud Build. And now we have almost finished the support for Prout so that we can record the Kubernetes projects as they build. So you ingest the source code, well, the build system uses the source code, builds, and then you can point it to record the artifacts from an S-BOM, which is to me the ideal part, from an OCI registry. So you'll notice the new images that show up in our registry. Buckets to find for search for archives or binaries. The native artifacts of those build systems, like if you have a GitHub Actions run, it will look for the artifact server, file systems or the stations. I need the most important part, it will generate the station and sign it with six or after it's done. Okay, then the other tool we have is the Publisher Release. It's like a very tight tool that does help you to push all your artifacts that you build to a GitHub release for now. Eventually we might add a GitLab and others as well, but since we work very tightly to GitHub, it makes more sense for now. Contributions are welcome to extend this as well. And this also generates the S-BOM generation and they can import the release notes that we generated from the other tool. And then, as you can see, this is another example for the release notes that it's pushed on the GitHub release. The other tool we have is the release notes itself, like everybody that contributes to Kubernetes and the CNSF, almost all the CNSF projects, they have, when you open a PR, they have like specific section in the summary PR that you put like release notes. You write that thing, like you write your release notes, you can write your breaking changes or action items that the user might need to take action. And together with this piece of section, including with the labels in GitHub, like the kind feature, kind bug, kind API change, these kind of things, the release notes fetch all the information from one reference to the end of reference, let's say for the release one, two, three, to the release one, two, four, and they fetch all the change between those chains and generate the release notes. To this happen, like if you think about your own project, your open source project or your own pet project in GitHub, you can only, you need to add this section in your PR because the release notes gonna fetch all the PRs, including that time range, in that range, and fetch and look for this specific section. And if you don't enforce people to use that when you generate the release notes, you're gonna be empty. That's the thing you need to do before, like making force. In Kubernetes, we have a pro that enforce the release notes to be written or we skip the release notes in some cases, if it doesn't need it, but you need somehow enforce. We are working in a GitHub application that people can use and just add to your repository that gonna enforce that thing. Then you can use the release notes as well. All right, so for a while now, we have, C-release manages several projects, not just Kubernetes, but we have also our repositories that generate artifacts and all the things. So until now, we had had these binaries floating around and whenever I wanted to cut that new release, I would go and just build it and sometimes create an action or a tag and always getting bad and my dad here would get really angry at me, like what are you doing, this is all wrong. So we decided to come up with a better solution. And then we are introducing, like we're creating some GitHub actions that help you to have all these tools. If you use GitHub actions as your CI CD stuff, that's gonna be easier that you can just use that action, install the tools you need and use in your pipelines. That's the things we are working here, like we have this repository for now, call it release actions. And this includes, for now, right now, we include Bounte, Tehola there, and Zeitgeist and these three tools. The other tools, Cosine is implicitly installed because we are like all our binaries is signed by Cosine. And then when you use this action, it also uses Cosine to verify the signatures for that binary. Then we add that as well, like it's implicitly installed. And the one that is missing, we are working to put this as the publisher release and the release notes. We didn't talk about Zeitgeist here because Zeitgeist is more for checking your dependencies and alert you in the dependencies you have, you need to track. That's another tool that is not like tight related to this talk. Okay, that's great for the gods of demos now. I'm gonna show we have... Okay, this is the repository for the release actions. And as you can see, we have the three I talked about and then you can read and start using it right away. We created this repository here for like to show how to use those tools in a really, really simple Go project. And we're gonna use Go release for building the project and push the artifacts for us to get a release. I'm gonna cut a tag here and have the job running while I explain to you all how the workflow works here. Just I'm using it sign to sign my commits. That's why I have this workflow here. And as we can see here, it might kick. Yeah, it kicks the release. Okay, let's take a look on the... Is this size good or better? Okay, this is a simple Go release configuration file that just builds the project we have in different like architectures. And there's a session called sign here that signs with co-sign the binaries. And this is the important part for the tool we have. It also generates the S bonds for us using the bomb utility. And this is the arguments we have. It's gonna generate the output called example.jzone.xpdx. This is the most important part that you need to know from the Go release section. For the release pipeline, it's pretty simple actually. It's really straightforward. We, it listens every time we push a tag that's gonna kick our release. And we check out the code. We set up the Go as this is a very standard step that we need to do. We explicitly install co-sign here because we want to make sure that it's really installed but it's installed in the next one. And then this installed the bomb utility for us. And then we run the Go release here. And that's gonna build the binaries for our release, push and including the build materials. The next session is that the station part, the provenance part that we do like we also check out. Then in this time we install telehote to generate the provenance for us. And then we run here the binary to generate our intoto.jzone file that have the provenance there. And in this case, right now we are using like another action for another person in the GitHub marketplace just to push that to our GitHub release. But as soon we have the published release we can use that one here as well. So one of the things that I want to show here is the way that telehote registers the build system and how you point it to record the runs and the artifacts. So it has, this is the binary telehote and then it does a test. This is like the simple run when you just point it to an already run run. And then so you specify where it will read the artifacts. So the build system is going to put the artifacts there and then we created this URL's with the different providers that the whole of the support. So it's GitHub for the artifacts and also GitHub here for the run. So you point it to the GitHub actions run and to the GitHub actions artifacts where it's gonna, to the release page artifacts where it's gonna pick them up. If you are using Cloud Build you're gonna switch GitHub for Cloud Build or eventually Pro in the future. Okay, I guess the release is out. It's finished. And if you go to the release page we're gonna see all the binaries we generate. I like the go release for us to generate the checksums, all the binaries and the signatures for the binary signing. And we have the in total the provenance attestation here including the S-Bone SPDX. Let's do a quick look if, okay. This is the build materials for that we just generated and it contains all the files. It contains all the dependencies and everything that it's expected for S-Bone. And here is the provenance attestation. It's sign it as well with cosine. And if we decode it's gonna show all the attestation like the information we record from the build system. Okay, this we went through and. Okay, so now we have, if you saw the examples you may have noticed that they were all on my personal GitHub repository and this is because we have been working during this year to ensure that it can actually build our projects and seek release and we can now use it externally, but you don't want to be running it from our organization. So what we're gonna do is, donate it live here with you because. And just one comment while he's doing the donation. Our tools like the bone that they are holding is we are dog fooding itself. Like the binaries that we generate for bone is getting the view of material for itself and get the attestation as well. Here it goes. So this is our work for the last month, so here it goes. So what we're doing now is donating the new repo to letting off our house to live now on the Kubernetes organization. So once we bug enough people maybe it'll get accepted as part of seek release and everybody will be able to join it. So that's the issue of the nation. So if you wanna go and see it, you can take a picture, maybe it was wanted so that people think that we didn't like drive you live. So, but yeah, so this is where we are gonna be starting scaffolding all of our tooling so that you can reuse them easier because we had had some feedback that it was kind of difficult to start using the tools that we put out. So we now have ensured that they work well with GitHub actions. We chose to call it the seek release toolkit because why not? And yeah, it's ready for everybody to use. So this is what we had and we have like five minutes for questions. So if you think this is useful and would like to use it and we're missing something we would like to definitely hear from you. We are like, if you have like when you start implementing your project you have questions you can ping us in this leg or open an issue or smoke signal or whatever. We have to help you. Like Java for releasing the software. Yeah, I mean, once you build them, yeah. The one limitation that we have is our S1 generator only works with Go. So we're working on the other language plugins which were, so the SPDX open S1 generator is a project that was spun out of the SPDX project to make it more neutral. So other tools are like other contributors trying to get it fit so that we can have like a common base of libraries that understand language dependencies and then it will be able to do it. But the other things, yeah, you can, you can. No, no, this is good. So for context, we have been adopting these tools for to secure the supply chain for VTES. Well, I have a question and it's, what happens if I generate S-bombs with another tool? Am I still able to use the HoloT as it is? Yes, yes, it's independent. Sweet. Yeah, well, so the HoloT is the Salsa processor. So it creates Salsa testations. It understands S-bombs. So if you run a build and it outputs an S-bombs, it's, you can use that document to tell the HoloT that the actual list of items that you want to have this. And if that tool creates a valid S-bombs, which is not as common as it sounds, it should be able to parse it, understand the release artifacts and create that solution of a, yeah, well, it doesn't matter if the S-bombs is signed or not. So, oh, okay, yep. Yeah, yeah, absolutely. There's one. Thanks. So did it, is it using the GitHub Actions token somehow to, to using OIDC or something that it is the right kind of builder using Cosine? Yeah, it used the OIDC token from GitHub. Then it generates the token and then go the entire flow with SiegStore that generates the temporary certificate with that token that a test that belongs to that repository and then sign. Okay, and there's different keys for each binary or was it just a- Every time you generate, you're gonna generate a new certificate and stuff and then gonna be like, but everything is pushed to the transparent log then you can validate and check how the certificate is. Yeah, I was thinking when it was different. Yeah, if you don't want to use the OIDC token and you want to use your specific key, you need to provide like the, you need to generate the key with Cosine, generate key and then you can pass that key to, during your build. Okay, cool. And there is, there was protection. One needs to put protection that the tags are signed, of course in GitHub before otherwise. Yeah, yeah. Any other? We're gonna be around like a few minutes here and there, but you can find us around in the conference and in the booth as well. Thank you. Thank you.