 So hi and welcome everyone. Thanks for attending the session. I know it's Fridays and the conferences are the best the worst Days to have a session in the morning My name is Salo for Garcia, and I am one of the tech leads with Kubernetes to release So just a little bit about me I worked for my whole life for my own company and then they until the pandemic hit and had to Switch jobs, and I am now a senior DevOps engineer at Marmos I have two little girls, and I also like to Travel the world with my bicycle one of our can and so with the beginning that this year I let the effort to produce a nest one for Kubernetes All right, so let's start a little bit to talk a little bit about sick release and what we do so sick releases the special interest group That does the Kubernetes releases whenever you see a new Kubernetes patch release or a new version coming out We are the guys that work to make that happen so we also form the Kubernetes release team that is managed by the Kubernetes Release leads joining us today here. Hi, Ray and we also manage the actual Decision of what goes what goes what gets shared picked to the supported branches of Kubernetes and The the other thing we do that is maybe worth Concerning today is that we are in task of continually improving the Kubernetes release process to keep it up today to practices and best Best practices and standards that can keep coming up so I Wanted to talk a little bit before we go into the as one thing about what is in a Kubernetes release? I wanted to go briefly into this because the Kubernetes release has many different kinds of artifacts many of different ways of distribution and so we think it's a good example of why Having an S bomb is a beneficial thing for a project of this size so the most people are used to How consuming Kubernetes by its container images we produce Five different images for five different architectures and If you cross that together you get 25 images, but there's a lot more to a Kubernetes release than just the images We those images are produced. We support three different operating systems Linux macOS windows we Have we built some of the artifacts and the images for the what we call the supported platforms Which are currently the 386 AMD arm arm 64 the PPC 60 and as the ms3 90x platform we As I said before we have 25 images with each release We produce 53 naked binaries the naked binaries are what you whenever you download for example a cup a copy of cube ctl How cube call ready to run? that is one of the Then that's what we call a Naked binary same thing for QB by the end et cetera We also have a bunch of tarbols Which bundled together some of the components of the whole Kubernetes system? Which are called client node server and tests and for example if you download if you want to some of those the client side of of the tarbols includes QCTL and the node contains the binaries that Drone the cubelet and so on Then we produce a source code tarbol With the the source code up to the point where we chose to the commit that we chose to cut the release The RPMs and their packages and also file with release notes So our mission is to provide guidance and tooling to facilitate the production of automated releases if you take a look at this Statement mission statement. It doesn't specifically say Kubernetes in it That is why we Try to produce some of our tools as and may make them as general purpose as possible If you may you may be familiar with the release notes, which is the probably the most popular Can call these things popular Project and utility that we release it's used by a bunch of other projects to generate their the release notes We also publish as a general Purpose tool the the builders are controlled so cool with cloud builds Then we have a thing called publish release which allows you to upload and sync your releases with github and Finally the bomb tool, which is think that we're talking about today so the S-bomb definition, so I think this is the Previous definition right now. This is they prefer it has been updated since But I wanted to more than take a look at the actual S-bomb definition Tell you a little bit of what I think how I see the build of material So when you see a bill of materials you have it's a document that lists The components in a in a in a software release and it allows you to determine the relationships Among them. So if you think about the work have warehouse like this A bit of materials would would really quickly allow you to determine where things are if you if you want to ship a Complex a multiple item order you could easily find out and pick them up from from the boxes but the way An important thing to note here is that we are sick release. We are not sick as one so what Well, our focus is more on how we can actually describe a more complex Release that spans many distribution channels and can so far to facts So I tend to think about the release a little bit more like this like a market So if you think about a market, you probably can find some of the items that sale there in different Presentation, so if you think for example, you may have a tomato and that tomato come maybe available In its raw form for you to buy by the by the way, or you can maybe could find the Transform to a salsa inside of a sandwich or different presentations and for me a software release is kind of the same thing So if you remember the the can so far the fact that a Kubernetes release produces We have the same components package in different ways. So we have the if you think about Cube CTL It's available in an RPM package as a binary inside of the notable or even in source code if you Want to consider it like that like like a consumable? so There's a thing to consider when you start building your S1 You the first thing that you will do you start thinking about the S1 is So you take your documents and the first thing that you would add to it is your source code and Now project like Kubernetes the S1 lists about 24,000 files of an if I remember it correctly so Just by listing all the files in the source code you get a long long list Then you start adding for example your container images and then you add your binaries then you add more binaries and your packages and then the dependencies the transient dependencies and Then all of a sudden you've got everything in there except for the kitchen sink or maybe you do have the kitchen sink like in the picture and So what I what we chose to do is that to start thinking so we the first says one that we produced was a single Monolithic file so we've been thinking about other ways that we may split the S1 into more consumable documents more linear ways of doing that so I Tend to think about a Way that we could produce linear s bombs for what could potentially consume them. So if you think about The source code maybe that would go into some kind of source code tool like code search things like that that could could actually Take advantage of the large number of items that they're in there. So the same thing with containers got the container images Could be better suited for things that will actually consume the container images So if you think you could have like an admission controller in Kubernetes and would check the the Contain images coming to make sure that everything is in place or that to match them against a certain for an ability so and so on and Same thing for the binaries the binaries are intended for the operating system. So it would make sense to check them in there rpms so on So our If you were at the keynote this morning, you're aware that there are two great efforts going on to Produce s bombs. We went the Linux Foundation way, which is as PDX which is now an international standard Of course, I'm going to repeat a lot of what was said this morning in the in the keynote, but I'll try to give you SPDX looks like this It's That's the header of the SPDX document and a fragment of a file listed in there It has information about the the document an external reference to another one And then the file entry with the sashes the license It's her So how can we build a better Kubernetes s bomb? You have a if you have a monolithic S bomb like we had at the beginning The first thing that you can leverage is the ability of SPDX to Do two things relationships and packages Using those constructs you can start splitting parts of your S bomb Apart from each other and the first the first the first thing to that user package is your source code and we inside of the Kubernetes had been treated like a separate package of that Bundles all of the source code together and then we simply point all of the artifacts as being built from that From that package Then another thing another thing. So this is not what we do in Kubernetes But it's like a some of the ideas that we've been having in the towards the future of how we can handle it The next thing that we can do is split out the container images And so if you think about the container image, it's simply a bunch of star balls, right? And you could think of those If they were packages in themselves and then if you think about it there is an index manifest in front of them and that So what we the way our tool works is that it marks each of the Platform-specific images as being variants of the base package, which is the index manifest That's the way we try to start structure container images when described in them here And then after that you can start making more complex decisions like for example here. We have a Potential S bomb detailing the binaries and the RPMs Spends one into a different document and in there we have the dependencies bundled into another package so the idea is to make this Units more consumable and available and better suited to process by the tools that could potentially Use them in the future. We're still Understanding who is going to use them. Are you going to use them inside of your processes outside to make matches with other systems and so on? So how about our tool? all of the code that we wrote to generate the Kubernetes releases derived into a original SPDx implementation Which is which runs inside of the release the parental release process every time we cut a release but we we brought those libraries and built these This this general proposed utility that you can use to to run and process your your repository so the the easiest way to to add that to create a bill of materials for a project is just CD into your repository file run bomb generate in the path of the directory and it'll it'll speed out your bill of materials describing that of course a bill of materials to be complete and useful you have to add more things to it so you can pass and other kinds of artifacts To to be to get included into the into the below front here So you can add directories you can add images that are stored in the registries and You can have Tarble the the tarble flag down there is a container image, but exported as a Docker archive in your file system and as well single files you can add all of those kind of artifacts to your to your Bill of materials now the other thing that this thing does is that you can have a declarative as one definition inside of your repository and now the bomb utility as a step in your CICD pipeline, so it will read the the definition that you're that you just do right in the yaml file and Construct your S1 based on the under artifacts that that it finds and are defined and there will see you have them all you know it Okay, so let me switch screens to My demo so okay It's it like a big enough Little bigger so This is a test project, it's a little project that I wrote Call mini proud which is emulates the proud interfaces in GitHub actions, but the project itself is not really important What I wanted to show you is a way to structure. So this this project is like a single it's a go module standard go project with its tests and and Files in there, so let me remove this S1 because that's Not needed now. Okay so For example this when you compile this project it spits out in this output there it through here It's binary. So if you wanted to So the first thing I wanted to demo is the thing where I was saying that you can just point it to the directory run it and have your album so If I do Bomb generate So if I just pass it that a directory as a as a source and run it It will run and process everything inside. So let me give you a brief overview what sort of what it's doing there so it starts by processing the files in the directory listing everything it it goes and fetches the SPDX license lists from the SPDX website So SPDX has a set of the fine open source licenses that the Linux Foundation produces and maintains and They the S1 the our libraries have a classifier in there that Read those licenses in which it's downloads in JSON So that it can interpret any any files licensing files that that it finds inside of your project and dependencies, which we'll see in a bit so downloads the licenses and writes them to disk and It scans your directory and says oh, so probably this is the license file for the whole project And if it finds one of those predetermined By predetermined means like I did not quite scientific study of the license names that people use But they're kind of hard-coded in there for the moment But it will it will use them as defaults and it will actually look for other Files or licensing information inside of other of other files as well But it looks for so it asks itself. What's the license that I would use as a default for this project? then The next step is that the tool will actually read and honor your git ignore files so if whatever you Set in your git ignore it the it'll honor those patterns and keep them out of the s-bomb when you're running in directory mode like this and Then it scans the files the whole file tree and adds them to the directory It right here. It's detecting that the director contains a a go project Oh, by the way, the only Since this is this was written for Kubernetes the only language that we fully support and for the moment They'll say it like this is go It'll it will allow you to index Container images binaries and things like that from in Derived and compiled from other languages, but if you want the the dependency analysis, it only works with go so It reads your go mode and lists those are dependencies as direct And then after that it will calculate the full dependency graph and add the transient dependencies as well to your project So once it's it's got the information it needs to run It will actually go and fetch all of the go modules download them or use them in from the local go cache To actually go inside of each of the directories of the programs and look for licensing and information to include them in the in the final output And it found called dependencies and It wrote So it wrote the file so The the first new feature for others that have Perhaps previously look at this tool or not. I I want to showcase a new feature that I have been working on for the last month or so is that you can now take So if you see the help for the bomb utility, you'll see it and it has to Uh, two main modes of operation. Oh completion there. Yeah, so yay for supply chain and features getting added so the the the two modes of operation or for operation are generate and document so the first one is generation of documents and there are there is Ways of working with documents and currently what you can do with uh document is the following You can actually outline Your s1 Tamed it so that's okay so Here is So if you take a look at the actual spdx code It's like really hard for humans to understand what's going on there Of course, if you take some time and try to visualize what's going on that you can certainly make sense of it especially in a simple s1 like this one, but I thought that this Kind of visualization should bear on this help People understand better how their s1s are structured so You have the root document here Which is describes one package if you remember When we were talking at the beginning how you should probably list all of your source code inside of a single package It it it's what it's doing there and that that package has 31 relationships defined to Which are the actual files of the source code in my little project here That that that's the first ones and then it lists the the next set of Relationships which are the dependencies of the go of this go source code At the end this it it says that it describes zero files because what it means is that you have the root null of the document and from there Then the next derivation is that the next branch or yeah branch you you could see it as Having packages or files attached to it and what it's saying there. It's that it doesn't have any files so An important thing to notice We are defining we're like Showing here the s-bomb in a tree like fashion But it's just a simplification to make it easier to understand if you think about the many many many One-way relationships that an s-bomb contains it would be I think better suited to to visualize it as a graph more than a tree like this Since Since one package one file can span multiple multiple The relationships and with other things inside But I mean certain helps as you can see here we can kind of understand the how the s-bomb was built so Now the next thing we want to do is for example This this little project is designed to run as a github action. So it produces a container image Which are already oh, maybe we can choose one of the errors. Let me So Okay, so this this one is interesting So what I'm going to do here is run the bomb generate command and I'm going to pass it a One of the distro list images to see how how it will actually generate the next one for A container image by reading it from the registry. So first thing the first like this Okay, so what is what it did there is that it went fetch the image and its layers from the registry downloaded them and Generated the s-bomb from there So if you take a look at this one We can take a look at the outline of this the way it's structured it So The this s-bomb has one package which points to the actual tag of the image and then itself It's a package which contains all of the images in there as I was saying before Things can be related to other things and in this case You can see here the same the same It's the sbdx element listed twice by because The container the actual container images are variants of the tag. That's that's the way we structure them so Now I want to let me see the time okay and What I want to showcase next is the The building the s-bomb from the file. So this here is the The yaml of initial for the for the project I'm running Probably if you take a look here What we are defining here to be included in the s-bomb is the directory where we are running A single file which is the the compile binary that gets written to the output And then the container image We give it the license of Apache 2 then The other thing is okay, so there's a new A new feature that I also wanted to to showcase today is that we've added We've been working on enhancing the Kubernetes Supply chain security to a little bit and we've started working on producing in total attestation files For some of our artifacts. So what I've done here is that I've actually included Those libraries into the s-bomb tool so that whenever if you if you don't if I don't know if if you're familiar with the attestation file, so The attestation files are a list of components similar to an s-bomb But but they can it contains information so that you can trace those those items back into the CICD chain Back and actually find out where those will build so if you think about an s-bomb you're already Tracing all of the artifacts that get produced in the in the s-bomb And you already have that information So what this tool can help you achieve is produce like an empty attestation file that includes all of the files So you can reuse them and maybe build your whole provenance from there So this is highly experimental and secret code. So let's see if it works because I'm I'm not sure. Let's see So let's see what happens So I I run the tool and I pasted the pasted the Configuration file so that it detects and reads the ammo configuration In it's doing the go dependency construction and it's Adding here. Oh, this is this is the actual provenance conversion of the artifacts And the container image. So if we were lucky we should have the The s-bomb ready. Okay, it's there. So let's take a look at the At the outline So here's the output So as we saw before here's the image That got listed as the as a package there then the source code It will always grab the the name of the of the code module as the name of the module and then here's the The the single binary attached at the end So if those were to be my artifacts that I'm releasing my s-bomb ready to ship And then the new the new file that should have it produced is this attestation file right here jq Here to make it more visible. So This is the attestation file that uh, this is the intoro attestation file. So If you if you see here, it has the the the part that concerns and relates to the s-bomb The metadata is is the is the subjects here. So since we already computed the hashes for all of the files in there the It's virtually free for us to include in the attestation file. And so if you have this file And you simply go and complete the predicate section down here, which is basically the information about the process that run the thing the dates Who did it and things like that the recipe which arguments went then you You if you completed the information to this file you have the s-bomb and as well the the necessary attestation to make uh, potentially make your your Your release salsa level one compliant because if all you're missing is the provenance metadata, which this think Makes you create and Yeah, I think this is those are the main points of of the tool that the way they work Yeah, probably those are the ones and then A little bit of a future the direction of this this tool We are as I said at the beginning Kubernetes releases a bunch of RPMs and dev files The packages with the release we are not currently listing those Because we are still working on the actual analysis of the rpm and and the packages then we been in talks with the linux foundation s-bomb generator folks to maybe see into merging our efforts and that way we could maintain the project and Uh, we would benefit by gaining more languages in our tool could be an An option and then the final one. Well, not the final one, but spdx allows you to output not in just that format that you saw there, which is called tag value It also allows you to output in many other formats from rdf json YAML and we only currently output to to to the tag value This is a validated one, but if you need it and the one of the others are still missing and then At some point I would certainly like to add some validation and verification. There are still Divergencies in spdx tools in the way they interpret things. So the way you saw For example container images listed here or some of the packaging that goes on in there different vendors Toolmakers have different interpretations of how they see and things we are trying to to to Make efforts like the s-bomb upcoming s-bomb or amma Where we will try to compare and make more Consensus on the output, but what I would like to add to this tool in a not very distant future Is the capability to ensure that if it produces an s-bomb it can be Reverted and to make sure that the actual Output corresponds to the things it it understood and so on your artifacts and yeah, and that's That's the that's it for for for what I had prepared and if you want Those are my my deals if you want to contact me ask me in a way Just I usually go by puerco in most most everywhere and If you search for The blue link there it will be when I upload the slides you can you can use that one I did a version of this presentation where we demonstrate the how to integrate the s-bomb tool inside of a tecton Release pipeline and having it as a task And you you can see it there. There's also a ripple A ripple with the with that content in there And yeah, I think we don't have time for questions, but But if not, I'm happy to take any any on the hallway or