 Hello everyone, nice to see you all. Thank you for coming as announced. This is a practical guide to CIG release the tooling for Kubernetes subprojects. I am Marco Mudrinich. I am a senior software engineer at Kubernetes. I am also a teaching associate at the University Union in Serbia, teaching cloud native and Kubernetes and all that stuff. And some of you might know me as a Kubernetes release manager and contributor to CIG, CIG, CIFRA and some others. So a quick disclaimer before we start is that this is not a deep dive. It might seem so at some point, but the plan is that we try to do a high level overview of many tools as many as possible. You might ask for the beginning like how are the Kubernetes releases created? Like what we do exactly there? And the core Kubernetes, speaking of Kubernetes slash Kubernetes releases are creating using Krel. And it stands for the Kubernetes release toolbox. While to some degree, Krel only supports cutting the core Kubernetes releases. There's a reason why is it a toolbox, right? But Krel utilizes many different components that are to some degree standalone and that you can also use in your projects. And we're going to see how to do that today. Before we start, so what is like our mission, vision and values? And like in an interesting way is that we want to create a tooling that can be used by every Kubernetes subproject to do releases. Like you can take the tools from the places where they are, put them in like CI CD pipeline for you and have the magic happen. And why we do that? What's our vision is to make releases reliable, safe, secure and trustworthy. And we are doing some, let's say a lot of jobs to make that happen. And speaking of values, like we want everything to be transparent. Everything that you do, we do is single release. We try to make it like followed by proposals that you can review, that you can take a look, that you can provide feedback. We are trying to be as open as possible. Even if you're not a release manager, if you're not contributed before that to single release, everyone is welcome. You can join our meetings. We will find something for you. You can provide feedback. You can work on something like we welcome everyone. And as the final one is that we really pay attention to visibility and modularity. We are going to see about that today. Like we want to make sure that everything we make is, if possible, make it possible to use it for every project out there, but at least for the Kubernetes subproject if that's not possible. And I left a link to our charter, which describes that in a little bit more of a detail. About a 10,000 feet over you, what we are going to see today is I'm going to start about how to host artifacts for your projects releases, speaking of container images, binaries and torbles and WN and RPM packages. And then we are going to see how to enrich your project releases with tools like release notes, bomb, telephoto, side gaze, then similar tools. And let's start about how to host artifacts for your releases. So every release needs some artifact, so users can use that release after all. Yeah, that might be said by me. Maybe not. I don't know. But there's a little problem over there that we want to make people aware is that serving artifacts costs a bunch of money and we need to be careful about it. And we need help from call providers, the different structure providers, and we need to be creative about how we host and server artifacts and shut out to all of them for who helped us all these years to make everything that we host available to all our users. And you might ask, but there are many free services and we need to be super careful about that because those free services can be go away at any time. They often have limitations like rate limits and we need to pay attention to start being secure and trustworthy and sometimes it might not be super easy with free services. And SIG release and CK are providing both guidance and tooling and infrastructure that you can efficiently publish releases and your artifacts. So let's start for container images and you probably know about Registry KSIO as the Kubernetes registry for all the images. But what is the interesting fact here is that it is in fact available to any subproject out there and you can just submit a request to get the repository there and this is actually the recommended way to host your images for your projects. And basically, you have a doc here that explains it. It's really simple as like submitting a PR with some information, having CK to see if we approve your request and reconsulate adding relevant benefits and enable automatic image builds. We are not going to speak much about those freeze. It's very well described in the document, but the last one tends to make some trouble for projects getting good work. So I'm going to explain it about how it all works. First of all, all images are first published to a strategic repository, which is a kind of a temporary repository. Then images are basically built and published via Google Cloud Build. So this is a very important point is that that's the only way to do that stuff like you have to use Google Cloud Build. There's no other way. You just add Cloud Without YAML to the root of your project's repository and then you add the post-summit jobs that's going to trigger the Google Cloud Build when you do a release. And this is how it looks like. It's very basic. So you basically just install Google Cloud Build what to run on your project to build the images and to push them. And this is basically the job that runs when you trigger a release, for example, if you want to each push to main or to release branches of stuff like that. And there is like a special hint here that this job must live in a specific place, which is like config jobs, image pushing. And what's another important thing to do is that after you push your image to the strategic repository is that you need to promote it. And this is where we have some pretty handy tool, which is called K-Promo. And this is a very important step because if you don't do that, images are not going to be available to users. And this is often forgotten by projects ending up in a situation that users see your release on GitHub. They want to use the release and that is a problem that image is not available, but it should be there. And it's often because image promotion was forgotten. And how that all works is that Registricate SIO is replicated across many locations and regions to save on bandwidth as much as possible because it's super expensive. And it turns out that storage is not that expensive. And we don't really want to ask maintainers to push to all locations that we have Registricate SIO on. So we have the image promotion process. Like what you do is you take the KS, KSIO repository, you fork it, you install the K-Promo tool that we have. You export the GitHub token and you do like this command here, which is like K-Promo PR. You provide like what's your project that you got from the CK to push your images to. You provide the tag and it's going to create a PR for you that you just need to hold cancel, get it approved, merged. And after that your image will be public to the users. You just need to make sure to watch the correct job. It's post KSIO image promo job on proud.kss.io. There's like asterisks here about successful. The job might flake, especially when it comes to replicating signatures. We are aware of that and we are working on fixing it. And after the job is done, your image is available to users on Registricate SIO. And now this is like about container images. Now let's talk a little bit about binaries and torbles. And I actually don't have a lot to talk about it. So at this time, we strongly recommend that the Kubernetes subprojects use GitHub releases to host the binaries, torbles and files. There are many reasons for that, but this is the recommended way at the moment. And we also have the LKSIO and it is backed by Fastly CDN. And thanks a lot to them for the generous donation and support from them. But it's not really easy to get stuff on the LKSIO for subprojects, so it's recommended to use GitHub releases. But if that doesn't work for you, you can reach to see Kate Simfra and they can see what they're going to do, but that's mostly on case-by-case basis. We will figure out something. And now we are down to Debian and RPM packages. So as of August 2023, not so long ago, we provide an option for the Kubernetes subprojects to host Debian and RPM packages at packages.kts.io. Before, this has been strictly limited to the core Kubernetes releases when we use the Google repos, but now we have this possibility and we are opening that to all the subprojects. The way that packages.kts.io works is that it's powered by donations from OpenSuzi and AVS. Packages are built signed and published by OpenSuzi's OpenBuild service, shortly for OBS, not even for recording and streaming, but this is another OBS. And they're hosted on AVS, which could be changed, so you don't really depend that much on it, but it is that way right now. And this is how it looks like the OBS platform, right? So you have your project that you get, you have some packages and you have some builds. It's pretty much nice, but you don't really have to interact a lot with it. And there is a catch of we are not really happy about it, but we don't have a formalized process for requesting packages and package repositories for your projects. But that shouldn't be a roadblock here, because you can just reach out to us to release managers, like SIG release or release management or Slack. And we will make sure to set you up for success and like to give you the repositories, to give you the instructions and to help you get everything running. And this is a little bit about how you can do that. So you can just take the spec files from us. We have the QR code and link here with that is pointing like QCTL ones and QBlitz ones are one that I recommend taking a look into. And one thing that I would just point out here for folks that are interested is that we build only RPM specs that are automatically converted to Debian specs. Thanks to some super awesome tooling that we have in OBS. And we don't build binaries from sources inside OBS, but instead consume prebuilt artifacts. So there's lots of curl. And that's actually the recommended way because it's easier to maintain your pipeline and to ensure that you have it consistent everywhere. And also you can publish pretty much any package, any type of package that is supported in OBS. We talk about Debian and RPM here because that's what we use so far. But if there's anything else on OBS that would work for you, please let us know when we can give it a try. And how it works with our tooling. So what you would do to publish packages as we have repositories. This is an exception where you would use CRL for your project. You would do like OBS stage, provide the packages version project that you got from us. Template the OBS specs file and like that would trigger building the package. After that is done, you can make the packages available to your users by using CRL, OBS release and provide the packaging and project name. And that's you basically don't interact with OBS at all. You just use CRL in your CI CD. It can be proud, for example, you just run those commands. And that's all you need to do. Nothing to fiddle with OBS. It's super simple. And yeah, that's the way it works. And now we are to the part about enriching your releases. So I don't know, maybe a little no effect or maybe it's a very well known fact. I don't know, but Kubernetes C release maintains a lot of tools. And it's like a lot. And I would like to highlight some of them. And the first one will be K-Promo, which we have seen before. The release notes tool, the Zygreist, BOM, Telehota and a bit more of them. And I'm again pointing to CRL is charter because it's like a nice place that you have all of our sub projects and projects that we maintain and that you can see and use eventually. And I would like to start from the first tool and this is release notes because it is one of my favorite tools, in fact. So what is it doing? It's a tool that parses PR descriptions. So your PR description here and collects release notes from a special release node block in the PR description, right? So you create, you put this release node block. It must start with those backtyx and release note and you put something here, right? And then you might ask, but why? GitHub has its own automation as of recently and you can generate nice looking release notes. There's a little bit of a problem there is that GitHub PR titles and commit titles are not sufficient for change of entries for many reasons. First, you're limited in terms of land, like, for example, 72 characters or something like that. And that's often not enough. And PR titles often focus on cold changes, not on user impact. Sometimes you want to highlight something especially for users. And there's also that some PRs might require multiple change of entries, which is hard to do that with PR title or commit message. And we actually saw that with the release notes where you have the special syntax like you can use none if you don't want to include your PR. For example, you did some insignificant update. There is no reason to include it. It doesn't provide value for users to include it. So you would just put none in the release notes block, or you can put, for example, action required to, you know, in the Kubernetes change log that you must like your detection of grade notes, you must really look at this. This is how you get it with action required. And there is in fact no other formal syntax than this to make it simple. Like we really want to use release notes as simple as possible. And it also has some pretty nice other functionality like the maps functionality that I also pretty like is that you can override release notes, provide the JSON files and say like this is the PR number and this is the message and you have like keep track of it. If you are not, let's say, maybe not happy with the release note in the PR and you can't edit it, or you want to like, you know, do some editorial work and make it look better. You can use the maps functionality to override some of them. And this is a project that can be used on any project on GitHub as long as it uses the release note block. So it's not just to be used by Kubernetes and its sub projects. It's really in fact the any project on GitHub. And how you do with it? It's very simple. Is that you do like install the tool, do the release note generate, provide the information and you get the file with release notes. And then we have the next tool, which is like decide guys this a language agnostic dependency checker that keeps track of external dependencies. So let's keep it simple. Is that if you have one same dependency that's used across many different places, like in go code, the ML Manifest, JSON Manifest, how to make sure the version is consistent across all those places. You can define them using this tool. And it also has some additional use cases like that. It can also check for the latest version on upstreams like GitHub, GitHub, hell. But Kubernetes uses it mostly for the static validation purposes. So you basically have the CNI version. You provide it here and you provide all the files that it's appearing. And then you just do like the validate comment. And if you don't get anything, it matches everywhere. You're good to go. If you have a mismatch in any of those files, for example, you forgot to update it. This is going to fail. And you know that you're going to fix it. And some of another tools that we have is like bomb, which lets you create view and transform software builds of materials, which is still a pretty hot topic. And it has been used for creating gas bombs for the cure Kubernetes releases. It's still being used. And it works for container images for Go projects and files on upstream. And it works and supports with tag based SPDX format. And how do you use it? It's very simple. Like simplicity is one of our values as well. Is that you just do bomb generate with dot and it like generates the as well for you based on your current working directory. Or you can use, for example, from the image, like for Cube API server example here, or you can also provide manually files to be included in the S bomb. And you just run the command. You get the S bomb. And you can also use it to visualize as one, which is pretty useful with a document outline command and to see what's actually in the resulting S bomb. Or if you want to like to see how to use it, what's there. Yeah, you can use the bomb command. And there is one more tool that I would like to highlight, which is also pretty awesome is telephoto. And there's official definition of it, but we don't have a lot of time. But in short, it works in a way that you can give it your build information. It will observe your build and give you information about each build step and what artifacts were created where they're stored. What about checksums and all stuff? You're going eventually to sign that file and provide it with a project and it's like attestation of what happened in your build process. And it's going to provide you a JSON based in total attestation. I hope I didn't get that wrong. But yeah. And like this is from the read me. You just like provided it works with Google Cloud build and GitHub actions. And we plan to support proud at some point. And finally, you might ask now, how do you use this? If I don't use proud, but something like GitHub actions, we have release actions tool or action. And it is at this link here. It provides all of these tools that we mentioned now to be used in the GitHub actions. And then you can use stuff like bomb as simple as this screenshot. And do we have time for questions? Not really. Okay. But you can find me on Slack, XModery, and you can reach out on single list channels and release management channels on Slack. And we can talk about anything. Thank you for your attention.