 All right, we're starting for real this time. Welcome. Thank you for coming to the talk. I know it's the last day of the conference. We're all from SICK Release, and we'll be sharing stuff about the release team and release engineering. So my name is Grace. I led the 1.28 Kubernetes release that came out in August, and I've been on the release team for over two years now. Hey, I'm John Anderson. I'm an SRE at a company called Ditto, and I'm the newest member of the release team. I've been on the last two releases, 1.28 and now, 1.29. And my name is Savos Fogarcia. I'm an open source engineer with Chingard, and I am one of the technical leads with Kubernetes SICK Release. And as Grace said, we're so honored to have you here as in the very, very last minute of the conference. So I guess thank you to all of you and to the thousands looking at us on YouTube. So cool. So let's begin with what is SICK Release? So as some of you may know, Kubernetes is organized itself in several special interest groups. You have special interest groups for all sorts of things that compose a project. We have one for storage, which we have one for networking. And we happen to be the group that takes care of releasing Kubernetes every month. We have a clear and defined mission, which is establish a consumable, introspectable, and secure supply chain for Kubernetes. But I guess the way you can think about this is that we are the SICK that groups together all of the contributors that are working on the release tooling and the processes that bring you Kubernetes every month. SICK Release has two main sub-projects. One is the release engineering sub-project, which I'm one of the technical leads together with Carlos here, and I'm going to be talking about that. And the other one is the release team, which my colleagues here are going to talk about. This is our main entry point into the city. We're going to have links for it at the end. So you could say that SICK Release is sort of two parts. One part, a wonderful group of volunteers that rotates constantly with every release, and the machinery that powers all of the release processes. And release engineering is a part that takes care of that. The release engineering team takes care of the tooling that powers the Kubernetes releases. We also manage the release managers team, which are the group of contributors that take care of executing the tools to release Kubernetes, to cut the releases every time we have a new patch release or when the release cycle is up. We also build all of the supply chain security enhancements for Kubernetes. And we also handle some infrastructure that hosts the release artifacts. So specifically for this update, the big thing that we wanted to talk to you about is that we now are using the OpenBuild service from SUSE to build and publish our system packages. So our system packages, every Kubernetes release comes together with a number of artifacts, container images, binaries, some tar rolls, the source code. But we also package for you RPMs and devs. And if you have been following the Kubernetes news recently, there has been a huge effort from all corners of the community to move from the infrastructure that Google was kindly providing us to the project for many, many years. And we're now moving everything to community-owned and controlled infrastructure, which not only is better for us because we have more control, but also lets other companies chip in to help with the cost of running and serving Kubernetes. So I'm going to give you a super brief overview of the release process so that you understand a little bit how, what improvements are involved. So in three vignettes, the release process works like that. The first stage, the first phase, is the stage phase. And during that, we build everything. We build the container images, the binaries, package the sources, the tar rolls, and everything that's going to go out with the release, except for the OS packages, the next phase that the release managers execute is the release phase, where we push the images to the registries, all of the binaries to the bucket where the final is served. We push all of the Git objects to GitHub. And again, not the OS packages. And then when everything is done, we publish out the release to those thriving fans that consume Kubernetes every month. But this is not exactly the way it works. There's a small, there used to be a small detour in the middle. So the way it used to work is that we staged everything around the release process. And before letting the world know that Kubernetes was available, we used to call our friends at Google to help us with the packaging, the OS packages. And so when everything was ready to be shipped out, we called Google. And just to make it less informal, less impersonal, I'm going to slap some random guys face in this. So we called them and said, OK, we need help. And they would literally build the OS packages on their laptop. No one was looking. And then push those packages back to the repository, which is still owned by Google. But we are not publishing there anymore. And then once those RPM and devs were in the repository, then we could announce to the world that they were ready. So we changed things a little bit. And the way that we do it now is that we have during the staging process in the release cycle in the release process, what we will do is we kick off the builds of the packages. So all of the devs and RPMs get built using sucess open build service. Then when we release, we use their API to publish the newly built packages to the community repositories. And then finally, we announced the release is out. So there's a bunch of benefits of doing this, because now we don't have to abuse them and others in the middle of the night to help us package the system packages. Now we have a fully automated process that runs as part of our pipelines. The signing key, we used to use a shared key with Google. Now it's a key that's controlled by the community. And in general, it brings a better life quality for the release manager so that we can do things more efficiently. It comes with some challenges. Like for example, we now had to rotate the key because we were using the Google key. Now we have a community key. And it also was a challenge communicating the changes because we had to do it in a really fast pace. Do the changes really fast. So if you happen to be around the Balkans and you find my friend Marco Moudrinich, you really need to thank him. He single-handedly took over the job of making sure that this project happened. So the community is now should be really, really thankful to him. Also, big thanks to Susie, who has been really helping us and kind answering our questions. And of course, to the Google admin teams that over the years helped us package those system packages. And we have a bunch of other highlights from the backstage models to help us build the tooling to power this project. C-release actually applied, I think, for the first time to have an intern from the LFX program. And we've been working to stabilize the release tooling so QPKG and Rapture were the tools that used to build the system packages before are now gone. We now serve the checks and files from the S1 location from Kubernetes. So we are helping other projects move to use the OBS builds. And this has the benefit of also helping other projects under the Kubernetes umbrella use that same automation. Have some updates to our S1 generator. And of course, there's the heavy lifting that the team does every month. Like make sure we are bumping the go dependency updates. We are updating the, we updated the base images that we used to devian bookworm. And of course, the hard work that the release managers put into to cut the releases every month. So with that, I'm going to pass it to Grace to talk about the release team. Yeah, so as mentioned, the release team, it works more with the community. We worked with them to pull in caps and enhancement to gather them for deadlines. We also work on a bunch of different things, like creating documentation and communication with the rest of the community. So 1.28 was released in August. And two things that stood out for me, although we shipped about 47 features. And some deprecation mostly related to removing Seth out of tree. So two things that stood out to me. The first one is we are allowing three instead of two versions skew between the control plane and the note down. So hopefully that will make it easier for folks to upgrade. And then the other features that I think a lot of folks in the community care about is API awareness for psycho containers. That went into alpha and 1.28. I believe they opted in to beta for 1.29. And that will include more cool things like ordering container termination. So I linked the release block there. If you want to check out more. Yeah, so 1.29 is now in progress. And it will come out on December 5th. The mid-cycle blog is scheduled for post-coup con. But the PR is up if you're curious and want to poke around. Cool, let's get a little bit meta and talk about what we're thinking about inside the team. So before 1.26, we used this giant monster of an Excel sheet to keep track of all of our enhancement. And it got close to about 100 enhancement at some point. And 30 people on the release team plus SIG leads were looking into that all the time. And it was like a scary thing. And so 1.26, we migrated over to the GitHub project board. And that has worked really well for us. So great in fact that now we are eliminating the Bukhtriyaj team, not elimination, but we're merging the team. So the task that the Bukhtriyaj folks were doing can easily be handled now with the board. So starting next release, we are merging those two together into a team called the release signal team. And they will handle the stability of the test and the bugs in the release. We're also working on a process regarding removing inactive release team member. So this is something that we kind of expect to happen every release team. It's a time-sensitive role to be on the release team and it requires you to be responsive. And sometimes folks resign and sometimes they don't and they just kind of drop off the radar and that creates more work for other folks. So this is just putting things on paper about the knowledge we're sharing and how to handle those situations for future leads. And we rely heavily on the folks over at SIG docs to help us with things like documentation or release block, feature blocks. And so we're also working on a new checklist such that the folks on the release docs team can better work with SIG docs on the future for weekly status. One thing that the release team docs folks were struggling with as well is syncing the docs branch from the SIG docs repo into the things that is relevant for that release and they have to do that once every one or two weeks. And it's just a lot of work for them and it's like one of those things that could be automated and has been automated for the general release. So that's one of the goals that they're working towards to remove that workload for the team. Now, John is gonna talk about how you can join us. Yeah, and so if all of that sounded super exciting or if you just wanna hang out with Puerco and Grace, joining the release team is the quickest and easiest way to do that. It is, I joined in 128. And so the experience there is I had been using Kubernetes since 113 but didn't know how the sausage was made. And so I was really kind of worried about joining that side of it because one of my coworkers had been adding time zone support to jobs in Kubernetes and it took them almost three years to get it through. So like, I don't know if I have three years to commit to contributing to Kubernetes right now. And so I found the release team and it was a phenomenal experience. It's a distributed team, fully remote, asynchronous. And so they have to document everything. It's a volunteer driven organization. So it's rotating in and out. And so the on parting has to be top notch because otherwise the next cycle isn't gonna be as stable as the previous one. And so I've enjoyed it. And I think everyone should be joining the release team, figuring out how the sausage is made, joining this amazing community and collaborating with everybody. And it really gets you in the door to collaborate with the other SIGs and the other projects because since we have CI signal, which is collaborating with every SIG depending on how stable the release cycle is or like docs and enhancements, whatever's changing, you're gonna be in there talking with those contributors and you're gonna slowly get in there and then you can start working your way towards more contributions. And the time commitment is not a lot but it really depends on the team because if you take something like CI signal where you're doing a lot of test quality, that's consistent throughout the whole release cycle. So you can kind of decide what team you wanna be on based on what you're capable of doing. If you joined the comms team for example, that's gonna be really heavy at the end. That's where all of that communications gone. And what we're looking for is just people that are passionate, people that are contributing to open source. And so when we're reviewing these applications, we do have a lot. In Kubernetes 128, we had 126 applicants and now we had 165 in the most recent one. And so it's ramping up every time. And so it is getting more competitive. It's getting a little more difficult to be on the release team because we can only welcome so many new people each time. But we do encourage everyone to apply more than once. You have a much better chance to get in your second or third time while you're applying. And if you are actually showing more experience over time. So maybe you don't get into the release team the first time. Join some of the other SIGs. All of our meetings are recorded. You can come, you can join, you can communicate with us. And that helps you during the application process because we're looking for people that will stick around. Like Grace was saying, if you join and then disappear, that leaves the people that are there to pick up that. And we're releasing every six months on a cycle. We don't like to be delayed. And so we want to select people who are gonna be there for the long time. And so yeah, as you can see, like it was a 16% success rate to all of our applicants in the previous release. And that's only gonna get lower as we have more applicants. And so yeah, I'd say like the best thing is to join those other SIGs. You can help out Puerto Rico on release engineering. OpenSUS Build Services, a joy to work with. So it's a great place to start if you need to. Yeah, and then if you wanna join us, here's some links about how we're going. Do you wanna talk about some of the links you have in there about S-Bombs? Oh yeah, well, these are some of the links to some of the tooling that we produce that can help you with your releases, even if you're not in Kubernetes. Those tools can help you generate S-Bombs, generate Salsa provenance at the stations. We also have some release actions if you want to automate your processes. And going back to the release team, I think that I would like to spend just two minutes talking about how proud I am of the success stories that we keep hearing and you can have the privilege of being a part of. Grace here is a great example. She joined, she did like a really good job and the release team eventually got to run the release cycle for one of the biggest software projects in the world. And you can do that too, you can join us. Marco, who I showed, is another example. I am another example, I have the privilege of having been mentored by team here, Sasha here, Carlos, and it's been, it's a great example. So if you want to start your Kubernetes journey, come with us and don't be discouraged if you don't get selected in the first try of the release team. I got rejected, I think, once for the release team, so it's natural, but we have lots of work to do and the release team is easier because it sort of guides you through the Kubernetes journey, but we also have things for you to do. So if you're looking for work, we have some for you. And yeah, hopefully you can get a good experience joining the project and helping us release Kubernetes every month. So, thank you. So lots of familiar phrases, but if there are any questions, please, we're open to them. It's a secret. One of my coworkers. All right, well, thank you so much for being here at the last moment of the conference.