 Welcome folks. Happy Friday. Welcome to Kubernetes release team, the wise and house ways and means. So we are, my name is Ray Lajano. I am the co-chair for Kubernetes SIG docs. I've also been a member of seven release teams highlighting as the 123 release lead, the 125 advantages advisor. I'm also a sub project lead for SIG security for the external security audits, which was just published on Wednesday. So please take a look at that. Hey everyone, my name is Priyanka Sagu. I am currently the technical lead for SIG Contribux and I have been part of the release team for five cycles now starting with 123 as enhancement shadow, leading enhancements in 125 and release lead shadow in 126 and 27. Yeah, so 123 has a special meaning to both of us. It's the release that I led and it's also the release that Priyanka started with the release team. So the Kubernetes release team is part of SIG release. There is a new release team for each release cycle. So there's three release cycles a year and we're from all over the world. As a whole team, we are responsible for the day-to-day work required to successfully release Kubernetes and the release team has a shadow program. The shadow program is a mentorship for folks without any contributor experience. There are some folks as well that are veterans of Kubernetes contributions. We also have folks who are also SIG leads as well and SIG technical leads as well who are also joined as shadows in the Kubernetes release team. So the shadow program is pretty popular. It's a structured program for folks to learn about the Kubernetes release process, to actively do work to help release Kubernetes as well. And folks learn how Kubernetes is released from generating release notes, creating the documentation for new or graduating features. They help to gather enhancements. So with help gathering enhancements, they learn about the enhancements, they're reading the enhancement proposals. And they also make sure that all the code PRs are merged before code freeze and lots more. So there's lots of responsibilities that the release team is responsible for to help successfully release the largest open source Golang project. And also one of the points of the Kubernetes release team is as well is that it grows the community. So our shadow is a mentorship program like I mentioned before. And other SIGs have also incorporated a shadow program as well. So it helps bring new folks as contributors to the community. And also it kind of has that structured contributor ladder in the shadow program. So folks who and we'll talk about the different roles in the release team in a few slides here. So the shadow program helps folks learn about the specific task that's part of releasing Kubernetes. And from there, they can become role leads. So we have role leads for each role. Then we have also a release lead as well. And even the release lead will have lead shadows who are who have been experienced release team members. So the release team is not only just brings folks to release Kubernetes, but also helps grow the whole community as well. So like I mentioned before, the release cycle, we have we do three releases a year. There are about a 15 week cycle. The last release cycle of the year tends to be difficult with holidays. We also take a break for KubeCon NA as well for the last release cycle. So it might be from 15 weeks, might be 16 weeks, but might be just 15 working weeks or might be a 15 week cycle, but might be 14 working weeks. We do like to take a two week break between cycles as well, where we do your retrospectives of the previous cycle. So we learn if we talk about what we could do, what, what, what well, what we could do better, what we can, what can or what can we improve. And we're always iterating on the release team cycle or process and the release process as well. Every single release cycle and that's two week period also gives us time and we'll talk about towards the end of the talk to open the application for new shadows for the new release cycle. So there are some delays as well that might impact the release cycle. I think for 124 there was a three week delay because of a near going version as well. And 119 was one, I think 119 was severely delayed as well because that's when COVID happened. So release dates are aspirational and that's what we like to say, but we definitely have release date targets for it. So I'll just highlight some of the, some of the release cycle as well. It's like I said, it's usually a 15 week process, but we actually start before the release cycle, where we assemble the team about about a month before the next release cycle starts and starts, then comes in the first major phase of the release cycle where we collect enhancements. So Kubernetes each release, there are enhancements that are introduced to Kubernetes. There's also enhancements that are graduating from alpha beta GA or being deprecated as well or removed as well. Like POTS credit policies was removed in 1.25. It was deprecated, I believe in 1.22. So there, so we start to, we, so we now we have a process where the 24 different special interest groups of Kubernetes or SIGs. So Kubernetes is broken into all the code and GitHub repositories are broken into 24 different special interest groups. And these 24 different special interest groups opt opt in into what their, their own enhancements will be ready for the release cycle. So the first part of the release cycle is when the SIGs opt in enhancements that will be, that will be parts of the release, like enhancements that will take features from alpha to beta, beta to GA. It might also, it might also be reinstated as beta as well, because they might have changed something, like if they might have changed a controller or something like that. So that's usually the first step. The first major milestone is the enhancements collection, and then we have an enhancements freeze as well. And with enhancements freeze, that's where all the enhancements wishing to be parts of the current release cycle would have to be what we call in, in implementable states. There is, they have testing plans, they have graduation criteria. So they have to satisfy us very specific criteria to meet an enhancements freeze. And we'll talk about all the different teams and their roles and what they do in the next few slides here. So next milestone after that is after enhancements freeze is code freeze. That's where all the enhancements going into the release must be code complete, including tests as well. And the docs for requesters. So every enhancement in Kubernetes, most of the time they're user facing, so they need some sort of documentation. So for users to learn and to know about how to use that enhancement or feature. So around this time, around code freeze, a little bit after code freeze is usually a few days, there's usually a documentation pull request deadline as well, what we call the placeholder PR, meaning that they intend to create documentation for that enhancement. It doesn't have to be ready. They just have to have the intent and clear and make some effort into that. They will create some documentation for their for their enhancements. Next is the test freeze when all and this state also aligns as well with the with the branch cuts as well. So along the way, the other teams will talk about what the other teams are in a little bit, they will continue to do their work as well, enhancements team with code freeze will make sure that all the code PRs emerge. Docs will start pinging folks about to make sure that if they do need to, if they need to update feature gates, then make sure that the documentation is, it's noted that they need to do those updated docs. CI team and the bug triage team will will do will continue to do their work as well. Next is test freeze. So test freeze code lines also with the branch cuts as well. So the branch cut, we will create a release branch just for that release here also with test freeze. This is pretty close to the release time already. So the other teams get involved. We start generating finalizing the release notes. We start to decide what the major themes are because once we've reached code freeze, we know which enhancements are going to stable. When you key enhancements are going to be removed or deprecated, they might be part of major themes. Also some key enhancements might be very big that they might want to write a feature blog about it as well, kubernetes.io. After that, after the branch cuts with test freeze, we start doing a release candidates as well. So we do release candidates for the last few weeks, then come actual, the actual release dates, where we also do we do co-thaw as well. And the release dates as well aligns with just the finalization of the release cycle, except for the comms team. We'll talk about that a little bit because the feature blogs will will be published after the release. So there's one team that's whose work continues on a few weeks after the release. And the release team is made up about 35 to 40 community members. And there's nine roles with the release team. Most roles have four shadows. So there's always a release lead who is responsible for the release cycle and release process. The release, there's also an emeritus advisor. The emeritus advisor is you typically most time a former release lead, or has been on many different release teams. The release lead will have release lead shadows. The release lead shadows will will also be veteran release team members. They will have have been most likely been have been role leads in the past. We have a branch manager and a branch manager shadow. The branch manager does the actual cutting of the branch does the actual work on the on the final release cuts. The branch manager shadow might cut the alpha releases, the beta releases, the RCs as well. We'll talk about the exact functions of each role, but I'll just briefly highlight there's bug triage, there's CI signal, there's communications, there's docs, there's enhancements, and there's release notes. So we start with first with enhancements, enhancements come active into the release cycle very early. Usually we start even before the release cycle has started preparing the k enhancements repo for new enhancements to come in new features to opt in. So how it works is enhancements team is the team responsible for tracking all Kubernetes enhancements proposal that will be opted in into a release cycle. How we do that is by opening and kept issue, kept stands for Kubernetes enhancements proposal. We open those issues on a repo called Kubernetes slash enhancements. Now if a kept has to be included into a release cycle, one of the SIG leads chair or TL would need to go to the kept issue and add a few labels. And usually we have some, we have put in some automation now to recognize those labels on the kept issue and dump all of those tracked enhancements on a GitHub project beta board that looks something like this. So here during this enhancements, till enhancements freeze time, enhancements, shadows and leads would be talking to reaching out to SIG leads, different SIGs, asking them to finish their kept issue templates, add all the metadata that are required for the proposal. Yeah, that's all. And these kept issues would be helping the community to know which features would be moving from alpha to beta or beta to stable or maybe deprecations. Next we have CI signal. So this team starts to come in once we have the enhancements freeze in place and folks are now working on implementing code for the track and enhancements. So what this team does is we have a tool called test grid. CI signal team tracks all the tests that are running against a new code are Kubernetes, Kubernetes, Kubernetes slash Kubernetes master branch. They track the status of our master blocking and master informing boards and keep giving the community a weekly update. This signal also helps us to cut various release cuts during the release cycles, alpha, beta and RCs and eventually leading to the actual release cut. Next we have bug triage. This team is responsible for triaging all issues and pull requests that are targeting the release. They also track any potential issues that might be a release blocker. Next we have docs or documentation. So like I mentioned before, every new feature or new enhancements or any graduating enhancements needs to have the appropriate documentation ready when on release day. The docs team also helps generate the reference docs. So they help generate the API docs, the CUBE ADM docs and CUBE control docs as well. They also make sure that the feature gates page is updated as well. And also we have a deprecated API migration page as well. That started about a year and a half ago because we're out of this maturity level now that we're actually deprecating and removing APIs like 1.22 removed, ingress beta ingress APIs, beta CRD APIs, pod security policies removed, was removed in 1.25, etc. So now we need to create a place to keep track of those deprecated APIs in the documentation. Next is comms or communications. So comms helps create the official release blog. So the official release blog is published when after the release is cut, after some GitHub branching things are done as well. So once the release blog is cut, we communicate to the whole community through various mailing lists as well, that the release is official. There's also a theme to each release. So each release actually has a theme, a name, since 1.10, I believe, and also has a logo if you didn't know. So 1.23 was the next frontier to take on Star Trek. So there's a lot of Star Trek themes in Kubernetes. So 1.27 is the chill release, I believe. So there's always themes for each release since 1.10. The comms team also helped create the feature blogs. So a lot of new features, a lot of six want folks to use those features in one way, not just the documentation page, but they'll write a blog about it as well. And also they coordinate webinars about the release. And the release webinar will do an overview on the enhancements that are part of the release. Also coordinate interviews and press releases as well. And usually the interviews are done by the release lead. And there's release notes. Release notes help generate and edit the release notes. So there's a tool called Corel that helps to release Kubernetes. So there's a sub-command for Corel to generate release notes. And the release notes team actually created that tool itself. And I was part of that team in 1.18. I was a release note shadow. And my fellow release note shadows actually created this tool in Corel to actually scrape and create the release notes. And some of the teams afterwards helped to iterate on that tool to scrape all the PRs to create the release notes. So it's not just, you know, it's not just, so there's various levels of work that can be done on a release team. Like Docs, I didn't mention four, tends to be more of a Git heavy role as well. Because there's a lot of Git, there's lots of Git syncs. So it's lots of merge, there's always merge conflicts that have to be fixed. So it's not, so there's kind of a, each role kind of has their own like traits that it does not, it's not as obvious as one might think. So like Docs, you know, it'll be helpful. You don't need to, you don't need to have Git experience, but it'll be helpful to have that Git experience. Release note, since you are using a Go program, you'd probably, it'll be helpful if we do have documentation as well. But if you want to help iterate on the tool, you know, you would need a little go-ling experience, but you don't need to as well. Like you can just use a tool as is. So there's various levels where folks can help within the release team efforts. And then we finally have the branch management. So we have a role called branch manager, and they also have shadows who actually help us cut the release, different release cuts, and actually the final cut as well. So this is a diagram for how actually cutting any release cut looks like. So we start with a tracking issue. One of the branch manager or somebody who is responsible for cutting the release in the moment starts with creating an issue on Kubernetes slash Sig release repo. An issue looks something like the first picture. After that, they also put the same notifications on different Slack channels we use on slag.ks.io. So our release process is divided into two parts. We start with a mock release cut, which is also two parts. We start with mock stage, and then we do a mock run. And if everything goes well in mock stage, mock phase, we move to our no mock run. So meanwhile, when we are running our mock stage and mock run, we would be tracking all our logging, all our progress, whether something is green, something failed, what version of tools we are using at that moment, everything, all information will be logged in our tracking issue. Once we have a green mock stage and mock run, we'll move to no mock run. And that is followed by promoting images. What promoting images means here is whatever container images, whatever artifacts we have created during our no mock stage run would be moved to production buckets, and then we'll have our no mock run step. After that, once we have our no mock steps also green and running successful, we would be needing help from one of the Googlers, because right now, currently, as of this talk, our tooling for building RPMs and debt packages is under Google infrastructure. So the release branch manager or shadows, they coordinate with one of the Google people and move the responsibility to them for the RPM and debt packages. And once everything is done, same is again, logged on our tracking issue and an announcement is said into the dev mailing list to the community. I do want to make a note that there is work to remove the reliance on a Googler for the dev and RPM packages, so where we've done tests on using open build system or OBS to build the dev and RPM packages. So that is some work that is being done, that was done last year, will continue this year as well. So lastly, we have the upcoming release cycle for 1.28. So the shadow application program is actually open until May 2nd. There's a QR code to apply and folks are notified by May 9th. The official start date for the release cycle is May 15. The plan release date for 1.28 is August 15. Grace 1 is a release team lead for 1.28. Leo Pauquet is the Emeritus Advisor for 1.28. Leo was the release lead for 1.26. Anyway, so that's all for our talk. I want to thank you very much for attending on a KubeCon Friday afternoon. This is the QR code for our talk for any feedback. And any questions? Yeah, there's microphones on the sides here if you have any questions. Hello, thank you for the talk. I'm just asking about what are the requirements top life for the shadow program and the working hours all over this kind of detail. Yeah. Yeah. So there's no requirements at all. You don't need any experience if it gets. You don't need any programming experience. You don't need Kubernetes experience at all. So that's the beauty about the shadow program. You come in totally new. We have playbooks or runbooks to help guide folks on each role or we call them handbooks. Handbooks for each role as well. And if you need help, we are very friendly to show folks as well. I always do sessions on how to sync branches for the docs team, how to do merge conflicts. I actually have YouTube videos on how to fix merge conflicts for the K-Website repository. So you don't need any experience at all. I think the only thing you need is just a GitHub. You don't even need a GitHub profile yet, I guess, but maybe that's what I don't think you need. There's no other requirements except for an email address. Just to add, if you are looking for what is expected out of different roles, more on what we discussed, we have road handbooks, which you would find in github.com slash Kubernetes slash Sig release. You would find dedicated handbooks for each of the roles that we discussed here and also that would help you to answer the time expectations questions. Usually shadows are expected to work around maybe two or three hours or two to five hours a week and not really expected to work during the working days. They can work during weekends, but it vary from role to role as well and also what part of the release cycle you are in. So there is more information about that on our GitHub repo. Yeah, I just have seen that this is a very long running big process on the release, of course, because it's very complex. And what do you do if you have a serious issue after code phrase during testing where you say, ah, maybe we should not roll this out. Do you pause then and fix or what do you do? Different situations. They could be, we've had PRs reverted before. We've also have had to cherry pick in changes as well. We've also had to cut in for 123. We had to cut an extra RC. We had to do a fix for 123 was released on Tuesday, the Friday before there was a regression. We had a PR to fix it. We had to create a new RC, but yeah, on Friday nights. So yeah, so we do try to fix it. And, you know, and we do delay as well. We have delayed 118 was delayed a few days for an issue. 119 was delayed months. 124 was delayed three weeks. So yeah, so we do all of it. Yeah. Yeah, good. Thank you. Welcome. All right. Any other questions? If not, thank you very much for coming. Have a wonderful QCon, the rest of the QCon.