 All right. Hello. We're so happy to be here in person. My name is Veronica Lopez. And my name is Saddle for Garcia. And we're part of the big release engineering family for Kubernetes team. We're presenting today a little bit of the work that we have been doing for the past few years, but that obviously wouldn't be possible with the whole team. And yeah, also, Navarune will be joining us from India. It's the last moment he couldn't come. So I hope that you enjoy this weird hybrid format. And let us know if you have any questions. All right. So first up, we're going to play back Navarune's message. He's probably talked to. So we're really excited to see, to just let you know, the advances we've been implementing into the release and the Kubernetes release process. And we'll start with some past history about what we've been working on the past. Well, I am Navarune. I work at VMware as a software engineer focusing on the Kubernetes upstream project. I am a release manager associate in Kubernetes single release and currently serve as the branch management shadow for Kubernetes 1.2. Previously, I have been the Kubernetes 1.2.1 release team lead and have been in the Kubernetes releases since Kubernetes 1.17. In addition to my release duties, I contribute to areas like SIG architecture, APM machinery and contributor experience, and also serving the Kubernetes code of conduct committee. What I'll do right now is I'll start by giving you a brief history of the evolution of tooling which powers Kubernetes releases cycle after cycle. So until February last year, a lot of the tasks done in a release were performed using a tool called Anago and some other helpers written in Bash. It was amazing to see all of the heavy lifting done by code in Bash, but there are problems as well. Although the tooling work really good, really great, there were some downsides. A particularly big one was testing. It is comparably harder to test code written in Bash. Using this set of tools also involved some really nasty war stories while cutting releases. So what did the release engine in project two? Sasha, along with other members of SIG release, ventured into rewriting all of the tooling in code. Now, this is a humongous job, but all of that was not taken at once. The team took a really pragmatic approach into moving one component at a time into the new way of doing things. And as such, there was an iterative development process. For example, in this tracking issue, you can see what all micro tasks were done and what steps were taken in order to port the code into code. When you get the slides, you can go to the link and check all the work that was done and see what kind of challenges the team faced. And by November, 2020, the efforts finally converged into the removal of Enago as the release tooling. Here, you can see Sasha committing all changes. The final ones, which removes references of Enago from all scripts and documentation in the release process. Now, what exactly were the benefits of Crayol? Let me go through them. Number one, it's the ability to run tasks in parallel. We can run seamlessly all of these complex things in parallel, saving us time. We can also submit these jobs into Google Cloud Build so that we don't have to run it on our own machine. Resiliency. And not just a few places all across the port. The tooling now can handle situations like network errors. It handles them with appropriate amount of retrace. If things fail, we can start from the same place. Security. That aspect is also enhanced a lot. We can now reliably filter secrets from outputs. We can also use key management services if we want to store some secrets out there and then retrieve them in the pipeline. And at the same time, ensure that the logic works well. Since all of this tooling is written in code, we also get faster execution times. Remember one of the problems I told earlier, a particularly big one was we can't reliably test our code when it was written in bash. When the code was ported to code, all functionalities were mocked and can be reliably tested using mocked helpers. And last but not the least, the most important benefit is that Krell sets up a foundation for new features and improvements to existing features. In later parts of this talk, you will see what kind of things we are doing in terms of improving transparency in our release processes. And all of that can get built into Krell without much issues. It's very much customizable. It is now far easier to add new functionality and improve existing functionalities in the required in the release process. Now look at how things are in real life. It got so much easier to perform tasks like fast forwarding release branches. Now, I would like to pass on the pattern to Veronica and Adolfo, who will take you along the journey of what we did in the last year recently. What is on our plate now and what is our vision for the future? Again, for those who are late, Veronica Lopez, an release manager, recently promoted from associate. And well, I have been part of the release family for a while. We have internal jokes about it and I have been serving in different roles for like around seven releases already. And the change has been real. Change has happened before my eyes like in real time. I remember as part of other roles or other sub teams and the release team how sometimes the wrangling and fighting the tooling was more time consuming and labor intensive than the actual tasks that we were trying to achieve. But you know, like that is systems programming and that is Kubernetes programming. So yeah, so with part of the things that we're presenting today that Navarra already mentioned are like the evolution of how we are here now that is the part that I'm going to talk about and how has the past work and the present work set us for success, hopefully for the future and all the efforts that that will involve. Okay, so we have to go a little bit back to the, yeah, sorry, I cannot see. Okay, back in the day with all of this description that I told you of the tooling that we had, even though we had amazing teammates and who knew their job, like the palm of their hands, sometimes these processes didn't inspire a lot of confidence to like people outside. So little by little, we have been able to provide tools that have provided the community with more certainty and guarantees about the products that we released, you know, some examples of this are artifact verification, the container image digest checks, binary tagging, binary platform checks, and like we could give a talk about every single part of this, but if you have any questions, happy to talk about it outside. Another very important aspect of the evolution of our tooling is that we are now owners of the container image promoter. This is just like an example of how we have been adopting or inheriting certain parts of the release process, that in the past, we needed people, for example, for in this specific case from Google to like when we were done with a process, like we needed someone from Google to do the next step, you know? And that was, I mean, even as we love them, sometimes they were super busy and like just we have people in Europe, we have people in India, we have people in America, in the American continent, and just it was madness. So being able to now own this process and others as well, has given us more freedom to just move forward, you know? So one of the main things, personally speaking, wins of our team, and Stephen is here, so if you won't let me lie, I was stripping Bazel from all of this tooling. We love Bazel, we love Kubernetes, but probably not together, yeah, anyway. And yeah, no, but that like laughs and jokes aside, like it was a huge, huge, huge effort, because I mean, we complain a lot about this synergy of Bazel and Kubernetes, but truth is that it was the best tool at some point, because we couldn't find anything else and people were not going to reinvent the wheel. So being able to finally achieve a phase where we could get rid of it at different points and under release, it was amazing and a huge technical effort. And well, yeah, we have other elements that you can see here, but that I wouldn't like to repeat or that Adolfo will mention it more detail, but yeah, so all of this amazing work from all the team and like we explicitly are not mentioning names except for like a few because we are very aware that we will be missing like people. So this is a love letter to the release team because like it definitely wouldn't, all these huge efforts wouldn't exist without the coordination of a lot of people. So with this in mind, and the more exciting process of all of this story, I will give Adolfo a chance to tell you in a little bit more detail on what the efforts he has been leading, again with all the release engineering family, but it's exciting. Yeah, thanks very much, yeah. My name is Adolfo Garcia. I am a technical lead with Kubernetes Released for the release engineering project. And yeah, if you saw the history and evolution of the tooling that powers the Kubernetes releases, it basically are, it constitutes steps to be where we want to be right now. So this is like the beginning of a platform where we now have in place the necessary improvements and enhancements to our tooling where we now can start working on improvements that not only constitute the releasing of artifacts, but to actually build on top of them. And if you recognize some of the items listed on your screen, these are part of an overarching effort to basically secure the software supply chain where Kubernetes is involved. Now Kubernetes as most of other open source projects is not the complete software supply chain by itself. It consumes and gets consumed in either open downstream of the artifacts we released. So I want to break this effort in three just to give you a quick update on what we are working on right now. So our first aim is to sign whatever of our artifacts we can. The most artifacts and the actual, every file and container image that gets consumed by someone downstream, we want to sign that. We want to give the community the guarantees of authentication and non-repudiation of the artifacts that we are producing. And this involves several changes that we need to put in place. But now, as Veronica was saying, now that we have stripped and re-owned the container image promoter, and it's now in a ready state too so that we can start iterating over it. This project, by the way, is not one that originated within state release. So it originated, I think it was in Kate's infra. Yeah, and then now that Kate's infra evolved into a proper, from a work group to a SIG, we ended up having the owning the project. And this will allow us to implement the signing code and necessary features to have sign container images at some point. And the other one we've been working about is the Kubernetes Bill of Materials, that effort's mainly done. I mean, it's still, of course, like every evolving software project, it's still open to suggestions, comments, whatever. But you can already use and consume the Kubernetes Bill of Materials, which goes out with every release. Then if you, okay. So, let me see, I can see the, all right. And yeah, so we produced two Bill of Materials with each release, one describing the artifacts and the other one describing the source code. And these are the URLs for the Kubernetes 122 S-bombs, which was the first release that actually shaped with a Bill of Materials describing its artifacts. And if you're interested in the code that powers all of this and maybe interested in using it for your own projects, we're gonna be having a deeper dive session on Friday, where you can hear more about the history of this project and the tools available for you to use. Then the final one is we are trying to push towards salsa compliance in our part. So, as I said before, Kubernetes is not the whole supply chain software supply chain, but certainly the most important artifacts happen in the links that fall under our responsibility. So, we are trying to push and secure them as soon as we can so that we can provide guarantees to the community. So, salsa, we're not going to deep dive into what it does. It's a framework to ensure some practices and requirements about information and human aspects of your software releases. And we're trying to get to the first level of salsa, which is basically providing documentation and information about what goes on inside of the release chain, of the release process. We, most of the practices to achieve salsa were inherited since the early days from the bash script that we showcased earlier. So, this is not a new thing. Simply, some of the security features that salsa requires came already, were already sent from the beginning, I think. So, we are just trying to match up and have like a benchmark to see where, how are we, how are we going to to match with the salsa level one requirements. Of course, we would like to push forward to further levels, but we'll see that it requires a little bit more work to achieve there. So, we currently have an open tracking issue where we are gathering ideas. We are planning, of course, an opening details of the implementation, two in CEPs so that the whole community can weigh in. But what I would like to assure you is that we, based on the whole majority of things that we already had in place, and work that has been going on in the last month or so, we are already, we already have goal to reach salsa level one. And as a prototype, and we are going to be testing this in the coming releases, which is just providing the basic attestation files of provenance of the artifact. So, provenance in the salsa speak means that you should be able to take one of the artifacts and trace it all the way back to the release process to find exactly where it was produced. So, we already have that code in place which was the big missing piece to achieve salsa compliance as a level one. And we are going to be filing a CEP to see if we can push it all the way towards the higher salsa levels. Now, the thing about reaching higher salsa levels is that it requires more changes than to reach salsa level one. The first, the most important thing is that we have to, if we were salsa level four, some people claim that it's achievable for some organizations. Could be, I don't know if it's possible for the Kubernetes community to achieve salsa level four. Certainly, we can and we should try to achieve at least two or three, which means that all of the information that travels through the release process from the beginning to end has to be signed, properly authenticated, and so on. There are lots of aspects of this. I think the most challenging right now is the human part of the problem because we have to modify, if we were to achieve higher levels of salsa compliance for releases, we would have to actually go and modify some of the practices so the way contributors authorize changes to the code, the code reviews, and things like that. So I think the main message here is that we need to have the conversation. We need to have some discussions about how are we going to handle the Kubernetes release process within a world that is constantly demanding more and more security around the software supply chain. We're going to be opening up CAPS and most of their alternatives for all of the implementations from the attestation files to the signing of the artifacts to the way we store them inside of the container registries. So we want to hear from everyone. We want to hear from everyone and we want to collectively reach and find the best solutions for each of these problems. And salsa is simply, let's say, like a bunch of checkboxes that guide us to see if we are complying with some of the things that are expected from us. So if you are going to have a takeaway from this talk is we need involvement, we need opinions, and we need more eyes on the code and the things that we're doing because while we are responsible for cutting the releases and the things that you and your customers consume, it's the people. It's the community that drives everything behind it and we don't want to appear like we're pushing into one direction without talking to everyone. So please come and weigh in on the issues on the CAPS and let us know if you have any suggestions and we are, of course, happy to hear from you. So... What I want to say about that, sorry, is that it has been very interesting for me to see that all of these formalities of salsa and the compliance and the provenance and the security obviously inspired by the recent attacks in the industry and not necessarily on the Kubernetes side but in systems in general. It has been really interesting for me to see that this evolution was very organic. Like, as I said, from a very specific point of view of serving at the release team and different small roles but that did different parts of the puzzle. The achievements, the collection of this work and achievements has been like tooling that we have needed for our own work to do our own job in a faster and more efficient way. So like when we achieved the bill of materials and now are heading towards this, it really feels like a natural next step. Like we have been building like block after block after block. And yeah, so in obviously what Adolfo says is very important like involving all the community. But and that is important, but also we won't be able to move forward. Like especially for salsa three or four and or four if it's achievable and we'll see. But even considering if it's achievable, like those processes cannot happen. Like and let's say if I'm building or cutting a release like in my computer and that's it. We would need like processes in place but like timed and scripted and automated and verified and everything like that. And that is definitely not possible without strict coordination from the community. And a lot of resources of course. And I guess just one final remark for me is that things in the software supply security chain the security side of the software supply chains is evolving very rapidly. And if you have experience, if you have, if you want to help the project in this rapidly evolving space, please come join us. We really need your help. So we are, we try to keep things as current as we can but we are a very small team of volunteers working on this. So please come and join. And from either just making your voice heard or actually submitting PRs to help a release process. You are welcome. You are encouraged to join. You are absolutely, we absolutely need you. And yeah, so if you, I think we could open for some questions if there are any. And thanks again for coming and listening to our, to our experiences. We added a bunch of references in case you want to see our slides with all the work that it's like a lot and super interesting both the code and the documentation. So please feel free to look around and share your opinions with us. So that's it. That's the talk. Thank you very much. Are we taking questions? Yeah, if you have some, we reserve a couple of minutes if anyone has any questions for. Yeah. I don't know. We talked about this a little bit yesterday in person. And this is probably more a question for CNCF as a whole than just for SIG release. And that is all of the work that you've done for SBOM for Kubernetes. How can that be used by other CNCF projects? Right. So this has been a common request from, from some other projects. We tried to distill like general use utility. So if everything we do. So if you want to run your own build of materials for your project, this case we test, you can download our tool and run it. And we have one issue right now published where the community, specifically the K-Native folks approached us and asked us to produce our utilities in a more relative run state. So you can right now download the code and compile it and have the utilities. But we are also planning on releasing the utilities as part of pieces that you can plug together into your CSVD pipeline to generate a little bit of things. Right now you can download the release notes application. You can reuse some of the code that we used to launch the builds inside of Google Cloud Build and the build of materials tool as well. And some of the provenance attestations, attestation files that we are currently producing to reach Salsa One are going to be built into the build of materials tool soon so that you, whenever you run the build of materials for your project, you can actually output to several file the predicate needed to have the provenance data for your project and to make you closer to reaching Salsa One. And yeah, so yesterday we were discussing about if there was a way that other projects could come and reach out to us to have some either help or pointers of our software to run the releases. Absolutely, we're open. Everything is open source. Everything tries to be as Kubernetes agnostic as possible. We even have now a new repository where we're trying to make sure that everything that lives in that repositories has zero Kubernetes-specific content. So if you go to Kubernetes Releasing Uniting, it's still Kubernetes-specific, but we have another repository where I'll share the links later in the presentation where everything there is ready to run for any project. So. Yeah, a quick example of this. Like without going super deep into the process, someone recently told us that the tool as it is right now has been very useful for some teams just to get the list of container images in a release and then to use them for their own purposes in their own companies. But it's already being useful for people without the fully-fledged features. So definitely we love knowing cases like those to see where people would be more interested to port it to their own, for their own benefit without needing to have it as a complete project, for example. Any other questions? I think we don't have time now. So yeah, if you have questions, tabs, whatever, please contact us. The thoughts are handles and come join the release engineering meetings every Tuesday, every other Tuesday, and we happy to have you there. Yeah, now more than ever, we need people from different backgrounds and engineering side, like not only people from release engineering backgrounds, but security would be greatly helpful. So yeah, join us. Thanks.