 I'm Matt Butcher. I'm one of the Helm Corps Maintainers. I'm super excited because we have a bunch of Helm Corps Maintainers on here today. Karen, Bridget, Martin, and I will all be talking a little bit and then during the workshop portion you'll get to meet some other Helm Corps Maintainers. So here's the here's the agenda. All the time zones here on Pacific Time, which I am not so I apologize if I make any mistakes here. So we'll kick off now, do this brief welcome deck. Then I'm gonna dive in and talk for about an hour about about Helm, what's in Helm 3, how Helm 3 changed. I'll give you some history. I'll tell you some hopefully amusing stories and give you kind of the conceptual grounding for the next talk. After I'm done, Martin, one of the Corps Maintainers and the person who is instrumental in handling the and building the Helm 2-3 migration plug-in will dive in with a really practical talk about how that works and how you do migrations with Helm and we'll walk through kind of the the entire process start to finish and then we'll take a short break after that and then the part I'm really excited about is we'll do a hands-on workshop. We've been working to prepare some curriculum to basically make it easy to walk through the process of doing a Helm 2-3 migration. And we've got a whole bunch of people standing by to help out. We'll break into some breakout rooms and we'll we'll do that all together. Really looking forward to that. So that's our that's our agenda for the day. Awesome. So next, just a few housekeeping items before we get started. Everyone should be muted upon entry since this is a regular Zoom meeting, we want to give our speakers the time and attention to present first. And so with that, if you do have questions, please go ahead and drop them into chat. I and I'll be going through them and we'll we'll do them at the end of the presentation. So again, just drop your questions into chat and we'll do them at the end. And then please make sure you adhere to the CNCF code of conduct. This is an official CNCF event, so just be respectful to everyone here. And then lastly, just kind of a huge thank you to all the people who've helped put on this workshop. So, you know, the people listed are helping with the hands-off workshop or doing the presentations or have just contributed to the event in some way and also a huge Thank you to CNCF for their support with our events. With that, I guess, Nat, do you want to get started a little bit early? Sure. Okay. Well, I'd like to introduce Hapuchur, a principal and software engineer at Microsoft. And he will be talking about what is in Helm 30. Okay, slides up. Okay, Karen. All right. Okay, so I'm really excited to be here today. What we're going to what we're really focusing on today is the differences between Helm 2 and Helm 3 and how to migrate from one to the other. So in my session, I'm going to talk a lot about how Helm works and kind of what the design decisions were. I'm going to talk about what some of our mistakes were in earlier versions of Helm, how we tried to fix those in Helm 3 and really try and lay the foundations for what Martin is going to talk about. So that's what I'm going to talk about very focused on sort of the way that things work and the abstract notions that we have guiding Helm. Then I'm going to hand off to Martin and Martin is going to take a much more practical approach and really focus on what it means to upgrade how things work, what to look out for. So that's how we'll break things out in these first couple of sessions today. So the first thing I want to talk about is November 13, 2020. For us, November 13, 2020 is a very important date. So I would encourage you to keep in mind the date November 13, 2020. I'll just keep repeating that date over and over. What happens on November 13, 2020? Well, that's just a little bit. That's about a year after Helm 3 came out. And this happens to be the day that Helm 2 becomes un-maintained. So what does that mean? Well, there are two major components of this that you should understand. The first one is that the piece of software that we call Helm and Tiller will no longer receive any updates at that point. So currently we are in the RC phase of Helm 2.17.0 which means it'll be released within a few days. This is our last Helm release on the 2.0 branch, our last planned release. So for a long time, we've been doing only security fixes on Helm 2. But all good things have to come to an end. And in this case, in order to devote full attention to Helm 3 and start the planning for Helm 4, we need to stop supporting Helm 2. So 2.17.0 will be the last release. There will be no additional work on Helm 2 after that. There will be no updates after November 13, 2020, not even security updates. So that's one part. The second one, which has been, which surprises people a little bit more, is that on November 13, the stable and incubator chart repositories will also stop receiving updates. Right now, they're in maintenance only mode. If you go look at github.com slash Helm slash charts, you'll see that people are still making fixes to things, but they're small fixes, right? We're not accepting new charts, I believe. But the plan is that those repositories will no longer receive any updates. And from November 13 onward, they will be marked as archived and no more security updates, no more new charts being added. Furthermore, as of November 13, 2020, the Google cloud storage bucket that holds a bunch of those charts will no longer be available. So we're working on migrating everything over to charts.helm.sh, which will be the new endpoint for your Helm repositories, but those will just be mere static archives of the Helm incubator and stable repositories. So very quickly then, what does that mean as far as how you get charts now? Here's kind of the deal. When we first started this model for Helm 2, you know, back in 2015-2016, we never expected Helm to grow as quickly and as expensively as it has. And so the process we had for accepting charts was very editorially based, where we had a group of people who managed the stable and incubator chart repositories and examined every single chart that came up. Well, it has become an un-maintainable situation. There are just too many good charts out there, and it's too stressful to try and be the sole gatekeeper for the public Helm chart world. So over the last year, and you probably noticed this, we have been starting to spin off separate chart repositories, and we've been switching to a model closer to NPM or CPAN or any of those sort of distributed package manager systems. So now we have Artifact Hub, which is an official CNCF project. If you are looking for charts, you can go to Artifact Hub, search for something in there, and it will point you toward, it'll give you instructions on which repository to configure and which charts to fetch from there. And so behind the scenes, Artifact Hub is sort of aggregating from dozens and dozens or probably hundreds at this point of package archives for Helm charts and surfacing those in a nice easy to use search fashion. In Helm 3, we actually have the Helm search client wired up to talk to Artifact Hub. So if you do Helm search Hub WordPress, it goes to Artifact Hub, finds all the WordPress charts on Artifact Hub and prints them out in your command line terminal. So there's a lot of integration between these two. But going forward, stable and incubator as centralized chart repositories will be gone and this more distributed approach where people, organizations and individuals self publish their charts. That's the way we'll do things going forward. So November 13th is a big deal because we're making all of those changes will finally hit their very last phase on November 13th. In these last couple of weeks before then you can still expect security updates and very, very minor changes on the charts repository. But if you don't upgrade by November 13, 2020, you will be on your own because the Helm maintainers will officially stop all of our development efforts. So that's something to keep in mind. And that's actually the real sort of driving force for why we wanted to do this webinar right now. So that you would still have ample time to finish up these migrations and so that we could help you, you know, sort of proactively prepare for this and find any any potential gotchas right now up front when core maintainers can be here and help you. Okay, so that's kind of the preamble. Alright, so what I'm going to cover in this particular talk is I'm going to start by giving you sort of the background story for where Helm came from, how it developed along the the particular path that it did. And and and point out some of the hiccups along the way, one of our really bad assumptions that we made that we had to correct for with Helm three after that I'm going to talk particularly about the the the brief life and tragic death of tiller and and what that means in the context of Helm two and Helm three. And then I'm going to spend some time talking about releases. So this might sound like a technical detail, but it's actually the most important concept to understand when we're doing these migrations. And then finally, I'm going to wrap up by talking about the differences between charts and Helm two and and in Helm three and then I'm going to pass it on to Martin, who's going to take that practical approach and actually walk through stuff with us. So Where did Helm come from. Maybe the story has been told too many times, but I'm going to tell it again anyway because it's Helm's birthday and it's fun for me to tell the story. Five years ago. We had a I was I was working for a company called Deus. It had just recently been purchased by Engine Yard. And we have been doing some R&D on this brand new platform that was called Kubernetes. I think at the time Kubernetes was maybe at version 1.1 maybe 1.2 by that point. And so we had been doing all kinds of crazy stuff. You know, all kinds of you know experiments and and you know edge leaning things trying to figure out What we could do with Kubernetes and we had become convinced that Kubernetes was going to be the next big thing. So we decided to have during our company all hands meeting everybody flew into Boulder, Colorado, and we decided to announce there that we were going to pivot a lot of our development efforts to Kubernetes. So I was asked to present a Kubernetes in a nutshell kind of thing. To explain to not just the other developers that were in my group or in the day as part of engineering, but to all of Engine Yard, including marketing and communications and the executive team and everything. Explain in 30 minutes what Kubernetes was. Now I'm sure some of you are laughing already because we all know how hard it is to explain to anybody what Kubernetes is and what it does. So, so, and I drew the after lunch slot too. So I knew I was going to be doing this right after lunch. So came back and Arranged my kids stuffed animals around the house and took pictures of them and giraffe and a gopher and stuff like that and and wrote this silly presentation silly PowerPoint presentation called the Illustrated Children's Guide to Kubernetes and Karen and I later partnered up and and did the little book based on that. So Later that day we decided to kick off a hackathon project at stake was a $75 Amazon gift card. And so my team, Jack Francis and Remus and I Jack Remus and I We decided that we wanted to try something Kubernetes oriented because we had just kind of announced that was going to be our big shift. So we We came up with this idea to do a package manager for Kubernetes. We called it Kate's Place K8S. We thought it was really cute. And and we spent the next two days, you know, at every every possible moment between sessions and late at night and stuff like that, hacking together this little demo of a package manager for Kubernetes. So we won the $75 gift card. I know that's all of you really. Yeah, we won it. We split it three ways. I think I spent mine on food. Unsurprised probably coffee knowing me. And then we thought that was the end of it. Well, the next day, which was a Friday, I got into the office and the phone rang. Yes, actual actual phone rang And I picked it up and it was the CEO and the CTO and they said, Hey, we were talking last night and we think that this package manager for Kubernetes thing might be a good idea. So we'd like you and and Remus and Jack to keep going on that will will give you a couple more engineers and we'll see if it see if it goes anywhere. And I went, Oh, great. That sounds great. That'd be a lot of fun. It's just one thing we'd kind of like you to change the name or not. We think Kate's place might not be the right, you know, The right name port. So Jack and I sat down with a nautical dictionary and flipped through it tossing words back and forth until until until Jack said, Hey, wait, how about helm and And I went, Oh, that's great. And then we came up with charts as the metaphor for what the packages would look like. And that was the birth of helm. So so Adam Reese and Michelle Neurali and I really spent the next several weeks just hammering away on code. It was, you know, Go and that was a new language for for some of us. And and we were just kind of working and working Really exciting various engineers would drop in and help out and then drop out and on the very first kubecon Which was in San Francisco. We announced helm Which was I think helm 0.1 or 0.2 or something like that. And and we showed it off now the first kubecon was probably smaller than this Than this workshop today. I don't remember how many people it were But I remember being able to make eye contact with pretty much everybody in the audience So it was pretty small and we had a great time and it was a lot of fun But we didn't really expect it to go anywhere So that was about I think November maybe early December of 2015th well January google called us and said hey, you want to fly out to google and in Seattle and chat with us about helm. We've been working on a southern project We think the two can collaborate So adam and I and and gave fly out to google in seattle meet with some engineers there And they proposed that we merge together helm and they're They're About to be released or maybe it would had already been released system called kubernetes deployment manager So we started working on that with them That became helm 2 the basis for helm 2 so helm 1 never really saw the light of day I think the helm classic which is what we call it now maybe made it up to 0.13 But because google had already released what they called their 1.0 release of deployment manager We had to skip to the next number and that's why we went from helm classic to helm 2 But never really released the helm 1 So we worked together on that and then released the new version of helm And we were shocked when people just started jumping in and contributing code I think we have now hundreds of companies and thousands of developers who have contributed code to helm And probably an order of magnitude higher than that who have contributed charts and developed charts and stuff like that It's just been phenomenal. It's been Fabulous, but as we went we realized that some of the assumptions we made back in the kubernetes 1.2 to 1.3 or 4 Times we're incorrect assumptions And what we really wanted to do is we really wanted to follow semantic versioning and semantic versioning says, you know Version numbers are major dot minor dot patch So if you fix a bug you increment the patch release If you add a feature that doesn't break anything you increment the minor release But anytime you need to make a breaking change you have to increment the major release number So we knew that as soon as we broke something we'd have to go to helm 3 So we deferred for a really long time and collected all of our breaking changes up And then started this big development effort to do helm 3 and then when we released helm 3 We knew it had a lot of big changes. They were going to necessitate things like today's workshop, right? But and then with helm 3, you know, again, we're we're following the same pattern We're not going to introduce any breaking changes during the lifespan of helm 3 helm 4 will be the first release where we break anything But what this meant for us practically during this sort of history of helm thing Is that we we did have to kind of buffer up a lot of changes and wait for quite a while and then You know adam who ran the the helm 3 development cycle said, okay Here we go Everybody get ready and we started doing patch after patch after patch breaking change after breaking change in the helm 3 branch And then eventually got that stable and got that released So that's kind of the story of how we got from point a to point b You'll hear me mentioned here and there helm 4 helm 4 is really just an idea at this point. There's no ongoing development for helm 4 Um, so so don't you know, don't worry that you're going to migrate to helm 3 tomorrow and then have to migrate to helm It's still probably a year to two years out But we are trying to open up the opportunity now for the developers who have been Currently maintaining helm 2 and helm 3 to be able to work on helm 3 and start buffering stuff up for that helm 4 bucket All right, so that should give you a good background of Of where we've been with helm and and what it's done But I want to pivot now and talk for a little while about tiller Uh, and I so I like to call this section the life and death of tiller and overly dramatic telling of the story Um, so it all starts in that January meeting at google Uh, so we were at the google campus having great time. Um, we took a break at lunchtime went to the cafeteria All together we're standing in line, you know in the in the uh, the the You know scoop food on your plate section of the google cafeteria And just kind of chatting and i'm chatting with one of the engineers there and he says, you know, I really like working on kubernetes This has been great, but i'm a little concerned because uh, you know, some people seem to think that You know, we're going to be talking about thousand node kubernetes clusters with lots of different development teams working together But you know, look at kubernetes. We're we're not going to be able to reach that He said, you know kubernetes is never going to be a multi-tenant system It's always going to be the case that a development team stands up their own kubernetes cluster We have three maybe five tops maybe 15 nodes on this thing And and you know each team has their own And and you know, so that became the topic of lunch and so we discussed this thing the whole the whole of lunch I think, uh, you know, how much can you stretch kubernetes? Was it foolish to try and do thousand node clusters with no role-based access control and and no real strong notion of authentication? How are we going to do any of this stuff anyway? Um, and kind of at the end of the lunch We had kind of basically convinced ourselves that yes indeed kubernetes was going to be a single tenant system and small teams would work on it And every team would have their own cluster And that actually became Sort of a maybe maybe from that point on unspoken But sort of a driving principle for how we developed helm from then on out now. Keep in mind at that point There really weren't any r-back mechanisms in kubernetes and admission controllers were not a thing uh Deployments weren't even a thing at that point crd's were not a thing at that point So to some to some extent any kind anytime you jump in early you make some guesses in this case We were not we were not Minimally wrong. We were not kind of wrong. We were actually really really wrong, right? I routinely see clusters with hundreds or thousands of nodes and with dozens of development teams or even hundreds of Different teams with hundreds of different applications running inside of a single cluster And it works and it works really well and it's turned out to be one of kubernetes's strengths But keep in mind this little assumption We made that might not have been right as I start to walk through the architecture that we're going to talk about because that one that one Flawed judgment. I think changed the architecture that we did And is what ultimately caused the biggest change from helm 2 to helm 3 as we corrected for that So this is how helm classic worked helm classic Basically was a manifest uploader if we were really honest. I mean package manager is a very generous way to say what what helm classic did um It for the most part had a chart dot yaml and a bunch of static Yaml files that were just kubernetes resources and it could bundle them up together Send them all to the kubernetes api server at the same time wait for the api server to say, okay, you're good and then Report to the user that it was that it was working That was about all helm classic did We relied heavily on conventions for labels and things like that for people to be able to figure out what they just installed And we didn't really have a strong upgrade story or a strong rollback story or even a team management story for helm classic But again, it was 0.13. So, uh, you know, it's fairly early on One of the things that a lot of people have not taken a look at is uh, is what deployment manager did and how So we were taking that model that helm classic model and combining it with the deployment manager model And that's where helm 2 came from the deployment manager model And i'm I gotta be honest I don't actually remember the model and all the names correctly. So i've kind of fudged some of the names here I think the client was named kdm Uh, and I I I can't remember what we called packages and deployment manager They might have been resource bundles or templates or something like that, but I just left chart here for the sake of continuity But essentially the way it worked was the client was would upload a package To a server that was running inside of a kubernetes cluster And that server was called deployment manager. It was a json api server We'd send data up to it. We'd send, you know, a a chart. We'll keep calling it a chart And deployment manager would unpack this chart and it would send the templates over a network connection to another service running inside of the cluster That was called expandy bird Now expandy bird was sort of like a multiple template renderer. It had it could render python, uh, code it could render, um Some template languages one of the one of the python template languages I forget which one it was. Uh, it could it had experimental support for json it Uh, but it was basically like its job was to expand templates and then return back yaml And then dm would store the yaml in mongo db and then upload a copy of the yaml manifest to kubernetes Wait for them to come back successful and then return to the client that it deployed So this should look a little bit familiar because uh parts of this, you know made their way into helm um Side note here on occasion people ask us, uh to include support for more template languages in helm And the main reason we did we don't and we've reconsidered this multiple times But we still kind of stand fast on this one is because our experiences expandy bird indicated that it would get very very complicated because what we were experiencing was a single render right a single chart, uh Might pull in, you know some some python code some, you know, fairly trivial templates and some json it And then it would render something and it would stick it in the cluster And you'd go and look at the cluster and go well, where did this thing come from and then you'd have to sort through a combination of python code and uh templates curly braces templates and json it and try and figure out how they interacted together In such a way that they produced that thing And then, you know, we were talking about adding more languages We were talking about adding go templates and other languages And we realized that this was going to be such a mess that it would be That a sizable chart could be practically impossible to debug So one of the early decisions we made Was that we we did not think it was right by the user or the chart developer for us to support multiple template languages Even though that meant that Some people would have to learn a new template language It prevented the case where all people would have to learn nine or ten different languages in order to just Read a chart or look at their install and know it happened But uh, so we took this deployment manager model and that earlier helm classic model and we sort of combined them And we decided To reduce the complexity and deployment manager and increase the complexity from helm classic and kind of meet in the middle So what was the dm server got replaced with what was tiller? Where tiller's job was to sit as a g rpc server the helm client remained relatively straightforward It would basically take charts and values and upload them to tiller tiller would then render the templates itself Perform any additional operations load that stuff into the kubernetes api server store a release record of that release inside of kubernetes And then return back to the user and say okay. I deployed your thing And this worked well because it introduced for us over helm classic the ability to do upgrades and then roll backs But it was still simpler than the deployment manager model Which uh, which was definitely on the on the complex side and difficult to operate so This seemed to work well And it met many of our obligations because we had one single tiller that was almost like a root user and And since we were working for you know single tenancy clusters It was a great solution for us and we really liked that But over time we realized and i'll talk about this, you know coming up soon that tiller was not holding up to expectation In fact where we had assumed kubernetes would go one direction kubernetes really turned a very different direction and because of that we had to adjust course and We again waited for a while But when we did finally do this with helm three what we decided was we needed to remove tiller That tiller was actually had become more of a hindrance than a help So really in effect we rolled back almost all the way to the helm classic model But we there's one notable change and this is the change that we will spend the the A good chunk of our time today talking about and that is that we kept the notion of storing releases inside of kubernetes We just took tiller out of the equation and tiller was no longer the sole authority of those releases But but we kept the object the release objects there We had to tune them up and that's what causes this whole migration thing is that we made some changes to the release object So we'll talk about that in just a moment, but before we're going there. I wanted to talk about why we removed tiller there have been speculations are a lot of loud opinions and stuff like that, but In all honesty it boils down to one thing It boiled down to authentication and authorization So people will say oh, well, you know tiller was a security nightmare or tiller was a stability nightmare Or I had to run nine of these things in every cluster or 64 of these in every cluster And those are the superficial problems But the core cause of all of this was actually we hit a boundary with authentication and authorization And we could not figure out a way to solve it Excuse me So what was what was the boundary? Well, here's how it goes. Um in helm two I'm I'm use I'm you know, I'm matt right my username is matt b martin is martin h right So we're both working together on a cluster hypothetically And uh, and I log in to kubernetes using kubectl, right? What does it do? It authenticates me against the kubernetes api and then grants me permissions based on what kubernetes knows about my roles on the cluster Now with tiller I could use those credentials to tell kubernetes that I Was allowed to proxy through the kubernetes api server into tiller But once I got to tiller I could do whatever tiller could do right So if tiller had cluster wide access to do anything Then I could upload a chart and say hey put this install this wordpress thing into dev Which I have permissions to but if tiller had permissions to install into martin's staging environment Then I could say hey put this in martin's staging environment and tiller could do that Even though I personally would not have had permissions to log in directly to martin's staging environment Um, that's not necessarily a it's not a bad feature if you're working on a small team of users And and the trust model is that everybody on the team is trusted But as soon as you get into multi tenancy situations, that's a bad security model to have right? Uh, and so we tried to fix it And what we wanted to do was to say okay I want to tell tiller. Hey, I'm I'm the kubernetes user matt b and have tiller say okay Well, let me see what your kubernetes responsibilities and roles and permissions are and then I will do this act on on your behalf So you won't be able to install it into martin's staging Namespace because you don't have permissions there. You'll only be able to install it into your area um We could not find a way to do this. We tried all kinds of things We had one branch that lived on the helm two For I think six months where we were trying to figure out, you know ways to finagle around and embed stuff in g rpc headers That would authorize users for things There was no way for a pod to contact the kubernetes api on behalf of another user It was always only itself. Uh, and we we tried and tried we tried Probably near a dozen different ways some of which were you know Ridiculous at face value and others of which, you know, we worked and worked and worked and then ultimately decided they couldn't Couldn't function the way we needed. So we ended up with a choice We had two different models to make this work model one we could take tiller and we could write our own authentication and authorization system and uh users could set up their own permissions and we could have a helm tiller administrator who managed the user and permissions database And that administrator would also manage the permissions on the kubernetes cluster and would have to manually, you know, keep those in sync Or we could remove tiller all together and use kubernetes itself for the authentication Authorization and have it all be negotiated between the client and and the kubernetes api server So the first one obviously would have been very very complicated, right? We would have introduced new roles, right? As you need a tiller administrator It would have introduced massive amounts of new code and it would have introduced the problem that we would have had to manually synchronize the permissions you had in kubernetes with the permissions tiller gave you and We decided that was a non-starter And so we began the process of tearing out tiller So in the new model helm three connects to the api server as matt b or as martin h And when I connect to the kubernetes server the kubernetes server says, okay You only have access to your dev namespace martin has access to staging and dev But you don't so anytime you try and install something it can only go into dev And you can only install all these particular things So use the kubernetes arback system in helm three directly by contacting and negotiating that directly with the kubernetes api server instead of having tiller as a as a mid person that had to Receive everything and then act on behalf of someone else So i'm acting on my own behalf instead of relying on tiller and that's how the new Authentication and authorization model works and it's proven to be I think the right step forward I think we did the right thing in adjusting course kubernetes is indeed a multi-tenant environment and In order to facilitate that we needed to integrate as fully and completely with the kubernetes way of doing things that we could Now unfortunately the upshot of a lot of that any kind of pivot of that size is that It's going to require some strain as we migrate from one to the other and that's you know What we're here today to talk about right and nowhere did this issue show up more clearly than in the concept of a release So releases have tended have turned out to be the biggest impact on the helm two to three migration Because we had to change the way this system worked in order to follow the the new tiller structure I think the introduction of releases which came about in helm two wasn't present in either deployment manager or helm classic I think that was probably the single best design outcome of helm So what's a release well a release is a record of an installation that's a living record that uh that updates each time Each time an installation is updated. So here. Let me give you a practical example so martin and I are working on the same cluster and We both want to install our we're both working on separate projects But we both want to use say dripple to have our internal website for these things So I want my dripple instance martin wants his dripple instance Uh, so we should each be able to install separate versions of dripple his for his stuff mine for my stuff And then when I upgrade mine it shouldn't impact his and when he changes the values configuration on his it shouldn't impact mine So we need to track two separate installations of the same software And then as I upgrade mine we need to track how my version was upgraded and what changes happened between the last time and this time That's how the release system in helm works That's the problem. It's designed to solve tracking those releases over those installations over time So I create a new release when I install it I create a v2 release when I upgrade it the next time then a v3 when I upgrade it after that And I can roll back to v2 and things like that um The way that it worked in helm 2 was that I would send for my helm client. I would send the chart and the uh and my values files up to tiller But tiller would manage all the release stuff for me The client could be basically blissfully unaware of how tiller was managing all of the releases now behind the scenes tiller was storing these things inside of the kubernetes cluster And was managing the versioning and checking for race conditions and doing all kinds of things like that Uh, but when we did helm 3 we had to figure out a way to go to this architecture where helm was managing the releases from the client side Without anything necessarily running inside of the server to manage the state for us So that was a little bit tricky And the number one problem we wanted to solve is the problem that we we call the friday release problem And this is the one we needed to make sure that we nailed if we were going to make helm 3 work I'm sure so this is just to be clear definitely not based on personal experience. I've definitely never had this happen to me I'm sure all of you are saying the same thing. You have never had this particular incident occur in your organization, but imagine hypothetically we had this case where you know Your your your uh, colleague, uh, you know, sam has is getting ready for a vacation now Sam's got some major deadlines. Sam's got to get this thing out the door right away Works and works and works and works on it. It's you know getting behind, but they think they can get it out just in time It's 456 on a friday They got to leave work and make their way to the airport and catch their plane They pushed that release out at 456 they take off and they're gone for a couple of weeks So you're there, you know wrapping up for the end of the day looking forward to the weekend and that pizza party You're going to this evening when all of a sudden The cluster, you know catches fire the release that sam did uh broke And now you've got to fix it because sam has gone and incommunicado for some amount of time And so you're looking at everything going well, what did sam do? I need to figure out what broke how it broke and how I can fix it So with helm classic There was no workflow at all for this scenario, right? Basically, you'd have to find sam's workstation Log in and see what sam did from sam's point of view because there was no record whatsoever In helm 2 then, you know, you could contact tiller and say okay show me what values were uploaded Show me what what chart was used, you know, show me what the rendered version of the values was We needed to preserve that particular behavior But we needed to do it without tiller which meant our release record had to be robust enough That it could handle that particular scenario, but it had to be able to do it Without introducing the race condition that happens when say Martin and I are working on the same particular release And we we can't risk that if we accidentally both release very close to each other We corrupt the kubernetes installation or worse we corrupt the release record So we have kind of the race condition thing going on at one side On one side while we were dealing with the friday release problem on the other And helm 3 had to solve all of that while Recognizing that there was no central authority no tiller. It was in the middle saying okay Hang on martin. Matt's release is going out. Okay. Matt's release is out your turn martin and stuff like that So that necessitated making a number of changes to the release To the release object, but along with that like I said, we buffered up a lot of changes, right? We decided to make some other changes to the way release is worse so that we could correct a number of other things we did So in helm 2 releases were all stored together So, uh, you know martin installs into his staging thing and I install into my dev Dev namespace, but both of our release records go in the same particular the same place inside of kubernetes By default we were putting them in kubes system. A lot of people are like, why would you have done that? Well, again, let's rewind to the core assumption. We made that turned out to be false It was we thought we were building a single tenant system So it totally made sense to put stuff in kubes system because kubes kubes system was where you put stuff you didn't want users to look at and We we never thought about the fact that when you had a hundred different development teams all using the same thing You could end up with tens of thousands of of release records inside of kubes system Uh, so again, you know, it was a mistake made based on our faulty assumption But uh, even even though you can change where they're stored when you change them It still is the case that tiller would store all the release records together regardless of where what namespace they were put into Um, that introduced a number of management problems particularly when we took tiller out of the equation because tiller was no longer We didn't want to have to give everybody Access to the same namespace so that they could store all of their releases in there and then assume that these development teams Wouldn't accidentally stomp on each other's releases So we needed to fix that The the the helm 2 era fix was to just tell everybody to run lots and lots of instances of tiller But that seemed like not a good solution in the long run. And so in helm 3 we just changed it so in helm 3 now The release records are stored side by side with the releases that they describe So when I install wordpress into my dev namespace The release records instead of being written to kube system or somewhere else the release records are also written into the same namespace They're also written into the dev namespace And when martin installs his version into his staging environment His release records get written into the staging namespace Now this has turned out to actually have some great benefits to it. First of all, um, you know You can use our backs and things like that to limit access So we didn't worry about necessarily polluting that that namespace with things that uh, the users could Change around when they shouldn't uh, but but moreover Say you say you You know, you get you get that friday situation, right? And um And you have to drop into a namespace you're unfamiliar with and you do, you know kube ctl get pods and you see a Zillion things running and you do kube ctl get secrets and you see another bunch of things running And you're going what are all these things and how do they tie together? Well, now you can just point helm three at that namespace and say Tell me all the things that were released into this namespace, you know Helm ls this namespace and it'll say, okay. Well, you know, here's here's a drupal instance And here's a chart museum instance and here's you know, three ha proxy instances. That's what you're seeing in this cluster So it really localized the management of your releases into just one namespace Which again really fits the multi-tenant model that we've seen kubernetes mature into Uh, so it's been great. We really like that Um, the really kind of the only caveat we had to give people in this case was just don't create multi-tenant namespaces You don't want all of your teams working on the default namespace You want to sort of segment things out and then the the built-in security features of kubernetes and the RBAC system Will help you protect each namespace and grant people access to just the releases the helm releases that they're supposed to be able to see So we're actually really excited about that particular change in helm three Because it really opened up a security model that better fit with the way kubernetes Now works right the way kubernetes had matured its security model Then we made one more change To releases that's a notable change um When we wrote helm two we stored our releases inside of config maps Why did we do that? Well, there were really two reasons to store them in config maps at that point um reason number one Was that config maps were just ever so slightly smaller than secrets. We had one less space 64 pass over them ever so slightly smaller um Not not actually a very good reason But the second reason is even worse We used config maps because config maps were new and shiny and we were like hey new and shiny This must be the way forward secrets are old. They're going to go away. We're going to use config maps Well, we were really kind of wrong on that count too um And and you know the the excuse of the time was well a secret's not really secret at all It's just a base 64 encoded object But since then again things have changed the security model for kubernetes has matured And the usage patterns for for helm and for kubernetes have changed and now we're in this situation where It's more important to protect the information in that release Uh than it is to save on size or anything else. Um And and so they really ought to go in secrets and furthermore Secrets in kubernetes now can be backed by vaults and other storage systems that actually store encrypted secrets So there is actually a legitimate safety and security story behind storing them in secrets Okay, so now if we add all those things up, right? We've made several big changes to releases. We've Changed where they're stored. We've changed some of the formatting of how the how the record looks We've changed where the kind of object they're stored inside of in kubernetes Those are all big deals and those are all things that when we move from helm two to helm three They have to be renegotiated, right? We have to actually migrate that data from an old format in an old location To a new format in a new location That's why secrets have been really kind of the focus of the of my discussion so far I mean, sorry secrets, uh releases why releases have really been a focal point is because this is the number one thing That uh that we have to deal with and deal with very carefully when we're doing migrations like later on in the workshop I didn't want to cover one more thing. Um as we as we talk about the differences between helm two and three Uh, but this one will have less of an impact than you might suppose on the migrations that we're going to do today and on your real world migrations When we introduced helm three, we also changed the the structure of the chart It was really our biggest change ever to the chart format Um, so again another embarrassing story, um I've been working on a book with matt farina and with josh dalitsky about helm three Uh, it'll be out. I think probably december or something. So we're done writing everything So if you ever worked on on the book writing process You write all the chapters and then you send them to the publisher and the publisher gets some volunteers from the community And and from the technical Technical arenas to read the book and give you early Give you an early critique of what was clear what was unclear What wasn't covered what was over covered all that kind of stuff and uh, I was shocked when I got back a technical review that was just like Angry we'll just say angry And the reviewer is going. Hey, I don't understand this I thought you were going to write a book about helm three and instead you're writing a book about helm two and Over and over again, you're talking about helm two and you're talking about, you know How this should work and I don't even understand why you're bothering to cover any of this because by the time The book comes out. Helm two will be fully deprecated And I'm going I only wrote one paragraph about helm two and it was basically Migrate off of helm two by november 13th or else Then I realized to my horror That I had been talking about charts Charts v2 and the reader had Thought that charts v2 were part of helm v2 And we had I this is the first time it actually occurred to me. Oh, no Charts v1 worked with helm v2 charts v2 work with helm v3 charts v2 don't work with helm v2 But charts v2 and charts v1 both work with helm v3 We created a naming nightmare And I very much apologize for not noticing this until just recently but The upshot of this is that it's going to be confusing, but we we introduced a new version of charts called chart v2 that is part of helm three and these charts have a couple of of structural differences We changed the version string from version one to version two. That's always a good thing But we also we moved from the requirements yaml content directly into the charts yaml file So there is no more requirements yaml. We added a crd's directory We added the ability to write json schemas that can be used to schematize the values yaml file Martin wrote the support for a library chart a library chart is a A chart that doesn't actually install anything on its own But can be used by other charts for it to provide common tooling It was a pattern that we saw emerge out of the Out of the community building of charts But which we thought we could sort of like further Ken and i's and then make it easier to support as a first class thing So these are some of the changes we made in charts v2 for helm v3. I'm going to trip myself up here any moment now It's good to take a look at these things You may have inadvertently run across this when you tried to install a v2 chart with v2 helm because those are incompatible But the the best part uh for this particular context is that you don't have to actually Worry about the v2 versions of charts in order to do this migration Helm 3 shipped with full support for the older chart format And so as we do this migration, we actually don't have to change anything about the charts The focus is really more on the releases now as you have time and needs that are better handled by the new version of charts You know, I'd urge you to migrate Because the new version does have some niceties and some Some forward-looking features that will make certain other things easier But there's no rush to take care of that particular thing during your migration, so uh I think I would like to give a chance for people to ask questions if we feel like we have enough time for that Otherwise we can go straight into martin's. I'll let bridgett kind of call the shots here Oh, hey, we had a few questions in the chat, but um, I think we Are mostly up to date perhaps Just telling folks if they are getting started taking a look at the helm 3 book. Can they start there? What can they start with? If they were taking a look at the helm 3 what the helm 3 book that you wrote learning helm Can they Yeah, so it'll come out from o'reilly in uh in december of this year Um, and I I don't actually know what the publication date is in the meantime Docs the doc site for helm, you know covers Really a lot of the the same material the book will be sort of more obviously bigger narrative form with a lot of explanation of how it works um and more more stories about Mistakes we made in helm 2 and how we fix them and other things like that Um, but definitely for now the best best sources the helm.sh docs and then coming in december You know makes makes a lovely holiday gift. Uh, that's probably not true But uh, that's when that'll come out. Did that answer the question? I think so we had one other question in helm 3 the version of the charts is v2 This is this is the thing this is i'm so sorry about this in hindsight We should have just skipped v2 of the charts. So charts really only had one version Um up until helm 3 came out, you know, we we the compatibility level for charts has been very high from Helm 2 alpha 1 up to the present Um, but when we wanted to change some things in the format of the chart We incremented the version number exactly 1 which brought us to charts v2 So charts v2 are for helm v3 and charts v2 do not work in helm v2 So again, it was a naming faux pas on our side. Um, and we'll try and I guess I guess with helm 4 We'll have another shot at it. So we'll try to do charts before we'll write a throwaway version of charts v3 So that we can iterate the version number and and have the move in lockstep Just one of those things you learn in hindsight. That's hard All right, any other questions we had one other asked Um, is there a way to watch or observe or release outside of helm? I'm particular to be notified say in a controller When an upgrade or installation is happening So that is an interesting question and I like it because Um, it is possible and I don't know of anybody who's tried this before. So here's what you could do Um, you could write a controller That would observe the release records as they're written to kubernetes So you basically write a controller that watches for secrets with the helm type attached to them And the the release record actually gives a fair amount of information about the current state That the release is in And many of these things are you don't necessarily see surfaced all the way to the client because they're so quick that the client wouldn't necessarily see them But you'll see it go into its pre install and then it's install and then It's installed status or it's upgrading status and then it's upgraded status So you could actually write a controller that would do it be a really cool Controller if anybody wants to give it a shot. I think it would be really interesting because you could write some monitoring Tooling that would say hey looks like seven people are upgrading their helm charts right now And you know, this one is taking a really long time But the other four seem to be on or other other four I can count other six seem to be on track So yeah, that's a great question Great So summarizing, um, if you're migrating a chart not a deployment It's just the chart are only a couple of changes needed like Move the requirements.yml inside the chart.yml bump the api version anything else And martin is answering in the chat saying you don't need to convert the chart to api version 2 only if you want to use Use new capability. Do you want to kind of clarify on that topic? Yeah, do you want do you want to take a minute or really answer that one go ahead? yeah, so And I'm going to touch on it in a few minutes, but Your api version v1 charts which were used in helm v2 and v1 Are still render renderable in helm v3 without any changes except around CRD install hooks So what'll happen in that situation is it doesn't install the crd's if you're using crd install hooks Um, and also it won't create an in space on the fly unless you give it an extra flag But apart from that, they're still renderable because we wanted to maintain that capability But if you want to use new capability like You want to use the type whether you want to specify it's an application on library charts or you want to use the new way to Um To use dependencies for example in the chart at yaml Then you'd you'd bump it up to api version v2 and you'd make the changes there and I suppose going down the line it would be Uh before we get to helm 4 if helm 4 Comes out someday then you probably would want to have moved up to api version v2. Uh, does that seem okay matt? yep Thank you marten and I think we have time for one last question which is uh So helm 3 doesn't use its own dedicated crd's just secret resources Yes, yes, um We we explored the crd route very carefully. Um There were a couple of security things that caused us to go against uh, the that called that weighed against it But ultimately at the end of the day There's one feature of feature of crd's that we've realized could be so utterly catastrophic that we would not do it um crd's by definition are are modifiable by Cluster users right whereas secrets are not If for any reason you delete a crd it deletes all the resources of that type Which means with one accidental typo you could wipe out all the helm releases on your cluster And we went that's not any problem We ever want to force anybody into secrets actually have been perfectly capable of of accomplishing what we've needed Uh, and so there wasn't necessarily a big Big requirement that we move off of secrets The the security consideration for secrets was because secrets are auto backed by a vault on some systems And the different customers can choose easily which particular security back end they want Secrets ended up being a having some highly desirable features that would make the security model quite a bit stronger But at the end of the day it was that a scary scenario in which you could wipe out all your releases that ultimately I think convinced us that was wrong. So that's why we chose secrets and why we didn't choose crd's