 Good morning. Good afternoon. Good evening. Wherever you're hailing from, welcome to another episode of GetOps Guide to the Galaxy. I am Chris Short, the one with unstable internet today. Also the executive producer of OpenShift TV. So it's been kind of a stressful day for me, but I'm joined by two of my friends, Christian Hernandez and Scott Rigby, who both I believe were named to the Open GetOps Group, or at least Christian was. So congratulations to Christian. Yeah, Scott was. Yeah, Scott was as well. Yeah, Scott's co-chair. Yeah, Scott's co-chair. Look at that. Congratulations to both of you being elected and nominated. I don't know how that process worked, but you're both part of the Open GetOps committee. Crew, yeah. Whatever we're calling it. So yeah, seriously, congratulations to the both of you. Cool. Yeah, thanks, Chris. Yeah, I think I was kind of nervous that you weren't be able to be on today. You know, you're having internet issues, Chris, because I don't have that cool intro that you do. Right? Good morning, good afternoon, good evening. You could totally do that on your own. Yeah, I need like a my own catchphrase or something, or my own. Right, right, definitely. Like my own intro. He doesn't want to clone yours. Yeah, yeah, exactly. Well, clone and fork it. We're open source. So yeah, we're joined today, by Scott, Scott Ripley. Welcome, Scott. Scott is I'll let you introduce yourself here, but he's he's all he's part of, part of a lot of things, right? He takes part of the flux upstream, also Helm. He also is again, like Chris Short said, the part of the Open GetOps working group that we're all that we're all taking a part of. And actually on that note, and I literally got an email, Scott didn't even know. I got an email that GetOpsCon 2021 North America is happening. It's a co-located event for for for KubeCon North America in LA. So it'll be my backyard. So I know I'm coming to visit. You don't know. OK, so all right, all right. So I'll get my couches ready. You guys got there. So CFPs are open. I'm going to put that in the chat. CFPs are open. Please submit a CFP. We got a lot of great participation last time. This is going to be a hybrid event, right? So we'll have some recorded pre-recorded stuff. There's going to be some live some live events as well. So it'll be a lot of fun. So but with that, Scott, why don't you introduce yourself? Yeah, sure. I'm going to go ahead and share my screen. Do it. Well, I introduced myself because my second slide has all this info. All right, one second. And it's very exciting. Well, there we go. Can you hear? Can you see me? OK, both here and see you. Yes. All right. OK, yeah, so not that I necessarily needed this to outline myself, but it's on my slide anyway. So yeah, I'm Scott Rigby. So where I work is we've worked. And I am on the DX team, the developer experience team. It is an awesome team for people who are interested in joining that kind of thing. So yeah, I am a like, so yes, I am a Helm org maintainer. I'm part of the Helm team. I have been for a number of years. I primarily focused on the org side of thing and the governance org side of thing, as well as the I'm on the charts team. We used to manage and co-maintain all of the stable and incubator community charts. And now we maintain the charts tooling automation and all of the other things that help people do distributed charts. So I'm also on the flux community and security teams, and I'm generally part of that team. And like you said, yes, I'm helping to do co-chairing work, and I helped to maintain the CNCF Open Get Ops working group, and also the Open Get Ops project. And if anyone doesn't know what that is, I'm totally happy to stop sharing my screen and just start talking about that. You're more than welcome to. Yeah, I know I should. Why not? I think we should be. Yeah, because I actually don't have a slide for that. So it's like, let's see my face instead of a blank slide. Yeah, so OK. So for those who don't know, last fall, a number of different organizations, including Red Hat, Weaveworks, Amazon, Microsoft, there are a number of different orgs that are listed on the Get Ops working group, Read Me, as having put this together. And in that time, there was a sandbox project created by the same name temporarily, just as a placeholder, while the working group decided what that was going to be named. After a big public discussion or a series of discussions and a lot of back and forth, we all decided that Open Get Ops was the thing. So that's that project. That's a CNCF Sandbox project. And why do we have both of those? Because the Get Ops working group is responsible for putting together the information and the specification, the definition of what Get Ops is, as well as really partnering with other people within the wider Get Ops ecosystem that may or may not even be connected to the CNCF. For example, the CDs Foundation has a lot of Get Ops-related things. In fact, they had so many proposals for CD-Con that were related to Get Ops. Kind of last minute, they were like, let's just make this a Get Ops Summit. There's really a lot going on out there. Even though they're connected through Linux Foundation, there may be other projects that aren't even connected through LF or CNCF that the Get Ops working group is still. Part of our purview is to help people know what's going on out there, but also to have a firm set of principles on which people can understand whether they're using that phrase in mostly a buzzwordy kind of way, or whether they do in fact adhere and in what ways they do and donate here to certain to foundational principles. So Open Get Ops is the project where that information lives. So the lasting documents, any lasting programs, such as forthcoming certifications, conformance related things, et cetera, and any code that may come along the way, which has been discussed already, for example, for interoperability between projects, we'll all live in Open Get Ops. And the Get Ops working group is a working group that lasts as long as it needs to last, just like all other CNCF groups. Yes. So that's the history behind that. And it's actually really, really, really cool. Again, like we said before, Scott actually worked. We have weekly meetings, if anything. I see a lot of Scott where it's a really cool group. So if anyone really, we're always looking for members, always looking for opinions, always looking for help. If you want to be part of something and help out, we're more than happy. And also, if you have an opinion, we have a lot of those. And so we have a lot of those. And we love to hear more, because as we're ramping up, we want to take, like Scott said, kind of just share a piece of that pie for everyone to have a seat at the table. Yeah, it's not a closed group. It's definitely an open group. And we want to have you join us. It is where the cool kids hang out, though. You know what I'm saying? That's all I'm saying. Yes, you're awesome people. Those two, I know for sure. Yeah, at the very least, at the very least, there's a working group to define what cool kids mean. But we're going to use it for now. We'll use it for now. OK, yeah, so the topic of the day is talking about helm workflows and how to get out of space. I'm wearing the shirt. So I wore a shirt. I wore it for a reason. So yeah, so I'm ready to talk about helm. I love helm. I've been actually recently getting more into helm as of late, where the trying to, what's funny is that when you're starting in your get-ups journey and you start doing all these workflows, I actually came to a point where it's like, you know what? A lot of the stuff that I'm doing, helm would be really, really good that fits in. So a lot of people don't really understand that that that relationship is like, do I use helm? Do I use get-ups? It's like, well, no, it's like, yes, you're going to use both. So yeah, so let's talk about helm workflows with get-ups. All right, cool. Yeah, so let me share my screen. Full screen again? Yeah. So basically, I just had, I know this is not supposed to be a formal forum for just like a long presentation. And then when it's done, there's a few questions to the end and we're gone. This is more about discussion. But I thought, hey, why not have a little slide? Some slides, you know? And some visual aids conversation. Yeah, yeah. Visual aids, you know, I mean, I don't know, maybe looking at us talking is more interesting for folks. But I don't know. So we've got the slides and we can always stop them and start them whenever we want to. We have a little bit. We have a little bit of everything here. So yeah. And also in terms of in terms of format, I know you asked the guests if they want questions along the way or at the end, right? No, we get the questions no matter what. Yeah, you get it. Take it or don't. You get it's like a mic. It's a pie to the face. Right. I'm fully expecting that. I know, I know. Well, but I mean, audience questions, right? Like, because we have the Twitch stream and then we have YouTube, right? So people can actually ask questions along the way. OK, cool. So yeah, throw them at me whenever. And basically, I was just I wanted to say the outline was basically like I was going to just quickly you know, what flux is what what helm is and the flux helm controller. And then I wanted to talk a bit about my current favorite helm controller feature. And at the end, give some flux fun facts for people who are interested in that and like even what the hell you can do with it. So so yeah, all right. So first of all. Flux intro first go to flux cd.io. And this is really just a screenshot of that page. So this is like the weakest of all possible slides, but it is awesome. It's actually the coolest slide because someone else added this. I just mean it's my favorite. It's the coolest slide on the deck, but it's you know, it's still just a screenshot. So go there, check it out. But basically flux really ultimately is is for continuous delivery and progressive delivery. So since flagger is part of flux, that's that that takes care of the seat, the progressive delivery side and the rest of flux and its controllers handle the the continuous delivery side. So it is worth pointing out that flux is a CNCF incubating project. Yes, indeed. Indeed. Yes. And we're and it's over here. And also I put a I put the link to the operator. So the flux is an operator for open shift. So you're you're able to use give that a world to if you'd like. So cool. Awesome. Yeah. That's that's great. Yeah, it's actually I think I'm looking forward to doing some some specific open shift flux demo stuff together with you all sometimes in that would be really, really cool. Yeah, so so basically flux is the thing that lets you, you know, it's one of it's one of the tools in the GitOps ecosystem. It's, you know, it's the one that I work on. And it's, you know, for those that don't really know what that means. I think just at the highest level without even getting into what Kubernetes is and and and helm and flux and all of that. GitOps we probably should just mention is a set of principles that allow that really says that users should or it'll it's a set of principles for operating a system or a part of a system. So in short, we do actually in the GitOps principles section or we do have a pre-release of the GitOps principles and that we could read that word for word but I'll just summarize it quickly is that in short you you want all of your infrastructure and applications or really whatever part of the system you're trying to manage the declarative. So in that way, it's very, very similar to infrastructure as code. It's not only infrastructure as code even that single principle because it also can include applications and other layers that are not purely infrastructure. So workloads and so on. So it's really just any part of your system that you want to manage with GitOps. And for a lot of people, the Holy Grail is kind of managing everything from, you know, bottom to top with declarations. That's number one, the second is that you can get right over something that's versionable, some version control. And make sure that your versions are like a hostess and you know it's a history of what happened to change one thing and so on that measure, that gives you proof out of the box. So you can see who decided to change what description of your desired state of your system. And then the third principle is that you've got automated, some automation, some reconciling your declared, your version declared desired state from the first two principles into your actual system itself. And that those happen automatically without you necessarily needing to do any type of action. So it's not just add a bash script that says, could control apply inside of your CI pipeline because there could be other things that require reconciling your desired state with the actual state that didn't start with a change that you made yourself. You know, for example, something could just be failing in your cluster or something could, some drift could happen that has nothing to do with user triggered CI pipeline. And so that reconciliation should always be happening on some type of interval. Doesn't mean you can't also ping, you know, like poke that with a web hook or something, you certainly can, but that's not really the main highlight of GitOps. It's very acceptable and a good idea, but the main thing is that this is doing it all on its own. So you're really just as an operator interacting with Git, you know, guys, I probably should have stopped the share while I was talking about that. So you could see my face describing it, but I didn't do it. The, I actually posted in the chat everyone, if you take a look, I posted the version 0.1.0. The principles, this was a combination of, as Scott was saying, we have all these organizations getting together to define these principles. And this was a combination of a lot of let's sweat tears that we put together. So check those out. Feedback welcome, right? As we say, you know, this is an open group. So go ahead and take a look at that and see what we actually mean when we mean GitOps, right? Like, you know, yes, it's a buzzword, yes, but it's also a real thing, right? Two things can be true at the same time, right? So this is, it's an actual real thing, right? So don't get caught up. It's not a fake thing. It's not a buzzword thing as, you know, as a lot of people might, yeah, not only, right? So two things can be true. Take a look at the principles. It's actually there. It's an actual way of operating a system. So yep, there, Scott has got it up there as well. Yeah, so I didn't follow my own instructions in my own mind, which was going to be not just pasting links, but also showing them. So since I'm sharing my screen, here you go. So for folks who happen to just be looking and can't easily browse, this is the OpenDash, for now, OpenDash, GitOps, GitHub organization and the documents repo and the documents have, well, they are all the lasting documents and currently it's only the principles and licenses and read me right now. The principles contains a glossary. It's got the version, the release. Excuse me, the release. It's marked as a pre-release, so it doesn't show up on the right yet because it's not 1.0.0 yet, but it has all the description about how to get involved, all the things I said about what OpenGitOps and the GitOps working group are, how they relate and all the authors on this first set of principles, which is a pretty long list of people who are very involved. So thanks to everyone. Yeah, one other thing I want to mention just in the OpenGitOps project is that we use GitHub files for, if you want to see anything related to security support, how to contribute and so on, all these community health files are here and you can check them out just the way GitHub wants you to. So they'll show up inside of the UI. So for example, in the security tab, they'll actually show up in the security policy and they just try to use Git. Yeah. Yeah, they'll try to use Git for everything, essentially. They make it an open process. I also like the icons you use, by the way, Scott. Oh, thanks. I picked, they're in the commit message, but I picked OpenMoji, I think is what it's called, because it's an open source emoji set. That's right, that's right. It's properly licensed, so we just picked that. It gives it a little flair, right? A little fun for the reader. Fair. All right, cool. So a couple of questions in chat here, all right. And Kristin, you can answer this one pretty easily. I feel like when should I use OpenGitOps and Flux together? How do I answer that question? How do you answer that? How would, I don't know, because, OpenGitOps uses Argo currently, right? Yes, that's Argo CD, and yeah. And it'll, I think there'll be a point of contention, right, because I don't know. There's some delighting there, I feel like. Well, I mean, the, I guess I would need to know like what are you, what are the end goals there, right? Because the multi-tenant nature of Kubernetes makes it where you can technically point Argo CD to only one namespace, and then I guess Flux CD, correct me if I'm wrong, you can point it to yet another namespace. And I guess you could just, you can just have a multi-tenant, meaning like you have like a Kubernetes cluster and team, one team wants to use Argo and another team wants to use Flux, you can technically do it. Absolutely. And in fact, that's not unlikely at all. There are times where there are certain features that you might want from Argo that Flux doesn't currently have. Or maybe you like one of those features better and there may be other features, like say, let's say for a lot of your project, you wanna use pure Kubernetes Argo instead of, you know, like a custom ACL system like Argo has, then you would use like say Flux for those things and then where you don't want that, you could use Argo, et cetera, and just kind of mix and match them. The main thing about conflicts is you don't want both, you don't want two different sets of controllers or controller systems trying to reconcile the same thing. The same thing, yeah. The same thing. The same space, yeah. If you configure them both exactly the same in the ways that they make sense together, at least where compatible, you could do it and you would just have like reconcile, reconcile, reconcile, reconcile, reconcile, reconcile, but if you, but there's a lot of room for error there. So yeah, I think Christian's totally right. Name, space, separation, and then just look at the feature sets and talk to us all and who knows, man, maybe a lot for Flux at some point as part of your offering, I'll see. Yeah, we'll see, yeah. Yeah, so part two. Yeah, there's a question about Helm actually. Yes, how do you manage your Helm charts like governance and lifecycle? Like that's a great question, you know, because Helm is so new, I feel like it's, what rigor do we put around it? That's a super excellent question. And it's one of those questions that can be in part answered by get-ups, which is one of the nice things about this combination. So in my little outline, we started getting into some of this stuff, but I never even introduced what Helm is for folks, but do you mind if for a second I just give a super quick intro? Yeah, just because the folks that asked these questions, they're pretty aware. Sorry, Chris. No, no, I'll say, you may answer the question during a while while you explain, you know, the rest of the slide deck. Got it, exactly. We could hold off to that. We'll put a pin in that question because that's definitely related to this. It's a good one. It's a good one, yeah. So just for folks who may be tuning in and aren't even sure really what Helm is, you might even know not know what Kubernetes is. I tried to give a quick explanation of get-ups just so that at least we've level set for people who have some sense of what version control, basically the assumptions so far are only, you know what version control is, you know what a system is, and I think that's it. That's the only thing we've really assumed so far, like really at least in the description of get-ups. So maybe to really understand what it means, you'd have to have familiarity with certain systems. That's really all you need to know for that. As far as what Helm is in general, Helm is a package manager or the package manager or whatever for Kubernetes. Sorry, A package manager for Kubernetes. And Kubernetes, I think I'm probably not gonna describe that right now, except for just saying that it's a thing that helps automate infrastructure on the internet, right? Yeah, you can spend like eight hours talking about how a Kubernetes is, so. Yeah, so for like say if my mom were watching or whatever, basically I would just say do something else, you know, or no, I'd say basically it's a set of tools that helps the internet run. Let's just say something like that, but basically that helps to automate the infrastructure that used to be people kind of in rooms just like plugging and plugging and plugging and plugging. You still need to do that if you work in a data center, but you don't need to do that anymore for your infrastructure automation. So yeah, so basically Helm allows all of the, let's say Kubernetes uses manifests to declare what we want Kubernetes to do. So Kubernetes in itself is extremely compatible with GitOps. It wasn't always because there wasn't always a way to spin up Kubernetes clusters with declarations, but now there is using the cluster API project. So really from, as long as you have like at least one bootstrap cluster and that cluster has access to create other clusters using some sort of cloud account, you can, you know, nuts to bolts, you can really do everything with declaration. And so what Helm does is it allows you, it basically allows you to do two things. It allows you to, initially envisioned for applications, it allows you to simplify the grouping together of the resources, the Kubernetes resources that are all required to give you the pieces of infrastructure you need in order to run your application. It does that by creating a kind of, if you think in Kubernetes, now I'm just gonna assume that at this point users are gonna need to know Kubernetes to really get this, but like inside of a diagram of Kubernetes, you're gonna see kind of a dotted line around the resources that make up all of the things that you need application to run. And that dotted line would be a release. So it's not a native concept to Kubernetes. It is a concept that Helm introduces. And so with Helm, you create these releases. The releases are great because you can then have a release lifecycle. You can keep track of those releases. Those releases can be versioned in the case of incident management, even with GitOps, if you're primary mode of interacting with your applications and your infrastructure is through your Git repo and not really just through like manual commands. What happens when one of your Git provider APIs goes down? What happens if you have, yeah, like a fire? So, and you have to call in the troops for incident management. That hasn't lately happened at all lately. Yeah, so that's one of the reasons why it's really like, yes, we could, and I know that there are some tools, and again, I'm not gonna like do a full tool comparison right now, but there are different tools that use Helm in certain ways. Like for example, to template out those and then use those templates just raw. One other way to do with that is to keep, but if you keep Helm functionality that continues to keep the releases inside of those, inside of your Kubernetes cluster, one of the big benefits is that you can use Helm rollback functionality natively. Outside of GitOps for incident management, when you have to do a break glass scenario and move outside of that workflow, which can happen. So, that's just one other note, but basically that's what Helm is. It packages up, it's a package manager, it packages up manifests for Kubernetes because for Kubernetes, what a package is, is a set of instructions. So, that's basically what Helm is in a nutshell. It also helps you to template things because it uses the go templating language to help you avoid a lot of accidental mistakes because you have to type in the same matching string 8,000 times, things like that. There you go. Yeah. That's the one thing I really like about Helm is the templating feature is essentially, you know, you can, you know, like a lot of the times your manifests are generally the same across multiple clusters, but like there's like slight differences and so like that templating is really useful because you can essentially, you just have one set essentially of manifests, that thing you're just changing on the fly as you're installing the Helm. So, I think that's also another big plus of using Helm. Exactly, yeah. And you know, a common use case for people who may be asking the who cares question, super common would be, you've got different values for the same application and dev station prod, you know, or let's say even in production, you use the same application for your company, maybe 10 different times for different things, different instances of that same application or maybe 8,000 times, you know, depending on how big your company is. So, yeah. Yeah, there's different applications, right? And when you talk about like, there's application in the sense of like, like a customer facing application, like I have an application that takes like orders or I have a website or whatever, right? Like I have a customer facing application, but then there's an application inside the company, inside your organization that people use, right? So like there's also developers for developers, right? Like you're building tool sets for other developers to use and those are very, and then you have your own release cycles and your own way of deploying it. And sometimes, you know, your other engineers or other developers are consuming that as well. So, you know, that's like the other way of using Helm is you can package that up and say, here, install this, that's how you use my app for your app sort of thing. Cool, yeah, totally. I love that description. Yeah, so basically that kind of like leads me into the Helm controller, the Flux Helm controller. What it's primary design with essentially Flux, I skipped Flux, but I did say generally what it is. I just didn't give a whole architectural diagram and I think that's probably fine for this call that those is on Helm. But essentially, I can say that Helm, or excuse me, Flux is a componentized distributed application that lives on your Kubernetes cluster. It runs on top of Kubernetes. It can control, there can be controllers that control things outside of Kubernetes. Currently, the controllers that are bundled with it do not, they control only things within the same cluster that Flux is running on or other clusters. And so therefore, Flux can also do multi-cluster work in the sense that you can have one management cluster, say you have an instance of Helm controller running on this cluster where you wanna manage things, but you've got these other clusters and you really wanna keep them light, super light. You don't want any additional controllers running on them that you don't need. Well, that's great. You have the controller in your management cluster controlling the Helm releases in those other clusters. You can even have, through CAPI, through a cluster API, you can even have that management cluster create and spin down those other clusters as needed. So you can really see how that works when configured properly. You can manage multiple clusters that way as well. But I'm focusing primarily on the charts, on the Helm side of things for now. So that's really cool. It uses the Helm SDK primarily as part of its design so that it's not really shelling out to the Helm binary. And what that means is that we are able to take care, excuse me, to keep the imprint very, very low, the memory usage very low because we're just pulling in the packages. Okay, so for those who don't know, much of this work is written in Golang or the Go language. And so these are no exception. It pulls in the Helm Go packages and uses that functionality through their SDK. And that's why the Helm version three and above was designed to be SDK first. And the Helm client uses that SDK and the idea is that other tools will use that SDK as well. So that's what this does, that's part of its design. The other nice thing about that is that it is feature rich in the sense that pretty anything that you can do with Helm, at least it's non-experimental features, you can do with the Flux Helm controller. Nice. Yeah. So, and for those who may be asking me about OCI, feel free to ask questions about it. If we have time, I can get to it at the end. Yeah, the quick answer is that it's still an experimental feature in those packages that are not public packages yet. So therefore that's not pulled into the Flux Helm controller. But there is work upstream in Helm for that, actually upstream at OCI. Things have, yeah. Someone asked about that. It's funny that you should bring that up earlier internally. Someone like, funny enough, pinged me about that. I was like, that's a good question, I don't know. I think since it's experimental, I don't think any of those are public currently. They're not, yeah. And there's some work being done on that. In fact, the short answer is I can link you all to the Helm dev meeting notes. And even as recently as last week, there was a discussion in the last several weeks, there's been discussion about how to move forward with that. In fact, the last, quite a long time, but over the last month, there's been some headway because the bar has finally moved in the open container initiative. So now that has been moving upstream. We have, the short answer is this, that there was not, for the longest time, there was not a full release spec. A full release spec. The spec was not solidified. So for tools that adhere very strictly to backwards compatibility, such as Helm, and also Flux does as well, we couldn't provide that functionality and still claim for backwards compatibility if, if, I know I'm talking about Helm now, but like Helm can't actually open those packages and say, oh, we're still going to follow Simber if the upstream spec itself is going to change. So that's why it was here. That's why it's been behind an experimental feature flag. You can still use it, but we have to guarantee that backwards compatibility for that graduated CNCF project. It's extremely important. It's one of the reasons why it's so widely used is because it's so highly trusted. And the same thing with Flux is we won't pull that in until we can get that solidified in the Helm SDK. So there's work being done and please follow it if you want to follow the Helm Dev meetings. That's probably the best answer I can get for now. Yeah, so this is part of the fun part. I was actually hoping to do a live demo, but my demo was breaking like moments before this. So I'm going to walk you through it. Yeah, yeah, it's totally cool. I'm going to walk you through it though. Through how this works, because we essentially have all the things, I just can't hit the command line and say, it worked. I can do it all the way up until the point where it says it doesn't work. Where it works? And it's just because I waited too long to actually form a correct patch. So unless I want to get in there and someone wants some strategic merge patch expert wants to say, oh yeah, let's fix it. I know you guys like demos that kind of fail and then fix or fail and then fail, but I like that. It sounds like a challenge almost to me. I was like, now you got me curious if you want to post the... Yeah, I certainly can. Yeah, whether it's just making that little simple repo public and posting it afterwards or whether we have some time at the end to just say, hey, let's poke around at this like you sometimes like to. We could certainly do that. Yeah, let's do it. Yeah, but let me show you what it is. We never had homework before, but there you go, right? Yeah, all right, cool. All right, so let me show you what the deal is. I might zoom in just a little in the way here. There we go. Okay, so you've heard of me talk about this topic before and it almost seems like a little bit of a broken record, but I think it's a really important feature and I'll explain a little bit more why, not just because it's cool, but because of what you can do with it. So ultimately we were talking about how more close and one of the big workflows that I wanted to focus on was what happens when you're working with a chart and the chart, say it's a public chart, it's a very widely used public chart. Let's say you're working with some of the Prometheus charts, right? They're Prometheus community blessed. Let's say you are working with the cert manager chart that installs that application. That's one of the tools that Christian was talking about everyone that developers use the tools to build their tools. So cert managers help with TLS certificates with third parties and inside your cluster. So it's a very widely used tool and it's not really a good idea at that. I mean, you can, but I would just, if you can get around it, I would discourage someone from forking that just to add in their own configurations inside of the Go templates of that chart and then using them when they're in your company because immediately that's pretty fine, but if your needs don't have, even if you do follow the open source, let's say program where you say, all right, well, I'm gonna upstream, I'm gonna try to upstream my changes. I'm gonna fork, I'm gonna make my changes. I hope they're useful for the wider community and I'll make a pull request for that. Well, it might take a while for that to be reviewed depending on the community and depending on the other priorities in place. Not only that, I mean, that community is very good by the way. So definitely that wasn't a silent dig. They are absolutely excellent on top of everything. I mean, for real, but it may be that, so some other charts may not be that way. Some chart authors might be on vacation or they might not really be, they may be very, the cadence may be very spread out because it's a very older application that doesn't really need a lot of perpetual maintenance. So the second reason why it might be, it might not really suit you is because your change might not be something that the authors of that chart agree with. They might not think it's a good idea and they might not have a better recommendation for you because it might just say, well, listen, that's a very, excuse me, that's a very idiosyncratic use of this tool and that's not something that we wanna clutter up this chart with all the configurations for all, for your special use case, right? Yeah, and there's also, yeah. It's also, yeah, you're definitely spot on about there are certain situations. I know I maintain a couple of things and I'll get a PR and I was like, there's just no way we can support this for your specific use case. So I always feel bad because I understand, because I have snowflake installations as well, but it's like, you know, it's just like, I just generally speaking, there's no way we can support this or it doesn't even make sense and it breaks other things, so sorry. That's right, exactly. Yeah, and say it's a chart where you, you know, we don't, the chart authors don't, or maintainers don't really want to maintain backwards compatibility to certain older API versions of this Kubernetes resources and otherwise there may be no changes within your manifest, just instead of V1, it needs to say V1 beta prime or something like this, right? Yeah, yeah, exactly. Well, some charts do that and they do it backwards to a certain point and it's to certain versions of Kubernetes, just to, you know, maybe like one or two versions back, but let's say you're on an older cluster, but you still want to use this and you know you've assessed the risk and you know that you just need to change that version of that manifest, that's not gonna make it upstream, you know, because that chart's not going to be backwards compatible forever, you know, even if you've cleared it, you know. So here, so this is one of the reasons why, why that part of the workflow means that you end up forking that and then you kind of have to maintain that work as all the new changes come from that chart. So if you want to take advantage of any of the new changes, too bad, you know. So that's one of the really nice things about, not too bad, but just, you've got extra work on your hands basically. And generally there's a lot of, speaking of drift, there's a lot of difference at that point between new features that for instance, for example, in the Prometheus community where new features come out weekly or sometimes daily, you know, you may not want to fall too far behind in order to keep up with new features in the tool. So the Helm post renderer, this is built into Helm itself. This is not something that is unique to Flux, but the Flux Helm controller does implement this well and I wanted to show you how they both work. So, so here is, come on now, there we go. So in the Helm docs, and we can post this link later, in the Helm docs, there's a section under, I think it's under advanced Helm techniques, yeah, boom, is post rendering. And it basically what happens is, of course you can, the other way you can do that is what I mentioned before is you can just template things out, modify those templates, whether it's through customize or whether it's through any other string manipulation tool, bash. There are other, yeah, yeah, there are other, well, he said, there are other good tools out there that I'm just not gonna list them all right now because I'm trying to stay on the main point, but that allow you to make customizations against raw Gamble, right? But like I said near the beginning here, if folks didn't come in late, that one of the big features of Helm is maintaining those release, those release concepts within your cluster and that release metadata. And if you template things out and only use Helm for its template features, you're losing all of that. So this is why we actually added the post rendering functionality within Helm because it allows you to take advantage of whatever a chart does, but also have a defined change to those rendered YAML manifests that still make their way into a Helm release. And you do that within the CLI tool by passing a post renderer flag. So yeah, and we all admit that this is a bit of a hack because it allows you to, but it was the most flexible approach and it solved the greatest number of issues. And this is primarily for edge cases. So that's why that made sense. It means that for security, you need to be really trusted to this executable because it can just point to any executable. So that's good because you can make a shell script that just simply does some piping to customize and whatever. In fact, there is an example of that Taylor Thomas wrote linked from the dots here. An example using customize. It's a nice example. And in fact, why don't I show you generally what it is? So the example chart is a very simple one. It's got, it was really just created using the Helm create command at the time. The customization, here's the customization YAML. It really, it brings in all and then a secret. And then it uses a patch YAML to target this deployment. So here is the, so excuse me, here is the customer, excuse me, the, sorry, let me show you the YAML first. Here is the secret. It's a super duper secret. The deployment is created by, the deployment is created by the chart itself. So here's the template and here's the, excuse me, yeah, here's the deployment or whatever. You just, it just uses, it just simply uses the, the render templates from the chart. And then here is the patch object. Oh, I'm sorry. Yeah, so here's the patch. And it just uses, it specifies the targets. The name really doesn't matter because you're just simply, in this case, the, yeah, yeah, you're targeting star. Yeah, exactly, exactly. Thanks for remembering that, yeah. So you're targeting that and then you're essentially, you're adding a new security container. And this is what you're doing. And then you're using that secret that you're adding. So this is the binary. It's just a shell file or a bash file that literally just, it takes, It takes standard in, right? Exactly. It takes standard in, does your modifications. It pipes to standard out. And since customized build already sends out, this fits the bill, but you don't have to use this. So this is just a really good example of adding some security feature to Helm charts that may not have them, but they're still important to use. That's, and that's linked from the Helm docs. What, how flux does this is a little bit different or no, it's not different. Flux does that in the exact implements that properly, but instead of using a binary, you know, for security purposes, we don't want to run as of flux version two and above, do not want to run user specified binaries inside of the cluster. Also, we want this to perform well, you know, and we want it to be secure. So, and we all love that, right? So what, what, what flux does is we implemented the most, the initial standard post renderer and set up a tooling to be able to, to more easily allow additional post renders if they become, if there's a demand for them. So it supports, basically supports customized patching. So patches, strategic merge patches, JSON 6902, rather than calling the customized binary, it just uses the customized SDK, the customized go packages in order to add your patch. So you just basically provide the patch in the, in the manifest itself. So you don't have to worry about doing all that. All right, nice. Exactly, this is how, it's not a full on instruction step by step, step instruction of how to use the Flux Helm controller. We have a lot of videos on that. I've done demos in the past, Leica Pelle's done demos, Kingdon Barrett has done a bunch of it. Actually, now I think about it, there's like five different people who work in the Flux project, at least who have done a lot of demos on how to do that end to end. But this is really just focusing on this feature for now. But because I'm not showing you end to end, I just wanted to tell you what we're looking at. This is a custom, this is a custom, an example of a custom resource whose custom resource definition is controlled by the Flux Helm controller. So both your Helm repo, Helm repo kind and a Helm release custom resource definition are both defined by this controller. And this is how you do declarative Helm with Flux anyway, as you use custom resource definitions. These are all, this is just an example of things that you may, something that you might want to use, for example, the metric server chart. And here's an example of things that you might want to change about the metric server chart that that chart doesn't support. And I hope that makes sense. Yeah, yeah. So essentially you get all the, everything that you get with Helm, you get with Helm, and then everything, all the patching and all these kind of snowflakey things that you need to do, you have the ability to do it there as well, right? Instead of doing like Helm templates, raw manifest, you get everything you do with Helm. Exactly. And you can use custom, so essentially is what I say all the time, Chris, is this is a chocolate and peanut butter section is like, you put those together and you get like this, this is some amazing thing, that's really cool. Yeah, exactly. I was just going to stop the share for a second so we can share it. And I can also share it right back again when I get to the flux fun facts and stuff. But yeah, that's how it works and that's, and we can link those docs. So you can send out if you want to, I don't know if you're all. Yeah, just share them there and share them and zoom for you so you can paste it if you want into the end user community. Yeah, so I have a question. So I don't mean to bore got the time from the users here. Oh, thank you. It looks like Chris is on it. And so what I like, so the question I had was, okay, so we have this workflow of Helm, right? And installing Helm charts and using Helm charts, right? And then we have the GitOps and then we have the GitOps principles, right? Where is that single source of truth in terms of like my values, right? Like I have a values file that I use. Is that, do we still store that on Git? Is that in the manifest directly, directly, what's the, what are some of the patterns that you've seen in order to be like, okay, hey, I have a values.yaml file that we use that we want to keep as a source of truth. Exactly. What are some of the patterns you've seen? Yeah, thanks. That's like, that is one of the biggest questions that I was doing consulting work and even just like, say, moving into a team to help them set up their workflows in a company, which I've worked in different companies before like big ones, kind of small ones. That's the number one thing because that's what people want to know. Like how do we set this up, right? And if you're using, it used to be that really, really before these GitOps tools came out with Flux support, excuse me, sorry, with Helm support, for example, Flux. Again, I'm focusing on the tool that I work on the most, that there are other tools out there as well. Really one of the tools that allowed people, like for example, Argo, right? They also integrate with Helm and so on. But one of the tools that people used to do declarative Helm in the past was a project called Helmfile, which I still think is a fantastic project. One could use that to do, for example, one could write a Flux controller to do a Helmfile Flux controller, so you could still continue to use a Helmfile syntax. I actually at first thought that would be a great thing to do, but I quickly abandoned that effort just because you really can do everything with the Flux Helm controller custom resources as they're defined that you can with a Helmfile. It's really just different syntax. And if you use a custom resource syntax, it's actually, you're able to define that in a structured way through the CUBE API and through the documentation of those API resources. So I'm not saying, oh, it makes Helmfile obsolete. I'm not saying that at all. I think that people can certainly use that. And I could imagine someone writing a controller just because they want to continue that syntax, but I believe that would be the only difference is that it would almost be like a mapping project, a mapping tool between that and the syntax that the custom resources, excuse me, that the Helm controller custom resources are looking for. So one of the things that I normally see is that with Helm you can pass dash, dash values or dash F to the Helm install or Helm upgrade commands. And you can do that multiple times. So you do that in layers. And then you can also pass a, excuse me, a set command on the command line itself, which will then even override those. So the nice thing about that is that Helm, Helm isn't built this way, but it has the capacity. Once you were allowed to have multiple values files, Helm now provides the capacity for a concept of layering. When Helm was first written, it didn't. It was really just, there was some string replacement in there for your values and that was it. And then more advanced needs required more flexibility to be built into the tool. What you see out there is normally you'll have something like a common values file. If you're using a chart that you own, you can actually put what your normal common values are right in your values.yaml. But if you're using a shared community chart, either it's shared between teams, kind of inner source for your company or open source generally speaking. So it's really just same defaults. You normally will have something that you use almost all the time and that will be your like kind of common dash values.yaml, right? For that application. If it's a community chart, you can't store it inside the chart because you're using it as a package that someone else built. But you don't have to, right? That can be, it's usually inside of your Git repo and you'll have a folder. I mean, this is just an example. You might have a directory called application one, or metric server, or a cert manager, or WordPress or MySQL or whatever, whatever you want to install with a chart. And you'll have a common values file in there that you use all the time with very few overrides. Then for, you might have a dev stage prod dash values.yaml for each one of your environments, whatever it is, stage one, stage two, et cetera. And for those specific environments, you may have specific needs and overrides. Maybe I want a lot of replicas in production and I even want an HPA, a horizontal pod autoscaler, but I don't want that in development. Say I want an HPA in stage to make sure it works, but I want fewer replicas because I don't want to pay extra money unnecessarily, et cetera. And in dev, I only want one replica and no HPA, right? So that's just one example for a workload. And you'll put those in the various things and then what you'll say, hell, install my chart, my name, dash file, common, dash file, my environment, dash file, my teams overrides. And if you really want to be super specific, you can set additional overrides on command line. So let me share my screen and show you how that works with the help controller, though. Yeah, that's really cool, by the way, that you can just layer them like that. And you know, overwrite, overwrite, overwrite, overwrite, and like you can, you're maintaining less, right? As you're taking these values in. So that's really cool. Yeah, I think so too. So this is, there's a section on values overrides here within the, I'll paste it into the zoom chat again, just so that. Anyway, I know we don't have a ton of time here. And I do want to answer that person's question or at least try if I can before we end. But here's what it looks like inside of the home release object, the CRD. You can specify your values directly there or you can specify your values and values from and have those stored as a config map or a secret. So that's one of the. That's really cool. Yeah, that's one of the. Yeah. Sorry. I think that's really cool. Well, you can load your, you can save your values in, you can have it in get and then have it slurped in and then from the, from the map, from the actual controller, you're gonna say, hey, look at that secret for your values.yaml file, that's really cool. Yeah. And so, and so basically you would do that with getups by having encrypted secrets. If there's something that should be secret or if they don't need to be secret and they're just configurations, you can just, you can just have a non-encrypted config map resource inside of your directory that your getups just sucks into your cluster. Yeah. Nice. Yeah, in fact, okay. So the fun facts, I'll actually just leave them on screen. I think the main one that I wanted to highlight before was that we do multi-tent, multi-cluster. Flux also does multi-tenancy using native Kubernetes RBAC. So that's a very interesting thing for folks who need that. If you want to ask me questions about that, just ping me afterwards. And I just wanted to say as well that we're really looking for more contributors and our, for instance, Kingdom Barrett just led a bug scrub that was very successful and now it's going to be a weekly bug scrub where he or someone else from our team will guide people through just kind of stepping into what it takes to start getting involved in the project. So I wanted to send out that invite as well, as well as the invite we already gave, yeah, for the getups working group. So, so yeah, I've got some links in the links section and I'll send them to you so you can pass them to the community. Oops. And I know we don't have any real time left, but do you sometimes go a few minutes over? I feel like I remember you did. Okay, okay, we don't have a hard stop. Sometimes you have a hard stop, sometimes we don't. Yeah, there was one question that I said we'd get to. Can you remind me what that was? Yes, so how do you manage your Helm charts governance and life cycle? Right, so let's say you weren't using, I'll keep a quick time on this for folks who are just like, I think I have five extra minutes, let's do that, maybe say five minutes, right? Yes, okay, yeah, that's good to give people an indicator so that they're not wondering if they should hang out or not. Okay, so if you got five extra minutes or four now, let's stick around. So basically say we're not using getups to answer this question and you're just using, say you're not even using version control at all, but you are storing your Kubernetes resources or sorry, your Helm charts and those values files that I mentioned that people often use for Helm charts and anything else related to your Helm releases, inside of, I don't know, let's say inside of a Google Drive, right? Do that or whatever, or inside of some sort of a- Dropbox, yeah, let's say. Dropbox, could be anything. Shared, I'm just trying to be very vague even though I think for most people industry standard is to use version control and most likely get and most likely one of the major get providers or a pure self-hosted get to do your version control, not everyone necessarily does. So sometimes people have third-party tools out there that just maybe keep your configurations inside of a database and then your team members can update them. That's pretty common when people use a tool like say the old school Jenkins to deploy certain kinds of things. It's not uncommon at all, right? There might be some bash around it but that's really what we're talking about because that's all stored in the database. I'm not shitting on Jenkins because you can certainly be very declarative with Jenkins at this point with the Jenkins file and it's a really awesome project. Believe me, I was using Hudson back in the day too. Yeah. Oh wow. Yeah, there you go. I have touched things that were labeled Hudson before, yes. Yes. Indeed, I was very excited when that came out. So again, this is no shade. It's an old monolithic project that is just one example of how one could do that where I know people do in fact, still do that without a versionable ledger like Git. Okay. So clearly Git is not the only answer to how you deal with governance but even without GitOps, if you are using a version control system, I would say that's number one. And so let's just assume Git for the moment, okay? It could be anything but one of the, when you're talking about Helm specifically, the most important thing I think to realize is that with Helm three, with at least with the beginning of Helm three, charts are names-based. So the nice thing about that is your governance can be one-to-one mapping with your Kubernetes RBAC. Yeah. So you really- It could match your organization, right? Like your organization, it could be scoped down to like, now you don't need cluster control. It's like, well, if I only care about my own little world, my own little namespace, right? Multi-tenancy sort of thing. So that's really cool about Helm three. So- Yeah. So really we're just talking about, whether we're talking about service accounts or we're talking about cluster roles and cluster roles, or sorry, not cluster roles, roles and role binding that are names-based. Let's pretend you didn't even have Helm in the mix. You really are still doing the same thing. You're just using the features of Kubernetes that allow you to be flexible in structuring how your manifests are organized and you usually do that and you usually do that to fit your organization exactly like Christian said. And I think that's one of the big features of DevOps, not generally speaking, not GitOps, is a way to integrate your organization with your technical workflows, not just mirror one to the other, but try to ideally have a kind of a play back and forth as you discover what you need as you move at the speed of business. And so with Git and with Git and Helm, a simple example would be separate your manifest. If you have different resources that one team should be responsible for and not another, put them in separate Git repos. Hey, guess what, flux and handle that, multiple repos, easy. I know other tools can too. So that's one simple answer. Another answer could be using the same repo, but using the built-in access control that that Git provider offers you. Like so for GitHub, there's a code owner's file, for GitLab there are other ACLs, et cetera. So that's one other way to do it. The most important thing I think to keep in mind though is what are the Helm charts controlling? They're just, it's just a templating engine that allows you to package, not just, it's a templating engine that allows you to package resources that the Helm client then creates as those releases that with versioned releases, like I mentioned before or the Helm SDK, whichever case, right? So if you have a chart that really extends beyond what it should do, so for example, it's managing cluster-level resources, if the agent who's running those Helm commands doesn't have correct RBAC access, they can't install the chart. And that's not Helm's problem, that would be the same as if you had just plain manifests. So you just have to kind of look at your, basically look at your, at the access model you want and then make sure that your Helm charts conform to that and one thing about the post-render feature I showed you, if you've got a chart that you look at every aspect about it, except for one feature that it does that assumes access you don't wanna give someone, post-render it out and then set those, set your alternative manifests yourself. Cool, that's great. Yeah, fantastic. All right, I think we are, we are, yeah, we are over a little bit. Thank you everyone for joining. Yeah, that's, it's cool. So thank you, there's a lot of chatter here. We posted some links, right? I guess to the next bug scrub. So thank you for posting that, Chris, and all the various links are there too as well. So yeah, I think that's it for this week, right, Chris? Or Scott, unless you wanted to cover anything else? Yeah. No, not for this week. I mean, you extended a few extra minutes, I think seven minutes at this point. So that's like, you get some extra, that's cool. Yeah, you get a little bit extra. That's right, bonus get-ups, that's right, bonus get-ups. You get a little bonus get-ups for you here. So cool. So yeah, so. And I guess just thank you both and all, you know, I posted some links to, you know, Fluxy did IO, I posted links to just really just to the Helm docs and to the get-ups working group and open get-ups. When I say posted, I posted them in Zoom so you can share them with the community. Yeah, we're sending them over over here, yeah. And also we got the link to the CFP for get-ups con. So hopefully you guys can make it, it'll be fun. It'll be in my backyard. So I'll show you guys all the best bars and restaurants here. If you guys come down, if you guys come down in person, I'll definitely, it'll be in downtown. So it's actually kind of a, I'm on the West side here. But anyways. I'm super excited to hang out with you all in person again. Yeah, exactly. It's crazy. It's crazy that we get to do that now. So yeah, so be there. Braided Silver's challenge accepted. Yeah, so. So it'll be fun. So yeah, Chris, I don't know if you wanna talk about what's coming up next. Well, I mean, there's nothing left today, but tomorrow at noon Eastern, 1700 UTC, we're doing an OpenShift Commons gathering, talking about data science and application development. So that should be pretty interesting, I think. That should be fun. That should be interesting, yeah. I should round out the week and send this off into the 4th of July weekend here in the US. Yeah, have a long weekend here. And Canada is Canada day-to-day, by the way. Oh yeah. So they're off. So it's kind of North America shutting down for the weekend. Yeah, pretty much. Yeah, the redhead is off. Redhead is shut down Monday, so streaming that day. Yeah, all the Canadians with the day off, so. Yeah, it will be back at it, though. First thing Tuesday morning, sounds like, so yeah. Cool, yeah. And by the way, good job, Bobby. No one knows. No one knows who's behind the curtains here, but yeah. No one's behind the curtains. We had the intern run this show today, so. Congratulations. Yes, congratulations, Bobby. Made it all the way through, no worries. We made it all the way through. So yeah, whenever you're ready, Bobby. Yeah, we'll see you guys next time. Have a good one all.