 We have our next participants here. So we have Hilary Lipzig and Christian Hernandez, both from Red Hat and the topic will be Helm and Back Again, an SRE Guide to Choosing. All right guys, you can start. All right, so I'll start again. My name is Christian Hernandez. I am actually a Technical Marketing Manager, Senior Principal Technical Marketing Manager, probably one of the longest titles I think I've ever had. Over at Red Hat, and today we're talking about Helm and Back Again, trying to decipher when to use what, whether to use one over the other. Kind of a little bit of background first. I work in P&T and in product management and engineering. So I work with Red Hat customers who are using our products to keep them happy and to help push their RFEs upstream. And a lot of the times I test Alpha and Beta software before anyone else. So it's broken, hopefully with me first before anyone else. So I spend a lot of time talking with them about operators and Helm and when to use what and giving my perspective on that aspect. So with me is Hilary. So Hilary, if you could introduce yourself. Sure. So my name is Hilary Lipzig. I am the team lead for the Cloud Services SRE team here at Red Hat. We are the SRE group that supports several of OpenShift dedicated managed services. So Red Hat Open API Manager and some others. And actually part of the reason for this talk, this talk is all my fault. Christian sends me his black message one day. He deals with the customers on the sales end and I deal with the customers on the support end in a lot of ways. He says to you one day, he says, I'm trying to decide if I should write a Helm chart or an operator. And I said, well, what is the operator going to give you in exchange for all that overhead? And he says, nothing, I just already know how to do them. I said, great, sounds like you're writing a Helm chart. And that turned into a much longer conversation that turned into the slides that you're about to see. Yeah, yeah. So this is basically a conversation we had turned into a presentation. So this is kind of a little background for everyone. If you ever have a really cool conversation, just turn it into a presentation. And so a little, little over an agenda, right? Some of the things we're going to want to talk about here, we're going to, we have like about 50 minutes. So we're going to use all that time. So we're going to talk about what our operators, right? The pros and cons of operators, what's Helm, right? Like, what is Helm? Kind of a background on that, the pros and cons of Helm. And then pretty much what you probably came here to see, when to choose one over the other. And then talking about how to transition between the different types of deployment options out there between Helm and an operator. So for those of you who know operators, for those of you that know what Helms are, sit tight, right? It'll probably be an overview for you first, before we get into like the deep weeds of, you know, why to choose one over the other. So I believe I'm going first. It is what is an operator, right? And I apologize for the background noise. Again, everyone's remote, right? But we still apologize for the background noise. And so, first of all, what is an operator? So before we get into what an operator is, let's talk about what CRDs are, right? So I'd like to like start there to kind of lay the groundwork. So firstly, CRD stands for Custom Resource Definition. So Kubernetes at its core is really just a collection of controllers that you interact with using an API. And these controllers act on objects and resources that are scoped into its own property domain, right? For example, the pod controller manages pods and the replica set controller manages the scale of that pod. So, et cetera, right? So they have their own particular domain. They manage their own particular domain. So these controllers are declarative in nature. So desired state versus running state, paradigm, right? So like if it looks at the running state and if the running state is different, then it does not... If it's the same, it does nothing. And if it's different, it tries to reconcile that difference. So these primitives allows us to manage containerized workloads. And in layman's terms, it uses Kubernetes really as a resource controller. So what if you wanna manage other resources, right? What if you wanna manage other things beyond what's in the Kubernetes API? So for example, I want Kubernetes to recognize my VMs. I want Kubernetes to manage more abstract things like database cluster, right? So in order to do that, you need to submit a proposal for the Kubernetes code base. You know, that was kind of like the past, but not anymore, right? So now you need to create a CRD, right? So by creating a CRD, you're basically telling Kubernetes that there's a new custom resource type. For example, like here in this example, you see fancy DD, right? So you can pretty much name it whatever you want. And then when you inquire, you know, against that custom resource, Kubernetes understands your language, right? Kubernetes value kind of defined of like, you know, what is Kubernetes, you know, what is fancy DB and Kubernetes understands what you mean when you start speaking that language to Kubernetes. So now when we create a custom resource using the type defined by the CRD, custom resource objects gets created, right? So now you can say, hey, get me everything that is fancy DB. So this is fine for simple CRUD operations, like, you know, create, update, delete, you know, objects inside of Kubernetes, but what if you want more complex things, right? What if you want things that only operators know, you know, how to manage, right? System operators know how to manage, like how to recover from failure, how to back up the object, how to restore it, storage management, right? Like, et cetera. What if you want a little bit more complex things for Kubernetes to manage, right? And this is really the idea behind operators, right? So operators isn't like something that's necessarily new to Kubernetes, right? This is really based on the core idea of CRDs. So operators, it's really to codifying operational knowledge, right? As you see here, you know, you're taking your sysadmin, you're taking your SRE and you're kind of putting them in the box, right? You're codifying them there. And it's really, it's a method of packaging and deploying, managing specific Kubernetes applications, right? You know, so I'm talking about applications instead of controllers. So this has operational knowledge of upgrade, patch, recover from failure, tune application and services, right? You can really go into fully auto autopilot here. Operators, you know, by extending that, that CUBE API goes, you know, beyond that and kind of takes care of the application for you. So you're taking this, the admin knowledge, this operational knowledge, it historically has just been taken care of by the admin and you're kind of encapsulating that in codifying it, essentially, right? You're democratizing the apps, right? So now where either, you know, an SRE or a developer can go and, you know, request to this application, the application now kind of takes care of itself and kind of it becomes a self-servicing, right? So this is a really, really, really long-winded way of saying that I'll replace you with a small shell script. There's that old joke that says, say, you know, I'm gonna replace you with a small shell script. This is kind of that on steroids, I guess, is what the saying goes. What's really cool is that you can get started with things like the operator SDK, right? And so I think for any good operator, a good operator requires a story. So I'm gonna hand this to Hillary because Hillary I think has a really good, this is basically her slide, her idea and I think it's a great description of it. Yeah, Christian once told me he wants to see a good operator requires a story on a t-shirt. So maybe someday we'll actually make that happen. Yeah, so basically what Christian said, right? Operators are super powerful because it basically takes that stateful paradigm and applies it to the application layer. And the most effective operators are operators that are released by the community that develops and maintains the application that the operator deploys. Ideally, this community understands the who, what, where and how of their software. So who are the end users? What are they doing? Where are they running this? How should it be managed? You're gonna use your operator to do more than just install software or even just to upgrade software. It's gonna be responsible for other maintenance tasks that you write into it. And sometimes the operator deploys a software but sometimes the operator actually is the software. So SREs, especially within Red Hat, we develop and maintain operators to automate many tasks, not just deploying and maintaining SaaS products. By and large, what we're about to talk to you about is the SaaS products packaged as operators. A lot of the stories you're gonna hear some of the perspectives, some of the advice is specifically applying to that use case. But there is the additional use case. And if you go to any SRE convention, you'll get to hear a bunch of great talks, some of them presented by friends of mine on using operators to solve problems as standalone software. It is a piece of software. It requires a great story. It requires a good use case, planning it the way you would plan any application. And it has a life cycle of its own. It both operates the full life cycle of the application that it's going to be packaging, deploying, maintaining, and it has its own life cycle. So that's really important to remember when you're looking at operators, when you're using operators. It's not just tooling. It's way more than that. I'll go ahead and next slide, Christian. Find that button. Okay. So yeah, so it's important to know, right? I mentioned the operator SDKs, right? And it's important to know that there are different levels of operators. So as you're hearing us talk about specifically Helm, and in this particular talk, we're comparing really Helm in the Golang one, but there's different levels and then there's different types of SDKs, right? So here, as you can see, you have Helm, you have Ansible, you have Golang. In the operator SDK, being in PNT, I talked to pretty much everyone at Red Hat. They're also thinking about having a Java and a Python one. So it's really, this is just the beginning, right? And don't think you need to know Golang through and through just to get fully operational, right? So you talk about the different levels, right? You see here in the slide, there's essentially goes from, you have the idea of basic install, right? Essentially it's just like this operator just installs the software for me and that's pretty much it. To all the way to what we call autopilot, meaning that once you instantiate an operator, it takes care of itself. And I know it sounds like magic and it kind of almost is, right? There's not a lot of operators that are fully level five. Some of our ISVs at Red Hat, you know, like something like CrunchyDB or CouchbaseDB are actually at fully autopilot meaning that if you ever get a chance to see a demo of CouchDB, you'll see how it's autopilot. Basically, when you scale it, all of a sudden your data's everywhere and it'll balance that database out and all of a sudden you just scaled out automatically which is really cool. But as you see here, there's different levels, right? So you have the ability to do just install, the ability to install and upgrade, lifecycle management, you do deep insights and then really double five is kind of like the dirt of Juana as we've been talking about. It's just kind of self-healing, self-taking care of. Basically that picture of the operator going into that box is essentially the whole thing autopilot. So, Pro, no, go ahead. Just to interrupt briefly, the managed services from Red Hat that I support are typically a level four with some level five capabilities. So these are typically operators that are, I don't want to say enterprise ready but they're also not enterprise ready either. So that deep insights, the metrics, et cetera, that really enables not just for your operator to be very useful and to help the people who are leveraging it to understand what's going on with it when something goes wrong but allows you as the developer and maintainer of the operator to continuously improve. Having those insights, having those metrics, having those alertings, you can do as the reactions like create SLOs and SLIs based on that and continuously improve your software because you have implemented that in your operator. Yeah, and I think, no, it's fine. No, I think you touched on an interesting point where the insights are really important, right? So like, I think that jump from level four to level five is I think has a lot to do with all the insights that you're getting continuously improving. So yeah, that's definitely a good point to bring up where that's very, very important, right? Beta is very important and knowing how to codify something depends on what is happening on the system. Yeah, exactly. It's very much a crawl, walk, run thing in terms of leveling up your operators. In terms of like an SREA supporting it, I don't love supporting below a level four. There are lots of great reasons to have below a level four. There's lots of reasons why you might never go up that high. That's a different talk. I think we've actually kind of touched on that again later in the presentation. But as an SREA, what I get from those insights metrics and alerting, what I can send back to the community. And a lot of these are very much their open source projects. So is an SREA supporting them in a managed way, but they're open source. Everything that I encounter operationally, everything that I suggest pushed back on get changed, that goes up to the entire community. Which is more about how Red Hat operates than operator specifically. But because they started packaging and selling things as an operator way, we're able to put back more operational knowledge into the software and benefiting everybody. Yeah, definitely, definitely. So let's talk about the pros and cons of operators, right? And so, so, you know, it's sunshine and rainbows. It may be not all be sunshine and rainbows, but we'll talk about how, you know, some of the good things that operators give you, I think Hilary touched on a lot of them. So it's, you know, it's a community of freely available and open source software, right? And with the operator, I'm gaining all the advantages of technology without the need to like really deeply understand it, right? So really as a consumer, you know, if I just need a reliable database, right? I go back to the CouchDB example, I just need a reliable database that, you know, I don't need to take care of, right? An operator is, you know, of level five, level four and five above is extremely valuable, especially for those teams that, you know, there's those teams, there's the, you know, we talk about DevOps, there's those teams that do a no ops kind of a thing where it's just kind of like a group of developers, you know, maybe using AWS or something and they need something, they don't have a DB admin, they don't have administrators, but, you know, they'll use something like an operator because all that operational knowledge is codified and you gained the experience of the overall community, right? And as really for me as a, like, if you're on the developer side, right? Or if you're on the ISV side, you know, I'm making my application more accessible to the world, right? So for example, you know, I have an application that automatically takes care of itself, more and more people want to use my application and more and more people are, you know, as it gets more and more reliability, more and more people contribute, right? And this is a whole open source ideology as like, hey, cool, I have a lot of contributions and the software keeps improving, my software keeps improving. And I can drive that adoption to my application. Even internally, right? Even if like I'm doing this internally with my own, at my own company, you know, I'm driving that adoption, you know, you know, within my own, within my own company, right? And so, so the reconciliation loop, right? In the operator, right, we talked about that declarative approach within my own domain of my operator, you know, if something isn't ready yet, right? So like if, you know, if something isn't, you know, as applications growing complexity, you know, certain dependencies come into play, if it isn't ready yet, well the operator can just take care of it for you, right? And so this is kind of the things where scripting becomes a lot easier, deployments become a lot easier because the operator's taking care of a lot of the stuff that custom scripting would have taken care of. And you know, there's no snowflakes, right? And so, you know, one of the cool things about it is that, you know, everything's the same when operating a system that's built on operators, right? It's just declarative. You have that contract, right? We talk about Kubernetes operating on like promise theory where it's like, okay, you know, I declare that I want this application stack to be there, it'll just be there, right? And you operate that the same, whether, you know, your private cloud, public cloud, when you're an Azure, AWS, right? If you're using Kubernetes as that common fabric and operators as how to operate that, you know, you get that consistency across everything. So there's no snowflakes because it, you know, everything's pretty much managed the same way. So, so I'm gonna pass it all over to Hills here, talk about the cons. Thanks, yeah. So I obviously I'm supporting operators that are, you know, packaging and deploying and delivering SaaS products in the field. So I'm gonna maybe make some enemies amongst the operator community here, but they have some serious downsides. That friendly neighborhood reconciliation loop, you'll notice it's both on our pros and our cons side. The fact that it allows for retries and waits means that you're constantly retrying the same thing over and over again, and it will never fail out, even though a failure would be more appropriate. So there have been several incidents that I've worked where the Kubernetes API server was being flaky and that was reducing customer access to their managed cluster and managed services. So the customer complained that they were having connectivity problems. And so I take a look and it turned out that the issue was a reconcile loop that would never finish. So the operator lifecycle manager or OLM, which is an operator that runs to help basically keep all of the other operators on a cluster on fresh versions up to date. Yeah, meta version, right? Of the operators. Operators for operators is what I like to call it. Yeah, operators for operators. The operators for operators is great, but it's also where things get a little dicey. You have to be careful. So OLM was trying to upgrade a community operator. So this is not a managed service. And to add a little bit of context, if you have a cluster with OpenShift dedicated, you can go and download community operators directly through the UI, they're right there, they're available, they're free, right? And it'll just like automatically get it up and going for you and OLM will install it and help maintain it. So the problem was that the deployment configuration for this community operator was not backwards compatible. So what they had done is they had changed a field that was previously immutable. So not only was it no longer immutable in the new version, but it was also different. Well, the recon style loop is trying to upgrade the exact, that exact value, right? And so you hit this condition where it's like, I'm gonna upgrade this, okay? Well, I need to go through and change these values. Oh, I can't change that value. It's immutable, I failed to upgrade. The recon style loop happens, there's an upgrade. I gotta apply the upgrade, I can't apply the upgrade, fail. And this ended up creating so much pressure on the API server and at CD that it actually created issues where the connectivity was down, the ability to access a cluster went down for a couple of minutes. And it was intermittent, right? Because it happened every time the recon style loop was looping. So what we just did is we just went and deleted the old install plan and then the new one downloaded in and upgraded fine. So that brings me to my next con though. There wasn't any alerting for this, right? This wasn't, there was no, no insights, no alerting, nothing going on to tell us, hey, this is failing. This is why it's failing. It wasn't written in, it didn't exist. So without corresponding alerting and logging, watching the operators and the operands, operands being the things the operators operate. I wonder how many variations of operate I can get into one talk, we'll find out. I'm keeping track. You're not gonna be able to detect failures, right? So with something like Helm, because it does fail, that would have been really readily apparent. It would have been super obvious. It would have been like immediate and you wouldn't have ended up with that pressure on the cluster causing problems. Which makes me to kind of my next con, right? Operators have more overhead to write. So a typical operator is gonna have more logic in it than a Helm chart. And there's power there, but it has trade-offs. So developers writing operators must realize that they're writing software with a software lifecycle, I said that earlier. It should be versioned, tested and released as any application would be. That means unit tests, folks, unit tests. Since it's not just tooling, it's not always going to be the right tool. And something or someone still has to operate the operator. So another operator like OLM, operators for operators like we discussed, a person or both in situations like above where OLM steps on the operator and you have to fix race conditions by hand. And finally, no snowflakes. And you'll notice that this again is on my pros list and my cons list. And I say mine because I did the first version of these slides. So as an SRE for a managed service offering, my fleet is not uniform. These are customer-owned clusters with customer-owned workloads. And it creates an ecosystem, not a monolith. I, SRE, an ecosystem. It's a very unusual version of SRE, right? So they're buying OSD clusters and they're buying AWS instances and they have budgetary and size constraints on resources. So like CPU and memory and so forth, right? So one of the things I see a lot is that customers are pushing the limits on their cluster resources, right? They're trying to get the most value for their money. And I can't blame anybody for that, right? That you're paying it. So the problem is though that in operator land you have installs and upgrades tend to spike the resource usage by the operator, right? It's a temporary spike. But if I have multiple applications running, it's possible that there might not actually be enough resources for that installation or upgrade to complete even though there would be enough for the application that the operator is deploying and managing to just run in normal runtime. So it's easy to get around this, right? You just scale down some workloads temporarily so that, oh, we lost Christian. Anyway, you just scale down some workloads temporarily so that that bulky installer upgrade can finish. But operators are managing operands. So if I have replica sets defined, my operator is going to realize I've scaled something down and scale it right back up. So it's gonna undo my changes before I get the benefits of my change. So that also means that I have to scale down the operator. So first I'm scaling down the operator, then I'm scaling down the workloads that I don't want or don't need temporarily. And that begins to get to an uncomfortable amount of room for human error. So if operator B is also being managed by operator A, operators for operators, like we said, I'm gonna have to start scaling down a lot of pieces before I can have my otherwise fairly simple change stick around, and this is not too great. And if your operator story isn't good, if your operator isn't delivering value, there's not a lot of benefit that accompanies that extra toil. And this can hit just, you know, customers who are administering their own workloads, the same as it can hit me as an SRE. And I actually see it quite a bit. Like I think I said that one of the services that I support is an operator and it actually installs four other operators. And some of those other operators that it installs, excellent, super great operators, like the Prometheus operator gets installed, amazing, couldn't live without that. But some of the other ones are not really doing a lot of those day two, day two being operating the application actions that couldn't be handled by the first operator. And so in this situation, that cascade of operators can be a little bit difficult to work with, whereas some sort of mix and match of like Helm charts with operators would be preferable. And really the takeaway here from having common points between the pros and the cons list is that these are double-edged swords. So you need to be careful, be mindful about how you use them. Next slide, Christian. All right, cool. Now we'll talk about Helm. By the way, that was always great. I always like to listen to Hillary actually talk about what actually goes on with customers, because we like to talk about how everything's all great. There's this old saying, everything goes according to plan until you get punched in the face. That's always a fun thing to hear. So let's talk about Helm. So Helm, one of my favorite topics here. So would it be nice if managing applications on Kubernetes was just like any other framework? So for instance, if you're using like a Red Hat distribution, you're doing like YUM install something, or app get install, brew install. So what if installing and managing applications were just like anything else? And so what Helm gives you and what Helm is really at its core is a package manager, package manager and a templating engine for Kubernetes. So it can be used to repeatedly and reliably deploy cloud and cloud native applications with the correct configurations. So ideally the software you want to run already comes with Helm, with Helm chart supplied by the maintainers. But if not, you can do that yourself. With Helm, you have that ability to configure and deploy software across your fleet of clusters. So just like an operator, just like Kubernetes, where we live in a Kubernetes land, I think I saw a joke somewhere, such post somewhere that someone on the resume put a YAML engineer. This is essentially what we've been coming so far in the cloud native ecosystem with Kubernetes. And so just like Kubernetes, just like with cloud, just like with operators, Helm is declarative. And so you provided YAML or JSON configuration. But yeah, so for a time I used YAML for an API, I don't think anyone thought that would ever catch on. Just like, look at it now, right? Yeah. We were in the days of like, XML was still King, right? Yeah, yeah. First time I saw YAML, right? Especially because you and I worked together in this and then I did several other companies like this where it was like, web to print. XML was like the heart of how that worked and how it worked reliably. And so we start seeing JSON and YAML and we're looking, and of course, one being a subset of the other, it's a different story. But so we're starting to see JSON and we're like, okay, yeah, that's pretty good. And so we see YAML and we're like, you know, I don't know if I like it as much as JSON version of things and we're just kind of, I know my team, we have back and forth like, yeah, well, no, it's not gonna be a thing, right? And it's literally yet another markup language. It's almost like the creators didn't take it very seriously themselves. Yeah, yeah. It is, it now runs my life. So just like- Yeah, now it's, now I can, you know, before I used to have problems now, I think I can indent in my sleep, right? Like now it's just, now I can like see indentation problems like from a mile away. I'm like, oh, that's indented wrong. Yeah. It's how weird, how fluffy. But yeah, like back then it's like, why would you ever use this to post a request? Like there's JSON so much easier, but, you know, look at us now. Yeah. So, so with Helm, Helm comes with like, you know, a few different components, right? So it comes with the CLI, something called the chart, something called templates, values, revisions. These are kind of the nomenclature you'll hear when talking about Helm, right? So you'll hear these over and over again. So the CLI is really just a binary, just like anything else is a goal line binary. And it provides, you know, it's basically your interface into Helm, like the ecosystem. It's basically how you interact with Helm, ecosystem as a whole. So charts. So charts is nothing other than YAMLs, right? So templatize the YAMLs. So it's YAMLs where you get to plug in different parameters. And, you know, templates, it provides you with that dynamic capability. It's really under the hood, it's goal line templating. So if you're familiar with goal line templating, it essentially uses that, right? You have conditionals, you have loops, you can do all kinds of things. It's a really fancy way of managing your templates. And your values is basically the parameters you pass into a chart, right? And so you have a chart with, you know, different variables and in order to plug in those variables, you provide it the values file, right? Foo equals bar, for example, right? Let's use something simple. That Foo inside the template will be replaced with bar in the chart and it'll render your Kubernetes YAML. So it's basically a way to templatize Kubernetes YAMLs. What it spits out is a raw YAML, essentially. And, you know, when a running instance of that YAML, right? So when you merge the two, when you have your Helm chart, when you have your values and you merge them and then you get your raw YAML for Kubernetes, we call that a release, right? You have a release inside a Kubernetes cluster. And then you have different revisions, right? You know, version one, version two, version three. So these are the things that you'll hear with Helm, right? I kind of don't want to spend too much time on Helm background because there's really not a lot to it. And when I say that, I actually mean that in a way that is positive, right? For me, I'm pretty sure for Hillary, but I don't want to speak for her that I love it when software is simple. I love it when it gets out of your way, right? It's, you know, a lot of times I know myself, I'm guilty of this when I'm writing something, you're trying to be clever. But in actuality, you want it to be easy to use and easy to consume. So nothing too majorly complex with Helm. It is basically a package manager. Think of it as like, yum or DNF for Kubernetes. This is basically what it is. So, Hillary, I don't know if you wanted to add to anything to that or if you want to go straight to the pros and cons of it. No, yeah, let's go straight to the pros and cons because it's basically just drilling down on what you said. Yep, cool, all right. Pros and cons, so here, I think Hillary's going to play the bad cop here. No, I'm the good cop. I was bad cop last time. That's right, that's right. You were a bad cop, yeah. I was bad cop last time, yeah, yeah. You need more coffee, sir. It's early over here on the West Coast, so for those who don't know, it's early for us. Yeah, we've just hit 8 a.m., so. So there you go, yeah. There we go, yeah. But as an SRE, I'm used to getting waken up at weird hours and having to be on and smart as sales usually gets to work during normal business hours, so that's the difference. So as we go into the pros of using Helm, I could have actually made this list a lot longer and it's still probably a longer list than you might have expected for a tool that's not as intelligent as operators are, but the title of this talk is Helm and Back Again. So arguably, as Christian said, it's a lot easier, actually I said it too, it's a lot easier to get started with Helm than with operators. Simply put, there's less overhead into generating Helm charts than there is an operator and especially because, as I've said before, you don't need to have your day two operations defined and since it's all YAML, you don't need to know go line. So a really nice feature in Helm is the config YAML file which you can use to dynamically control the particulars of your deployment. This in conjunction with a GitOps tool can be really powerful as it can be leveraged to execute deployments with business logic attached. So from the persona of an application developer, this could allow me to execute AP testing with the ability to rapidly turn it off through GitOps pipeline integrations with Helm. So contrasting that to an operator which could be programmed to do AB testing, but then we need to be repackaged and re-released in order to change and remove the AB test. Operators are just not designed for this type of task, it's not the problem they set out to solve. If you're an Ansible user, you can integrate Helm and Ansible if so desired. So this combines the benefit of Helm charts with the power of Ansible playbooks, but this isn't super dissimilar to writing an operator. And in fact, the operator SDK is Ansible under the hood. It could be a really good intermediary step to writing an Ansible operator which is also YAML driven instead of Golang. And I think I mentioned it later in the talk, but as a note, and you saw it earlier too, the Ansible operators are exactly as full featured as the Golang operators. So if you don't have Golang experience, but you need an operator and you know YAML, easily can learn Ansible from there, just Ansible operator all the way, right? So Helm is better for deploying third party software and this is important. When you don't control the code, you don't control the consistency in the operations of that code. So using or creating a new Helm chart and taking care of your operator, operational considerations separately is a better strategy when you don't own the application. So Helm has strategies for rollbacks. So if an installation or upgrade is failing, the test rules, which is like, you know, I've put through my operation or my application, but I've deployed it or upgraded it and then I've had Helm do some post checks to make sure that things are working. If that's failing, you can rollback. It has that functionality built in. Operators don't have that, right? Operators, you would have to kind of go with the rollback by rolling forward. So you make a fix, you patch it, apply it and roll forward. And that's pretty common within the industry. If your team is mature enough that you're at the point where you're looking at an operator, you're probably mature enough to also do that, but it's something to keep in mind. It's, again, it's possible to convert a Helm chart to an operator. So while the operator will remain more limited, it's gonna be around a level two with a couple of level three functions. These are great for when you need to take, when you need a reconciliation loop. And like I said earlier, as much as I don't love working with operators that are below a level four, they have their time and place. And using it as a crawl, walk, run approach is exactly that time and place, right? And lastly, and this is really uncommon, especially because we're talking about Cloud Native most of the time here, K3S support. So I've used K3S for IoT servers. So that built in Helm compatibility for Kubernetes on bare metal is absolutely game changing in that world. And if you might even have to think about going to that world, Helm is going to be your friend. I'll send it to you, Christian, for bad cop. Yeah, I'll be bad cop. I'll, Hillary's leaving the room. I'm entering the room here in the interrogation room. So, so cons, cons to a Helm chart, right? So trailing CRDs are a real issue with Helm, right? So the cleanup of a Helm, a Helm chart isn't perfect, right? It's, there's still some, there's still some bruises and warts there. And often you need like some clever tooling around that, that shortcoming, right? So there's still some, with Helm, there's still the need for that custom scripting that I talked about where operators kind of kind of shine a little bit. Helm is needs that custom scripting and that kind of brings to the second point where you still kind of need to write a level three or above operator to, you know, basically from scratch, right? That's to say that the work you've done is wasted when you've done a Helm chart, but the Helm operator implementation doesn't extend Helm, right? So it's essentially you're wrapping operator-like functionality around Helm. And so it doesn't actually extend anything that you get with Helm. So to get that as digital functionality, you have to kind of wrap that in a go or an Ansible operator. These limitations are due to the limitations of Helm itself, and then those are done with intent, right? They're what kept Helm simple, right? So kind of our good points or our bad points kind of idea is that Helm is intentionally simple and it's intentionally to solve one specific use case, right? Being a package manager for Kubernetes. So anything beyond that, it's gonna take a little work. So, you know, a big advantage actually, you know, kind of, I know this is the cons, but a big advantage, it kind of, it makes you wise up a little bit. Kind of makes it a little smarter on the installs for the CRD issues. But if your application upgrades come with a database schema upgrade, Helm won't handle that for you, right? So, you know, if a lot of the times, you know, we talk about upgrades and you know where operators kind of shine a little more is that what if you, you know, a schema change, right? That's, you know, people try to compare Helm and operators like, well, they just install software. What's the difference? Like, well, there's a pretty big difference between installing software and managing that software, right? Like a schema change to a database, that's a big event and there needs to be a lot of operational knowledge of that Helm doesn't really do that for you. You know, if you wanna keep from repeating yourself or have kind of a dry code mentality, you might find that the inherent models for sub charts is less than optimal for first, right? At first. So if you're using sub charts or like an umbrella chart where it's basically a Helm chart that references other Helm charts is what a sub chart is. Typically, variables that are defined for the parent chart must also be redefined in the sub chart, right? It's like, it's the inheritance kind of weird with Helm in that aspect. This again is a piece that creates an uncomfortable amount of human error, right? It's like you miss an eye over here and you misspell it down over here. You may lint the YAML as much as you want but it won't catch spelling mistakes, right? So anytime you introduce the human elements to something that there could be a human error. And so that's kind of what operators try to solve there. Whereas with Helm, more human interactions is needed. And so kind of quick with the Helm cons because I know we're running short on time because what we really want to know is who wins, right? Who wins this battle, what do you do? I think Hillary has big opinions on this, right? She does this every day for a living. I just talked to customers. So I think I'm gonna hand it off to her to kind of talk about who wins here. I'm curious who wins. Let's see. All right, well I have big opinions on everything. Yes, yes, true. We could spend another hour just on my big opinions. But anyway, I've mentioned this before. Helm is a great starting point in your operator journey, right? Helm provides a way to install a workload on cluster. It's easy, it's low barrier to entry. This installation process has the ability to tap into pre and post hooks like I kind of talked about earlier. And that way you can do some sort of like ordering of rollout, sanity checks, et cetera. And Helm also provides you, it's a templating engine. You said this. So you're not storing the same YAMLs over and over and over again. Which makes it a lot easier to just kind of roll out to many different clusters, right? You write some script that replaces some variables and you're good to go, right? Off you run. And the upgrades are smart-ish, right? Like we said, it can double, it can check some things. It has some limitations, but it's still somewhat intelligent, and this is honestly what most people need and want to begin with. So it's where a lot of people start. It's a great starting point. It answers the question, how do I get this workload onto clusters? And then you have room for refinement later. Do you wanna add anything to that, Christian, before I move on? I think really the only thing I wanted to add is the fact that going back to our initial conversation about when I said, I don't know whether I want an operator or a Helm chart. 99% of the time, if you're asking yourself that question, it's gonna be a Helm chart. And it's true for me, and I think you mentioned it before, crawl, walk, run. Just because you start with Helm doesn't mean you can't just move on. So really, that's really the only thing is that you probably want a Helm chart when you ask yourself that question, when you initially start. Yeah, if you don't know your day two operations yet, start with a Helm chart. If you don't control that software, right? And maybe I talked about on the next slide, which I'll have you move to real quick. If you don't control the software, right? People who do control the software could be implementing things like floating tags, such as latest, which means that your reconciliation loop would just automatically pull the latest image of whatever that software is, and you could get a failed upgrade that you can't debug easily. And I've seen that happen. So really, Helm charts is your crawl. It might even be your run, right? There's a lot of ways to make it your run, other tooling, integrations, et cetera, that you can do. And then when you don't own the software, like what I would recommend stack-wise is GitOps, Helm, and Ansible. Put those three things together and go. That's your safest way, you're never accidentally getting something you don't want and can't debug well. Next slide. So how do you know it's time, right? You do control the software, you're thinking about operators, how do you know it's time, right? So you actually can use the Helm operator SDK to automatically convert your Helm charts to some operators, and that gives you the reconciliation loop. Again, it has limitations of Helm that we talked about, but it's the great next step, right? And it'll allow you to take some time to sort of live with your software as an operator so that you can start pushing the continuous improvements. Like I've mentioned, once you start getting to the operator thing, a big reason for that is it drives continuous improvement really effectively. So when you're ready for a level four or a level five operator, you're gonna need to write a new operator basically from scratch. If you want to use Golan, you'll have to do it completely from scratch. If you want to use an Ansible operator, there's actually some guides for how to do it, somewhat automatically, it's a little bit of some shell scripting and a little bit of some copy paste, but you can get there with some ease, right? It's not world-endingly difficult. Ansible operators use JINJA 2 templates. As a Python developer, I really like JINJA 2. I've been using it since before it was part of the standard library. So that dates me a little bit. I'm really comfortable with it, right? You're like a JINJA hipster, right? I was using it before it was cool. I was using it before it was cool. Yeah, Python 3 brought JINJA 2 into the standard libraries along with what, pandas and some other stuff. That's whole digression. And I was like, oh, wow, all these things I've always used are now part of the standard library. Awesome. But also now I feel kind of old because I talk to newer developers on my team who are like, that didn't used to be part of the standard library. No, my friend, it did not. We're almost over time here. And basically the heart of this is going from helm charts to operator, this crawl, walk, run approach, it allows you to do things iteratively. And I spent 11 years of my life in quality assurance. So I love doing things iteratively. And as you get, when you're reaching your level of maturity with building your application and you're starting to understand your application and the operational pieces and you're able to define when you actually have well-defined day two operations, like what are my service level indicators? How do I know if I'm meeting my service level objectives? When you're to that point, you're ready for your operator, right? That's your time. When you know what you want to measure, that's the time. That's when you're ready to move forward. And so like I said before, you can use operators to install other operators. There you can use operators to start install helm charts. And honestly, what I would always recommend when you're looking at operators, when you're looking at helm charts, when you're looking at an ecosystem, when you're looking at mixing and matching, it's really never fully one over the other. It's really use a helm chart to install an operator. Lots of helm charts do that. Use your operators to leverage some helm charts to do some other pieces and then leave everything in the main operator. There's great use cases for that. You can strike a balance, you can mix and match these. So when does one run over the other, right? Like Christian said, 90% of the time, you're probably gonna want a helm chart, at least to start. When you're ready, when you're mature, when you know your day two operations, when you know what you want to measure, when you know what your objectives are, it's time for an operator and you can get there. That's it, I think we're at time. Yeah, I think we're at time. I think, Michael, I don't know if we have some time for questions. I don't know if we have some questions. I could stick around maybe for a couple of minutes. I don't see any questions from the audience. So I guess no questions. Cool, cool. Well, thank you everyone. Thank you, Hilary, for waking up early. Yeah, thank you. Thank you for waking up early. Thank you everyone. Yeah, if you have any other questions, I'm on Slack, Kubernetes Slack, Christian H814 or Twitter, you can hit me up there. If you're a Red Hatter, I'm on Slack as well, so cool. You cannot find me on Twitter. You cannot find me on Open Slack. You can find me on LinkedIn. I respond to messages. I will answer that. I don't have a lot of public social media though, so LinkedIn's pretty much how you reach me or you ask Christian how to reach me. Yeah, that's me and I'll ask Hilary. All right, thank you guys. Thank you so much. Thank you.