 I was going to run through a quick intro of what Helm is, but we have a great demo that Andy's put together about signing and stuff, so my name is Karina Angel. I'm with Red Hat, and Red Hat does love Helm. I know there have been rumors out there. We do like Helm. All right, so we're going to run through Quick Intro, which, and then Paul's going to get into building charts, and then we're going to talk about publishing. And if you want to get involved, go to Helm.sh, go read through what Helm really is, and all the community stuff is there, enjoying the community, they're actually really nice. There's rumors about that, too. All right, so we all know Kubernetes can be difficult because now you have so many different resources to think about, your deployments, your config maps, your persistent volume claims, really any resource, any Kubernetes resource you can think of. And now your applications are not only containerized and hopefully in a cloud-native manner, and hopefully it's not just the lift and shift, but they're highly distributed. And while deploying your containerized applications in a cloud environment yields so many benefits, and that's where we're all here, right, you have an increased level of ownership and your work required to manage this entire set of components. And Kubernetes also contains a vast ecosystem. You've all seen the ecosystem page. It's kind of crazy how many different projects and resources we can all use now, and it's amazing, but how do you manage all of this effectively? And, of course, wouldn't it be nice to be able to just do a brew install or just use app, and that's what Helm really is for. It's a package manager designed to pull all of those resources in together, and it is a cloud-native, or it is a graduated project. It graduated in 2020, joined the CNCF in 2016, and very active, I think we all love all the AV stuff going on. Okay. Thank you, Scott. Okay. So, Helm, let's talk about the primary components of the package manager for Kubernetes. There are other package managers, too, so we don't want to say use one or the other, because you have to decide for yourself what works for your environment. However, your Helm charts is an amazing package manager, and we'll talk about the primary components that we have. So your Helm charts are packages that represent your Kubernetes deployable resources. We just talk about all the different resources that you have to worry about, and your values are the configuration variables that are injected into your templated resources, and your release is a consolidation of those resources, and it's, and your values, and is your deployable resource into Kubernetes. So there's your very brief intro to Helm, and now we're going to have this talk. We're talking about Helm charts, and so are the user profile, I'm not going to say persona because it's a user profile. It's your application distributor. You're worrying about packaging for people that are consuming your applications. And now, Paul is going to walk us through the building. A chart. Hi, everyone. My name is Paul Czakowski, and I also work at Red Hat, and I kind of just want to get going. So we want to create a chart. How do we do it? Like, when you're creating a new project, there's a ton of boilerplate, and you're doing copying and pasting and stuff. And we didn't want everyone to have to worry about boilerplate. And so we built what we call, well, actually, let's talk about the actual anatomy of a chart. I forgot about this one. So we have the chart.yaml file, and that is where the metadata for your chart lives. So it's name, it's version, any dependencies it might have, et cetera. There is a templates directory, which is where your templates live. And inside of that is a couple of special things. You have your helpers file, and that's where you can create reusable blocks of code to use as functions inside of your charts. And you have the notes.txt file, and that doesn't get rendered into your cluster. It gets rendered into the CLI when someone does helm install or helm upgrade. So you can provide breadcrumbs and hints to your application users on how they can access the app once it's deployed based on the various decisions they made when they were creating the values file that says how they wanted to look. And that leads into the values file, which is kind of a well-written values file as the contract between the helm, like the chart creator and the chart deployer. So it contains, it's a YAML file, it's a list of keys and values that are rendered into the templates and contains the defaults. And so the user will then change those defaults to suit their environment. And if you document that really well, it does really become that contract between the person who writes the chart and the person that runs the chart. So we don't want you to have to do a bunch of boilerplate because, you know, any programming language, there's tons of boilerplate you need to go through. We just want you to be able to get to a running chart that you can modify. So we have helm create, and this will give you that functional chart. And that's what gives you this directory structure that is your chart. And it's just enough to deploy a basic nginx application. So you can do helm create, helm install, and you'll get a running application in your cluster. So then you just start changing things in the template, like the image name, you'll change some ports. You might add some resources like a config map or a secret, but you'll just use the same boilerplate at the front of the other templates and copy them across. So there's still going to be some boilerplating, but it's a lot less. So, yeah. So this is kind of what the templating looks like. It uses the Go templating language, and there's a series of functions in sprig, which I've brought in. And there's a couple of helm specific functions as well. So the double curly braces is the templated bits. It is pretty straightforward, but also is templating. So it's also incredibly complicated when you do complicated things. And that's just the nature of this. I have discovered that GitHub Copilot is really good at filling in these sections for you if you happen to have access to GitHub Copilot. So I do recommend it. It has made my life easy. Okay, I'm going to give you up on that. And so this default set of templates are really good, but chances are you have your own opinions. You have your own processes. You have things that are unique to your organization that you want to embody into your templates. And it's the same pieces in all of your charts, right? And so we have a couple of ways you can do this. You can create starter charts, and then you can have library charts. And we'll talk about the library charts in a bit. Let's start with the starters. So Helm starters lets you create your own boilerplate templates. So instead of doing Helm create, you do Helm create and specify your starter. And that is more or less just a regular chart that has all of your opinions in it. Say you always have a specific set of, like, Prometheus service monitors, or you have a specific way you want everything to have Istio endpoints. You would put those into your starter, and that way any time you create a new chart, those bits would show up and just be part of the chart. And that saves you a lot of copying and pasting. And the Helm ecosystem has some really good examples of starters if you want to figure out how to get going. And then out of Salesforce, we have a really good Helm plugin for managing Helm starters. So it becomes pretty easy and quick to, you know, embody your own thoughts and processes and best practices into the defaults for whenever you do a Helm create. So I mentioned library charts as an alternative to provide opinions. So let's talk about our two chart types. So this is a chart.yaml file, which I kind of mentioned earlier. The Helm 3 chart spec, apart from the name, version, dependencies, has added a type. And that type is either application, which is the default, or it is library, which is this type of library chart. So the library charts define primitives or definitions that can be shared in other charts. So library charts don't render into templates that get applied to Kubernetes. They become things that your chart can consume to make your life a little bit easier. And so you can use them across charts like I have these opinions about PVCs. So instead of writing this massive block about PVCs, I can just call it my own PVC function and let it kind of build that out. And so you're able to start codifying your own opinion standards into your charts. And Bitnami, any time I'm like, how should I do this? I go and look at the Bitnami Helm charts because they're kind of, in my mind at least, the way to do charts right is the way Bitnami does them. And they have an excellent common library chart. So if you want to get a better idea of good ways to do library charts, I highly recommend looking at the Bitnami chart, library chart and how they do it. And also any other things, you're thinking about best practices. Reading documentation is great, but sometimes seeing really well-written charts like in practice is better. So I always look to the Bitnami charts. Any time I have questions myself about a particular way I should be doing something. So back to our chart.yaml file. So this is a chart.yaml from a Bitnami chart that is using both a library chart and an application chart as dependencies. So basically you write your chart, you want to use your library chart, add that as a dependency. And then if you want to have something else like MySQL database, you could add a bunch of templates to run MySQL inside of your chart, but chances are someone else has already done that for you. Chances are your vendor has done that for you if you're getting a database from a vendor. And so you can include their chart as a dependency and take all of their best practices and all the things they do and have it deployed. And then you can also say no thanks to the dependencies. You can see the database chart there has it enabled through. You can set that default in your values and that will allow you to use your own MySQL chart if it's MySQL or whatever that's in there, or MariaDB. So that's still giving a lot of power to your end user, but it's also sick giving them a lot of good, sane defaults and you don't have to redo a bunch of work. A couple of Helm lifecycle features I want to talk about, hooks and lookups. So hooks allow you to provide some control over the order in which things happen when you're deploying a chart or upgrading a chart. But Helm is not an orchestrator, right? So don't think you get full orchestration. You're getting some ability to say, oh, I need to make sure this config map needs to be created by God does, or more importantly, I'm about to do an upgrade and therefore I need my database to be, like, migrated to the new schema so I can have a hook to say, before I actually do the upgrade, run a job to migrate the schema, then do the upgrade. And they are defined as annotations in the templated resources, so they're pretty easy to slip in there and they can be done pre and post certain events through the lifecycle. They're pretty straightforward and there's not a ton of them. That's kind of the list. One here quick mention is there's a test one and so when you want to do integration tests, if you annotate resources with the test, they won't be rendered unless you run a Helm test and that way you can deploy your app and then you can run a Helm test and it will run jobs, pods, whatever you've asked for to actually validate that what you've got is what you asked for. And that's how we can then look at doing integration tests when we're actually wrapping it into our, like, software delivery pipelines. Now, lookups, maybe they're not exactly lifecycle-specific, but they provide a way for you to actually look at your Kubernetes cluster and find information on it. So there are times where maybe you need something that's in a config map but you need to actually put it in a manifest because your app doesn't know how to read a config map, so you can look up a particular key in a config map or you can look up anything else in Kubernetes and then render that into the template. And that can be incredibly useful at times, but it's kind of a last resort thing if you can't do it another way because if someone runs Helm template and just applies that, they might not have a Kubernetes cluster like that they're connecting to and so there's nothing to look up against and you'll get an error, right? So if you do something like, if you do lookups or you do hooks, you want to really think about that and use this use case and if they may be doing just a Helm template, make sure that they can still get a fully functioning application even if the hooks and the lookups don't function if you have them. So it's just something to keep in your mind. If you know that your users will never use Helm template, then of course you can get really deep into your hooks and lookups. And then of course we all want to be secure. None of us are security experts. Well, some of you might be, but I'm not. But I do know some things, right? I know that every pod should have a security context that says, don't run this container as root. So we should think about how we can integrate that with our charts to give users the ability to turn those on and off because some people will be deploying into insecure Kubernetes clusters and they won't care if it's root or not. Some people will be deploying into clusters that are inside of their enterprise that have very strict rules. They're running OPA, they're running Gatekeeper. They're running something that will actually break the chart from running, but they also have a good security context. So you want to provide that way for those people to use your chart. Otherwise it's going to fail and they're going to move on and they're not going to use your chart and they're not going to get the opportunity to use your work. And then things like config map secrets think about which values should be in which and also think about how people might use external sources like Mozilla SOPs, sealed secrets, or Hashicob Vault. So if you're providing secrets, provide them an alternative way to use something else so that they can use your app even if they're using, say, Hashicob Vault. Operators. So we're not at war with operators. There's not this big, like, operator-versed Helm thing. Deploying operators and their operands with Helm is such a common use case that if you go to Artifact Hub which Scott will talk about in a bit, you'll see a ton of operators and then operands in there that you can deploy using Helm. So Helm works pretty hard to support operators and that workflow. But as I said earlier, Helm is not an orchestrator and so it can't really be aware of the special activities that you might be done to upgrade, say, your specific operators' CRDs and the resources they manage, because sometimes when you're upgrading an operator, you have to do things in very specific order and Helm isn't really built to be that orchestrator. So we took a fairly naive approach and we said you can have a CRDs folder and anything you put in there is a straight Kubernetes file that is holding your CRDs. They're not even templates and we will deploy them before we render the templates and apply the templates to your Kubernetes cluster. If you need something more than that, you may need to reach outside of Helm and that's okay because we should be able to pipe these tools together. Some chart authors will actually put the CRDs into templates and just hope that they happen in the right order. Some people will do their CRDs as dependencies and, again, hope they happen in the right order. So you can do that but your mileage may vary and think about what you're doing when you're doing it and if it makes sense for you. So quickly under testing before I hand over to Andy and Scott, so we can do some basic testing locally and then we really want to push most of our testing to GitHub actions or to CircleCI or whatever. We've got Helm Lint which is going to basically check for templating errors and anti-patterns, anti-recommendations. So it'll shoot errors and warnings depending on how egregious your mistakes are. And then you can act on those and then we also have our mentioned Helm tests. So that's where you do integration tests. So you can deploy your chart, run tests, make sure everything is there and everything is how you want it to be and, again, you do that locally but you also want that to be part of your software which Scott will talk about. And I think that is about all I want to talk about. Oh, no, there was one other thing. Don't make clever charts, right? You're a human writing a thing for other humans to use. So they need to be easy to read, easy to understand and easy to customize. Otherwise you'll lose people. So always think about any time you're getting some complicated logic in there think about if that's really necessary or if it can be solved another way. So that is the end of my section. He has my mic on if you want to take my mic. So anyways, now you can all hear me. Let's start over. So once you go ahead and you have your file system with all your chart contents how do we go ahead and publish it and package it appropriately? We can use the Helm package command to take all those artifacts and put it into a compressed archive. That archive is very good, very simple but it's only step one. Step two is looking towards the future. The future is adopting a lot of the paradigms that are out there in the Kubernetes and cloud-needed ecosystem and being able to publish it using OCI content. How many of you have heard of image registries or registries at all that support additional content aside from traditional container images? How many of you are actually publishing any content whether it be Helm charts or any other content? How many of you are using the experimental feature of Helm? Okay, once again, it's experimental not quite ready for prime time it's closed, it's almost to the big leagues but not quite there yet. So the bigger question is for security, I'm very security minded I'm sure you're going to hear a lot about security in this little conference how many of you are signing your Helm charts? That's about right? For those of you who aren't signing your Helm charts is it either because you didn't know that there's a feature in them or is there another reason? Yes, it's hard It is hard because PGP the only supported mechanism for signing Helm charts is a pain, it really is I'll be honest I did struggle a little bit when I was preparing the demo on the show getting the keys set up, it's a pain it's okay to use the supported mechanisms of signing charts now all you have to do when you're packaging a chart is pass in the dash-sign flag point to where your key store is and you get a brand new signed Helm chart what it will do is it will create a providence file it's a separate file right next to your artifact a .prov file it's going to contain your detached PGP signature and some metadata about your chart check some as well as the chart.yaml file that's step one that's what's supported right now let's go ahead and glaze into the magic future, the future is going to be how do we make signing content easier and one project that aims to do so is called SigStore, how many of you have heard of SigStore? SigStore is a Linux foundation sponsored project that aims to make signing and verifying content easier and the Helm community has a plugin available for you to go ahead and take a signed chart that you use with Helm package dash-sign and upload it to what they call a transparency log, a transparency log allows anyone to go ahead and see if any content has been signed and verify it in a supportive way this is very common in the certificate world we want to extend it beyond certificates to artifacts as well so we can go ahead and take the Helm package that you created and publish it up to Recore Recore is the SigStore project that is a transparency log and we're going to walk through that in a demo in a second it is currently a first version of it right now out there in the community we're going to be working with the entire community as a whole to talk about how we can adapt, enable and advance the technology to potentially support additional mechanisms of signing content that's the goal so we might be able to get away from the pain of PGP let's go ahead and try to do a quick demo and the question is, we'll play there we go, I'm going to go out here because I can't see it so this is a little sample that is a tecton pipeline that enables you to publish, release and then use tools like chartrelease which we'll talk about to sign a content this is just a tecton pipeline that will do a complete release process basically take an existing chart package it up go ahead and push it out to github which is going to be our hosted chart repository and then use salm6 store to publish to the recore transparency log as you see we've gone ahead and published it out we have a release out in github we now have our chart repository using github pages so it's now available for verifying the helm chart there are two ways to do it, one is the supportive mechanism first thing we need to do is add that helm repository that we now have published to github we'll go ahead and add that we'll then pull in the latest updates to our local cache we'll then pull in that chart as you see we have the two files the helm archive and the provenance file then we can go ahead and verify that chart we'll go ahead and use the helm verify command that's built in supportive tool to verify the chart now let's go ahead and let's look into the future and see how we can do it with sigstore the first thing we need to do to enable sigstore verification is to add the helm sigstore plugin so we'll go out add the helm sigstore plugin we'll then go ahead and use the helm sigstore verify command and point it at our archive that we have locally it's going to go to record say is there content up there if there is great see if it's there and then do a quick verification of it using a lot of the same things that helm does we can then go ahead and take that uuid that came out from it this is a universal id in record and go ahead and actually see the data in record in record we capture the signature the public key and other key content so you can really understand what content out there so you can verify so nothing's been tampered with or modified as part of your software supply chain this is key being able to do this extend it to because we've already done it for images let's go ahead and extend it to helm content as well so as you see we went ahead and provided an automated packaging signing and release process using tecton we populated a helm chart repository we've published that to record we've gone ahead and used this helm sigstore plugin to be able to verify this as well as using the integrated helm solutions I'm going to pass it over to Scott who's going to talk a little bit more about testing as well as some publishing mechanisms awesome thanks Andy Scott take your mask off um okay is that a little bit better how about I try no it's not better is that better okay mini mic mini mic time okay I'll just hold it like this we'll just we'll just move on because I know we have five minutes thank you Allison um hybrid events are weird um what happened here is it still okay no here we go ah alright um uh hello hello hello okay thank you Allison let's give Allison a big hand and the tech people oh my gosh okay perfect alright so um Andy was just talking about sigstore this is our transition slide uh Paul earlier talked about chart testing that you do during your chart building and just for a time check I think we have five minutes I'll try to keep it under that so that we might be able to squeeze in a question to and also if we don't get to that just remember we are here we can chat with you afterwards there's also Helm folks online but everybody in the room come chat with us if we don't get to too many Q&As questions okay so um basically uh that's the that's the stage you want to use helm testing you want to make sure to do that it's really great um just remember uh you know even if you don't have helm tests even if you haven't written helm tests you can still use helm testing because it's a great it's a great tool um there is once you get past the just the helm test command that you run locally uh there is a great project out there um that members of the helm charts and chart tooling team have been managing all this time including Reinhardt and and other fine folks here uh called helm slash chart dash testing use that when you get to you can actually use that locally not just for automation but especially once you get to the automation phase it will run other it will do other things besides just run tests you may not have written so if your charts don't have tests which many don't it will actually help by installing those charts like you know spinning up a kind cluster installing those chart or at least the action does you can do this yourself in any way you want it's try to install those charts um it even has flags to help you um if you have any values in the values file that are uh uh star dash values it will it will automatically run those sequentially so you can actually have dev dot values uh or any other test cases that you want and the helm testing tool automatically run those for you um and the last thing is really important that I wanted to just mention before moving quickly from from testing is um uh the helm chart testing tool can allow can help you make sure to automate that upgrading between versions of charts works so you can pass that flag and it won't be the ability to upgrade and it will do that just go to helm chart testing uh github repo all the commands are there all the flags are there um I won't like get to into the details here but please just like if you walk away with anything around testing do it please it's really awesome um okay uh so um uh let's see wait did I actually mention um the github action I did okay so so here just some links and we'll make these slides available to everyone but um that's where you find the stuff and then also there are there is a demo I want to point you all to that some people haven't noticed that's in the helm uh github org called um charts repo actions demo and uh yeah so that is something that we wrote to be able to help people just see an exact working example um if you find it's behind open a PR but it's usually kept pretty up to excuse me pretty up to date um and uh you can pretty much just emulate that and you're already on your way so there are lots of other cool tips and tricks you can do Paul mentioned annotations earlier the the um the hook the helm hooks for testing um just one tiny tip I want to mention before we move on is that um you can use Helms post-delete hook to uh mixed with the test annotation to have specific things that you want to as Paul was mentioning you can have specific uh resources that you only need for testing if you use both of those uh with or without chart testing tool it will automatically clean up for you when you run your test so that is a really sweet thing to do okay um so just let's see you turn check here 1134 okay we got one more minute so publishing charts uh I just want to mention what it is it's a core concept for Helm um this was already discussed a bit packages were discussed um I think the most common thing is using well what we're seeing is using github pages to package them uh but there's other methods there are just the main thing you need to keep in mind is it's just an index file with metadata that points to a chart excuse me a helm uh repo it's just an index file with metadata that points to locations of your packages that's it um and so it can just be an HTTP server it can be uh there are other awesome projects in the ecosystem like chart museum um the still experimental OCI um hopefully not for very long um you can have public repose private repose etc it's just a matter of controlling access to those um uh HTTP web servers basically however you have you make them work so um uh the nice thing that to keep in mind is that we have um we have another tool that's very similar to chart chart testing for the releasing site called chart releaser and that can that right now integrates with github because let's face it when you when you uh one of the old ways as you you upload your index file uh you also upload a ton of binaries um you can see that on um if you go for an example to the Prometheus community helm charts repo you'll see that there's a lot of chart binaries in the github pages branch that's not the way it's working now that's just for backwards compatibility with older with older charts before we move to that we now use a whole chart releaser which also has an accompanying wrapping action that will uh use the concept of releases you know github releases where you download each release for for software we attach that excuse me that tool attaches the binaries the tar balls to the release that it creates for that version of that chart for each chart so even if you have a repo with multiple charts there will be releases prefixed by default with the name of the chart the version of the chart and that release will have the the chart artifacts connected to it so this will do it for you again we're talking not to chart consumers right now or you you all are people who are interested in publishing charts and distributing them so this will help you a lot if you're not already using it um yeah and uh yeah so i just want to uh end i think by saying um i'm sorry we are over time aren't we okay do we have a moment okay we'll just end by saying that what do you do when your your chart repos are distributed right please check out and use artifact hub it's a central cncf project that is there um to uh what do they say to find install and publish kubernetes packages of various kinds so it includes support for helm charts and plugins it also has many other types of artifacts including excuse me including cube control plugins etc so please check that out and there are really great features that as chart publishers and chart distributors you really want to make use of um the only few that i want to mention that you just want to have in your brain is um uh there there are artifact hub supported annotations that you can add to your chart.yaml file under the annotation section and that really can help you drive your users to important information about your charts for example where's the source code and things like that uh related packages and other and other such things um i'd be really remiss in not saying that uh if you all remember the the github.com helm slash charts repo that's now archived a lot of people were concerned where did all that stuff go well all the history is there if you read the helm blog posts uh luckily github has been very generous picked up the baton from google and is now hosting all of those all that chart package history but uh for ongoing chart work new charts the way to find it is artifact hub and so as chart distributors it's your responsibility to make sure to list that in the centralized cncf artifact hub just like you would if you were supporting um a yum or an apt package you would have to make sure that that is listed um and aggregated for discoverability um there's a lot of great features uh for end users that i will not mention but things that are extremely important for owners is if you are or want to be an owner of one of the helm charts that is currently archived in the chart the helm slash charts repo it doesn't yet have a home right you want to work on it your company is working on it you use it all the time you're making improvements maybe you even have a fork of that consider owning that and consider making that available for others and if you if you want to take ownership of that there is a feature in artifact hub to claim ownership because we migrated all of those old packages to artifact hub but most of those are under the helm org right now because no one wanted to claim them um i mean many of them did but there there's still a lot that haven't yet to be claimed so please do that talk with your colleagues talk with people who are supporting those applications that those charts are distributing and make sure if you can please to contact us ping and slack whatever you need to do but let's just keep that ecosystem rolling because the reason helm is so important is because of people like you um obviously the end user is the most important user but if we don't continue to keep the momentum of the charts in this ecosystem then they won't have anything to install it'll just be a cool templating tool for most people and not really a package management system so thank you all very much and um let's see i think that's it thank you