 Howdy, folks. It's 3.50 in the afternoon. You must be relatively tired. Let's all stand up for just one moment, just really quickly. Just do some stretching. Yes, that is good. That is good. Now you can all sit down and promptly fall asleep. Howdy, my name is Vic. I'm a Cloud Solutions Architect with Google. I live out of Santa Barbara, California. Some contact info for me. It'll be on the last slide if you need it. We're here to talk about what we've learned from the Helm charts repo and things that you can use in your own home charts to make them more flexible, more scalable, just better for your users in general. So how many people have used Helm? It's a lot. How many people use Helm in production? How many people have used Helm charts from the charts repo? Wow, that's amazing. And how many people write their own charts? Yes, sweet. OK, you guys are committed. I like it. So I've got this Kubernetes problem. I'm into it. I get into Kubernetes mode, and it just makes me go all crazy. Today, we're going to go a little bit, Helm 101, for those of you who didn't get an opportunity to use Helm in the past. But as you can see, we've got a good community of users, and it's a useful tool. It can be a useful tool for you. Then we're going to go through what the charts repo is, why we started it, and that kind of thing. And then talk a little bit about the important pieces of charts, and then a couple of tips and tricks that we've seen in some of the charts that we have in the chart repo, some of the patterns that we've seen emerge for making them better charts for your users. So Helm 101, Helm is a tool for installing and managing your Kubernetes application. So it helps you with the lifecycle management of an application within Kubernetes, and we'll talk about what that is. It has two parts. You install a client on your laptop, and then inside your Kubernetes cluster, you have a server called the tiller, and that manages receiving the commands and doing the actual work of deploying manifest into Kubernetes. So Helm, if you're looking for an analog in the operating system world, it would be like apt-get or yum. And then you have charts, which are more like your packages, so your RPMs or your Debian. And chart has metadata about what's in the chart, what it is, and then a bunch of manifests that are templated so that you can have ease of use and pass some values through it. So what is a Kubernetes application? This is something that's kind of been tough to figure out over time, like what is an application? And generally, it's going to be a set of things that you install into Kubernetes. They may relate to each other. You may have a config map that needs to be attached to a pod. You may have a volume that needs to be attached to a pod, and then a pod that needs to be load balanced by a service. So these things are kind of all interrelated. You can't really update one without thinking about the other at the very least. And they're all written in YAML for most of us. How many people do YAML for Kubernetes manifests? Yeah, most of us. So Helm allows you to package these things together. So you can templatize them and use the same values across a bunch of these manifests, and then install them all together. For your users of these packages or these charts, they can see just a simplified config, right? The things that are important to the application. They don't have to worry about the internals of how your services are mapped to each other or how your deployments work or what variables you're using. You can provide them kind of a higher level abstraction API via the values file or the configuration of your Helm chart. So you deploy your Helm chart, and then you deploy a new version, and that's awesome, and you're stoked on your new version, and then you hit a problem and you're like, wait, we need to roll back. So in this case, Helm helps you because it can roll back everything as a whole, right? You don't just roll back the config map. You don't just roll back the deployment. You roll everything back together and get back to the original state you had before. And then you can roll out version three and keep going along the way. So I've talked about templating. I've said templating probably 50 times already. In Helm, you use Go templates, and those are passed into your, you have values passed into your templates that you can kind of parse out some values there. So for example here, I have this replicas that I've provided to my users where they can change the number of replicas by using the values file. You can also use similar names for things. So if your application is called VIX app, I can call my service VIX app and my deployment VIX app with just these templated values. Some of the cool things that you get out of using Helm on top of just regular Go templating is we have these things called sprig functions. And these are little helper functions that make it easier to do some of the more complex things like have a default value or writing a checksum for a particular config file. So here are some of the things that sprig allows you to do. So on Go template, we have all kinds of things we can do like loops and stuff like that, but sprig allows you to do math functions or maybe you need to add a date or a timestamp somewhere. Maybe you need to base 64 and code a config file for some reason or another. So this gives you a lot of flexibility as far as how you're writing your configs and your services and stuff like that. Cool. So the charts repo, Kubernetes slash charts on GitHub. It's the centralized repository where we're hosting and maintaining the charts that you as a community have provided. How many people have interacted with the charts repo in here? Awesome. We love you. Thank you. And it's basically just a place where we can get together and take our knowledge of applications and how you configure them and how you use them and so that everybody has at least a starting point. I don't necessarily expect that you're gonna have production workload running off of one of our Helm charts, but maybe it's the first step for you to get there. We provide continuous integration, so we have a testing subsystem that will run your chart and make sure that it passes the linter, make sure it actually installs properly, make sure that pods go running. And then we have code review provided by some of our maintainers. Do we have any maintainers in here right now other than myself? We have one right here. That's it. Oh, one in the back too. So they caused all the problems. And it's the default repository when you install Helm. So the stable repository when you install the Helm client or you do a Helm search, those are the charts that we have there. So why did we do this? I don't know if you guys remember in like a year and a half ago if you did like Postgres, Kubernetes, Google search, you'd end up with lots of results, lots of different repos that had similar ideas on how to do things and were maybe had trade offs as to what they did. And so you'd have to look through 10 results and decide which one was best for you. We wanted a place where we could just all work together on the same thing and make it as easy as possible to have a starting point for those common apps. So if there was a bunch of things, then we added one more thing, creating a new standard. But this time I think it's stuck around. We have 130 charts now. A year ago, I think we had nine or something like that. So it's grown pretty fast. We've merged 2,000 PRs over that period. So that's improvements to charts and new charts. And we have over 100 contributors which are the folks that you saw raise their hands, which is super awesome. It's not slowing down in any way, shape or form. We're still kind of hammering through lots of PRs. About six months ago, we had like this high watermark of about 100 PRs being like we need to drop everything and figure out what's going on here. Now the high watermark is 200. I expect in another three months, it's gonna be 300. We'd love to have more maintainers if you're interested to keep those watermarks down, but it's growing really fast and it's awesome. We have 10,000 unique visitors every week to the charts GitHub repository, which is a lot of people. And the number of forks, we can no longer see our fork graph, which is a really great display, but we do have the number. And this is cool for me, not like as a vanity metric, but this means that people are actually taking the code and doing something with it. Maybe they're changing it, maybe they're contributing back, but at least they have that starting point and we're kind of the known place to go for those starting points for these applications. Cool. So next we'll move on to kind of the chart structure and things that are important with the files and charts. So the chart structure, there's a certain set of files that make up a chart. One of the most important ones is the chart YAML, this is the metadata about what your application is. What thing is this installing? So here's the example from the Spinnaker chart. You have an API version, that's the API version of the chart right now, we're only at V1. You have the description, which is a human readable way to describe what you're actually installing. This shows up in the Helm search output, for example. You have the name of your chart, then you have two version strings here. So version is the version of the chart and app version is what the application version is. So for example, here I have version 1.1.0 of Spinnaker and the chart version, the iteration of this deployment mechanism is 0.3.8. This is important, we've had some chart maintainers try to pin these things and what you find is over time, your application doesn't change as much as your chart, you're adding things that make your chart more flexible or a bug fix in just your chart and then you're adding bumping versions. Mike, goodness here, was one of our first ones to try that with the Prometheus chart. I think he went about three months with that methodology and then we just decided to break from it. You have a website here, which is just a URL to point people to, but I think the more important thing to have in your chart YAML is the sources. Where can people go to rebuild the pieces of this? So for example, usually people put the GitHub of the actual project itself, a lot of the charts that we have are open source, so where's the GitHub for that thing? And then did you build like a custom image? You have a custom image point people to that Docker file so they can maybe customize it and move on from there. You have an icon, which is mostly just a visualization tool. We have kubeapps.com, which uses this to show your chart icon. Then we have the maintainer, so the name here is actually the GitHub ID of that user and their email. So as maintainers, we look at this and if there's something that's a little bit overhead, something that's not just a trivial bug fix, we try to ping those maintainers to make sure that this is a good change that they want in their chart. Now this is the values file. The values file is basically the API that you're providing to your users of your chart. That's the way to look at it. Changes to this that are backwards incompatible, you should probably bump the major version of your chart to show people that you are breaking the API. Here's where you can abstract away some of the complexity of what you've done in the Kubernetes deployment methodology and make it simpler for your users to configure things. You can add comments, generally people comment in how to do certain things. For example, here, this is again the Spinnaker chart. I allow people to change the images that are deployed for each component and also, for example, allow them to configure how mail works. It's disabled by default, but I have just the parameters that they need and they don't have to go look in the Spinnaker configuration to figure out what things to configure there. So with the values file, generally what you'll find is you're on this spectrum or just charts in general. You're start out kind of relatively reproducible. There's not that much that can change, but as people use your chart and your application, you'll find that you're going more and more to be more flexible, which means that you're opening up some trap doors for people to change things that maybe you can, maybe you can't reproduce on your own, and that's just kind of a natural progression at this point that we've seen. Another thing that we have in a chart is the requirements YAML, and this is what defines what other charts you may want to deploy alongside your chart. So in general, what we've seen is this is used for data stores. You have a Postgres dependency, you have a Redis dependency or anything like that. And the version here can also be a Semver, so I can say, you know, give me anything in the zero for X series of the Redis chart, and you'll update to those things. Then you have your templates folder, and this is kind of the core of the chart or the kind of simplest place. You have all of your Kubernetes manifests that are templated out or can be templated, and this is what will be kind of cube controlled applied to a certain extent into your Kubernetes cluster. Next you have the notes file, and this is one of my favorite kind of experiences in Helm or using Helm is that the notes you can, as a chart maintainer or a chart creator, you can provide to your users a getting started experience via the notes file, and the notes file is again a templated file that is templated out and spit out after your chart is installed. So when you do Helm install chart, they'll spit out this templated thing that can maybe tell your users what to do next. Instead of just saying, yes, it's installed, tell them, hey, here's how you connect to it. Here's how you get to the web UI. Here's if you're gonna use it inside of your cluster, here's the service name that's currently configured. So it's just getting people that much closer to being kind of interacting with your application. Next is a read me, and this is generally kind of more in-depth documentation on what people can do with your chart, and it may have methodologies for configuration or things to watch out for or disclaimers, things like that, it's just a markdown document and kind of gets into more in-depth information. So now we're gonna go look at some tricks that you can use inside of your charts and things that you can look at to kind of make your charts a little bit more flexible or a little bit more useful. I think the first thing that most people should be using is when they're first gonna create their charts to use the helm create command. How many people have used helm create or use it regularly? That's good. So helm create scaffolds out a chart, an example chart for you. So you can start with this and then tweak it from there. The actual application that it deploys is just an nginx web server with no config, but it just shows you the things, the elements that you're gonna have to have, and we try to keep that up to date with our best practices. So if you're gonna get something into the charts repo, it's best to start with this and then you know you've got something that's gonna match our expectations. Another thing you can do is to add starters. So let's say in your organization, you have certain patterns or certain things that you wanna have in each helm chart. You can create your own starter templates and then pass them to the helm create command with the dash dash starter flag and have that be the scaffolded thing that gets put out there for users of your charts. So super useful. One example is you have a certain paradigm for how you do staple sets or for your services you have a certain way that you wanna do your deployment rollouts or your deployment strategy. You wanna do max unavailable always. So just set that for folks so that they have it by default. Maybe you have a certain way that your values files are laid out. This is where you can put all of those semantics in place. This is a trick that I really like. I've used it in a couple of charts and I know that some charts use it as well and it's slightly complicated but it's a very useful tool. So what I wanna do here is I want to be able to change a config map in my application, in my chart and have it automatically update my deployment. So what I'm gonna do here is add an annotation to the deployment and I'm gonna use sprig functions to put in a check sum of my configuration. So here you can see I have the, I'm printing a full templated config file that spinnaker-config.yaml and then getting the check sum of that contents. When I get that I'm putting that into an annotation. So every time the config file changes, it's gonna roll out a new deployment because that annotation changed. So it's kind of a little trick to get config maps to automatically update. Some applications will just detect that the config map has changed and reload themselves but some aren't that smart. So you can do this to kind of hack around those applications that don't know about config maps. Another super useful tool that I've used in some of the less than cloud native kind of applications that I've had to install is the Helm hooks. So Helm lets you do some things as it releases or as it installs or deletes your application. So here I'm doing a post install step. So let's say you deploy an application and you just need to like run one command at the end. Like for example in this one, this is again the spinnaker chart and I need to create a bucket in the Minio object storage after it's installed. So Helm allows me to add an annotation to a job. That job is gonna run until it completes part of the job controller and all I'm really doing in that job is just running the Minio client in order to create the bucket. So as Minio is coming up, this thing's gonna fail and Kubernetes is gonna keep restarting it until it succeeds once and then it's done. So now I've gotten those kind of post init kind of steps automated through my chart. The types of hooks that you have available are pre and post install, pre and post delete and pre and post upgrade and pre and post rollback. So you have a lot of control over your application life cycle. You can imagine doing migrations and those kinds of things in one of these hooks. All kinds of stuff that you might have to do at install or just normally throughout the process of managing your application. This one is pretty useful. Something we saw I think first in the Prometheus chart is to allow people to basically provide you with a volume for data. And so you let them provide you with the name of an existing persistent volume claim and if they provide that to you, you use that rather than one that you would have originally provisioned in the chart. So you might have the need to decouple your chart life cycle from your data life cycle. Maybe you don't want those ever to, you don't want someone to accidentally delete data or you have a developer who has a PVC that they're like special data set PVC that they wanna reuse with your application chart over and over, you can give them that ability here. Another thing that we've seen often is web apps being submitted to the charts repo. And with those web apps, we generally ask that they allow someone to add an ingress really easily. Like pre configure this ingress. You wire up the service and port already for them. You can have the ability for them to list house names or maybe a TLS secret to include in there but try to have that as part of your API if you are a web app, because most things will want to do an ingress. Another side of configuring an ingress for people is to have the annotations be available as a thing that you pass in to those ingresses. So annotations on ingresses that are really common are for example, like which type of ingress controller to use, you might use the Nginx ingress controller, you might use the cloud default one, but you as a chart maintainer should not dictate that necessarily. Another one that's seen commonly is like cube Lego, go get my TLS certs. You don't want to necessarily manage that, you don't want to over abstract that, just let them add their own annotations. Annotations are kind of like Pandora's box anyway, all kinds of crazy stuff gets done with them. So I tend to just let people add those crazy annotations there. This one is near and dear to my heart. Michelle Narali wrote this helm test thing, and we're like please we need helm test because as maintainers we want to have the ability for the chart maintainer to send us a test that says this thing actually works. So now you can do helm test from the helm client after a release has been installed and it will run a step. And so here I add an annotation saying that I want this test to run and I expect it to succeed. Here I'm running a pod and it's just gonna run a bash script. Like most things the world is tied together via bash scripts. So here this is the actual test that's in the Jenkins chart. Not the craziest thing in the world, but it lets us know that like we can at least get a 200 from the login page. It's something simple like this that as chart maintainers we can bank on to think that your chart's actually doing something useful. In your organization you may have much more in depth testing and you can run this at any time. So maybe you just run this periodically. You have some checks that you wanna run as part of automation that runs every two hours or something like that and it just spot checks things. It's also templated so it's gonna be able to use all those values that you've passed around as well. So it's a useful hook to have. We hope to have more tests in the charts but I think right now out of the 130 less than 10% have tests which is pretty painful. It means we have to go and launch it and make sure that it actually does anything useful and that the PR that we're accepting doesn't actually break core functionality for that chart. This is one that I think is really useful but is also one of these trap doors that people kind of, once they go down this route you kind of, they're on their own. Some people will want to, if you have a config file for your application rather than have you template it out, for example for an Nginx config or something like that, they wanna just provide their own. So some charts have done custom config map parameter that allows you to overwrite or provide your own config map. This is like on the opposite side of the reproducibility spectrum. I mean you're not really gonna be able to reproduce what they did basically ever but it does open up more use cases for your chart which I think is useful. Another thing we've seen in the Nginx controller chart is not just having them overwrite the config map but to provide the name of a config map. So can be done a few different ways. One thing we've done recently is to have all of the helper templates in Helm charts be namespaced. So a helper template, if you look at a chart you'll probably have an underscore helpers file in there and this is where you can define little chunks of snippets of code that you wanna actually inject in different places in your chart but you don't wanna have a big stack of code sitting there. So these templates can be generally for names of things in Kubernetes you have to have things be a certain length. So if people name things in a strange way you wanna truncate that to something that's supported within Kubernetes so that's what we do with names and full names. Name spacing it allows us if you have this chart as a sub chart for example you can always reference the Jenkins name by Jenkins.name across the board. You don't have to think about how it's actually nested or anything like that. This one is also really useful. So conditional dependencies is a feature of the requirements file. A lot of charts, we expect charts to actually provide something useful out of the box without any changes to the values but you may have some data stores for example that we deploy for you as the chart that you already have somewhere else. So one example is the Spinnaker chart actually does needs object storage. So we use Minio by default because we can provision that within anybody's cluster and we get out of the box experience that's good. Within cloud providers though you wanna use their object storage. So in Google Cloud we have Google Cloud Storage and in Amazon you might have S3 so you might wanna disable the Minio deployment and you can do that with a value called Minio.enabled. So if you set that to false we will no longer deploy the Minio chart for you and you are on your own to configure the S3 configuration. Again on the kind of opposite end of the spectrum these are some more of the trap doors that we've seen I mean chart maintainers provide to their users is the using the two YAML function in Sprig which will basically take a chunk of the values file and just inject it as YAML into some other template that you have. So where this is really useful and kind of necessary is with like node selectors and tolerations specifically because these are gonna be practically bespoke across everybody's deployment. You're not gonna know how they label their deployment or their nodes or anything like that. So just give them the ability to add node selectors on their own in the values file and then propagate those through to the deployments you have similarly with tolerations. We also have seen this relatively often with resources some people wanna have requests and limits some people only want limits just give them the ability to set those however they want through the values file and pass that directly into your pod specs. So those are some of the tips and tricks. We have 130 charts out there. So if you have an application that you wanna deploy go look at the charts repo and see if there's something that's similar to it from a kind of deployment mechanism standpoint and try to figure out if you can kind of take some tips from some of the charts that already exist. For us in the charts repo the next thing we're gonna try to do is give the chart maintainers more power over their actual chart and merging their own PRs. That's gonna help us with our velocity so we don't hit that 300 PR mark that is almost inevitable at this point. And also to delegate the responsibility to maybe even a different get repo. So if you wanna host your get repo or your chart somewhere else but still get it into the stable chart repository we wanna be able to facilitate that as well. Now that we're at 130 charts it's time to look at how many of those are actually updated or useful and are still maintained. I think we'll get at least a 15% attrition rate on those once we really start looking at it and just remove those from the tree or put them in a folder somewhere like deprecated or something like that. And if someone wants to bring them back to life that's great maybe we'll call it zombie. And then we can go forward with it. The other thing is more functional tasks for charts. There's an idea being thrown around that will require a functional test or else we won't merge any more PRs in that. I don't know if that makes you scared or not but it should be scary for any maintainers. But the functional test is really important to us. It's something that gives us more comfort in merging things and will give us better velocity as chart maintainers. So that's all I've got. Thank you folks and I'll do questions out in the back.