 Hi everybody. My name is Gabby. My name is Phil Whitrock. And we are both engineers from the Kubernetes team at Apple and today we're really excited to talk to you about 535,600 CELIs. You don't want to hear us saying the entire rent soundtrack in this talk so we'll keep that part short. But the jokes aside we're really excited to talk about the variety Kubernetes CLIs out there today and to help demystify the space. While many of us are familiar with Kube-Cuttle, Kube-Cuttle has a lot of company among Kubernetes CLIs. Whether you're getting started running your first pod or running sophisticated applications in production, there are many CLIs in the Kubernetes ecosystem that can help you describe how you run applications. Though some of these tools have become more popular than others, the number of distinct solutions continues to grow. This means that staying on top of the latest developments and the most innovative techniques is next to impossible. Sometimes looking at the landscape of Kubernetes CLIs feels like looking at everybody's favorite CNCF landscape chart. It's overwhelming and it frequently changes. So rather than going through each CLI individually and telling you what it does, really the goal of this talk is to introduce a model for making sense of that vast landscape so that you can really navigate it on your own. And we're going to identify a couple categories of problems that CLIs typically try to address and then the ways that they try to address them. So the first problem we're going to look at is you wanted to find your application in some higher level construct that maps to some lower level APIs. So for example, what you want to do is run a Golang program rather than saying, I want to run this deployment service config map, etc. The second sort of problem we often see is you really want to run the same construct in varying ways across environments and the ways they vary usually are at a relatively low level. So for instance, you want to run your Golang program in staging, canary, prod US West, prod US East, all with slightly varying values. And the third problem we're going to look at is really about taking high level supplemental concerns that are loosely coupled to any specific abstraction or any specific invariant and really just cut across them. Right. So for instance, you want to set up logging or some sort of a sidecar container for all your applications, whether they are Golang or Spring Boot or Rails or Node.js or something else. And when you're building a production grade system, you're likely going to need to overcome all of these problems, right? So for instance, you're going to want to say, I want to run my Golang application. I want to run it with bunk logging and I want to run it in staging, canary and production, all with slightly varying values. A firm understanding of how to select among the ecosystem CLIs, as well as combine them can help deliver substantial outcomes. And these include architecting your first Kubernetes application all the way to building comprehensive continuous integration pipelines. So the CLIs approach these problems in a couple of different ways. And the way a CLI approaches a problem really has a big impact on what trade-offs it offers and how well it's able to address each one. So we're going to go through each of the kind of high level approaches that CLIs use to tackle these problems and show an example using a real tool. So templating is the first approach we're going to look at. Templating is a relatively simple approach for solving abstraction. You have a template that accepts some predefined set of input values. And then that produces a set of output resources. So for instance, in our example, you would have some Golang template that accepts some inputs and then produces a deployment service. All right, let's hop into our first demo. We're going to deploy something a lot of people deploy to their Kubernetes clusters, which is kubestate metrics exposed via Prometheus. The chart.yaml we have here is pulled from the Prometheus project's official Helm chart repository on GitHub. So looking at this chart.yaml file, which serves as kind of the overarching document over this entire Helm chart, you'll notice that this file describes all of the applications that are actually going to be deployed as part of this chart. As well as metadata about version as well as maintainers. And another element I want to focus on here is the fact that this is a Prometheus project, meaning that kubestate metrics is actually not part of this chart itself and is pulled from another Helm chart. So we have an example of cross cutting abstraction here. And finally, you'll notice that we have a templates directory, which includes templates for all of the YAML that will be generated as part of this Helm chart, and whose values we can actually modify using values.yaml, which just gives us a lot of variables for customizing what our settings are, our configurations are in this Helm chart. So let's get started then. I'm going to deploy all of these services through this chart to my local client cluster. So let's install this. We'll name our application cube metrics. Finally, we'll pass in the directory including our chart and all of its templates. And you can see all of the output we have here corresponding to actually starting all of our Prometheus and kubestate metrics deployments. And let's see if our deployments are there. And you can see they were all spun up and this required very little involvement from us. It was just all abstracted away and ready to use. Right. And so Gavin just demonstrated using Helm with templating. Some of the great things about templating is a relatively simple approach. It's relatively easy to understand what's going on. And it's relatively mature. And sort of the limitations of templating are that there's really only so much you can put into the template as inputs, right? So you can't supply every field that you want to maybe set in the low-level types. And it doesn't natively really tackle variants or cross-cutting concerns. The second sort of approach we're going to look at is YAML composition. And so an example of this is customized. With YAML composition, the inputs specified by the user are still in the native form of the API that the server supports. And so really it's about breaking up your YAML configuration files into composable pieces and factoring out either a common thing like a namespace from a bunch of different YAML files so you can only specify it in one place. Or factoring out some common piece of configuration from a base where you can use that to apply to different variant environments. In fact, you can layer customize on top of other CLIs more focused on abstraction like Helm. Using customize, you can post process Helm charts and create variants out of the box applications. And even better, you can generate these variants instead of having to manually fork and modify your application configuration. All right. So in our last example, we briefly covered how you could customize the configuration of your Helm chart application using the values.yaml file. But what about the case where you actually want to have two different variants of these Helm charts? One for a staging and one for a production cluster. And just to dive into this example a little more, let's say I wanted to have different metrics retention dates for my different Prometheus servers deployed to staging versus production clusters. So we can pull out the retention date we currently have by default by just getting the deployment and gripping out what retention is. So you could see that we have a retention argument passed to one of our containers. That is for 15 days. How could I make it seven days for staging because we don't care about staging and metrics retention as much, but also set it to 30 days for production clusters. We can do this using customize and a concept called overlays. And what overlays do is allow you to apply patches to create variants of a base source of configuration. In this case, we're going to use Helm as our base. So you can see that in this patch here, what we do is we just provide enough information for a customize to know how to map this patch to a particular pod in deployment. So you can see that we provide the necessary matching labels and we then provide an override for the argument that we want to set to seven days in our staging instance. And likewise, we have a patch in our production overlay. That's for 30 days. So to run this, we can create a small script around the customized CLI that allows us to supply these patches whenever we want. And we can invoke in this example, which patch we want to use by using a environment variable to say what environment we're deploying to. So we are going to set our deployment environment to staging. And then we're going to run Helm install similar to last time. But we're going to add a new argument called post renderer. And what this does is it allows us to pass in our customized script to apply our desired overlay. So let's run this and apply this to our cluster. You can see we got the same output as last time indicating that we got the expected deployment set up. Let's quickly go back and run and see what our retention policy looks like now. And you'll see that instead of 15 days earlier, now it's just seven days. So this overlay was applied as expected to our staging cluster. Thanks for the demo, Gabby. And so as we saw that YAML composition is great for producing like a variant or factoring out some piece of YAML from another piece. YAML composition allows multiple layers to be defined. So unlike templating, which tend to be binary relationships, you can stack more and more layers on top of each other. And they do stage true to the underlying API. So you can understand like how you, if you want to set a particular field, it's relatively straightforward to understand how to set up that field. Really, their limitations are that one, they cannot express high level distractions. You are operating at a low level, inserting snippets here and there. So if you just want to say set this field, and then it goes and generates a bunch of different other stuff, that's not something you're going to get from YAML composition. And it can't express any sort of dynamic or reactive logic. So that means if you are patching another layer, and that layer changes underneath you, your patch is not going to be able to react to that. It's just going to be static and that can be brittle. Sorry, I'm going to have to turn on my do not disturb. There we go. Sorry, there was a little ping. That's annoying. Okay. And starting. A third approach we're going to look at are configuration domain specific languages. And configuration DSLs are languages specifically designed for users to express configuration and solve problems related to configuration. They're optimized for expressing and composing and transforming configuration data. And they frequently use something like inheritance or unification as their primary technique. In this demo, we'll take a look at the BSL Q-Lang. And what Q-Lang does is it allows you to create this notion of a patch that's applied to base deployments. So let's actually take a look at this patch first. And you can see that this patch is responsible for providing default values, but also imposing types on certain fields that are supposed to be populated by the base values that I'll show you above. But you can see that replicas are meant to be integers and components are meant to be strings. And scrolling up here, we can see that we've actually been able to populate those typed values with the actual values of their own. So we have one for staging that has the seven-day retention period we're now familiar with. And we also have a production one with a 30-day retention day period. So let's take a look at running this. And you can see that what is spat out are our deployment manifests with our corresponding customer retention dates. And just to show this typing in action, let's set replicas up here to a string, which it's not supposed to be. And you'll see that Q was able to give us an error related to this mismatch between types. And so summarizing domain-specific languages, they're really a one-stop shop for expressing your configuration. You can do all your configuration stuff in them. They really are focused on providing a configuration-first experience. And they have a slightly different way of thinking than standard traditional programming where it's really more about something like inheritance. Some of the cons are that because they're a different model than imperative programming languages, you need to learn how to use that particular DSL. And they really need to be standardized in an organization. If you're having different sorts of DSLs in your organization and some DSLs are used for one particular type of workload and another DSL is used for another type of workload, then that can be really difficult to build up a good set of libraries. And then a tip with DSLs is because they are not general-purpose languages that most folks are familiar with how to debug and have all the debug tools that they're used to. You really want to avoid doing clever things that can make the control flow hard to follow. Moving on to the general purpose language techniques. So this is kind of following in the idea of why don't we do configuration as code. And instead of having a kind of new thing that folks have to learn, I trade that sort of customized model focused on configuration for a model that just uses a programming language people are familiar with. It's often part of a framework which can do more than just expressing configuration but can also manage stuff like rolling out your configuration or solving other problems for you. And oftentimes these frameworks are capable of pulling in configuration from other sources so you can use helm or customize or a DSL tool with them as an input source. And this example will briefly cover Pulumi which allows you to write go that is compiled into an executable that Pulumi runs to install whatever is defined in this executable to an actual Kubernetes cluster. So you can see that we can provide configuration arguments such as what destination or type of cluster we're deploying this to and then write imperative logic to change the value of certain variables depending on where we're going to be using these variables. And down here you can see that the Prometheus server deployment that we've been using throughout our demos is now represented as a classic ghost truck that makes use of the variables we set imperatively above. So this quacks like go, it looks like go because it is go and that is one of the powers of Pulumi. So looking at general purpose languages the pros are they're a familiar approach because they have reusable libraries, reusable patterns, IDs and debuggers folks are used to and they may come with a built-in support for tackling other problems and they can usually ingest configuration from other sources as well. Some of the limitations of general purpose languages is that the languages themselves are typically not optimized for expressing configuration so it's a little more challenging than the DSL would provide. There is complexity that comes with writing a program like the ability to just look at the program and know that this is going to output the right thing is more challenging when you have functions and you have for loops and branches and these sorts of things. And that when they are packaged with an orchestration solution that it may be difficult to tease the two apart and they may be tightly coupled enough that you can't just use the general purpose SDK with your arbitrary orchestration solution you're using today. And so the final approach we're going to look at is modeling client-side configuration similar to server-side APIs or server-side really configuration. And so the idea here is you write modules or functions which look at the holistic state of the system similar to how controllers look at the holistic state of the system and are capable of reading into the configuration, generating new configuration, transforming configuration or validating configuration based on some desired spec provided to that function. And so like one thing you can do here that's different than it's been shown so far is rather than trying to take a high level construct for an abstraction and generate configuration from it you can actually take in a low level deployment and annotate that deployment to say okay turn this into a high level thing and then it can promote it by attaching a configuration data to a resource you've already defined. All right time for our last demo where we're going to be showing off Kpt and Kpt is different than Customize and even DSLs because instead of patching a high level resource with low level settings Kpt takes an opposite approach in which it takes a low level resource and populates and promotes it into a Prometheus deployment. So in this root deployment yaml we provide information merely relating to what Kpt's reconciliation of sorts will match on to add for more fields to a particular deployment. And we use something called the Starlark script to handle that in Kpt land but we won't delve into that too much in this demo. Just know when we run Kpt function run and then try run it to get the yaml and enable star so we can run that Starlark script. You can see this deployment we have above is expanded out with the other fields like image arguments that we expected as in previous demos and here we again see our default of 17 days. Now to watch Kpt's variance support in action we can uncomment the spec here which provides a different first argument and then run the same argument as we ran previously and you'll see that the promoted deployment now includes our new custom retention time of just seven days. So that's just the taste of Kpt in action. The pros of the sort of controller-esque approach are that the architecture is similar to the Kubernetes APIs and controllers as you get kind of the same capabilities that you get there. If you are consuming functions that this is a relatively flexible and composable approach just as controllers and APIs you can match deployments in horizontal parallel scalars and these sorts of things can do similar capabilities. The cons are one that writing a controller for instance is more difficult than writing a template and that's true here as well. You're evaluating a much broader set of inputs and reconciling the state of the system so it's going to be more difficult to write those modules because they are more sophisticated. And a second limitation here is that it is a really relatively new and experimental approach. It's really only been come about in the last year and so there's not a lot of resources for what are the best practices and how do you best use this technology? All right now that we've had a chance to go over five techniques used by Kubernetes CLIs we can evaluate how each type of CLI can help us with certain objectives. So on the top two rows templating and yampling composition emerge as the ideal option for folks who want to keep their logic simple. They are useful for abstraction invariance respectively but given their emphasis on simplicity they don't attempt to provide all of abstraction cross-cutting abstraction invariance at once. So as a result you might not get all that you want from either of these options. Moving down the y-axis we can see that domain specific languages and general purpose languages offer similar functionality with less simplicity traded for more customizability. The choice between the two language-based options hinges on how declarative versus imperative you want your logic to be. DSLs are also great if you invest in your configuration system and can standardize on a single approach whereas general purpose languages are useful if you want to develop solutions in a language you're already familiar with or want to ingest multiple input types into an orchestration system with some last mile changes. And finally our relative newcomer controller-esque CLIs allows you to layer and combine logic. So for folks looking to mix and match logic on various types of inputs as well as try something kind of fun and new, controller-esque CLIs might be a good fit for you. On that note we hope this talk successfully opened you up to something you didn't know and empowered you to explore the diverse world of Kubernetes CLIs out there. Thank you for your time. Now we're going to take questions, right? Yes, question time.