 Okay, we'll get started. Quick shout out to the AV people who've just saved my bacon. My computer was not working, and they've turned up with a new one in record time. So we can actually get started on time. So thank you, AV people. So hopefully you're here to hear about developer tooling for Kubernetes configurations. So I work for Puppet, and there's a few bits of the background to this talk is rooted in a lot of my experience with, as a user of Puppet, and then at Puppet. But it's not really a talk about that. But if you don't want to talk about Puppet, come see me later. I'm Gareth R, basically everywhere on the internet. I do the DevOps Weekly newsletter amongst other things. So if you don't know about that, you should read that too. This talk is going to cover a sort of infrastructure as could and testing infrastructure as could and why that's relevant and prescient even for the Kubernetes community. I'm going to talk about specific types of testing that is useful to do for configuration. I'm going to talk through a bunch of demos and examples. They were going to be live demos. My laptop doesn't work, if you want to see them actually working, come see me later. So who in the room is familiar with the term infrastructure as code? Lots of people, excellent. Who's actually had experience with using whether it's Puppet or Chef or CfEngine or Ansible or maybe again, a similar number of people, about half again. And Wikipedia puts it like this. And it's a process of managing provisioning computers through machine-readable definition files rather than physical hardware. This is how most people are interacting with Kubernetes today. They're definitely writing machine-readable definition files. We'll come back to whether handwriting YAML is a good way of doing that later, maybe. Stepping back, so that concept has been around a while. That's not come from Kubernetes. It's not come from Puppet, predate to that. But one of the roughly 2011 something interesting happened. And actually it wasn't just aspect Puppet. There was ChefSpec as well. It was the same year. There was a few other tools. And aspect Puppet was a tool for using the Ruby aspect testing library to validate and verify your Puppet manifests. And ChefSpec was exactly the same. And this idea definitely does span different communities and the sort of Puppet lens and food critique from the Chef world. And a number of others, I'm less familiar with some of the Ansible testing stuff, but there's some tools there. And those tools have really stood the tests of time in those communities. They've been around a while, they've matured, they've ended up in the sort of the actual like the best practice way of doing things. They've added value. And I think it's an interesting sort of journey with respect to where we are today with like configuration files for Kubernetes. Because you start asking the same questions in those worlds as well as this one, which is fundamentally people like, but it's just a declarative thing. I'm just saying, this is the state of the system should be. Why would I test that? And I think it's a valid question. And I think as an observation, increasingly that configuration, once you're past that hello world, once you're past that first example, once you're past that sort of that first application with a few deployments, and you're into multiple teams and multiple bits of the organization, and you're into how many ammo files, you start seeing patterns emerge. You start seeing templating in some cases. You start seeing high level tools and others. You start seeing logic. You start seeing wanting to pass arguments. You see interactions between different bits of configuration. Interestingly, those points were not written for this talk. They were written for a talk I gave four years ago about testing puppet. And that's one of the reasons why I think it's this interesting comparison. And who in the room is actually using, or who in the room is just writing vanilla files? They're just writing the ammo files. And who is using a sort of helm, or case on it, or Kedge, or Captain, or Capose, and higher level tools? A few hands as well. I think this is the direction that at least a chunk of the community will go as these tools mature. And as an example of actually a really good sort of public example of a more real world deployment, the Bitnami folks have got a Kube manifest repo public for a bunch of their services. And if you go through, there's actually, they use JSON it, which is basically a sort of programming language for JSON data. But they've got 2,000 lines of that JSON it. That actually expands into more than 7,000 lines of JSON in their case. So you might knock a few lines off for Yamal, but same sort of order of magnitude. I think they've got a few clusters in there. It's not organization, large scale size. But that's a lot of content to not use something like JSON or a high level programming tool. And there's actually loads of those about. This isn't really a talk about that. This is sort of a side rant. But I don't think Brian Grant, as part of the setting of the application definition working group, went out and found, I think he has a list of 43. I can't list all of them. I presume no one else can. But there's definitely that number points to people with a sort of challenge, a problem. And I don't think it is that there's one answer. The variety of tools and the different types is interesting, sort of side conversation. If you're interested in that, come and join the application definition working group that's starting to get going and talking about this. We've had a few meetings, but not much else yet. So the sort of crux of this talk is that the lessons learned applying testing practices to infrastructure as code apply and will increasingly apply to Kubernetes configurations. And I want to talk about some concrete examples of tools you can use today to start off, start seeing if that actually helps your development. And I think, in particular, helps you scale Kubernetes across teams, across organizations. If you look at those tools, if you look at the sort of the reasonable immature sort of set of testing tools from some of the configuration management tools like Pappet, you sort of see they break down into sort of something that deals with validation. Is this going to be valid? There's sort of linting. There are practices that the community has come to that says, you know what? These are good default practices. You have unit testing where these are probably more assertions that you're making that are specific to what you're doing or your organization. And you often have acceptance testing as well in terms of just verifying it at a much higher level. And this is true as well. And outside those sort of declarative programming tools, realistically, if you look at testing tools for Java or Python or JavaScript, then you'll see this sort of set of things emerge. And I want to sort of start the conversation and maybe answer some bits of what tools to address these problems look like for Kubernetes. So the first one I want to talk about is validation. So really, it's starting to ask the question, is this a valid Kubernetes config file? You don't have to answer that. You probably don't want to answer that either. And this is just as much text as I could get on the slide of a reasonable size that hopefully people at the back can see. As we've just seen, even a reasonably sort of smallish sort of set, you're into thousands of lines of this. Are they valid becomes a difficult question. Is this valid? If people want to answer later, feel free. It's non-trivial. And it sort of gets worse when you actually introduce higher level tools at the same time as getting better. So Helm uses templates, or there's a number of other tools that use templates. And this is probably a good way of reducing the number of lines you write. But it doesn't change the number of lines that you output. And you start then asking questions like, well, is this valid? And the answer is, it depends what's in those variables. And you can go on and on through all of the tools. Is this cache configuration valid? Is this puppet manifest valid that generates the Kubernetes configs? Any higher level tool suffers the same things. Like, are these valid? One of the really handy things here and one of the things that really got me into Kubernetes to start with was the strength of the API primitives. Like, I'm background as someone who ran things, more recently as someone who's been building tools for things. I've not had to deal with any of the sort of, I guess, the stability and sort of the growing, like, sort of instability of Kubernetes two years ago in production. People have got battle stories. I didn't have any of that. I was running small clusters because I was building tools against them. So I got obsessed with the API really quickly. And you have these really strong primitives, like deployments and pods and services and controllers, and sort of the 100-odd more now. This is all described in basically using OpenAPI, what was originally Swagger. So there's a Kubernetes description of the entire API that's auto-generated from various different places in the Kubernetes code. I can't show that. I've sort of brought this up in slight jest because I think it's to now 85,000 lines of JSON that describes the Kubernetes API. Again, you're thinking, well, that's not something that people can read. That's something that tools can use. One of the things about the sort of the direction of travel with OpenAPI, there's multiple different versions. Kubernetes is using version 2. There's some good stuff in version 3. Talk to API machinery folks about that. OpenAPI descriptions use JSON schema internally to describe those types. So what is a valid pod? What is a valid pod spec within the context of something else is already described in JSON schema in the OpenAPI description. There's some fuzzy bits there that I won't go into because I'll run for 10 minutes. So what I went about doing when I was all thinking about how can I validate all these things? Because I want to do it client-side. I don't want to have a server for every version that I want to do something with. I actually extracted out of the OpenAPI schemers the JSON schema bits and then broke that down into separate files and broke that down by versions of Kubernetes. I need to automate this a bit better. Both ways, there's a repository filled for every single version of Kubernetes with an entire set of JSON schemers for each of the types that we deal with. And there's ones that roll out. There's a bunch of different flavors. Turns out that's a really large amount of JSON. This got out of hand quite quickly. I didn't really think about this. The SwaggerSpec for, well, the OpenAPI spec, it's called SwaggeredUpJSON in the repo is about 85,000 lines. That's 25,000 files in that Git repository, amounting to 7.5 million lines of JSON, that describe the shape of all of these primitives in Kubernetes for every version of Kubernetes in a bunch of different flavors. Again, that's knowledge you need to validate something. That is not knowledge that any one sane person could have in their head. So there's, oh yeah, and that's, on top of that, there's actually all the equivalents for OpenShift as well published. So they extend the API in a few different parts, on top. So there's separate specs for that. For those that are familiar with JSON schema, all those that not, it's just a way of describing a schema for a JSON document in the way that a database schema describes a schema for a database. And there's a whole bunch of sort of low level tools. There'll be libraries in most programming languages that you're using that adhere to JSON schema. So grab the sort of the canonical Python client and you can do JSON schema, point it out here, well, AcumenitiesConfig, hello.nginex. And I'm grabbing, I'm basically just the file off disk and saying, well, grab me the standalone deployment schema because that's the deployment. And it's saying, well, actually, that's missing a template. A template is required property. This is a useful integration point for other tools. So if you're building something and you want to have validation in your tool, then you can just use the schemers. From a user perspective, this is not very approachable. This works, but you need to know a bunch of information. Validation is useful for a bunch of different things. Like catching simple typos, fast feedback. All of this is happening on the client. It's all happening locally or it's all happening in your CI system. Checking against multiple versions of Kubernetes is a good one there as well. You might be running multiple different versions in different places. You might be looking to update. You might be looking to make sure you're not doing something that's going to cause a migration to fail later. That's all sort of valid use cases. And that's where KubeVal comes in. So this has been around a bunch of months. So there's a few people using this as well now, but I've not talked loads about it. It's a four months, according to this picture, if I took that recently. All this is a nice command line interface for validating Kubernetes configurations. So as I say, you can do it with the different schemers. This adds a tool on top that gives you a friendly UI. So you can see, you can sort of pass the different versions. You can ask whether it's OpenShift. You can actually host and generate your own schemers if you're adding custom resources. And it's very simple. And you can either just point it at a config file, and it will give you some information about issues that it finds. It will exit with a sensible status code so you can use it as part of a pipeline. And it will take things on standard in so you can just pipe things through it as a sort of chain. So it's hopefully a nice, small, well-behaved little testing util. And it works with basically like multiple declaration files. So basically, if you're putting multiple bits in there, it will do the right thing. It will tell you about each of the files independently and fail if there's any failures in one of them, et cetera. And I mentioned as well, you can run it for specific versions. It would be sort of nice to do this with multiple versions and pass a sort of comma-separated value. It doesn't do that yet, but that would be nice. Because I think this is one of the sort of useful features. It's implementing Go, and so it's actually available as a library as well for other Go tools. I think that's pretty new. I'm after as much feedback on that as possible because I'm not sure the interface is right, but it does work. So Coupval is useful now. I'll have some demos of actually tying it in with other tools a bit later on in the talk as well. So once you have validation, you sort of, well, you do find that that's not enough. A number of people have said, oh, that's handy, but. Because really, validation says something is valid, but it doesn't say whether or not it's what you intended. It will blightly let you do silly things as long as they're valid. So talking to lots of people, and some of my colleagues running Kubernetes, everyone ends up with this script, or a lot of people end up with this script that they run against all of their Kubernetes configurations in probably in that CI tool of choice that checks a bunch of different properties. So I did a talk to a bunch of people in person that I knew were already doing this. I sort of put some stuff on the internet and we talked about it at CIG apps. And a bunch of people sent me these scripts. And nearly everyone was like, oh yeah, we haven't shared this because it's really janky. It's like just this script. It seems like a good idea at the time when we added two things and now it's like loads of things with a lot of logic and it's really useful. We're like, we need it, but we're not sharing it because it's horrible. And that came up with like, I'm not gonna name any names, but like half the people said something similar. But to me it's valuable, like the fact that people are using it, they're doing it, it's an emerging pattern. What I'd like to see is basically a way that we can share and use sort of similar tools and not everyone has to go through the same thing. And it becomes a bit like, more there's more awareness of it. Because there'll be lots of people thinking, oh, I haven't thought about having a script that checks my code. And this was sort of Brian saying like their internal tooling includes something that lends their rendered Helm charts and validates certain rules. So talking to a bunch of people and sort of looking at a bunch of these scripts, you see a number of different types of things people can regularly do. And ultimately you have these data structures, but lots of people I say, well, I don't want peg pod specs without resource requests. Like my cluster runs into problems, the operators will hate me. Let's make sure that no pod specs don't have resource requests. I mean, Liz Rice did a talk yesterday about like building things on top of metadata. If you're building things on top of metadata and labels and annotations, you need to make sure your labels and annotations are there. So it's really common for people to say like, things must be labeled with these things. Maybe a label that says this service or deployment is owned by this team. Otherwise, I'm not deploying it. Not happening. Preventing usage of latest images. Again, sort of again, you'll read a lot of the sort of best practice type stuff community saying, don't do that. How do you make sure people aren't doing it? You have a script that reads your data structure. And there's a number of other sort of examples of the types of things people are doing. So with all of that, I wanted a tool that made that sort of both easy for you to write those types of scripts but also then hopefully share them with the community. And so I've been working on a little bit of this. This is, KubeVol is pretty mature. There's a bunch of people using it. It's sort of 07 or sort of it's had a few releases and a bunch of bug reports. KubeTest is brand new. I sort of shipped this a couple of days ago. So it will definitely have issues. But I think the approach is hopefully interesting. And this talks partly, if it has one takeaway is like, go have a look at this and tell me whether you like it or not. So what KubeTest does is basically allows you to write tests against your configuration. Basically writing assertions again for those types of things or others. And again, it's packaged as a small CLI tool. You run it against your config files and it will output information about passing and failing assertions. And again, in similar way, you can pass things to standard in and it will hopefully be well behaved. It will output status codes sensibly so you can use it as part of a pipeline. Tests themselves are written in Skylark, which I sort of considered writing this as basically with existing unit testing tools. What I saw with what sort of different organizations were doing was some people just wrote a script. Some people were like, oh yeah, this fits with how you write unit tests. And they would write unit tests and they would use the tooling around a unit test framework. The problem there is simply everyone prefers a different language. And it's less the language the problem, it's more the runtime. Ruby as a language might be easy enough to go like, hey, it's Ruby. If you can program a CLI language, you can program sort of Ruby a bit, like just for something that's very domain specific. On the other hand, now you need to know how to manage and install Ruby and no one wants to know that. Ask me how I know. So Skylark is actually, is a dialect of Python. So it's actually super familiar to anyone who's programmed anything like Python or sort of else like C bits pieces. So it's untyped, dynamic, high level data types. Well, I'll show you some examples. But the reason I've picked it is basically it's designed to be embedded in other tools. So you download Coup Test and that's all you need. That's it. Everyone gets the Skylark interpreter packages as a tool. There's no other runtime, there's nothing else to install and Coup Test is written in Go. So it's basically nice static binders for different operating systems. I think, like for me, and I'm interested in feedback on this, I'd like a tool that facilitates sharing. And I think the moment you go into different language communities, basically it's harder to share practices across them. So hence an embedded language. And I went for Skylark over something like RU or M-Ruby for sort of experimental reasons more than anything else. So what does that look like? What do tests actually look like? Well, here's a few examples. I mentioned that sort of idea of saying, well, like we don't want latest images. Well, here's an example. And so, well, again, you can sort of read the code, but we see if it's a replication controller, let's loop through all the containers in the template, let's extract the tag, and let's assert that it's not latest. Testing the minimum number of replicas, again, similar, we're saying, well, replication controller, grab some information from that structure, make an assertion. And that's the pattern. That's the, like, you might have some gates, you'll probably have some extracting data out of templates and you'll have some assertions. And I say, that's what it's got like as a language. The things to know are the sort of special bits that Qtests brings are spec. That's the Qtests object. That's the deserialized, like, from YAML, or JSON, or other tools. Like, that has the Qtests object in. And the assertions, basically, Qtests comes with a number, about sort of 10 or so, like common assertions for common things. So assert contains, fail, not equal, equal, assert true. You get, like, common assertions that you'll find in unit testing tools. And there's not much magic. So another example mentioned sort of enforcing labels. It's as simple as something like this. Also worth noting that, like, these are very direct. If you had a lot of them, well, you can write functions. You can write your own in Skylark to be used by others. So you could boil it down to your own DSL if you chose. For example, the sort of there, the spec template metadata labels, you could take that into a variable somewhere and then just use it everywhere else. So Qtests knew I would love some feedback if people are writing these scripts or not. Please have a look. I mentioned linting earlier, but sort of went to unit testing first. Cause I think organizations having their very specific rules. Like, oh, you must have a team label and it must be one of these teams is really useful. And you can't, that's not a general problem. That's specific to your, like, patch. But there are things where they're probably useful. Sort of like best practices for anyone starting out. And what I would like to see is basically if people like Qtests, can we build a Qtests suite that we use as that sort of common best practice starting the linting tools? And again, that becomes, rather than you have to write everything from scratch, it becomes, oh, I have these ones that everyone else thinks are good. Oh, and I disagree with that one and I've removed it. Oh, I've disabled it. Again, really similar to, like your typical sort of pylint or like RuboCop type linting tools. And that's part of this track. It's about developers actually using Kubernetes. And for that, we need developer tools. So yeah, that's sort of common. I think the idea of encapsulating those sort of emerging community practices in code is a worthwhile one to pursue. We probably don't have many today, but I would wager that over time we'll have more. And having them in code is better than a Word document. So yeah, I'd like to bake maybe sort of like an out-of-the-box experience of Coup Test or something similar, or a separate repo with a bunch of sort of common tests that is actually collaborated on by loads of folks. I'm gonna talk about a bunch of this stuff at CIG apps when I get a chance. I had to finish writing things first. So this was where I was going to dive into live demos. Unfortunately, I can't because computers, luckily, I have slides for everything, which was a good job in hindsight. So imagine I'm actually typing for a bunch of this stuff and it will be more interesting. So some of the design decisions for some of these tools around being well-behaved like command line tools, be like standard in, standard out, error codes was because there are lots of different tools in using Kubernetes for generating that configuration, from just handwriting YAML to templates, to custom template thing like engines, to helm, to case on it, to puppet, to cage. There are loads. Having testing tools for individuals, individually for those, is probably not the right answer. Some of them will have them, and puppet's already got good testing tools, JSON actually has a testing library. But the more you write tests for those individual tools, the more you're locked into those tools. If you can write your tests in something more general, you can swap the tools out at the same time as have your tests. And I think that is really powerful because the tools are gonna get better. You might just today be running Kubernetes in one team in a small scale and you've just handwritten everything so you're getting started. But imagine if you can write your tests and then port your tool, and you know your tool is working because your test is passing. That feels a useful property. So, lots of the tools in the Kubernetes ecosystem sort of follow that ideal as well. Yes, you can use Helm to actually chip your applications to Kubernetes and talk to the API. But you can also use the Helm template sub-command to generate them. So if we were to run Helm template, in this case, nginx from the Helm examples, it would output a load of JSON or YAML, I think it's YAML. And again, in here, if we pipe it through Kubeval, well, we can verify that that's valid. And nicely, it is valid. I was happy with that. I was thinking like, oh, I hope this is not an invalid one because that would be bad. We can also write tests in exactly the same way as we've been looking at for our Helm charts. And Kube test will work just as well with Helm charts as it will with handwritten Kubernetes configs. So JSONid and the JSONid library that basically provides a lot of helpers for Kubernetes configs is, again, a really interesting tool. I've done a bit of pieces with this. This is partly with the Heptio and Bitnami folks. If you've not seen this, I would have a look. There's been some talks on it already. You end up writing in a much higher level language. So you can see sort of variables and if you don't squint, you can go like, oh, yeah, there's a container and there's a deployment and the deployment contains the container. Ah, yep. And if we run JSONid against that, it would output the JSON file, the JSON for Kubernetes that matches the API. Interestingly, the JSONid library is actually built from the Open API spec. So there is actually less of a need to move like this here because this should always be valid. Having said that, it's because it was like there, it's built from a single version of Kubernetes, it's useful to be able to validate against maybe a different one. Gets into sort of a bit better. But KubeTest works exactly as well with JSONid as it does other tools. And again, JSONid has its own testing tooling, but having tests that are useful for all of your tools, irrespective of what anyone is using, I think, again, it has useful properties, not just the ability to move tools, but also the ability for different people in your organization to use different tools. Similar with Puppet, you can write Kubernetes conferencing in the Puppet language, Puppet ESL, with a whole bunch of abstract things there. You can use Puppet Kubernetes Convert to generate the JSON, and you can pipe that through KubeVal and KubeTest. You get the idea. This is, broadly speaking, should be true of well-behaved other tools in the Kubernetes ecosystem. Anything that can output to standard out or anything that can output to a file should work with KubeVal and KubeTest. As I said, there are a few people using KubeVal already. Brian from Sky in the UK made this observation on Twitter. Obviously, the demo's examples have been interactive. They're useful locally. But actually, if you're in a team, if you're in a group of people doing things over time, you'll probably want to integrate them into a CI system. I've got an example repo that I talked about this, and I thought, ah, and so an hour ago, I wrote a quick example. There's a Travis example of just doing exactly this. So all Travis does is pulls down KubeVal, KubeTest, and actually Helm. It takes one of the Helm examples and validates it against, if you squint at the bottom, there's basically a matrix build for different environment variables, for different versions, so this gets tested against a bunch of different Kubernetes versions, and then it runs KubeTest with some assertions in there. And so adding that to CircleCI, or Jenkins, or Travis, whatever, is really simple to do, and I'd like to make it a bit simpler. At the moment, it's sort of explicit. There's a whole bunch of we'll get entire lines. It would be nice to have one of those one line installer type things. So coming in concluding, I'd say if people want to see these things actually in action, come and see me later. The community is definitely still exploring different ways of writing Kubernetes configurations, and that sort of tool explosion is part of that, but Kubernetes, now we're starting one of the sort of themes of the conference, I guess, has been let's get a lot more conversations going about running things on top of Kubernetes, not just Kubernetes itself, the project over time should be the boring bit, and the things we can build on top should be more interesting. I think the better tools we have there, the easier that becomes and more people approach it. One of the things that I think I keep going back to, and again, led some of the design decisions, is this concept of the configuration complexity clock. Everyone sort of wants there to be one tool, and that idea of 43 tools is sort of quite scary, and over time, there'll probably be winners and the things sort of become a bit more niche, but it nearly never collapses to one or two or even three, and part of it's this idea of, well actually, hard coding sometimes is really nice, there's nothing in your way, there's not another tool chain, there's not another thing to learn, there's not something that's different to anything else anyone's doing, but that can be really verbose, and you add templating and variables, and it's sort of nice, but then runs into other problems, and you then end up sort of in a, well, can we have a sort of rules engine? Can we have a domain specific language? And you get cross all the way around. There are trade-offs, there are pros and cons, and it's often based on people's sensibilities individually, but also the organization they work in. One tool probably won't work for everyone, but actually, if you can have build tools that are composable with the testing stuff, that can potentially, and as I mentioned a few times, some of them do lend themselves to native testing. If you're using the Kotlin DSL, which is really nice, you could just write your tests in Kotlin. It's not as portable or as understandable for the bigger, broader community as something like, hopefully, KubeVar and KubeTest. So making them, one of the things I'm trying to do is build tools that are portable, and I'd love to know whether that works for people. So that's sort of two things. I think there are other, not two tools, there are other areas of testing for Kubernetes that I think if we can jump ahead and go, what does the development environment look like for where you're building things with Kubernetes, and if you are sort of totally abstracted away, it's the tool builders that need these tools, not you. But if you are building things natively on Kubernetes, these things become interesting, what other bits of testing do we need? Without good testing tools, you don't really attract the best developers. I know whenever I look at a new programming language, I'm always like, well, what's the testing approach? I want to learn that as well. I think once you get into more programming languages, that's a common pattern. We need that to bring more application developers to Kubernetes. Because there are, there's loads of inspiration we can take from, I'm not just the sort of, I sort of come from that sort of configuration management space, but what tools are really good in other spaces? And there's whole areas of sort of interactive debugging that I haven't talked about, but other people will have lots of experience with. There's lots of inspiration we can take from things that have worked. Can we jump ahead by just getting some of that done now rather than realizing we need it later? And with that, I'm done. And thanks for sticking around. And...