 All right, let's get started. So, thank you so much for coming and joining us this afternoon. Gareth and I will be talking about Kubernetes, metadata that you can use alongside Kubernetes and what that metadata can do for you. My name is Liz. I work with a container specialist company called Aqua Security. Aqua have a booth downstairs, so you are more than welcome to come and chat with us at any point over the next couple of days. I'm Gareth. Gareth are basically everywhere on the internet. I'm Gareth, I should go and I work for Puppet. So, if anyone wants to talk about configuration management, bits and pieces, confiners. This talk is hopefully the thing you've come along to see. We're going to talk a bit about... What do we mean by metadata? It's in the title, people have probably heard of it, but what do we actually mean at a pretty fundamental level? We're going to talk about what relevant features Kubernetes has for manipulating and managing and using metadata. We're going to touch a little bit about why we care. It's not just for its own sake. Who's it for? What can they do? The quicker we go, the quicker we will get to a whole bunch of live demos and examples that are there to wet your appetite for the types of things you can build with metadata. Hopefully that sounds interesting. So, we'll start out with, again, the least Kubernetes bit of the talk. What is metadata? So, heading to the dictionary is always a good idea. It says metadata data that provides information about other data, which is pretty abstract, but it is about having information about things you already care about. So, in Kubernetes farms, well, yes, you've got these deployments and these services and these pods, but you might want information attached to them that you can use later. Getting even a bit deeper into, well, there are different types and different use cases for metadata. We'll go and ask the librarians, because if anyone knows a librarian, they're obsessed with metadata and libraries are cool, so let's go there. The sort of understanding metadata paper breaks it down into three sort of main areas. You sort of got descriptive metadata and this is something that describes, it's for the purposes of discovery and identification. And so, thinking about those resources in Kubernetes, well, how do I find services that relate to this part of my organisation or are using this technology? Structural metadata is about the grouping of things. So maybe this deployment, obviously you've got direct relationships between a deployment and a pod, but you might have a number of different deployments that really make up a concept of an application. Like these things are all running, like having some metadata that groups those together, sort of structural metadata, and finally, administrative metadata, providing some information to help with management. And so, yeah, again, like, well, when was it created? Maybe you want to react to something being, like, that's been there for a year, is that a good thing? And there are solid use cases around those types of metadata, so you might be, like I said, that idea of discovery, organising, and one of the things we'll keep coming back to is the power of metadata for interoperability. Different tools can write metadata, different tools can read metadata. If we're all doing that in the same place with shared schemas and other bits pieces, we can build quite interoperable tools without direct integration, and that's really good from replacing tools, fighting lock-in, but building an interoperable ecosystem. Okay, so let's talk more practically about the metadata tooling that exists within Kubernetes itself. Kubernetes supports labels. The documentation tells us that labels are key value pairs, that are attached objects, things like pods. They're very much intended to be identifying those objects. They're supposed to be meaningful to you as somebody who is operating Kubernetes. The system itself doesn't do anything with these labels inherently. It doesn't imply anything meaningful to Kubernetes itself. But you can use labels for functional purposes, like selecting a group of pods that match a certain label, getting logs from pods that match a particular label, and so on. So that it's very useful functionality. I meant to warn you, this section could get quite confusing because different tools use the same terminology to mean different things. So labels, according to Helm Best Practices, labels should be used if it's identifying a particular resource and if, as an operator, you're going to find it useful to be able to query, filter on resources with that kind of identity. The other kind of metadata that Kubernetes supports is annotations. This is much more free-form data of your choice that you can attach to a resource. Pretty much anything you want can go into an annotation. Examples of things you could, and this is by no means exhaustive, information about the software that you're running, you could have links or references to things related to that resource, perhaps links to where the logs go or where the audits go. Even things like who is supporting this particular resource, who do you contact when this resource goes away? We'll look at a use case related to that later on. Here's another example where we've got... It's an ingress and it's got some information about time-out configurations, essentially. That can be useful to you as a human being to be able to see when should this time-out, how long should this go before it would time-out. You might be saying to yourself, but how do you know this matches what the software knows? This is annotation data, it's not part of the code itself. It turns out there's a useful feature called the Downward API. Gareth told me about this as we were building this talk. I hadn't heard of it before. The Downward API lets you make labels and annotations data accessible to the container itself. In that example of the Nginx, it could be reading that configuration from the annotations. The human readable version and the software information are one and the same, which is a good thing. This is where it starts getting a bit confusing. Docker, inside docker files you can define labels. These are not the same kind of labels. These are labels that are built into your container images, but they are key value pairs, just like Kubernetes labels are. You can put pretty much anything you want inside a label. Gareth and I were involved in a project called Label Schema, which looked at some common kinds of keys that you might want to use in those key value pairs for things like the name of the software, the vendor, maybe links to documentation, where you could imagine tooling built on top of that, whereas there's a common understanding of how you read the name of a piece of software from the image, that would be helpful for tooling purposes. That label schema work has actually been absorbed into the OCI image spec, but just to keep us all on our toes, for some reason this is called annotations in the OCI image spec, even though it's labels in docker file. I don't know why. Nerming is hard. Nerming is hard. So, we've looked at some existing tools and features that help us with metadata, but what they cover at the moment is, or what we've talked about so far, covers information that you can build into your images, and information that you can associate with your software when you deploy it. But there are obviously a whole bunch of other stages that software can go through between those two points. Examples of the kind of things that happen post-build, but probably pre-deployment, things like whether or not a piece of software has been approved for deployment, whether or not it's past tests, and vulnerability scan reports. All of these things have to be associated with a specific build of software. You can't have a test report sort of just floating about between different builds. It has to be associated with a specific build. And this information could change over time. Vulnerability scanning is a really good example where a piece of software doesn't change, but a new exploit could be discovered, and it turns out it existed in that image that you already had, so the vulnerability scan report could change after the fact of the build. So these are examples of cases where the current tooling doesn't really allow us to manage and manipulate metadata. So we sort of know what metadata is. There's some bits and pieces in Kubernetes that we can use to manipulate it, and some tooling or other areas of metadata, I guess, that are sort of outside the direct sort of Kubernetes labels and annotations. But why do we care? Who's it for? And sort of thinking about Kubernetes as software or the platform, and there really are, there are different types of users who actually use Kubernetes that are part of this bigger, broader community, and not who are going to be using it just as part of their job. They're not as interested in it to come to Austin. It's just something there. But what I'm interested in there, there's not really a really widely agreed set of sort of user personas or roles at the moment within the community. And so this is a bit of a side sort of story because we'll then come back to why we care in the context of metadata at the moment. So I ended up speaking to a number of people about this and sort of saying, oh, there's not a shared agreement yet, but actually what do other people think? I think I know what I think. And a number of people sort of responded and replied, and I'm not going to read tweets out, I guess really boring. But it was pretty similar to how we'd been talking about it. This idea of actually there being three sort of distinct, sometimes overlapping roles. So you've got the idea of really a sort of platform operator, sort of responsible for things like the API server and like installing Kubernetes, setting it up. They're going to care about HA and file systems. That's sort of the very classical sort of operations aspect. They might not care at all about the things that people do on top of it unless you bring Kubernetes down, they might get cross. But the actual, they may manage the underlying infrastructure, they may not do. They might be on the other side of a credit card, they might be a service provider, a cloud provider that's actually just running this for you. You've got the application developers. And people who actually, they're writing applications, they're writing Java, they're writing Node.js, they're writing Python, they're writing Rails applications. They just want them to run. They might not care whether it's on Kubernetes or not. They shouldn't have to care unless they're actually going to do fancy a thing. But they do want to benefit from joined, sort of joined up understanding with, like certainly the app and the sort of platform operators. And I think that, and this is something that's been talked about a little around the start of Kubernetes community. I think I see a Michael in the room. This concept of application operators, actually people who are probably going to be more familiar with Kubernetes are the sort of platform tools. They're probably the types of people who are looking at like actually helm packaging or case on it or higher level tools for configuration management within Kubernetes. There are like to set standards for others. They might be, they're operating the application on top of the platform, not installing Kubernetes. In some organisations that might be the same people, in others, in some teams within larger organisations that might be the same people, or it might be very distinct actually organisations completely. And often where it comes to sort of metadata, it allows us to facilitate communication between these different teams groups where they're not sharing the same tools because they're not doing the same jobs. They're not caring about the same things. But you invariably do want information passed from one to another. Okay, this is where we get into the fun bit and a bunch of examples that maybe cover some of these different personas. They may be more useful to one or other of these different personas. And we can just explore a few potential use cases for the kind of things you might do with metadata. And this is also where like basically we're going to break all the rules of what you should normally do in demos. So for a start I'm going to have to turn my phone onto not silent and hopefully get off and do the same. That will make sense soon, but okay. Right, so the first example is this idea of wanting to change who the support contact is. So as an application operator you want to know that somebody is going to come and sort out problems. There are many ways to solve this problem, but this is a kind of an interesting use case I think. And the key thing here is we want to change who the support contact is without redeploying software. So we're going to use a tool that I've been working on called manifesto. Now the idea behind manifesto is you have these container images and they're stored in a registry. The image is actually made up of a bunch of data blobs. And we're saying we want to associate some arbitrary metadata with a particular image. And the idea we hit upon was why don't we actually store that metadata as data blobs inside the registry along with the image itself? So that's what manifesto does. And it also puts together a little image that does the mapping between which particular image has which particular bit of metadata. Okay, so this is where it also turns into audience participation. There is a really, really crappy website that I've written here. Please feel free to reload it, keep hitting that particular IP address. I'll just say that a few more times so that you have time to forget that IP address. 52-170-392. While you're doing that, so well, earlier on I set up Gareth's phone number as the support contact. I put Gareth into the metadata for my application. And I'm going to run a little app here. Let me just show you what the gist of this app is. So the idea is, Kubernetes is using health checks to look at the state of this pod. And it will restart that pod if it fails. And it will fail because it's designed to be really crap. And at a certain point, it will hit crash loop back off status. And at that point, we're going to say, no, things are really bad. Somebody should come and pay attention to this website. So the contact details are associated with the image which has been deployed in the pod. And that Twilio app that I just started, the core part of it is essentially when it detects that it's using Q-Cuttle to look at the state of the pod. And if it sees crash loop back off state has been reached, it will run manifesto on whatever the image is for the pod to get the contact information and send a message to Twilio. You need to be refreshing that web page. They don't anymore, because I've just been paged. You've been paged? I've been paged with the... Elizabeth's Hello Health Check needs some attention, apparently. Right. We probably mean some people have seen the TPO. It just needs to restart. There's only one instance of it, and it is deliberately crashing. Did you know there was an HTTP status code for TPO? Yes. Pretty cool, right? So if you've been refreshing the page, you may or may not have seen a TPO on occasion. So now I'm going to... In the right window. Sorry, Gareth, you're going to get alerted again before... I didn't get there quickly enough. Yep, there we go. OK. So I'm going to take over support now, because it's not fair on Gareth. I wrote this crappy website, so I should really be supporting it now. It might take a minute to do this. Demos with network comactivity required. Right. And come on. It just takes a little while to actually do this update into the registry. It will... There we go. OK, it's finished. Right. And now we have to also wait for the failure time out to give up. It will eventually say that it could be alerted again. Keep hitting refresh. Keep hitting refresh. We need it to fail one more time. And eventually, hopefully, the alert will come to me instead of Gareth. It should have made this a shorter time. OK. So now, an alert could happen again. It's still failing. And hopefully, it will come through to me. If it doesn't, we'll just blame the demo gods and move on to the next one. Here we go. No, I've got a message. Didn't make a noise, but I have... Yeah? Yeah. Good. OK, so that's the first demo. And the idea there is obviously that we can update information without having to change anything about the software that was deployed. We just changed metadata. OK. The next demo uses manifesto and another different tool. And this is about keeping up to date with security vulnerabilities. So we all need to be aware of, you know, the latest exploits that have been discovered. And we need to keep our deployments patched with up to date code when vulnerabilities get discovered. There is a project called Grafaius. Anybody heard of Grafaius? A few hands, few hands. So Grafaius was started by Google, been open sourced by Google, and it's essentially an API for storing, managing, querying, metadata related to software packages, not just container images. It could be machine images. It could be all sorts of things. And the core concepts in Grafaius are notes and occurrences. So a note tells you about a thing, for example, a particular vulnerability. Vulnerabilities are identified as a standard way of identifying them with what's called a CVE, a common vulnerability enumeration, I think, is what that stands for. So every vulnerability has one of these CVE codes. And in Grafaius terminology there would be a note for any given CVE that we might know about. And an occurrence maps a note to a software entity. So there might be an occurrence for an image saying that that image has this particular vulnerability within it. And Grafaius lets you ask interesting questions across your whole deployment of images. So you might, for example, want to know which of my large number of images have this particular vulnerability included in them. And Grafaius will let you make a query to find that out. So in this demo we're going to use the output from an aqua vulnerability scan. And we're going to convert it into Grafaius notes and occurrences. We have an intern called Itai Gil who wrote a reformatter for me. He wanted to make sure he got credit. He got credit. So that will reformat the scan report into notes and occurrences. And I've also been working on making manifesto take alternative kind of back end storages for metadata where the idea is that you can use manifesto to manipulate Grafaius databases as well. And then we can run these kind of powerful Grafaius queries over the data that's been stored. OK, wish me luck with the demo gods again, right? So I don't need this window anymore, we've done that. So this is aqua and I'm just going to add, Mongo 321 is kind of well, it's a very old version of Mongo that I know has quite a lot of vulnerability in it. So there's a scan in process there. And here we go, just in time, to see the scan completed and the reformatter has just created a bunch of notes and occurrence files. So if I try and query, at the moment I haven't stored them in Grafaius so they're not there yet, I can, let's do put. So I'm going to store the files that are stored in that Mongo 321 directory at the end, I'm going to store those in the Grafaius database using manifesto. So now if I, well actually let's list what's there, make sure that you can see them, see package vulnerability. Yeah, so I have now got package vulnerability data and if I run the get there will be tons of it, loads of data there. But I can also do things like, this is kind of the Grafaius format API where I can say, can't find my cursor, there it is. I'm basically saying, tell me about all the occurrences related to the note about this particular CVE. I think, oops. Yeah, so let's do, this one is a fake one so there aren't any instances of that one but, and this one is a real one and there was an instance of it. And we get up here the name of the image, it was Mongo, it's identified by its digest rather than a particular label, which begins 1-9-1-4 and if I go back in here I can just verify, it's probably a bit small to see but down here there is the Docker digest and it does indeed begin 1-9-1-4. So that was the right metadata for the right image stored in Grafaius. If you're interested in Grafaius there is a boff this evening about it and there's also a meeting on Friday morning about it here. OK. This is one that I'm not actually going to demo just sort of due to time but there is, well Kelsey basically put together a Grafaius tutorial which does this particular admission control pattern. As an application operator there may be things that I want to check for before software is allowed to run in my deployment. The admission control pattern is basically saying before you actually go ahead with the deployment validate that some standards, whatever it is we want to check for in the metadata has been complied with. So if you look for Kelsey's Grafaius tutorial you'll find a sort of walk-through there of checking that images have been signed by an authoritative source before they can actually be deployed into Grafaius tutorial, Kubernetes running deployment. Over to you, Gareth. OK. So one of the things about metadata being so piecemeal, different types and different purposes is that you can have lots of it that's unrelated but often you can build really interesting things by combining different sources of metadata. So I'm going to look at a quick example. These are illustrative of the types of things you can build really easily on top of metadata. So as an application operator so that I can help teams with audits I want to know what software packages different teams are using. So I'll use a tool that I worked on at work but it's an open source called Lumigon. A tool for scanning running containers and collecting all sorts of different metadata from different sources about them. So it grabs a bunch from the Docker API as well as from what's inside that container, what packages happen to be there, what container happens to be there. It outputs a lot of data. It's a low level tool that's useful to build things on top of more than a fancy end user type thing. So the two bits of metadata I'm interested in to solve this problem are Kubernetes labels. They provide me a mechanism to store things. And in this case I'm going to store a mapping between deployments and the team that manages them. So I'm going to look at some of the things that I'm interested in to solve this problem. So I'm going to look at some of the things that I'm interested in between deployments and the team that manages them. Having those in labels is also handy because I can filter them. I can use any of the Kubernetes sort of native tools to go, oh, which, like, give me all the logs from that team, give me the deployments from that team when I'm talking to them. That's quite handy. And Lumigon provides being metadata about packages in the containers which are in the pods which were created by the deployments. I'd say I'd not quite gone of time, I think, looking at the clock to actually run through this demo. But this is sort of the output. And there's just a couple of, again, like small Python scripts, sort of 10 lines or so of code each. If you want to have a look at them, the links will be... Don't try and type that out. But now obviously this will show the slides later. But you can end it with a table like this where this actually lists... You run this against your Kubernetes cluster, it collects data from these different sources and says, well, actually, these are all the packages, software packages that we found across the entire set of containers running. These are all the different versions. These are how many times we found them. And these are the teams that we're using them. Obviously you might chop and change this in different ways. You might combine different sources of data. But the thing there is just how easy it is to do if you have good metadata to build high-level tools that ask quite interesting questions, you probably wouldn't invest in... This isn't big enough to be an application on its own, but it's easy and simple to build things on top of metadata. This one I should hopefully have time to dive into. One of the sort of things that will probably come to mind is, well, what about metadata quality? It's fine to say, oh, great. I can filter all of the deployment by team or route back packages to the right team or route pages to the right support contact. But what if... Well, they haven't bothered to add that data. So they haven't actually added the label for team. They haven't actually included any data about a support contact or how that should... someone should be paged. The sort of building things on top of metadata starts to fall down because the responses won't... In that example with the table, it would be missing some teams that might be doing something bad. If it was a paging thing, well, actually the page goes into the ether and no one cares that the application is down. That sounds bad. So actually having a way of enforcing, actually having standards around, well, yes, you have to do these things. The reality is a Word document or for modern people, a markdown file in the Git repository saying you must have these labels, isn't it enough? It's likely something... it's a starting point. It's a part of the discussion about what should be required, what's optional. But actually it's a terrible way of enforcing something. The Word document and going round to meetings and going, like, you haven't added your labels. It's like that sounds a waste of time. So one of the tools I'm going to talk a lot more about tomorrow in the talk about developer tooling is CUBE test. So this is briefly a bit of a unit testing tool for Kubernetes configurations. You point it out a config file, you point it out a bunch of tests, you have assertions and they will pass or fail. It's sort of fairly straightforward. I'll show this working in a second. And you actually write tests in Skylark, which is basically a Python dialect that can be embedded in tools. So this is a test to actually ensure the configs actually do have a label for team. Again, like saying, well, actually, if it's a deployment, grab the labels, assert that the labels contain team. Or fairly straightforward. So let's go actually have a look at that. Am I ever here? It'd be all the way to the left. All the way to the left. One more. One more. Using someone else's computer. So I happen to have the test that we just saw in a test folder. Obviously tiny text, but you've seen it before. I've also got a deployment. So the deployment, if you're eagle-eyed, has the labels team. It's actually commented out. So this is actually not a good deployment. I don't want this deployment to pass through my CI pipeline. So I'm going to run Coup Test. Coup Test is just a little CLI tool. We should just be able to run it against that deployment. And we get a warning. We get saying, well, actually, this doesn't contain team. Labels do not contain team. The output could be better. And obviously that should fail with non-zero access codes. So you'd put this into a pipeline and it would not work. It's going to tell you earlier that, like, no, I'm not going to deploy this until you fix that. So let's go fix it. Quick. We could pass it above us, and we'll see the passing assertions as well. And obviously that should exit with a zero access code. So we've got a mechanism to enforce standards about our metadata. And if you can have reliable metadata, then it's much, much easier to build reliable tools on top. So we've run through a number of different demos. Each of them individually, obviously, we could spend a lot of time just digging into and making a full thing. What we're really trying to do is say, like, look how easy it is to build things on top of metadata and the types of things you can do. And coming back to those sort of personas as well, and if you do sort of fit into the sort of the Kubernetes operator, think about what metadata would make debugging a platform problem easier. That's one of the times you really want those tools and the ability to ask questions across your infrastructure and discover just the right thing, not everything else. What would make that easier? As an application developer, start adding metadata to your applications. There probably is a time where you've added too much, but that's probably not to start with. That's a problem for another day. Start adding metadata. Whoever's running the application, whether it's you or someone else, will thank you later. I think this is the application operator. Let's start talking about tools, let's start talking about schemas, let's start talking about interoperability and solving the problems for application developers and for operators. Solving up metadata provides a really super flexible platform for building useful tools on top of. I think with that metadata, we can make managing Kubernetes systems and deploying to Kubernetes systems easier and more pleasurable. There's a few links there to some of the tools that we've shown this afternoon. They're all on GitHub. Some of them are pretty early-stage projects, so very much welcoming feedback and ideas around those. We're probably pretty much out of time, but we'll be around, so if you want to come and grab us outside for questions, that would be great. Thank you so much.