 Hello, welcome, and thank you for coming along to extending Kubernetes what the docs don't tell you. Funnily, I wrote this title a few months ago and the docs didn't tell you a lot of this stuff then. We're gradually getting better at actually documenting some of this stuff, so you might be able to find some bits in the documentation now. You should anyway, complain us if not. So, yeah. Who am I? Who a jet stack? So, I'm from a UK-based Kubernetes consultancy. We do a lot of, well, a lot of deployments for a lot of different customers of Kubernetes and kind of accelerating their journey to cloud native, as it were, Oswords. We write a lot of API extensions, so this is with custom resources, API aggregation, so on. So, yeah, I'm a software engineer, solutions engineer at Jetstack, sorry. And yeah, been using Kubernetes quite a long while, and all of this new extension stuff is really exciting. So, first of all, what is all of this and why do it? Yeah, Kubernetes has got a really nicely formed API, I think. It has its problems occasionally. I'm sure many have run into issues at points, but it's relatively consistent between types. You interact with everything in a similar way. You create your manifest, it's all nice and declarative, and things eventually happen, they converge. We have things like versioning, although occasionally there have been historically some problems with this. We can release v2 of an API and automatically migrate your applications across. Everything works fine. You don't have to upgrade everything at once, and that's versioning. It's got a lot of tooling for building all of these reconciliation loops, so you probably hear a hell of a lot. This desired versus actual state in Kubernetes. Yeah, the Kubernetes project itself has a lot of code helpers' utilities to actually help building all of this. Of course, we've got the CLI we all know and love, Cube CTL. We have a lot of things in the project, and we can reuse all of this. It's all well and good writing your own, and sometimes it does make sense, but it would be nice if we can integrate this as part of the standard experience that we've all come to know. But it is hard to do that yourself, so we can piggyback, we can extend. There's a number of examples of this out there. The NCD operator was probably one of the first things I remember hearing about, but now even the Kubernetes project itself, large parts of it are implemented through extensions. Gradually more and more, this whole track all day is about this. It's becoming a big deal, extending Kubernetes. It lets us do things like this. I think this is my mind the most basic way of looking at it. You can just integrate immediately and start getting and posting your resource, and I'm sure a lot of you probably already know this, given you are here. This control loop pattern that we see in Kubernetes, this is the way that you can start really getting into it and writing your own. You want to follow the standards that we've come to use throughout. Why? We get things out of the box when we do it this way. We get our back, authorization, all of this. It can all be folded into our types. We don't need to reinvent authorization. We can do this all with the standard Kubernetes native tooling we've got. You can go and create a role binding, a role that works and operates on your types. That's pretty powerful. It might seem insignificant initially. As you start to grow, you don't want 20 different authorization systems. You can start using extensions from all over, writing your own, and manage them all in the exact same way. That's pretty great. To me, it just feels natural. Cube CTL gets my type. That's great. How do we actually do this, though? We've got custom resource definitions. These are a nice, quick, easy way to add a new type. We've all seen it, I'm sure. I hope, anyway. It's good. It's quick. It gets you up and running. There's a few things we don't have with that. We don't have the versioning that we just spoke of just yet. Admission control, we do have these now with things like webhooks coming out. We don't have defaulting on fields and so on. If this isn't set, then make it this. We don't get that with CRDs right now, anyway. There's one other option, which is custom API servers. You've probably seen Cube Aggregator being able to put something behind the standard Kubernetes API and have it act as if it's part of Kubernetes Core. Behind the scenes, you've got proxies going round and about all over the place, but that complexity is hidden from the end user. To make a custom API server, though, it's a bit more work. We have to actually write our custom API server. There's a bit more of a learning curve than just Cube CTL create my custom resource. It is very similar to how APIs in Kubernetes are already done. The actual Kubernetes API server, it's based on the same code that you can build your own API server from. All of these similarities are right there. You get your admission control and everything else works as normal. We can do things like versioning. Downside of this, we do need to actually persist some data somewhere. We can no longer just chuck it at the API server and it goes away. We need EdCD or the like. Well, EdCD, that's it right now. To store this data has to go somewhere. There's definitely more of an onus on your cluster administrators, the people using your extensions. They now need to manage an EdCD cluster, or they need to open up their core one, which security-wise might not be the wisest. But it's a lot more flexible. That's my first tip, is API aggregation provides far greater flexibility. But they both do require some supporting code. That supporting code, however, we don't have to write all our self. There's some bits we do and some bits we don't. I really hope people are aware of all the generators in the Kubernetes project. Sig API machinery do absolutely great work putting all this together. It supports all of the Kubernetes project, I think, anyway. Everything works together. This is the API. This is a really core piece. Getting it right and consistently right between all of these different API groups, all these different versions is really important. If you had a bug in, say, one API group's client, that can get pretty difficult to track down. Generating code in a consistent way is important. A lot of you have seen Client Go. Again, out of thought. Client Go, ultimately at its core, is a client set. It's a REST client. It introduces informers and listers and all of these other utilities for building and extending against Kubernetes. We'll get into what these things mean and do. It's not just useful for building extensions to core Kubernetes types, but it's essential. You could just do it all through a REST client. You can even do it with Curl and Bash, but you end up with inconsistencies in how everything works. When we're writing our own types, we can generate these ourselves. Client Go, a large majority of it, is generated. We can do exactly the same. You can go from 0 to a project in 30 minutes once you get the hang of it. This is just a list. I don't think this is exhaustive. This is a list of some of the generators you see. We've got client generators for generating that REST client. Nice, typed Go. Conversion generators, when we need to do these breaking changes. We've got deep copy generators, which are more of a technical thing here, but if you want to just take a copy of a struct and start working on it again, you need deep copies. Defaulters, as we've mentioned, go to Protobuf, so we need to generate some Protobuf definitions for our Go structures. Go to Protobuf does that. It's kind of weird because you'd normally write your Protobuf first and generate the Go. We don't do that with Kubernetes. And some more. We've got informers, listers, so this is like a nice cache when you're writing your extensions. You don't want to be hitting the API server constantly for every single loop because it gets really expensive on the network as it doesn't scale well. And a few more. Open API is like the swagger specs. We can generate specifications for the types there. We write our types once and everything else just comes for free. So, yeah. To do this, we write our types.go file. We use comments to give it more information here. So like telling it whether it's a namespace resource or not and so on and saying whether to generate a client for this particular structure. And kind of just hit run on the generators and off we go. It's pretty convenient and pretty easy. Easy. So yeah, don't write this code yourself. I've seen a number of projects around where they have. They've either tried to write some parts of it. I've never seen someone writing for me yet, but hey, maybe. But yeah, don't go and write this yourself because honestly it's so much easier to generate it and it's so much more consistent. It gives you the runway for the future too. You can generate it against the latest version of the generator and you keep up to date with the latest things in Kubernetes. So I've put together an example application. Kind of demonstrate all of this, which is the API pager demo. I did this talk in London without the API server, which you'll see soon. And this is kind of an extension of it. So this is all on GitHub under my namespace. I'll put the link up in a minute. I think it's towards the end. So it's a small, simple pager for Kubernetes. This is not something to go and use as a pager in Kubernetes, really, I guess. You create an alert resource in the API server and you get alerts from PushBullet. If anyone knows PushBullet, it's a little paging application. And yeah, here's the link I did put on there. So I'll quickly show you that running just so we're all on the same page. So I've got a cluster. This is me making sure the internet works first. Yeah, so I've got a cluster, a GKE cluster. And I've written a helm chart for this pager. There we go. Cool. You can see there we've deployed our service accounts, our back roles, and our controller and API server. So when I go get pods in the pager namespace, I should probably bump this up slightly. We can see here now we've got our API server and controller running. We can see here, we'll see a lot of things. We can see we've actually registered this with the QBagGator. So here we go. v1l41.pager.kates.co. And now I can say Qubes ETL, get alerts. There's nothing there, but we didn't get an error saying you can't find the type. You can go ahead now, create dash F, docs. I can create an alert and should on my phone. You can see there we've got an alert come through. Yeah, I really hate that bit because my face appears. Very, very big on screen. Cool. So that's... I've lost my slides. Watching the keynote earlier. I thought I closed all my windows. Clearly not. Here we go. So yeah, that's the pager. That's what we're working with right here. So what have I used to do this? I've used the client generator, conversion generator, deep copy generator in former and Lister. So what is all of this? So first of all, we've defined our types like this. So we have our type alert. We've got our standard metadata blocks, type metadata, and then a spec and a status, just like you see with other Qubenetys types. This is a really simple one. We have our message as part of the spec and then the status at the end of it. The status just records a boolean, which is whether or not the message has already been sent because these types will persist in the API. They don't just go away afterwards. We could delete them maybe, but it gets a bit racy there potentially. So yeah, we define these types. There's some extra comments that go in here and we'll see those in a second, and then we run the generators. So follow the rules with these generators. If anyone's used them already, you might have run into random panics. You might have not got the code you want at the end. You have to follow the rules. These are what they're documented in, I think, the client-go repository. There's a few different places. We've got some links at the end that will send you out there so you can go and look yourself. But yeah, you do need to follow the rules. I've tried to fight it. It's hours wasted basically. Yeah. So client generator, first of all. What does that give us? Things like this are client, pagerv1l41.alerts and then namespace. And we can do an update. And under the hood, that's hitting a REST API, and it's actually going to go and perform this update for us. Informer generator. So I don't know how many people are using Informers right now, but you want to know when something in your API server has changed. That is the core of this reconciliation, the desired and actual. So here we add an event handler and we say when something has been added, when a new resource is added, we queue it to be processed. When it's updated, we do a quick check to see if it's actually changed because there's not a guarantee of, you know, only once delivery here. This can be called multiple times and multiple times, which is why you have to always look at the desired and the actual state of the world in writing these. And a simple delete function as well, so responding to your resource going away. Conversion generator. Now this is an exciting one, I think, if you get excited about APIs and maintenance and so on. So this is one example. Say we've written our V1 of our Elasticsearch cluster resource type and we thought we could just have a simple, you know, three data nodes, three ingest nodes, so on. And we want to change that to something more like what we see on the right. Conversions allow us to do this without causing massive problems for our users. The users can still go and create the old resource type and everything continues working with the next bit. And that's like super powerful. It means you can more confidently push out an API to get users using it without thinking that you're going to break everything next week when you realise that you've done something wrong. And we'll see an example of this in a minute. So let's actually take a look at some of this code quickly first. I did say it's a bit of a technical talk. So this is the repository here. Yeah, I've already linked to it. And you can see this is a very, very simple controller. I jump into it. I'll have to talk you through it. We can see here the actual creating our informer. So we grab an informer here from our shared informer factory. Now this allows us to build multiple different control loops that all use these same types and not kind of have to watch 20 times. It's only one. There's nice big shared cache. But here we just add our new function. So in queue, update, delete, and so on. The reason why we actually in queue things, we don't directly do stuff here, instead of we don't like send the notification immediately, we have a work queue internally to the application. And this is all part of client go. You can see just a little bit further down. You can see that we're actually reading messages off of a queue. Periodically we have like one, two, three, four workers. And that prevents, that stops you having to, well it stops you processing the same item twice at the same time, which is quite a big thing. You know you don't want to be, otherwise I could end up getting two alerts instead of one. Just them when I sent that. And it does a number of other things too. It goes on and going into it, there's a bit of logic here with push bullet where we're actually sending the message. And yeah, then our message goes out and we set a sent. You can see here in our sync function. So our sync function is called each time that a resource is read off of the queue. And the first thing we do is check to see if it's already been sent. And if it's already been sent, we can just kind of return nil here. We don't error. We're done. There's nothing more to do. Otherwise we prepare our new message and we go and post this note. And that's how it gets through to my phone. So it's pretty simple there. And at the end here, if we have just sent the message, we now need to actually mark it as sent in the status. So we set sent here. And this is where the deep copy comes in, because a minute ago I mentioned this shared cache. Well, we don't want to modify resources in that shared cache because it can cause all kinds of inconsistencies. So we use the deep copy function here, which was automatically generated by the deep copy gen, and then modify our resource, and then call update. Great. So where does the API server come into this? Well, sorry. First of all, that tip, don't modify resources in the cache. Bad things can happen, and you probably won't see the bad things happening for quite a while. And then they suddenly will, and you won't know why. And yeah, it's not good. So I was talking about these conversions. So creating the new version of our API, v1, beta 1. So I've had a breakthrough. I've realized message isn't quite what I want. I want content. And I want a title field too. So I'm going to launch my v1, beta 1 version of the API. Yeah. So jumping to that, just so you can see what I'm doing here. This is our alpha version here. I've got a kubicon alert. And this is our beta version. You can see the title has switched. Sorry, we've added the title field, and the field content has changed. So thanks to these conversions, this is possible. So I can show you here. Our custom API server is able to actually, if I can remember. Nope. There we go. So I found out I can actually request a specific version from the API just with this command. So you can put the full path in. So kubectl get alerts in a v1, beta 1 format. So first of all, that's that. We can see we've got content and title set. And now if I want to get this in alpha 1 format, I can too, just by changing that there. And you can see this same resource. It's got the same UID. Everything's the same. But the API server can dynamically convert that for us. So your old controller that is written against the alpha version of the API will continue to work if someone's created something in the beta format. So you get this kind of nice seamless upgrade process. You need to do everything all at once, turn it off, and then spin it all back up. It all happens for us. So, yeah. I'll quickly show you how these actual conversions are done, because I think it's something you don't actually, there's a lot of complexity in Kubernetes itself, and it really does make it hard to see. So here we have our conversions. A lot of this can automatically be generated, but some bits, content to message, a generator's never going to be able to work this out. So we do have to help it out here and tell it how to do it. So if I just jump in, sorry, into the beta version here, we can see first of all we're converting the alert spec, and we say out.content equals in.message. So it's a simple one equals the other. We're just mapping these two fields together. But how do we manage that? So there are external API versions, and you'll see this in Kubernetes. If you go into the actual Kubernetes repo, you'll be able to see we have external versions and we have internal versions, well, an internal version. So the external ones are the ones that you actually see. They're the ones that you can request and do things with, whereas the internal API version is used solely internally by the API server. Now, that is kind of a basis for conversions. You don't write a conversion between v1, alpha 1 and v1, beta 1. You write a conversion between the internal version and the alpha version, and then the internal version and the beta version and back, and the API server can then use that internal version as kind of a way to go anywhere. So if you need to go from v1, alpha 1 to beta to v1 or anything, it can just go from the external to the internal to whichever other external version you like. And we can see that here in the pager demo. We have our internal version to find here. There's a few differences that you'll notice. There's no JSON struct tags. So if we go here, we have all of this JSON. That's because we do actually convert this to JSON at some point. But the internal version, you don't. And this is one of the rules that you have to follow. If you go and put some JSON tags here, the generator doesn't even error nicely. It doesn't do what you expect it to do. It tries to make an external internal version and things start to fall apart a little bit. Maybe that will get better. I don't know. And you can see we actually have these tags. We say gen client here. So this is telling the client generator to go and generate the client. So, yeah. Cool. I've jumped ahead there. Take a look. We just did. So you define conversions between the external versions and the internal versions. That's it. Never between the different external versions. So, yeah. Wrapping all of this up. The API aggregation really is... It makes it... It's far more flexible. You can do a hell of a lot more with it. When you start writing your admission controllers and so on, so as a resource is submitted to the API, being able to change that, or modify it, manipulate it, or even reject it, you can get that with API aggregation. Obviously, don't generate your... Don't write your code by hand. There's no need. Generate it. And, yeah. So following the rules, not modifying the cache and defining your conversions between the different versions... between the external and the internal version. So, yeah. This is a nice big slide with tons of different things. We've got Stefan actually sitting at the front here. He's done a great blog post on all of these conversions. I'd highly recommend... Sorry, on all these generators. I'd highly recommend taking a look if you're going to be doing this. But, yeah. And, yeah. That's kind of wrapping up there. I'll leave this up for a second longer, because I know there's a lot of people wanting to capture this. I'll be posting the slides online so you'll be able to see. But, yeah. Thank you for watching. And I don't know if there's any questions. So, the question was, if you do code generation, does it work with an older version of Kubernetes? So, it does work with an older version... It does work with older versions if you're a... The remote API server is a different version. You do need to use the right version of the generators with the right version of the Kubernetes APIs. It's like dependency pinning is important. But, yeah. You can... It does work backwards. Yeah. So, if you do it for 1.8. Why not choose the resource version? Is it should I not be doing that for the updates, right? I just said resource version changed and updated. You did it. So, the question there is why did I do a, I think, the reflect.dp call there? I actually think you are right. We recently put together a sample controller and that was pointed out to me as well, that we can just look at the resource version instead. And it's, yeah, it is more efficient, because that update function could get called quite a lot. So, if you want to use the same repo and use them in multiple places, all the docs assume that you're making a controller and putting the generated types in that same repo. And I want to share it. I want to say here's all my types and then this project uses that type and that project uses that. And they just input it as a dependency. Yeah. So, the question there is, does everything all have to be in one repository? Do we have to have our controller, our API server, our types together? No, you don't. In that amount of repository infrastructure you need to make sure you've got to manage that, but it's just the same as every other project. There's nothing specific to this that enforces it. You could generate elsewhere. Yeah. The Kubernetes project itself does do that through a number of layers of indirection. Yeah, I don't know. Sorry, to say it again. Yes, I mean, what's the benefit to use your specific platform? So, you mean why you use the, like, generated one sort of thing, yeah. So, yeah, you're right. There is actually a generic shared informer that you can use. I think it's just a generic shared index informer. The issues there, it's not a typed informer. Ultimately your ad function is still hit with like an interface type. It's just, it becomes kind of harder to understand your code and read it straight away, because you are just using some cash.newsharedindex informer I think is the function. And your shared informer factory as well, that does rely upon like some typed, generated typed informers. But actually under the hood the shared informer factory does just return one of these kind of generic looking types. The informer generator ultimately is generating you these factories that, yeah, make it easier. I can quickly show you actually the output of that, because it is a bit of a confusing one. Because the informer is a shared resource. Here, so, ultimately it automatically, the generator automatically writes this bit for you. So, you can see we still have the cash.newsharedindex informer and then this list function and watch function. Yeah, this is written for you already, basically. Yeah, it's ultimately the same there, but you don't get that nice shared cash. Well, the shared list of informers, you can go and get the same reference to the same informer to save watching two, three, four times. So, yeah. Documentation in this area has been weak in the past and it's something we're all trying to improve. We actually had a meeting just yesterday to try to get some more things together. I think there's going to be some blog posts on it. We've got the sample controller repository now. I'm not sure if you've seen that. It doesn't go enough into depth, I think, on what each generator does and how and why. But it does give you a bit of a blueprint to doing it. There is also a new script which will eventually become a full binary, like an application which will take care of actually running the generators for you. They've all got different flags, different arguments and so on. So I think that's there's generate internal groups and generate external groups. I can't remember the exact name. You can actually see just here I'm no longer actually having to call these generators myself. I instead just call the generate internal groups script from the code gen package. Tell it where to generate the code into, so pkg slash client, and then we point it at my internal and actually the directory containing my external files too, and you can see we just give it a list of groups and versions to generate for. So, yeah, so the way I'm, yeah, because I need the actual, so this all comes from this kates.io slash code generator package. So I basically vendor this and using my, I think this is with depth, I've basically just got sorry, I haven't got a tumble syntax highlighting in Sublime. I list these as required packages and then I've got all of the dependency pinning below, so just making sure I stick to these versions. Is there any more questions? Yeah? How do we validate the CR? Okay, so this is actually one thing so by having this API server and writing it ourselves the API server is able to do that aggregation automatically and that happens as part of the API server library that you build your API servers from because it's actually got a copy of that type it can automatically like fail if there's any kind of an issue. So it's kind of done for free when you do API aggregation if you needed to do anything more advanced than that you can write an emission controller that could check things but for example if I come along here and force a typo or something so actually you know that. If I make this I don't know an integer so very wrong and then I do a create a create there internet's getting a bit contended clearly but it tells me that it can't be handed as an alert because there, read string well this is actually a much more ugly error message than I expected and it's telling me here that it can't be handed as an alert because the type is invalid on that field if you're doing this with a custom resource definition there's new API validation stuff that Nikita worked on and that actually works great it's not quite well you can basically do the same thing with it but obviously you still don't get your full of mission controller it's better in 1.9 so use it so it's a step up and yeah I actually had a play with this on the plane over and it is really good so yeah thank you Nikita shout out so any more? one more quick this slide the one with the links there you go I'm being told my time's up so if anyone's got any more questions though I'll be around here