 Hi, everyone. Welcome to my talk stop reinventing operator wheels This is an introduction to controller idioms, which is a go-lang library that we wrote to make writing operators and controllers a little bit easier My name is Evan Cordell. I'm a software engineer at off said where we're building a distributed permissions database Called Spice DB similar to Google Zanzibar system. If you're familiar I would do want to mention before I get too far into it that I am an assume some prior knowledge of Kubernetes and controllers and operators and go-lang So if those are not very familiar to you, you might want to check out some of the resources before you watch this But let's just get into it. So all operators and controllers do pretty much the same types of work They might ensure that some resource exists in a cluster with a particular configuration Maybe creating a deployment in response to some API call Making a job making a service Deleting duplicates that it doesn't think need to exist Or they might do something like wait for some condition to be present Waiting for a job to finish a deployment to be fully available Service to have endpoints things like that Or they might need to start watching a new resource because It's been referenced by another object that it watches and it needs to start watching the new thing so it can pull down updates and Update its local cache with a representation of that API So there's these really common building blocks that we use to build operators. Those are just some examples But we end up writing them from scratch almost every time There's a lot of great libraries out there that help with some of the low-level complexity of talking to Kubernetes and Building a controller things like controller runtime operator SDK cube builder But we still end up rewriting these same basic workflows over and over again, you know, create the service You know, we we use a client to do that things like that and operators are Essentially state machines. They read the Kubernetes state from the API They make some decisions based on that state and then they do stuff based on it What's kind of interesting about operators is that they're really really big state machines And I kind of think that this is what drives our lack of reuse because when you have big state machines like this You're often using the previous states the previous context to make decisions about what to do So you might decide whether or not to create an object in the kube API based on other actions The operator is already taken based on new information it gets from events things like that And that can make it really difficult to reuse bits of a controller elsewhere So that's kind of where controller idioms comes in. That's what we're trying to solve How do we make the steps in a state machine reusable for a controller? But you might be asking why I'm talking about this why we're talking about it now. What's going on? So I mentioned this at the top, but we're building an open source implementation of Zanzibar called spice DB if you want to check that out To run that we also wrote the spice DB operator to run it on Kubernetes, which is open source You can look through it and then our hosted services We call us a dedicated is built on top of a set of additional operators that are currently not open source A lot of us are former CoroS red hat and operator framework alum So we're familiar with operators. We really believe in their value and they let us deliver software to our own users quickly and safely but because we're writing all these operators we kept seeing these same patterns across all of them and Trying to come up with a way to share them between all of our implementations. So that's kind of why we're doing this right now We're building lots of operators. We have experienced building operators And we want to make it simpler for ourselves and share that with others so now we're gonna actually jump into The basic idea here, which is to come up with an interface to let us break down big controllers big state machines And just talk about little pieces of them at a time This isn't just for the ability to reuse. It's also really helpful for things like testing And composition and that kind of thing, but I'm just gonna start with this basic interface handle Which probably looks pretty familiar if you're familiar with Golang because this is essentially the same as the HTTP server interface and This is gonna be our building block for making large controllers I did want to mention that this really isn't any particular framework specific You could do this with controller runtime. You could do this with client go or SDK, whatever you're you're using It's really kind of at a different layer than a lot of those tools work So if all of our little bits of a big controller are broken down into these small handle functions These handlers we still need a way to pass Context between them because we don't want to always Recompute all the data that we need we can and Controllers often have access to huge swaths of cache data to recompute state But it's not ideal. Ideally, we're not recomputing things all the time in every handler just to keep them decoupled for testing And so obviously Golang has this great context package that's used everywhere But now with Golang 118 we can build some nicer abstractions around those using Typed accessors so one of the things that controller idioms includes is this typed accessor library typed context that lets you create Keys that index into a context and let you pop out values of this particular kind This is not really operators specific at all But it ends up being really useful when you're writing handlers and you want to transmit values between them So in this case, I have two handlers one of them does some expensive computation Stashes that into a context and then in a later Handler we extract the value back out So it's probably not super obvious that these compose well and I would say right now. They don't I haven't gotten to that yet But you can imagine some sort of synchronization function that's calling these handlers You do step one handler one do step two handler two But one of the problems is that if we're using context like this to pass values between handlers This second call handler to can't access anything that handler one added to the context because it doesn't have the value back out And since we haven't now using pointers and we're not returning the context We can't pass handler to the context value So what we could do is we could do something like this We could just say okay Well sync calls handler one and then handler one calls handler two Which does work because now handler two has all the context that handler one had and can grab values back out of it You imagine the earlier example with some expensive computation done in handler one Now handler two can pull that back out without recomputing it But this does make it a little harder to rearrange and it's not it's not super different from just having a bunch of Functions that are not a standard interface right because it doesn't give me anything new and I have to If I if I want to move handler two to run before handler one after you range everything, right? So this is where the builder type comes in which is a Function that takes in a set of handlers and returns a new handler, which is maybe not too exciting but what this lets us do is Think about injecting the next handler that a handler calls dynamically in a way that we can rearrange later I'll search some examples. So it's not as nebulous so here's an example using a function chain, which is part of controller idioms which will First run the first builder to create a handler then run the second builder to create a handler and it'll run the two handlers In order passing the context in between them so that they do have the the values shared And you can imagine rearranging these two you'd flip them back around and do second builders and then first builders So the second one is called first And I can rearrange this without rewriting either of them Now the only caveat there is that they may have different expectations around what Values exist in the context depending on where they are in the chain And so that is why there's lots of helpers around Ensuring that the context has what you expect it to do and in a bigger controller. It's Not every step in a chain will depend on every other step of the chain So it makes it a little bit easier to do things like rearranging when when needs change So another example is the parallel helper, which does essentially the same thing as the chain function But in this case both run in parallel with the same input context So in this case, they don't actually aren't they aren't able to affect each other's context at all So kind of we fixed the problem with chain and now we've broken it again with parallel We we don't get the values back out of context. So I'm going to come back to how this is solved But another way that you can compose Handlers is via wrapping if I have some existing handler I can stick it inside of another handler and call it and then I can check and see if there's been any errors there Now the thing here is that the context That is modified by the wrapped handler isn't accessible from the wrapper And so there's some helpers in the same typed context package that helps deal with this problem instead of Just putting a value in the context To start with we put a space for the object in the context And then when we call the wrapped handler, it fills in the space that we made Which is a pointer and then now after I've called handler. I can access the values from the handler again later So this is also how you could deal with parallel values You can run both handlers in parallel but then they can put values back into Context boxes rather than contexts and you can pull those values out when you need them and then one other way that we can compose Handlers is via branching In this case, we have these handler keys that identify a specific handler out of a big set so you could do something like In this example, I have a Sync function that first looks at the object to decide if it's been deleted or not and then branches into two different Handler chains based on that decision. So this would let me have defined elsewhere like a full chain of If it's deleted handlers versus a full chain of if it's not deleted handlers Let's me keep those things separate rather than all jumbled in one big State machine essentially So there are these helpers for dealing with sets of other handlers and picking out the right one and giving them identifiers and things like that So now that we have a way to compose controllers and build them up out of smaller pieces We can also more easily talk about, you know, common patterns and encoding those with these libraries And that's kind of what the rest of controller idioms is the first bit is the basic building blocks to make things reusable and the next bit is Some actual patterns that we've identified that we've used multiple times that we think are generally useful So the first idiom is adoption This comes up a lot when controllers are trying to be efficient with the resources they watch So for example, you could write a controller that watches every config map in a cluster And then when someone references a config map in an object You will have that config map in your cache already because you're watching all config maps Most people don't want to watch all config maps both for like security reasons and then also just for space reasons That means you have to have every single config map in cache in your operator to work So instead you when you open a watch against the cube API You can give it a label filter and that will only send the objects down to you that match that label But then of course you have a problem where if the operator is not the one creating the config map It can't put that label on it. So it won't be in its cache So this is an issue for any resource that is used as input to a controller rather than as output And this is what we call the adoption workflow when Some custom resources created that references another object that's already existing in the cluster we first check and see if it exists and then if it does we add whatever the special operator label is The operator is watching and that brings the object into the operators watch stream and cache And now the operator can know that it's there and continue working with that object going forward Another one that we use a lot we called the component idiom And this really is just when the operator is in full control over some external resource that it creates in response to user input so user creates a custom object and then we need to create a deployment based on that object or a service whatever In that case the operator is creating it and it's you know has some uniqueness property It's one to one or many to one with the object the user created Generally life-cycled with it. So you might have owner references. It's really not supposed to be Shared with other controllers. This is a relatively common pattern that some operators use and so we've encoded this in controller idioms as well so that It will handle the kind of the dirty work of creating the object if it doesn't exist You know checking for changes via hashing and then deleting duplicate objects if it's created in a way that there could be duplicates for example This just makes it easy to make Controller that just ensures that a bunch of stuff exists based on some input object Another idiom we're calling static resources Which is when a controller is given some definition of an object and it just makes sure that object exists and matches exactly This can be really useful for bootstrapping if you have a controller that you know needs to create some default resources when it starts up This can be really a useful way to do that could also be some sort of Operator global config if you have something along those lines that needs to just always exist when the operator is running Another one is the pause idiom Where it can often be useful to Telecontroller not to stop entirely but to stop processing a specific object It's gonna be especially useful if we need to go in and do some debugging Deal with you know human operator needs stop it from doing something crazy that kind of thing We also have a self pause idiom, which is maybe not a thing that you should use Much if at all that's really just for if an operator can Definitely determine that it has no way to make progress on its own Maybe it's too expensive to pull for state changes for some reason in an external system or its rate limit or something Doesn't have access To something it needs then it could self-pause the object and that Not only sets the an annotation that will tell the controller to stop Watching it, but it will also add a condition to the object if you're using standard conditions block on the object and Report that it's paused and why and what you can do to unpun. In addition to the idioms We also have a set of Kubernetes utilities I was kind of briefly go over these I think they're They're really in support of everything else, but we have some basic caching libraries that wrap hashing and validating hashes There's some utilities for work key management if you're if you're using Client go probably not something you'd want to use if you're using controller runtime It has some common metrics to admit if you're using standard conditions, for example, this will admit a mitt metrics based on the conditions It does come with a basic controller and controller manager implementation if you don't want to use one of the other libraries like client go or controller runtime for that it does have a File informer which essentially works the same as other communities API informers, but it reads from a local file instead This can be helpful if you're trying to bootstrap some Local config or or you want to allow people to mount volumes into your controller and control the behavior in some way And it also has some helpers for CRD bootstrapping which make use of The controller runtime libraries for the same thing So that's that's kind of it. That's the the overall that's the what the problem is That's how we think about breaking down the problem and how we've built some abstractions to help with this At Oth said if you want to know more you can definitely check out the controller idioms repo repo and if you have questions or Ideas or you know with still early days if it's a lot of stuff We haven't identified that would be a great addition to the library. So if you want to add something there I think it would be a great opportunity If you want to see how controller idioms are used in practice I have a link to the Spacey be operator here, which is open source. You can look through how we use controller idioms all over that And then if you just want to chat or just get in touch we have a discord that you can come talk about it Or you can get in touch on Twitter So thanks for listening and Have a nice day You