 Thank you. Welcome. Come on in. There's a lot more room down on the far end of this big wide space So come spread out make yourself at home. We're just thrilled to be here. I'm Michael. This is my colleague Austin We work on the operator framework and now and in the past in various capacities We're here to share some lessons that we've learned with you about operator design We're gonna tag team here a little bit I'm gonna dive in and I think Austin's gonna be back in a few minutes Let's get right to it Getting started with operators and operator development is pretty easy these days. We've got the operator SDK There's other tooling out there. There's cube builder a Variety of languages are supported now that you can use for developing some kind of operator And you can get something up and going pretty quick that does installation uninstallation maybe a little bit of reconfiguration And be pretty satisfied with that But we want you to have more advanced more useful operators that you can really do things with That really make day two operations successful and Day two as we've learned is full of messy challenges. These are just you know the first 20 or so that came to mind That you can face when you're orchestrating Operational concerns with your operator really solving operational problems kind of things SRE teams worry about So we have a lot of experience managing stateful infrastructure and that comes with a lot of those day two challenges You know things like you know bare metal server provisioning and these kinds of things We've helped make operators for cluster installation cluster upgrade a lot of Infrastructure related stuff that can take time It can be error-prone. It can be very configuration sensitive When you have a cloud API of course You know what you're gonna get in advance for example when you're deploying a cluster when you're doing that in a data center There's a lot more opportunity for for variation of environment, right? So we've we've done a lot of that and made a lot of these kind of operators in that space And yes, we've even made an operator for installing more operators If only it was so easy as as this This is my fo-go. I hope you'll Be okay with some of that along the way If we could reconcile and just say what's the current state? What's the desired state do a little bit of work to move the one a little closer to the other and then set some status and We're out of there success that would be nice, right? And that is largely what we want to do in our operators, but the implementation of that You can get a little more complex. So let's start. We've got five sections for you and this is number one We're gonna start with some API anti-patterns lessons. We've learned about API design Here's a pod set API shout out to anybody who implemented a pod set operator yesterday in our workshop But here I've added an annotation Called enable hot new feature true. I've seen this in the wild I've I've done my best to talk some people out of this on occasion It's very tempting to do this this kind of thing where you take a bit of information That really ought to be in the spec of a resource But you put it in in an annotation as more or less what I like to call a no consequences API Why would you do this? Well, sometimes you need a temporary behavior It's a workaround for something. It's not gonna be here for long. Let's just put it in an annotation You know, we know what the reality is of a lot of temporary solutions Sometimes there's a deadline pressure. We've got to get this out quick. We need to enable this new thing Maybe it's a feature gate whatever it is Don't really have time to go through a full API review right now or dealing with a CRD upgrade is not really in the cards Right now, let's just put an annotation on this resource We can get the feature out there and it will deal with the rest of it later Or sometimes a hidden feature You can actually use this as something of a feature gate You don't want to advertise it in your in your resource, but you still want it to be available So what's what's the problem? What's the downside of this? Well, you don't get API versioning You're not plugged into the versioning scheme of your CRD anymore There's no status to reflect feedback to the API user of whatever this is on the less Of course you put another annotation on your resource to give a little status. I have seen that Don't recommend that There's no validation. You know, we have access to like amazing validation now right out of the box and forced by the Kubernetes API server It's real easy to use but when you're using an annotation, of course We don't have access to that kind of validation quite so easily It's harder to document this Annotations are not somewhere that people normally look when they're trying to figure out how can they accomplish a behavior with your API and then of course For a user who's actually using that API. Let's say you're on a cluster cube cuddle get pod set You're not necessarily going to look at the annotation list for things that are going to influence the behavior of that API So it's just easy to not notice So what should you do instead? Obviously, you've got to make your own judgment calls about when the deadline is is really so tight that you just got a Got to swallow hard and deal with it But you're really going to be happier in the long run that you added a field to your spec that you just took the time to get it Right go through whatever you need to do and just put it there and document Feature gates for pre-release features can sometimes be a nice thing we've done quite a bit of that in the open shift world a Mix of documentation on the field itself combined with a Feature flag that's usually a configuration property of the operator itself not the API the operator implements But the operator itself that's a better place to put that kind of a feature gate that can then make a field either Be useful or not that that's a good way to handle that Great now here's another anti pattern outgrowing Booleans this one Borrowed the description right from the Kubernetes API conventions, which is a fantastic document If you have anything to do with designing Kubernetes API eyes either as part of Kubernetes or for your own uses and operators Highly recommend set aside some time Maybe for taking a flight home from this event read through this API conventions document It has a lot of insight and and guidelines that are you know Written in the ink of people who have seen the problems. They've been there They've lived through the pain and they're sharing them here and this is a favorite outgrowing a Boolean So here's let's imagine we we have a new DNS Provider new DNS service called fancy DNS and we're running it in a cluster. So we made an operator for it It's very simple now. You may know that DNS queries tend to happen over UDP, but also can happen over TCP So some product manager comes along says, you know what? We've got some customers. They want to just disable TCP Can we just add that feature just turn that off? Sure. It sounds sensible So we had a field called enable TCP and we can set it to false and now that's disabled easy done Well a quarter goes by and PM comes back says, you know, we've got a growing number of customers They've got some oddball use case. They actually don't want to use UDP at all They want to do just TCP for for their DNS queries kind of weird, but you know it happens Can we make that also configurable? Can we be able to turn off UDP but keep TCP? Okay. We had another field Let's be able to enable UDP optionally So now we've got a growing list of Booleans and we've got a circumstance where One point in our four square matrix is not really valid. We don't want both of these to be false, right? So now we've got a validation concern. That's gotten more complicated. Everybody's life is getting more complicated all of a sudden What should we have done? something more like this Reframe the question reframe the the API itself think about it differently not as a Boolean on or off But what listen protocol do you want? Is it UDP? Is it TCP? Is it both? Maybe instead of both I should have said all in case the third protocol shows up someday But in any case that I find is the secret to Having a better API field than using a Boolean as the type Reframing how you're thinking about that kind of setting look at it through a different lens in that way And then one more Kubernetes API is a data store. It's just not a great data store Now these are rough numbers more like an order of magnitude, but NCT is designed for a pretty small number of pretty small values You know, we've seen things like device inventories being tracked. I've done this device inventories of various kinds being tracked We we do Kubernetes native infrastructures We're very deep in the open-to-world at least philosophically into managing infrastructure using Kubernetes native APIs So we run up against these scale concerns periodically And when you think about something a problem space like a device inventory a good metric would be like Hundreds of devices that are related to to this cluster like the nodes in that cluster and the machines that are part of the cluster API These kinds of things bad would be like millions of edge devices. You probably should not be trying to track that As CRD is in your Kubernetes cluster Another potential gotcha is config maps and secrets for arbitrary input Very tempting to say, okay We need to allow somebody to provide a little bit of their own yaml manifest as part of installing this other thing Let's just let them provide that as a config map It makes a nice easy story and it'll work for a while until it doesn't because there's a size limit to how much data You can stuff into a config map And in the secret is the same And TLS certificates can be another one that occasionally into medecasing can get you It's easy to think oh certificates aren't that large Well, until they are, you know, you get somebody with a long chain of trust trying to put the whole chain into one secret And things can get rough. So don't use the Kubernetes API as a data store is a best practice. What should you do instead? Use a different data store find a database. I Love Kubernetes native APIs as much as it's about anybody you'll find It's great when it's the right tool for the job But when it's not the right tool for the job go find another tool and that's we'll all be happier for it KCP is an up-and-coming emerging Project that I think has a lot of promise for this kind of scale for managing with a Kubernetes native API a Very large scale of data and resources. That is the kind of project you might be interested in using someday For something like millions of edge devices, but stay tuned Highly recommend. I know there's probably a lot of KCP talks this week. That's a very cool project. Check that out That stuff out if you really need to use config maps to take in arbitrary data use an array. It's not beautiful But it works Okay Number two slow and imperative Kubernetes native APIs What are examples of this so cluster installation and upgrade this is the kind of thing that could take 20 minutes Could take an hour might take a couple hours Upgrade depending on what kind of constraints you have how big is your cluster? How long does it take to drain a node then upgrade that node in place? For example, if that's the upgrade pattern you follow and and get work was back on and then go on in the next node You know, we've heard stories of people Take days a week or more depending on their various constraints to get through a cluster upgrade especially with very very large clusters So these kind of things especially put a Kubernetes native API in front of this It's long Imperative errors could happen a long time after somebody declares what their desired state Provisioning bare metal likewise you might take 10 or 15 minutes for a server that you're trying to provision with the Kubernetes native API Just a boot and then you write some stuff to disk and then you reboot it. Okay. Now. It's another 10 or 15 minutes to see what happened Backing up data, of course back up seek a long time. So these are the kind of infrastructure concerns that are slow, they're pretty imperative and Are ripe for some kind of something to not go according to plan But your goal as an operator author is Really to take an imperative world and make it exposed and available through a declarative API So my my best advice to you on this is embrace using imperative patterns in your operator It's easy for us to get trapped into a mindset, you know a declarative API mindset and feel it would have to be declarative The whole edge and level based conundrum, but when you're writing your operator code Just remember your job is to be that bridge be as imperative as you need to be in your operator code But expose that through a really nice declarative API So think tips that you can use along the way when you're doing this Implement a state machine when you need one go for it on the right here. This is from the metal cubed project This is don't squint too hard. I didn't really mean for you to Actually be reading this. This is just like for you know for effect This is the actual state machine used by that project This is a project that interacts with BMC's base word management controllers on real server hardware to control it from Kubernetes APIs Turn saw you know turn hardware on and off provision it these kinds of things and They absolutely they wrote it. They have an operator. It's the bare metal operator and they use a state machine and it works well Many resources can have an end state, you know a job is a good example That's one that many of us are familiar with is built in It starts it doesn't work it ends it has a finite state You can embrace that and you can create your own resource in that same kind of pattern that is designed to have a final state and that's okay Returning from your reconciled function promptly. This is the best practice in general But try not to like start your reconciled function kick off and install Wait wait wait wait wait an hour later. It finishes and then you return from your reconcile Of course, you don't want to do that. You're gonna want to monitor some other way Return from your reconcile function in your controller We'll talk in a minute about some other ways to monitor that But you'll be happier in the long run The faster you can return from that reconcile And then I just avoid implementing long workflows in in your operator in general You want the as much as you can the implementation of those things to be somewhere else like a BMC that's rebooting a server that that behavior is implemented in the BMC You're controlling it. You're telling it Reboot and then you wait and you can pull or you know through some mechanism observe what that BMC is doing, but You don't really want to implement along like have like install Open-shift install logic for example like in your reconcile function. That would be a bad plan So what can you do you could spawn a job and actually use a job to implement? Some of that logic or a cron job would be very similar and that your controller can create those jobs and just watch them Until they can they complete it's a very nice experience because it's a Kubernetes API itself It had it generates events. You can watch those events reconcile whenever that job emits an event and and therefore be immediately plugged in When when that operation that you're interested in finishes another option You could use some kind of serverless or function as a service or maybe some config management Like ansible AWX is an example of that some external system That's not part of your operator that can It's designed for these start to end workflows that you can that you can trigger wait on observe the state of and and let them complete But this is an important point don't access the Kubernetes API from those external processes if you can avoid it because I think is Austin's gonna talk about in just a minute Using a caching client is hugely valuable. We want to avoid querying the Kubernetes API as much as we can Without some kind of caching layer in between controller runtime Cube builder operators to K all these use a caching client that that we want to use as much as possible And then you can use a channel source So if you're familiar with controller runtime at all, hopefully many of you are if you've ever Done any go base operator development in particular you can use a channel source and basically what happens here is you can Stick in this event structure into a go channel Anytime you see fit and that will now kick off a reconcile in your operator of a particular resource So you can either maybe you're using a go routine to pull some external system And whenever you get that system gets to a certain state now you stick that event into this go channel Or maybe you're you've got an htd callback that you're listening for and that sticks out of engine that channel But this is a best practice and a really nice pattern for bridging that gap And i'm gonna hand it off to Austin for our third topic Okay, so we've talked about Good patterns for developing your operator so that you don't get into trouble But now let's talk about how to be a good citizen on the cluster We've got to minimize the api load because all those operators have to live there together So step one you've got to collect metrics And it's a good idea to start collecting metrics early in the development cycle It's so easy to say, oh, we'll get it fully functioning and then we'll start measuring it But it's much better to know how your performance is being impacted as you're adding features Um, controller runtime, which is underneath Operator SDK will expose prometheus metrics to you Out of the box. So if you have prometheus installed on your cluster, there's no reason not to be consuming those metrics You can come you can create some custom metrics Completely on your own To measure whatever you'd like It's very easy. I've done it myself As your As your operator is running on the cluster Keep an eye on how it's changing Don't just deploy it and then assume that it's going to work forever Because operators are dependent on the cluster state And the performance can change as those clusters change And pay special attention to endpoint disruption And if it starts to get past a couple seconds, you know that you need to do something So how can an operator be a bad citizen? Well, the first thing is a watch Watches are much more expensive than you might realize And when you have lots of operators on there This can scale up very badly. So just remember watches are expensive Cube API server, uh, it can limit the number of watches allowed simultaneously on the cluster I know that OpenShift does this And Even if that limit isn't reached Those resource Utilization can continue to scale In a way that is not desirable So how do you avoid this? Well, if you've got some very tightly coupled operators, maybe they don't need to be two operators If you've got two control loops that are watching the same objects Consider putting them into the same operator so that they can share a cache That cache will emit a watch to the API server And any of the extra performance is going to be between your operator and that cache Second, we want to avoid the low sync periods So if you're not familiar, a sync period is the maximum amount of time That your operator will go without syncing again Um, most of the time We think of an operator as reconciling based on events And that's the way it should be you don't want your operator running every five minutes because it is expensive It defaults to 10 hours for a reason Um, and it's tempting to do because well, what if it misses an event? It doesn't miss events. It's very unlikely And even if it does the next event it'll get caught up If you do have some bugs a low sync period can cover them up And wouldn't it be better to just know that you've got a bug? And if absolutely necessary if you need to be Triggering reconciles more frequently than events are coming in Consider using req after This is something that happens as your operator is exiting You can say hey, I'm not sure about the state. Can you req like right now or Wait 10 minutes try again So we know that we can make our reconciliations cost less But how can we do fewer reconciliations? Well, you do not need to reconcile every single time Any event from anything that you're watching comes on in You can filter those events with predicates Uh, controller runtime provides a handful of very good predicates Just right out of the box that you can use just like i've done here With the with event filter It's also very simple to create your own custom predicates here And this will stop you From reconciling when it's not necessary So inside of your reconcile loop when you're hitting the kubernetes api Know when you can get away with a patch a patch uses many fewer resources than an apply And it's going to go faster But be very careful because if there are multiple actors Changing these resources and apply is safer Particularly a server side apply, which is the new way all of us need to know about that So if you're going to start playing with this Please be sure to check out server side apply as well as the various merge strategies So at the end of your reconcile when you go to update the status This is a just a nice easy way to stop using so many resources If the status hasn't changed Don't update it It's as simple as that All right, so now we're going to talk about the client cache Which is designed to make performance much better, but sometimes it can backfire So let's talk about a specific case This is a classic thing. We have people coming into the chat all the time saying, oh I've got this operator. It's Become totally unwieldy What happened? Well, here's how you end up in this scenario Let's say that you've got an operator that is watching a secret Let's say that it has a database password or something So of course if that secret changes if the database password changes you are going to have to restart some things So you do need to watch for that secret But out of the box and this may not be noticed on a small development cluster That cache is now watching all secrets And so if you now take that operator and put it on a large cluster with 10 000 secrets Every single time that any of those secrets change You're consuming you are consuming those events and it is expensive So how do we avoid this? Using the controller runtime We can now add selectors by object When we are establishing the cache And what this does is it filters out those events from ever making it into the cache Now this i've got it big in red here so that you will not accidentally do this. It's a big gotcha Those events don't exist to your operator So In this case, we're only watching My secret not yours So your operator cannot see those other secrets Which you might even consider it a feature And back to michael for multi cluster and the future Well, this is one of the more interesting areas. I think of operator design right now You we see the world changing to be far more multi cluster than it's been in the past More clusters that are smaller clusters that have more tightly scoped purposes And this this cluster does this Some other cluster does that as opposed to having one big broad cluster that does it all Clusters that come and go clusters of service. These are all things that that are growing in popularity quite rapidly But the pace of being able to access like have good tooling to access many clusters from one operator Um is catching up and and there's there's work to be done. So this topic is as much Sharing with you what the state of the world is and what you can do now as it is a An invitation to help An invitation to get involved It even if it's just sharing use cases Or come help build the next generation of multi cluster client Certainly in controller runtime at least So here's some use cases We do Again kubernetes native infrastructure management. So we do it with an open shift policy and governance through kubernetes apis You've got a hub cluster You get a many many spoke clusters And you can enforce policies about resources and and different things that exist on those spoke clusters, right? Well, that that requires you to interact with those clusters Um on other use cases just having a central source of truth Which is a related topic and propagating that out to clusters as those clusters come and go Maybe you've got in a uh some secure like robust data center type facility A larger management cluster and then you've got a bunch of edge clusters that are subject to change Maybe they're subject to to disasters or fire or uh theft or who knows what or just failure and you're just going to replace them So propagating out central sources of truth Kcp in fact is again coming up and uh and we're very relevant in that domain Management um scaling upgrading uh remote clusters That's a that's definitely a use case And uh having some kind of central controller One controller that itself is not just interacting and managing resources on the cluster that it's running on But maybe it's one controller managing resources and and operands Across many clusters. Let's say you have 10 12 clusters running all in the same facility Do you need to run the pod set operator just to pick on our own? Fictional experimental operator. Do you need to run the pod set operator on every single cluster? Or could you just run it on one of them and have that one process? Be watching and interacting with all with all of those clusters. That would be nice, right? That's not quite possible yet So here's what we do have This is in controller runtime It probably could use some better or more prominent documentation if anybody's up for that But basically what we're looking at here Is this new resource uh this new type called cluster? So what you can do here what i'm showing here is imagining You're going to watch config maps in some other external cluster And then be able to retrieve the config maps from that external cluster when necessary So in this first function We get a cube config. We set up this cluster resource with that cube config And then in that cluster resource we can get a client In that client now of course can be used to access those things But here's an important aspect. There's nothing yet in controller runtime that really helps us Keep track of that reference keep track of either that client or keep track of that cluster resource So you got to keep just store that somewhere yourself. So here where I've got that first red arrow I just picked an arbitrary key called name, you know, maybe it's a cluster name Might be might be something different. It's meaningful to you in your circumstance But the point is you're going to create that cluster resource and then store it somewhere where you can retrieve it later And now you can implement a watch So that exists you can do a watch just like it shows here You inject the cache as you can see there on line nine if you all can read that. I don't know if you can but You can inject the correct cache. And then you're probably going to need a custom handler That understands how to map That event from a different cluster From a secondary resource a config map on some other cluster to the primary resource of your actual operator Whatever that might be And then in the second function here Now it's time to access that config map. We want to read it off that external cluster You're going to have to retrieve that client. And so here again, you're inventing your own mechanism for getting access to that client the right client at the right time So it's workable And it's a step in the right direction, but it clearly it's not quite a complete story for all the use cases that we see in front of us Um, so things you might consider for your multicluster access. There are different patterns based on different needs Uh, do you really need to watch events? Do you need to do this list watch pattern or not? Maybe you're just very occasionally Doing a request uh to some external cluster in which case Don't worry so much about a caching client all this stuff. You can just stand up a client on demand and just do it Maybe that's just okay for your circumstance How many clusters do you really need to interact with? Is it just a handful? Is it a thousand? Is it a million edge devices running some small cluster? That can really impact The style and and the way that you're going to keep a cache the way you're going to keep a client and where that's going to happen How you might shard that out? And then will you be adding and subtracting clusters dynamically? so for example With this cluster resource in controller runtime the the classic example that that has shown there is You're mirroring a resource from like cluster a to cluster b You know about these in advance and that's the job. You're you're doing this this mirroring back and forth But in these other multicluster management kind of scenarios you start your operator and then After it's running you're telling it dynamically. Okay. There's a new cluster that i'm introducing I need you to start managing this cluster is has gone away now. It's been deprovisioned Um, so as these come and go that that adds a significant layer of complexity Right now at least today watches are a lot easier to add than they are to remove Um, so that that's a significant consideration now. Where is this going kcp is doing some excellent client work If you're interested in this, uh, we have at least one I thought we had two maybe still around uh of the people who are actually doing this work Are up here in the second row you come talk to them But there's a world developing in a fork of controller runtime by the kcp project Uh to to move this forward where Your reconciled function will get a request that includes now not just a name and a namespace But a cluster identifier as part of that key And then you can create your a context that has that cluster attached to it And now as long as you use that context with future client requests Your client will use the correct cluster the correct cache And give you that sort of seamless client experience we've come to to really love and and uh take advantage of in controller runtime So that's I think where things are going But it's clearly still in development and if this is something that matters to you, um get involved with probably the kcp community is a good place to go the What's the channel in the kubernetes? Well, I think it's on our last slide. Well, it was a good last night. Um No, it's not on on our last slide kcp dev in the kubernetes slack and then what's the operator one? kubernetes operators in the kubernetes slack both of those places are good places to ask about this All right, that's the end of our content. I think we have like a minute or two for questions Uh, especially if you like this, uh, give us a review at this qr code You can contact us twitter Email or come up talk and then we'll be at the operator framework booth At least between when we leave here and let's say four o'clock If you want to chat some more about that or just chase this down later So with that any any questions we got time for one question No pressure um You talked about performance issues and ways to avoid them and i'm curious what resources you would point people to who are Did not have not used the the the tools that you've provided and need to go Diagnose their own terrible mistakes I'm asking for a friend Okay, so the question is the question is uh, my operator has not been a good citizen And is consuming lots of resources What do I do? Well, it depends on, uh What what is going wrong if you're using a bunch of watches? There's uh some nice ways to uh Trim that back and if not, I would recommend you come by kubernetes operators in uh kubernetes slack And you'll find that the operator folks are very helpful and uh Just nice people All right, thank you very much for a very interesting talk. Let's give the michael and austin a last one