 All right, hello, and welcome to Cloud Native Live, where we dive into the code behind Cloud Native. I'm Itai Shakuri, I'm Director of Open Source at Apple Security. I'm also a Cloud Native ambassador, and will be your host of the talk for today. So every week, we bring a new set of presenters to showcase how to work with Cloud Native Technologies. They will build things and break things and answer our questions. Join us every Wednesday at 11 AM. Eastern, this week, we have Billy Click, which we've asked today to talk about improving Kubernetes experience, sounds very intriguing. Before we jump into that, I would like to draw your attention to a KubeCon, Cloud NativeCon, North America that is open for registration, in-person conference, and also virtual. So very exciting. You should see the link popping up right now. If you want to take a look and register, we really hope to see you there. And also, just a quick reminder that this is an official live stream of the CNCF, and as such is subject to the CNCF Code of Conduct. Please don't add anything to the chat or questions that would be in violation of the Code of Conduct. So basically just please be respectful of your fellow participants and presenters. So Billy, would you like to introduce yourself? Yeah. Hi. My name is Billy Click. I'm a staff engineer at Digital Ocean in the past group. But we build things like Kubernetes, our DOKS, Marketplace, App Platform. I'm sure there's some things that I'm forgetting. And DBass is another one that we've been working on a lot lately. And I've been writing go code, primarily go codes. So it's about the end of 2013, shortly before go1.2 came out. We talked about a lot of go today. I'm also the package main. I'm also the maintainer for VIMGO. So if you write go and you're using VIMGO, probably you're using something that I work on quite a bit. Thank you for being here. Yeah. Great. So actually, quite an intriguing title. How can you offer to improve our Kubernetes experience? Yeah, so I'm going to tell you a little story. So when I started Digital Ocean in 2017, they had started creating a platform on top of Kubernetes. So it's been said that Kubernetes is a platform upon which to build platforms. And that's exactly what Digital Ocean had done. The platform is called DOCC. And it provides a, an opinionated platform really simplifies the Kubernetes experience so that all the Kubernetes users don't have to understand all the particulars of all the YAML and groups, versions, and kinds. They don't have to worry about updates or upgrades or anything like that. That's all taken care of for them. And of course, it's opinionated. So it's going to do things like it's going to inject environment variables into people's applications. So they have information on what region they're in or what particular cluster in. We run about 14 large clusters. These clusters are somewhere between 15 and 40 nodes, typically. And they, all the nodes have between 24 and 48 CPUs. So these are large shared environments. And so we want to protect those large shared environments from people that might, might be doing something that would be dangerous to others, right? You have to wear out noise in neighbors and things like that. So there's a lot of protections built into DOCC. But DOCC oftentimes does not have, it satisfies the 80% case. People sometimes need to get outside of that box and go to more of a Kube-Cuddle experience where they're using Kubernetes manifest directly. But even in those cases, they've come to expect certain things to be available in DOCC. And the team that maintains DOCC needs to be able to change how things work, change a lot of the implementation details. And so what we found is using things like mutating emission web hooks and belting emission web hooks allows the DOCC team to make sure like, regardless of how you're creating your applications whether you're using the DOCC manifest, which is JSON and can take like, I've seen it take like 280 lines of YAML down to about 70 lines of JSON. So regardless of whether you're using that or whether you're using Kubernetes manifest, you still wanna have those environment variables injected. You still wanna have your Syslog sockets and things hooked up and mutating emission web hooks is how we do that without having to make sure that everybody in the company understands all how the nodes are configured and make sure that they don't have to worry about hooking things up like all those environment variables. All right, so to clarify this for the audience, is the topic of the call DOCC and how it can help them or how you, the DOCC team used admission controllers and web hooks to improve your experience? Yeah, so it's about how we use emission web hooks. So, you know, emission web hooks are really the fundamental building block to creating a platform on top of Kubernetes. You can also combine that with custom resources but we're not gonna get into custom resources today. We're gonna focus just on providing a curated experience so if you're maintaining or managing a cluster for some folks and you wanna make sure that they have certain features available, whether that's mounting volumes automatically or injecting environment variables into the applications that you can, you know how to do that and start building that platform. Got it. So maybe let's start by introducing the concept of admission web hooks and controllers and see how it goes from there. Yeah, sounds good. So I'm just gonna share my screen here and I believe I'm gonna share my, I'll just share, I'm bashing my terminal window for right now. I've stood up a DOKS cluster in DigitalOcean and also created a DigitalOcean registry and hooked that up to the cluster already. So just real quick, let's kind of go through. Can you see my screen okay? Not yet, I'm sure in a second. Let me make sure I selected the right one. See the terminal window? I did not, okay, now we got it. Okay, great. So we have here a small cluster that has three nodes. This is running in DOKS. I'm going to just show real quick that we have a deployment and we just call this one example. And you can see that this deployment is really simple, it has a single container and all that container does is prints out the date and the host name every 15 seconds. We put a CPU limit in place and a CPU, some CPU requests in place. And the reason we put those limits in place is what we're gonna do today is walk through how to build a mutating emission web book that will react to that CPU limit and inject some environment variables based on that. All right. Okay, so right now there are no pods in the default nice space because I've scaled the application down. I'm gonna scale it up so we get one pod. You remember though, when we looked at that deployment there were no environment variables there. So now if we go look at there's one pod that we've created. We see that there's a go max prox environment variables been injected. And the reason we did that is digital is primarily a go shop. So all the things we're gonna talk about today we're gonna focus just on this one environment variable because it's simple and can provide the path forward. But it's not overly complicated but you can do very complicated things with mutating emission web books to provide that platform that you want. In this case we chose go max prox because go by its very nature is highly concurrent and built for concurrency and parallelism. When we have users running their applications on these nodes that have 48 processors go automatically wants to spin up 48 threads to manage all the go routines. But when you have a CPU limit of one that means you're gonna have 40 threads but you only have one CPU. So you can't really take advantage of all the things you'd want to take advantage of for parallelism. So you want to instead of thrashing between all those threads we want to make sure that the level parallelism is the same as or reasonable according to what the user set the CPU limit to. And so that happened through a mutating emission web book. So to set this up, there's a couple of things that we need to do. First thing is you have to create a mutating web hook configuration so let's go take a look, see what that looks like. So mutating web hook configuration is pretty straightforward. It really consists of a web hooks property that contains an array of admission review versions and that admission review version contains a client config which has a CA bundle. So this is gonna be the certificate that the Kubernetes control plane will trust. So that all the Kubernetes web hooks have to be secure with SSL or TLS. So it's HTTPS only. And so in order for the control plane to be able to trust what it's talking to you put the CA bundle in the mutating web hook configuration. You also provide the URL for this particular web hook. A failure policy which can be either failed or ignore. A match policy, a name for the, sorry about that. A name for the particular review version which in this case is gonna be, we go up to this, yeah. We're gonna do object ignorable, go max procs. Then a selector which says what things, what resources this should apply to within the cluster. And then of course the kinds of resources and the operations on those resources that this should apply to. And then this, yeah, go ahead. Maybe we can describe in a second the high level flow of things so that people can visualize it in their mind. So someone creates a resource, let's say a pod. Kubernetes API server handles this request. And because we registered this web hook then we tell basically Kubernetes whenever something like this happened that matches this criteria, please call out to my web hook handler, my web server that I stood up in advance, right? And this is where for example the CA bundle comes in because how would Kubernetes know that it can trust that server? And this is the criteria. And so I just wanted to summarize the high level flow for the viewers. Yeah, that's exactly right. So what the mutating web hook configuration is doing is providing a configuration so that the, so the Kubernetes control playing kube controller can knows what to call, knows when to call it and knows whether or not it can be trusted. A web hook can be either a validating web hook or mutating web hook. Validating web hooks can be used to make sure that someone has the right permissions to modify a certain field or that they don't use certain fields. So like in earlier I talked about DOCC. DOCC for instance, because it's a large multi-tenant environment, make sure that people can't use host network. It does things like make sure that people can't spin up privileged pods, right? Because we don't want all those users getting access to the underlying notes. We also put restrictions on what volumes they come out if someone is using Kubernetes directly. But so yeah, this is all just explained to the control playing exactly how to connect, when to use the web hook and what things it should apply to. Great. Can you make the font a little bit bigger please? Bigger, absolutely. How's that? That's good, thanks. Okay, all right. So that's the mutating web hook. In this case, we are running the web hook in the cluster itself. And because we're running it in the cluster, we also want to make sure that if the web hook is down for some reason, or if certain critical things that might need to be executed in the cluster, like your CNI, if you're running that in your cluster, that the lack of the instance, if the instance is down, like if you spin up a new cluster, or if you're doing an update or something while someone else is trying to make changes to CNI, that we don't get into a circular dependency problem. So we make these match policies a little bit more complicated than what they would need to be. In this case, the base admission policy is this last one here, which is just go max procs, CNL, web hook configuration. Then we add two others. They're really function as escape hatches. The first one is NSignorable, go max procs with that name. And what it does is it makes sure like, hey, if a label CNL web hook allow web hook failure and this is customizable exists on the namespace, and it doesn't exist on the object, then we're going to use a policy of ignore. So that if the web hook fails, it's okay, the pod can still be created. And then we do something similar with the object ignoreable where we say, hey, if the namespace selector doesn't exist, but the objects label does, then that's okay. We're also going to ignore in that case. So all those critical components that we run in the clusters, we make sure they actually have all the things that are necessary and that the mutating mission web hooks don't have to operate on them. So that's the configuration. Real quick, I'd like to show the deployment itself that we're running here. It's also, it's a very simple web hook. It's a very simple container. I have a CNL web hook container that's hosted in the DOCR registry, where it's supposed to import 443 and I've mounted in certificate. And in this case, I just use a self-signed certificate. There's obviously if you're going to do this in production, you'd want to do something probably more robust than that. And so this runs in the coordinating space. And I also have this deployment has, all the pods has that label we talked about before to allow it to be ignored so that if the web hook isn't running, that's the web hook pods can still be created. So, people are now exposed to the concept of validating web hook, mutating web hook. And you've shown a few examples for how you are using it, I guess, digital ocean. Is this, like, do you have any recommendations for ideas more, what people could or should look after with those tools? Like, what possibilities, what are the possibilities for people to enforce or to change? Yeah, so if you're doing mutating a mission web hook, one of, like some of the things that we do in DOCC that are very useful is hooking up the syslog socket for people, because we have most of our services in order to get logging to centralized logging are going to be using syslog. And so we go ahead and hook that syslog up into all the pods for people automatically. We also do things like inject environment variables to let applications know which cluster they're running in, which region they're in, what their service domain namespace is, or sorry, what their service domain is, and the cluster name itself. And that all feeds into centralized logging so we can identify where something actually is. And we have 14 large clusters. If you have your application deployed into five of those, seeing errors in the logs, you need to know exactly which one is coming from. So we inject all of that. We also do things like inject the root CA certificate so that we can establish trust with all the other services that are running in the clusters. So that people don't have to worry about including all those things in their containers or in their application that just gets hooked up automatically for them. Yeah, it makes complete sense actually. And it's also maybe stressing a well-known debate or idea of the separation between the developer's role and the operation people role when it comes to Kubernetes, which is kind of in the middle. So the developer is mostly concerned with writing their code and making sure the application is functional. Where does the line go? Like do we ask them to also more about Kubernetes primitives to develop? And if they do, how far should their knowledge or familiarity with the platform go? And in this couple of examples that you just shared, I think we see a nice idea of letting the developer deploy their applications using Kubernetes primitive, but kind of a simplified version of it where you say, don't worry about it. I will fix up whatever I need, meaning the operation people in order for that to work in the bigger context. That's right. Yeah, this is all about making sure or trying to make sure that application developers can focus on delivering that business value. And they don't have to worry about all those operations in the particulars about how to configure their applications or operate their applications in these large multi-tenant environments. Especially as an organization grows and gets more complicated, you want people to be able to specialize. And you want application developers to be able to focus on developing code and delivering business value and operations folks to be able to just focus on operations. And these mutating mission web hooks can help bridge that gap and to reduce that trial and knowledge. It also frees up operations and administrators to make changes to how the clusters are configured. If we decide, you know, for whatever reason, we want to put the syslog socket into a different place where we want to start introducing new environment variables to explain or to allow applications to understand more context of where they are or what else might be near them, then mutating mission web hooks can help with that. And you can just introduce new mutating mission web hooks and let people know, hey, when you redeploy, you're going to have these new things. Yeah. So if it's okay, I'd like to start going into exactly how we build one of these mutating mission web hooks and start going to controller runtime. And so I'm going to start out very simply. We're going to start at a high level and go through the main, you know, just the high level of how this works. And hopefully by the end, we'll be able to talk about testing a little bit because controller runtime has some great testing features. We'll back up just for a minute. Kubernetes, if you've written Kubernetes code, you're probably familiar with controllers. Controllers are going to work with informers or watchers. Controller runtime takes informers and watchers and those concepts and reconciliation concepts and abstracts a little bit further. So they're even easier to work with. Controller runtime also provides the framework for working with web hooks more easily. So, you know, Kubernetes itself doesn't prescribe what a web hook has to be implemented in. It just says it needs to be HBS. You could implement your web hooks and Ruby if you want to. You can put your web hooks in and go or Rust or whatever feels most comfortable to you. But controller runtime is targeted go and makes it very easy just to start creating web hooks. So what we have here is we have a very simple application that is our web hook server. And the main function just calls real main so we can get back in error. It's a common pattern. It does really three simple things. It creates a configuration, passes that configuration off to the controller. In this case, controller is not like a reconciler. It's just, it's a control in the sense that it's gonna run. It's gonna have some live time. You could think of this as anything that's runnable. This code is borrowed from some of the things we have at DigitalOcean that I've pared down a lot. So this is kind of set up so you can build out from it and start making it much more complicated and introducing more than this one web hook. You start introducing reconcilers and things like that if you wanted to start dealing with reconciling resources in your cluster. So after we get config, we create the controller and we wanna hook up context handling so that when the pod needs to be terminated by Kubernetes, it will handle the termination signal and shut the controller down quickly so it doesn't have to be killed. And then we run the controller. So all pretty straightforward. I'm gonna go over here into new config because it has some, it sets up flags and some things that we don't need to worry about too much today. But one of the important things that it does here is it has to create a manager. And this is one of the key things in controller runtime is this concept of a manager. A manager provides the scheme, a cert directory report, a number of other options that you can add. If you're dealing with resources or if you're doing something that is mutating a custom resource or even some core resources, you need to add the scheme to your options. And you do that by just doing runtime.newSkeam, adding that to KubeSkeam and passing everything off to the manager. And that's a critical piece for a controller runtime manager. After we create the config, then we need to go create the controller itself. Again, this is really straightforward, pretty simple. We have a base URL that we need to configure. We have a number of options that we got from new config and it creates the controller. In this case, it's just the thing that has a run method. I remember from over here on main that when we're done, at the hookup context, when we've done hookup signal handling, we call run on the controller. This is where the meat of things happen. So we set the logger on controller runtime so that both our application and the controller runtime manager will be using the same log so everything will get to the same log sync. Are they gonna, we're then going to register some webhooks. We're gonna register the webhook configuration. And this might be controversial. I know that a lot of people like to work with straight YAML. And so that mutating webhook configuration we saw earlier is actually being created in this case by our process. So I personally don't like to work with, I don't like to have my YAML separated far away from my code and I want it to be more dynamic. So that if I deploy a new version of our controller that the muting webhook configuration also gets updated. So they're tightly coupled because they are tightly coupled. That allows us to move a little faster. It gives us locality of reference and it makes it a little bit more testable as well. So we register the webhook configuration. We'll go to that here in just a minute. We have a set of handlers and these handlers in this case is just gonna be a bunch of runnables. In this case, one of them is gonna be the manager. We run all those in a go routine and wait for them to return. And then once we're done we return back out of our run function, the main exits and the pod is done. So I just wanna clarify what you said like a moment ago about the YAML being coupled with the code. Does that mean that when I want to deploy this admission webhook instead of deploying the pod and registering it, I will just deploy the pod and it will register itself? That's exactly right. Yeah, so you certainly could have your manager YAML separately from the code that you're deployed but then the problem of course is that then you need to go deploy your code and then you have to go up, you deploy your update and then you have to go update the YAML and so there's this time period where there can be a impedance mismatch between what's actually running and what the configuration is. Of course, if you do the opposite order you have the exact same problem. So by putting it into the code we can be assured that like when we start running we update the configuration right away and immediately the handler kicks off. Great, thanks for the opportunity. Yeah, so again in run we're gonna register the webhooks and what we're doing here is we're gonna be registering with the manager. We're gonna register the webhook configuration. We're gonna for that we're actually gonna register it with the Kubernetes control playing. We register the handlers and so let's dig into what registering which registering the webhooks actually looks like. So this is again, it's a pretty simple method. We have a, we get the URL for our, for this one particular webhook or mutating webhook configuration can describe multiple hooks. So in this case we're just in the go max prox webhook then we register that with the hook server which is, sorry about that. We register that with the manager. So we grab the webhook server from the manager and that this is that controller runtime manager type. Register it, now all we do is we just give it an admission type and an admission is an interface that has a handler method and that's kind of it. And so in this case, if we take a look at a register server obviously it just takes an HTTP handler which if you're familiar with those that has a simple function that has a response and a request an HTTP response and HTTP request. So that's all you pass off to register. This admission type has a handler and it is a handler interface which just has a single handle method. And in this case, we are going to create a mutate pod strategy handler. So remember, this is really part of a larger piece of code. So you wouldn't have to have a strategy you could just have a single handler that will take care of what the strategy does here but we'll take a look at the strategy. You can see, again, it's really pretty straightforward. We hook up a logger and a mutate function. In this case, the mutate function is going to be modifying our DOMX prox. In the handle method, there's a couple of things you need to do. You need to decode your pod because that comes in on the web hook admission request. And then in our case, we're going to mutate it will be modifying the pod in some way, potentially. Then we marshal the result of that process and then provide a patch response. So what this does is this tells the control plane, hey, you gave me this Roth in your original. Which is JSON. I'm going to give you a JSON response back to that and what the control plan will do is look at the differences between those two things. And well, sorry, back up. The patch response from raw gives, compares the raw and the marshal, figures out exactly how to construct a patch that Kubernetes understands and passes that back in the response for Kubernetes to mutate the pod itself. So these web hooks don't mutate the pods themselves. It tells Kubernetes how to mutate it. And then once we've done that, we now have our web hook is registered and it's time to update the configuration. Before we go on to the configuration, you have any questions about how those handlers work and we want to get into looking at the actual handler for the Go Max Prox now or should we keep going to the configuration first? I don't see any questions coming up on chat. So yeah, we can continue just a quick clarification, actually a question of mine. So in the previous step, we got the raw object into the web hook handler. We did something and we returned a patch. Does the patch go through the same pipeline and again through the admission web hook or is there some kind of exception there? I'm not sure I understand the question. So if the result of this operation is a patch, meaning please Kubernetes do something, that is another like an operation against Kubernetes API and the web hook is validating or mutating that request as well or not? Yeah, so if, I mean, I think your question is like, if we modify the pod in some way, mutate it, does it then get sent back through again because it's being modified? Yeah, I mean the actual request to modify, yeah. Yeah, so we're creating a pod in this case. So it's actually part of the mutating web hook configuration about whether or not you can call multiple times or not. So you can either say, yes, call me, at least you can call me up to one more time or don't call me again. And so if you have a, if one of your web hooks might need to respond to changes that other web hooks might do, then you wanna make sure that you can call this web hook multiple times for a single creation event. But if you don't care, if you just need to do your thing and move on, then you can say, yeah, don't call me again. And that's part of the web hook configuration. Good, thanks. Yeah, that's a good question. Okay, so after we create and register the web hook with the manager, then we wanna go ahead and register these web hook configurations with the control plane. In this case, again, we're gonna need that web hook base URL because we have to put that into the mutating web hook configuration. So in this case, what we've done is we've created a set of web hook configurations. So what we're gonna do is we're gonna say, hey, give me all the mission web hook configurations. And we have a set of these, see if I can find them under there. Yeah, so it's just a string of them or a slice of strings of them or the ones that we wanna configure. And then we say, okay, give me the web hook config. And what we've done here is we've defined a type in our code that is really just the things that we care about on a web hook config. The things are gonna vary among our web hooks. And that's gonna be the path. So we're gonna be running an HTTP server. We wanna provide a path so we can deal with different web hooks being served by the same server. We're gonna describe a failure policy because this might be ignored, it might be failed. We're gonna have the re-invocation policy, which is what you were just asking about, which is how do we deal with this web hook if another web hook changes something out of that as well? And you can look at this documentation that describes what this particular policy type means. So again, it can be called at least one additional time in this case because we have if needed re-invocation policy. And in this case, because we're dealing with pods, we really don't care about Cree. I don't care if somebody is trying to update a pod or something like that. We're modifying the containers and the environment variables on it. So we don't need to worry about anything other than creates. You do need to make sure you set them to scope. It's either a namespace or a namespace scope because we're dealing with pods. They are namespaced. So we're gonna go through that set of web hook configurations. We'll grab the URL for each of them so we can build it. And so what we're doing here is we're grabbing the base URL from our controller and just adding the path from the configuration we just looked at onto that and make sure we resolve the reference properly in order to set that onto this mutating web hook client config is here. We also go ahead and include that sort of bundle for the server. We apply the roles in the configuration, the policy, all those things that we just talked about. And this is our initial handler. Now, remember, we said we wanna be able to make sure we respond appropriately if there are critical things that don't care about the web hook, it's okay if the web hook fails because we have those things fully configured. In those cases, usually what's happening is the operators of the cluster are making sure that everything is configured for those things correctly because these web hooks are primarily focused on making it easy for most users. What we're gonna do is we're gonna say, hey, if the failure policy on this particular web hook is fail, then we're going to add a couple of selectors. First one, we're gonna make sure the namespace that doesn't have the allow web hook failure label. We're also gonna make sure the object doesn't have the allow web hook failure. And then we're gonna add that to mutating web hooks. We're also gonna pass, in this case, it goes additional web hooks based on this particular hook. So we're gonna create the normals from mutating web hook. And this is, it's in the controller to make it easy. And a couple of things. We're going to copy our original hook because when you build Kubernetes objects, when there's so many pointers involved that you wanna do a deep copy that we can get a fresh full copy because you don't wanna modify shared objects or shared values on these two nested structures. So in this case, we're gonna say, hey, the failure policy is going to be ignore if the label selector for the namespace doesn't exist, sorry, if the label for the namespace doesn't exist, but it does exist on the object. And we're gonna do the same thing, but for the namespace where we say, hey, if the namespace has the allow web hook failure label, then they can be ignored as well. And then we return all those web hooks from admission web hook and fix. And then we just do a pretty straightforward absurd operation where we, yeah, went up a page and didn't mean to. So we check, see at the meeting web hook configuration already exists. And if it does not exist, then we wanna create it. Otherwise, we wanna update what was already there. Standard absurd operation. And then finally, we might register some handlers and this in this case is mostly just the runables you'll see here in a minute with when we get some of these tests about how this works with other handlers. So if we were running more than just the one server. Any questions before we go into the go max procs mutating function to see how it actually works? I don't think so. I'm monitoring the chat, so if something comes up, I'll let you know. All right. So we're going to register web hooks and we're gonna mutate go max procs. So we have a strategy so we can reduce the boilerplate and that strategy takes a mutate pod, which is just a function that takes a context, a namespace string or the name of a namespace in a pod. And so in this case, our mutate pod function comes from this constructor, new mutate go max procs and it passes back set go max procs. So this is the thing in our case that really it's gonna be doing the work. Let's figure out what the pod should look like. So first thing we're gonna do is we're gonna go through all the containers in the pod spec. Remember we already decoded this over in the strategy to reduce the boilerplate when we have multiple pod mutators. First thing we're gonna say, hey, there's a have a go max procs environment variable. If it does, then we don't wanna do anything. We just wanna continue because at that point we're assuming that the users have intentionally configured go max procs and we don't wanna overwrite whatever they've done in that case. And assuming that they don't have go max procs set, we're going to grab the CPU, make sure it's not zero. And then we're gonna see if it's less than the minimum CPU, if it is then we'll set the CPU limit to the minimum CPU. And then we're gonna set an environment variable for go max procs. And that go max procs is always gonna be a whole integer value. So we use an integer division here. So we set max procs on the go max procs environment variable add that to the ENV slice and then set the container back. So it has that new environment variable. One thing I feel like I should point out here is that at the top of this function, it does talk about I think it's in here. There are multiple, maybe I don't have it in here, but there are multiple ways you can set environment variables. It could be ENV or it could be like from ENV I think or ENV from, in which case it could come from like a secret or something. We don't mess with those in this for this mutating mission webhook. You certainly could if you wanted to cover all the cases but we're just trying to get 80, 95% of the cases here to help protect the health of the clusters. And that's what the go max procs webhook itself looks like. And so at this point, we have our controller and we've run it. And now we are simply waiting for the whole thing to finish before the process ends. And that's what it's waiting for. Nice message saying, hey, we're stopping the controller. So there's a couple of other things we should talk about. One is testing. Controller runtime has some really nice test environments where you can create very holistic tests that will actually use a subset of the control plane. It uses Kube API server and etcd. So if you have those binaries locally, you can run your tests against the API server and that etcd. So that you can actually make sure like, hey, I'm accepting to modify this pod, right? So this is getting beyond unit tests and getting into like testing integration, right? We wanna trust Kube controller do what it's supposed to do. We wanna trust Kube do what it's supposed to do. But we wanna make sure that Kube API server is calling our webhook and that our webhook is delivering the proper response. So controller runtime provides an ENV test. I believe that's the practice name. I always have to look it up. Let's see if I can find it real quick. Yeah, it is named ENV test. And we'll take a look at the documentation here in just a minute. But if you have those binaries locally, you can run your test against them to make sure like, yeah, hey, my pod, after I create it, I actually go grab it and it actually looks the way I expect it to look. And so we'll take a look at those tests. But before we do that, I wanna real quick show what this would look like, this one. Let me know when you can see the browser. Do you see the ENV test documentation? Yeah, now we do. Okay, great. So ENV test is pretty straightforward. You can control things with the Kube Builder assets and the environment variable that just specifies a location on your machine where you can find a Kube API server and an STD. You do not have to be running a Linux to do this. So although Kubernetes really only publishes the Linux binaries, you can get the source code for Kubernetes, go into the root directory repository and run make Kube API server. And it'll build a Kube API server for you on your system. So I'm on a MacBook and we'll see what that looks like here in just a minute to actually run these tests locally. Also gonna show, is that to kind of what using some of these tests looks like and how to configure something right to the terminal window? There you go. You see it now? Yeah. Great. So first thing we'll do is we'll just do go test here. Here. You're gonna see, not much, you're gonna see test pass. Right now we're gonna say, so in this case I have the test that used the EMV test environment tag with the sandbox environment because they require something more than what someone might have on their machine. And so we separate those from standard unit tests. So I'll provide the sandbox tag to go test. While it's running, there's a question in the chat. What's the best way to conditionally set GoMark's crocs because it's only applicable for Go applications. So you mentioned that you are a Go shop, so maybe you are not concerned with it, but do you have a general recommendation for people? Yeah, you know, so I am for go, so the specific question, I think the right answer is just set GoMark's crocs. Just as if you have a lot of Go code, just go ahead and set it, it's not gonna hurt other things. The general question of how you would control that, if there were some other kind of environment variable that you might wanna set for say, for whatever reason for a Java application, but not for a .NET application. In that case, I would, you would probably have to go with an annotation. And the value there too though, is that you can using annotations and providing a set of annotations for your users that you support, you abstract away the details of what might happen. So you might say, hey, in our particular environment, we know that we're gonna wanna do certain things. Those things might be like, oh yeah, I need to hook up logging. So one of the things that we do at DigitalOcean, and we talked about getting the logs that's centralized logging, and I talked about Syslog, but we actually have more than one way to get logs to centralized logging. We can have that application right directly to Syslog, but we can also read it directly from standard error and standard out and use and flip it and get those things to Syslog. So in that case, we provide an annotation, two annotations actually that allow people to kind of opt in to the Syslog experience or just to do the standard out logging to get their logs to centralized logging. And so when you're building a platform, you wanna provide something that you can support. And so if you have some kind of environment that people may be able to set, maybe it's not actually an environment available that you can set, they want them to be able to set an annotation. And then your web hook can look at that annotation and take appropriate action. That might be setting an environment variable and it might be something else, right? You might have something that says, hey, I want all of our folks to annotate their pods with kind of what kinds of things are running in this, like how they're being run because you might wanna hook up JVM environment variables or something like that. Great, thank you. So here you can see we got a lot more tests that came out of this or a lot more logs that came out of this. And the reason is because it spun up the UAPI server and STB. I'm gonna run this in verbose mode so you can really see all the logs because I'll show you in just a minute. The tests are configured so that when we run tests in verbose mode, we also hook up the API server and STD logs to standard out so we can see everything that outputs, which is really useful whenever you're having a problem. Maybe your server is crashing in control as I'm working or it's not trusting the certificate. So your tests are failing because you can't see why because it's really QBAPI server is getting a response it can't handle. So here we have a whole lot more logs and these all come from STD and from QBAPI server. Let's go take a look at those tests real quick. So in this case, we have a simple test function, test go to xprox. And what we do here is we create a pod so it doesn't have the go max proc set. But we do make sure it has a CPU limit. And then when we're done, we make sure that our ENV variable or environment variable for the container has a go max proc set. And the value is what we want it to be. Obviously this can be, you can handle all of your other edge cases, you can make sure like, yeah, if I have go max proc set that it doesn't override it, you can make sure that when you're dealing with parts of CPUs that it does the right thing. In this case, I just want to do something simple to show you how like we can actually test this. And the way this works is I've created this function with test ENV, little tags. We have with test ENV, which creates a new test test ENV and then executes the function that we pass in here to actually do the test. This new test ENV is a thing that actually sets up Kube API server and the control planning that we need for these tests. It's pretty straightforward. Again, you create ENV test environment. This again, this comes from controller runtime. You grab the API server, we make sure we configure it. You can pass pretty much any, you can configure all the API server flies that you want. Through this method, you configure returns back a set of process arguments that have an append method that you can see, you can add additional arguments to API server. And then you start it. We give back as a rest config that is, if you're not familiar with these, this is a type in, I believe it's in, I think it's in client go. Yeah, it's in client go. It contains the host API path, the credentials you might need, as well as a TLS config. So that allows you to connect to the API server that it stood up. And again, this API server is working locally. You can't connect to a real cluster if you want to, but you mostly don't need to because all you really wanna do in these cases, usually it's test like, hey, the pod has the right form. And then we're gonna trust Kubernetes to do the right thing with it. And then we passed that rest config off to our new config function that we saw earlier that configures the manager and everything. And so I really encourage people to use test ENV. You don't have to use that. You don't even have to be using controller runtime. If you're writing code in, if you're not using controller runtime for whatever reason, using watchers and formers, you can still use controller runtime test ENV to really write very robust tests. It's a good tip. Thanks. And also thanks for the crash course here on controller runtime, web book, admission controls. I think that it was very educated even for people who might have been familiar with the concept, looking at the code and looking at actually a real world use case sharing from your experience. So thank you for that. We do have like a couple of more minutes if someone want to ask some questions over the chat. If not, you could use the Slack channel that we have which is cloud dash native dash live on the CNCF workspace on Slack. So feel free to ask further questions there. I don't see any other questions. So I think we could wrap it up. Any last words, summary, Billy? Yeah, so a couple of things. You know, I just kind of want to re-emphasize that mutating with admission web hooks and validating admission web hooks are really the foundation upon which you can start to build a platform on top of Kubernetes for your users and start to get rid of some of that tribal knowledge and allow your administrators and operators to be more flexible in how they deal with everything, how they provide a consistent experience to their users. Thanks for having me. This was a lot of fun. I think my asking questions, feel free to reach out. All right, thank you. And just a reminder, this is Cloud Native Live. We're here every Wednesday. So tune in next time. And thank you, Billy. And see you next time, everyone. Goodbye. Bye-bye.