 Hello, everyone. Good afternoon. My name is Igor Valychkovich. I'm a SD for Amazon EKS, and we're going to be talking about what you can do with admission web hooks and admission controllers and also the dangers of those. And I'm joined by my colleague here. My name is Amin. I'm also an engineer at Amazon EKS. And, yeah, before we kick it off, I would like to ask you a question. So, by show of hands, have anyone here used admission web hooks before? A little bit. Have you ever broken a cluster or an application running in your cluster using admission web hooks or admission controllers? Nice. I already see three or four. Five. That's cool. You're in the right talk. So, today, we're going to be talking about exactly those problems and how to mitigate them. In today's menu, we have a quick introduction to mutating and validation web hooks. So, if you never used them before, it's completely fine. We're going to talk about them. And then we're going to switch to the dangers of using the admission controllers and web hooks. We will also, Igor is going to do a demo of web hooks and how to take down your cluster using mutating web hooks. We're going to also talk about a new feature that just landed in Kubernetes 1.27 and 1.28, called match conditions or the common expression language also known as cell. And we're also going to finish with a demo about how to use cell on leverage in your cluster for better security overall. So, let's state the problem. In early 2023, we stopped using KGCR IO. And we started recommending to use registry case to IO for multiple reasons that the community is very aware of. On the bottom, you can see them tweeting about this in 2020, like saying, like, hey, we should switch registries and try to spread the message across the community. Anyways, the reason why we're talking about this, like, hey, as an administrator, I want to enforce this policy in my cluster. I want my cluster users to start using the new registry. However, there will be always someone who's going to forget about it. So, let's say actor, or let's say Bob or Alice, someone who forgot about them streets. And they're still writing manifests, let's say pods or deployments. And they're still using the old registry. Let's call it A. They are using KGCR IO. And what's going to happen, they're going to apply that to the API server. The API server is going to persist the information at CD. However, I want the API server to persist the new DNS or registry address. One of the ways we can do that is by using mutating webhooks. Thinking about mutating webhooks is like an extension to the API server. I want the API server to do extra action on my manifest or the information of my application. And I want, for example, I want the API server to change from registry A to registry B. Another webhook that is not mentioned here is called the validated webhook. For example, you can enforce or reject any requests coming with a registry that you don't want to use. For example, if the actor who forgot about them tweets is using KGCR IO, you can reject any cube color apply or create in the cluster using validation webhooks. Let's zoom into the API server. So what's happening behind the scenes is that whenever the request is coming, there are like multiple steps or layers involved before accessing HCD. The first one is authentication and authorization. I want to know if you have the right to access the cluster and who you are and whether you have the permission to do a specific action or not in the cluster, which is also known as RBAC. The next step is called mutating admission webhooks. So once we know who you are and that you have the permission to do a specific action, we're going to take that request and send it to a mutating webhook. Say, like, hey, do I have, for example, an old registry that I want to move to a new registry? And the webhook is going to send a new response and say, like, hey, here's my new object, I changed a few fields in it, go ahead and validate it. The next two steps, they are mainly about validation and the last one of them is going to be the validation admission webhook. So we want the API server to validate whether my request is valid or not. If all those four steps are executed successfully, the object is going to be persistent in HCD. Here we have two examples. On the left, you're going to see the validation webhook configuration, and we're going to have, for example, on line four, you're going to see the name of the validation webhook configuration. You're going to see the rules on line seven. I want only to target the operations that are, like, creating pods and the API version v1. I also, like, I'm giving information about the webhook endpoints, for example, in line 13, I'm like, hey, go, call the webhook in the name, example service, and name space, example name space. On the right, we have a very similar example, but this one for mutating webhook configuration is not very complete, but you can see it's very similar, and it allows you to target specific operations and a specific workload in your cluster. So as we can see, there's a lot you can do with admission webhooks. That was just one example. However, there's a lot of dangers that come with them, and the reason for that is, when you use an admission webhook, all the operations that that webhook targets are now, your API server has a dependency on your admission controller, and that risk can be increased by failing closed, having too broad of scope, or too long of a timeout. So I'm going to talk about those three cases and what you should do and should not do. So failure policy determines whether you're going to fail closed or fail open. It basically says if there's any issues communicating to your webhook, from the API server, how it should behave. If it fails open, it'll let the request through. If it fails closed, it'll deny the request. The default is fail, so what you should do is think through and consciously select a failure policy and understand the risks and implications of that failure policy. There are valid use cases for failing closed, right? Say for security reasons, I only want certain containers to run on my cluster. Maybe I would rather have operations break than run an unsecure container. But don't just go with the default value and not think about it. Too long of timeout. So the timeout seconds field defines the timeout when calling your webhook. If a timeout occurs, your failure policy will be invoked. The default timeout is 10 seconds and you can use anything between 1 and 30. So do try to use a short timeout as short as is reasonable, because your webhook is going to be adding latency to the API server. Validating webhooks will all go in parallel, so that helps. But mutating webhooks are executed one after another. So if you have five and they all have a timeout of five seconds, you can add 25 seconds of latency to your request. And that can be really detrimental to clusters, especially at large scale. I've seen clusters with thousands of pods where GC garbage collection isn't able to clean up the pods for hours. And that's because there were several webhooks with 10-second timeouts taking 10 seconds. So every time API server is trying to do GC and clean up pods because they're supposed to be scaled down, it's taking hours. It can't keep up because of that latency. So one thing you definitely should not do is if you start seeing some timeouts, try to just use a really long timeout. You should try to root cause why your admission controller is taking so long. And don't just try to extend the timeout to get around it. And then too broad of scope. All of these risks are only, only impact what you target with your rules. So here's an example of too broad of scope, right? We have all wildcards. This is going to target all mutating operations on the cluster. This, these rules are the blast radius of your webhook. So do set the scope as tight as possible and do it in the validation webhook layer. Suppose I want to only target config maps, for example. I should not use wildcards and then in my application logic of the admission controller, say, okay, if it's not a config map, just let it through, otherwise do the validation. Make sure all of this is in the webhook layer. So now we're going to do a demo and we'll do a live demo because that's a bit more fun of breaking a cluster using webhooks. So I just have a local kind of cluster running here, have some components in CUBE system, and I'm just going to create a deployment. And this deployment has a service, very, very simple, straight from, you know, the basic Kubernetes example of a deployment. And it has three replicas. And we can go and look at our endpoints to make sure the service is working properly. Everything's good. So now what I'm going to do is I'm going to create a webhook. And I'm going to simulate a failure and do some bad things. I'm going to target everything. I'm not going to set the failure policy, so it's going to be fail. And there is no example service at example namespace. So I'm simulating a failure as if the admission controller was down. So I'm going to apply this. And now if I go and I try to create deployment two, which is the same thing just with two appended, oops, I did not apply the blocking webhook first. That's for the next step. Okay, now when I go and try to apply deployment two, it all gets blocked. This is pretty easy to troubleshoot, though. This is pretty benign. I guess you can't create things, but what's happening here is you get the error message right away. This will probably be happening in like a CI CD pipeline or Argo CD or something like that. But if you go into the logs, you'll see this and you'll know exactly what's wrong. So now suppose that I want to make sure that only our platform teams containers are running in KubeSystem. So I'm going to create a webhook that targets KubeSystem. And I'm just going to do that by changing this webhook. Okay, so now that I created that webhook, now it's only targeting KubeSystem, and now my deployment should go through. And there we go. It did go through. So I should have some pods, but I don't have any pods. Okay, so something is wrong here. Let's go look at the deployment. There's deployment two and we have zero pods. Alright, let's describe it. Let's look at the events and see what's wrong. Okay, there's no events. Alright, this is kind of interesting to troubleshoot now. Okay, let's see if there's a replica set at least. Only for deployment one. And we have no pods. Interesting. Now I'm just going to try to create a pod directly. Maybe that'll work. Okay, the pod went through, but it's not scheduling anywhere. The pod is stuck in pending. Okay, so I can't even launch pods. Maybe I just need a new node. Let's describe the pod and see what it's unhappy about. And also no events. Okay, great. Not really sure what's going on there. So now I'm going to try deleting the deployment. Let's just start over. We cleared out the deployment. Okay, and all my pods are still there. This cluster just is not working at all. Maybe I try to manually delete a pod. Let's see if that works. Okay, at least that worked. Hopefully now that should be taken out of service so it's not still serving traffic. Now that it's no longer there. I should have deleted the deployment. Yeah, so I can't create pods. I can't create deployments. This cluster is just hosed. And let's look at why. So if we look at kubesystem, we're going to see that the scheduler and the controller manager both have a restart. So something interesting happened there. So let's go and peek into them and see what happened. I'm going to look at the previous logs to see why it crashed and restarted. Okay, so leader election lost and shutting down. So it looks like it lost its leader election. And now let's look at the current pod to see what's happening. Okay, failing to update the lock. So what happened is this webhook ended up blocking the leases and these components crashed because they couldn't renew their lease and they could never grab it. So now they're just sitting in standby mode trying to get a lease doing nothing. As you can see this cluster is completely hosed. You can't really do anything with it. And those are some of the risks you run with admission webhooks. There's a lot of other things that can happen too, like for example if you're blocking leases in kubenode lease, you'll start to see your nodes going not ready. There can be issues with API server startup if you're blocking flow schemas or role bindings, API services. So those are some of the dangers. Now part of the problem is if you look at this webhook the rules are targeted in everything, but there's no negative selection. So there's no way for me to say I want to target everything except for leases. Until some of these new features that we're going to talk about next. Thank you Igor. So now that we've seen examples on how to break your cluster or your application overall using mutating webhooks, we're going to jump ships and talk about a completely different topic. There is what we call cell or common express in language. It's a language, it's a new language created in the early 2020s by Google, however it's not a programming language. It is a scripting language. Is it quite fast? I was very impressed about how fast and very like close to the performance of just pure go. It is very very easy to write and to understand. It's a little bit like Python or JavaScript overall. It is also easy to extend and to embed. Let's say like you want to bring some functions to this language or to embed it in JavaScript or Java, it is pretty easy to do. If I'm not mistaken, there is only one implementation that is in Go, but there is no different implementation but corrects me after from rock. I've never seen any. If you want to play with this language, there is what we call the cell playground. It's a website created by Matias. I don't know if he's here or not, but shout out to that very cool tool. And yeah, without any more talk, we're going to show you how it looks like. Let's say I want to validate these YAML objects. I want to say let's say I have this event named CubeCon, location Chicago, and I have a list of attendees, whether they have a coffee or have a funny hat. It's a Boolean to our faults. What I can do using cell is writing an expression like this. For example, in line one, I can say like, hey, I want to validate that the event.name is CubeCon and that the event location is Chicago. And then I can target, for example, the first element of my attendees list and say like, hey, do they have a coffee and do they have a funny hat? It is pretty simple. It looks like Python or JavaScript, basically, but it's not a programming language. It is only an expression language. It means that you can only validate data with it. And this is where it becomes very fun because we can use this kind of stuff in webhooks. If you want to learn more about cell, because we're going to stop here and start talking about webhooks again, please watch Jobit Stock last year or the year before about webhook fatigue introducing cell into the expression language features. And yeah, I really recommend that. Yeah, so now we're going to talk about how we can use cell in some new features that are coming out in Kubernetes that we worked on with the community to help make admission webhooks more usable and safer. So there's validating admission policies that are new beta API as of 128. And these are cell based policies. So rather than API server needing to call out to a webhook, it can just use cell to do validations itself. And they're beta in 129. And we have mutating admission policies planned for the future. Match conditions are new beta feature as of 128. And they allow for flexible selection of your rules and the scope of your webhook. You can now with cell define what your webhooks should and should not be targeting. And these are beta feature and they're going to be on validating admission policies and they're going to be on admission policies and admission webhooks. And they're beta, but they're on by default. So you can start using those right now as of 128. Here's an example of a validating admission policy. It has a lot of similar fields. It looks a lot like a validating admission webhook. That's by design. One of the things that you're going to see is different, though, is the validations instead of telling it a service to call out to, right? And there I have a simple cell expression that says for deployments, I want all my replicas to be less than or equal to five. And then you use a policy binding to apply that to, in this example, a certain namespace and tell it to do a deny on failure. This allows your API server to not take an external dependency and it's an in-process alternative, right? So the benefits here are less latency, like we said, cell is really fast and it doesn't have to make an external network call. There's no external dependency. There's a lot less overhead in setting this up as well. If you were to make a validating admission webhook, you would need to write some code, containerize that code, deploy a pod. The code would have to do TLS, HTTP request handling, a bunch of serialization and deserialization just to do the simple replicas less than or equal to five, right? And the other thing is you get immediate feedback. If you do use a validated admission policy and you do something that breaks your cluster, it will break it immediately and consistently, whereas if you have a validated admission webhook, it may be running good for a year and you might be fine and then all of a sudden there's a bug in your controller and everything's down. Match conditions are the other example we talked about. So this is where you can use fine-grained conditions with cell to define the scope. These are only for negative selection and you can have a list of them and it's an AND operation. So if one is false, then your webhook will not be called and invoked. So now we're going to go back to our broken cluster and fix it with the new cell features. So if we go back to our pods that are not running, yeah, so there's the deployment that should be gone, the pod that's not scheduled and the deployment that is not available, right? So we're going to go back to our webhook and we're going to take the same webhook and just apply this match condition and this match condition is going to say if it's coordinationk8s.io and the resources leases, we're going to ignore it with this not. So now if we apply this, our cluster should be back to a good state and our pod is now scheduled. The scheduler comes back a little bit quicker and we should see a cleanup of our deployment and our new deployment coming in and you can see that everything's working, the endpoints on this deployment have their IPs now, the old ones are gone, we're in a good state. Okay, so that's great, we were able to protect against, we were able to make one webhook safe, right? Now I'm going to transition, do a quick demo, validating admission policies and then I'm going to wrap it all together. So here's a very simple validating admission policy, very similar to our example but we're going to say it has to have greater than five replicas. And we're going to start by just showing the way deny works. So if we go and we apply this and we go and we try to apply the deployment, it'll say hey this isn't allowed and it'll give you the expression that is failing, right? There's a lot of other interesting uses here, so for example you can do a warn and audit. So say I don't want to actually block all the operations but I want to tell the user, hey you shouldn't be doing this and I want my cluster admins to have an audit of everywhere this is being done. That's also supported, so you can now update this and now when I try to create the deployment, it'll go through but I'll get a warning and there will be an audit log that we can go back and look at and see that hey this is failing validation and it's still being created maybe then you go reach out to your application team or something and talk to them about why they're doing something against the policy. Now that's all fine but my cluster still isn't safe, right? I protected against one webhook. Now how can I protect against all webhooks? Well we can do that with validating admission policies. So here I have a more useful validating admission policy maybe and it's going to target admission registration and it's going to look at validating webhook configurations and mutating webhook configurations. And another feature is you can have the special message, right? So with this message this is I'll walk you through this expression but this is a lot. If you got a warning with that giant expression it may not be clear to users on your cluster. So you can have a special message and you can maybe link that to some documentation that explains why and what. And then we're going to have this cell expression and this one's a bit more complicated so I'll walk us through it. First we're going to say object.webhooks.all. So that's an iteration operation in cell and as you can see the webhook configuration you can define multiple webhooks. So we want to make sure for all of those webhooks that all of them are safe. So what this is going to do is it's going to say for all webhooks and then this is the iterator like if you say like for item and items or for X and items I'm calling it webhook because that makes sense but for all webhooks every single item in that list it has to have match conditions and then for those match conditions we want to loop through those as well and make sure that there's at least one. So for all webhooks we want one match condition that has this expression to block leases. And then we're going to run a deny there. So if I go and apply that and we go to this webhook now if I take off the match conditions I get a denial and I get a useful message. And then if I add these match conditions back on it goes through. So now suppose I want I want my teams using this cluster I'm a cluster admin I want them to be able to use webhooks but I want to make sure they're safe and they can't take down my components. Now we have a mechanism for doing that. Admission policies are able to target admission webhooks right. Admission webhooks cannot target other admission webhooks because then you can get into a state where you're locked out of your cluster. So that's that's blocked. But admission policies can target admission webhooks. However they cannot target other admission policies but due to the reasons we talked about this is still a much better state because admission policies are much safer it's not an external dependency you get immediate feedback. If it works today it's going to work tomorrow. So yeah that that shows how we were able to safeguard this cluster. Thank you Igor. That's pretty cool to see live the the demo gods are happy today so can't really complain. So now that Igor showed you how to use match conditions. If you remark like observed at some point he's using request something like hey what are the objects you can use because on the runtime of the API server you have few fields that you can access not always request. This is from the Kubernetes documentation and I want a better place than Kubernetes documentation to read about a few things and here you have for example object all objects if you make an update operation you always have access to the new and the old one. You can have access to the requests the authorizer and the authorizer request resource for example if you access who is the user that is making this request is the cube system is a cube that you can have you can find those information information for example in in the authorizer. Anyways there is also discussions about adding namespace. I know you can find namespace in objects if you do object that metadata that namespace you can find what is the name of the namespace but there is also some work happening in upstream right now trying to bring the namespace objects as a full thing for example I want to see the annotations of the namespace and those few things that you might be able to access in one point 30 but I'm not sure it's going to make it to upstream or not there's a lot of conversation about whether we should do it or not but yeah maybe next year we're going to talk about this. So for example you can see here the third element the third element here was a request if you want to know what is what is really inside the request this is called straight from Kubernetes upstream Admission V1 types that go you can see all the fields you can access in the request. Igor was using I think user and the name of the object not the kind sorry it was like the kind to for example target the leases but those are all the objects you can access from the admission request. Yeah so in conclusion admission webhooks are super powerful we've had them for a long time but they also come with a set of dangers but now we have new cell features that make it easier to apply policies under your cluster in a safer way and please if you haven't go out try these features you'll need to enable the beta APIs for validating admission policy but you can use the match conditions today on 128 plus clusters on admission webhooks and please go try them out and provide feedback are there new features you want access to different fields any issues you run into provide feedback so we can make it a better experience and get these new features to GA. Thank you. Thank you all if you have any questions there are microphones there and there if any has some go in once. So if I do break my cluster because I have webhooks that don't let me do anything how do I unbreak it? Yeah so you just delete the webhooks so I touched on that a little bit but there is you're not able to target other webhooks with a webhook in order to avoid fully locking yourself out so you can either delete the webhook or I guess depending on what the webhook is blocking fix the admission controller but if I was in a situation with a broken cluster I would just nuke the webhook. Any potential performance problems? Yeah so we've built in some monitoring so you can tell how long these expressions are taking to evaluate there is a potential because as you saw we could do loops so you could like loop and loop and loop and loop and cause issues that way there is a cutoff in how much processing time an entire webhook is allowed to use and each match condition and how much compile time it's allowed to do so it actually compiles when it's created and then there's compilation and evaluation with cell so there's limits on all of those and if you hit the limit then the failure policy will be invoked similar to like if you can't talk to an admission controller. I can add to that we've done some benchmarks regarding Keverno cell Rego and pure go code and we were like fascinated of how cell is very close to go performance it's very very fast it's so far from Keverno and Rego so if you're worried about the performance of cell itself on like dealing with very very huge objects it's not gonna be really a problem it's really as close as to pure go code or compiled go binaries. You said we shouldn't use wildcards but I kind of have to you because I have six hundred namespaces and I don't know how to include 599 of them and there's no negative I can't say star minus cube system right so this is the tool that would allow us to do that yes exactly okay yeah so you can use the match conditions and say if namespace is not cube system then you're good bypass the webhook thank you you can basically access any field in your object so namespace, annotations conditions as well if you want to re-target only bots with specific conditions and like specific messages in those conditions it is possible to do any more questions? go in once go in twice and thank you so much for joining today I really appreciate it it's very cool