 All right, can everybody hear me? Thank you. All right, let's get started. So, hey everyone, this is the Sigoth deep dive for KubeCon NA 2023. So this year, we're gonna focus pretty heavily on cell and our usage of it in Sigoth. So I'm Mo from Microsoft. I'm one of the leads for Sigoth. I'm Jordan Liggett, also lead for Sigoth from Google. All right, so we're gonna mix it up this year. So normally we wait till the end to do this, but this year we're gonna do it at the beginning. So I see Anish, I see you in the audience there. Thank you very much for all your contributions this year. To hear, I know you're not here. Hopefully you're watching this. Thank you also. I don't think I saw James, maybe if he's here, but also thank you for your contributions. Nalek, I know you're not here, but hopefully you'll watch this and say yeah. So, Nebruun, I hope you were here. I know you're here at the conference, but thank you also for all the work you did. Rita, I know you're hopefully gonna watch this afterwards, and thank you for all your hard work. And Ting, I know you're not here, but we very much appreciate your contributions. I'm gonna go over some of the stuff we did generically in 129 before we sort of deep dives more into the cell stuff. So there was like six high level things we worked on. KMSV2 went GA and 129, so yay, it's over. It's over. Cluster trust bundle just landed in alpha, so I know I see some cert manager folks here. I hope that they will adopt it at some point. We made some great improvements to balance service account tokens that contain node references as alpha NV129. James helped with that. We've made a bunch of improvements to Authent and OCD that we'll talk about in detail, and we are continuing possibly the longest journey of any single thing, which is the reduction of secret space service account tokens. All right, so you've heard probably people in various sessions talking about cell. What is it? If you're unfamiliar with it, it stands for Common Expression Language. It's a non-touring complete language, and it's designed to be pretty intuitive to read and write with a heavy focus on being able to bound execution time, run quickly, run in performance critical paths, and be able to limit the amount of memory and CPU that's used. Here's just an example of what it might look like. It's pretty natural syntax. You sort of have dot accessors to fields. There's the ability to have macros or functions that the integrator provides to the expression so that the person who's writing the expression can make use of, but doesn't actually have the ability to define and do crazy dangerous things. So why did we settle on cell? We started this exploration a couple years ago on the CIG API machinery side, and we'd looked at a lot of different sort of extensibility languages, things like Rego and JavaScript and Lua and Wasm, and none of them really hit all the points that we needed. It had to be really, really lightweight. Some of those were not lightweight. We had to be able to ensure that the expressions had a bounded cost. So cell does that by limiting the complexity, but also allowing us to compute the cost at compile time, so we can just not allow expressions that would be too expensive. This was especially important for cell use in areas that were API facing, where sort of end users or maybe less trusted users were the ones providing the expressions. Because that was the first place where we were using cell in Kubernetes, it was critical to get that right. Another benefit of cell, we can do really, really deep type checking integration. So rather than just sort of accepting a blob and then at runtime, maybe it works, maybe it doesn't, we can actually do that checking at the beginning and tell you when you're starting up or loading the expressions, this is gonna work well, this isn't gonna work well. And then as we've used cell in more and more places in Kubernetes, we have the ability to customize what the expressions have access to in terms of variables and functions and macros and helpers for each specific use. So here's just some of the places that outside of sigott cell is being used. We won't go into them too deeply because these have been covered in the API machine or deep dive and I think at least one other talk from CC covered these. But I just wanna highlight one thing, which is all of these use cases are either directly or indirectly related to the usage of cell in REST APIs. Whereas the next two things we're gonna talk about are usage of cell in a file-based config that's past the API server. So this went alpha and v129. There's a lot of improvements that we made overall to how OIDC can be configured in this cap. But I wanna really narrow our focus on the aspects of this that are related to cell and how cell improve this. So there's a lot of advanced uses that you can imagine where you want a full-blown authentication webhook. But there can be significant problems when your webhook is down and just the maintenance costs of having such an integration point. So we wanted to see how far we could let people get by just having static configuration on the API server. That kind of means you have to make some hard choices. So you have to sort of define what kind of tokens you're gonna accept and what they mean. So if you had an opaque token, there's really nothing you could do with something like cell because there's no way to interpret that without making a database query or something. We're not gonna hand you network access in the cell environment. That kind of goes against what we wanna be able to do there. But if you limit yourself to something like JOTS, that means you have a mechanism for understanding how you can verify a payload, you have a mechanism for key distribution, which is basically OIDC style discovery. Now you've constrained the input in a way that cell can really help you. So let's walk through an example. So on the top left there, you can see the exploded payload from a JSON web token, right? It's just a JSON object. And you can kind of see the claims that are available. So as we talked about earlier, you can define a custom environment, right? So where you see the cell expression that claims variable maps to that object at the top that you can see, right? And now you can kind of start seeing what things you can do, right? So it just so happens that this user's IDP emits a string that happens to be a concatenation of basically their group membership. Previously, you would have to write some kind of authentication webbook to sort of explode this information out. Now you can just use cell functions, split that out. And in this example, we've purposely injected a static group into the list as well. You can imagine how you would use that, for example, if you want every user from your IDP to belong to a particular group, so that way you can assign policy to them, right? Another one of the things that the authentication config will let you do is just have as many of these authenticators as you want, right? So you can kind of mix and match how all these configs work out for you. So as we kind of walk through, the username is gonna just be statically suffixed with this external user. And we're also going to take the audience from the token and put it in an extra field, right? Now this means that if you had an admission webhook or admission cell policy, you can make decisions based off of that if you wanted to, right? So for example, if your IDP encodes in the token that the user logged in with 2FA, you can make decisions based off of that, right? You might not allow any action in CubeSystem from a user that didn't log in with 2FA or something like that, right? And we can keep going and take this further. It is unlikely that your corporate IDP has a concept of like Kubernetes system users, right? It probably does not have any concept of a Kubernetes service account or a Kubernetes KubeLit. And so a user info validation cell expression basically lets you say, I never expect that to happen. And you can sort of constrain that down, right? So this one is basically saying, hey, if for any reason I ever see a user that has a system colon prefix in their name, just flat out reject the authentication. This example doesn't cover it, but you can actually do the same exact thing with the claim payload as well. So you can say that, hey, if the user is not in like a particular group, they can't authenticate at all, right? So in Kubernetes, if you authenticate, you get a baseline level of access. Like for example, you can hit discovery and get all the metadata about the cluster, right? That's not great if you know that that user has no business being in that environment, right? So sometimes your IDP lets you configure these things. Other times the IDP could configure it, but you don't happen to have access to that. That's a different team. And you just wanna manage your Kubernetes environment. Now you can do that, no web books necessary. So that was the set of authentication configuration that we've added more flexibility to. In 129, we also worked on expanding what you could do with authorization config. So again, previously, the way you would configure authorization, you would just pass command line flags and say, here are the modes I want. I want node authorization and RBAC authorization and maybe webhook authorization. And then if you said you wanted a webhook authorizer, you had to give it a kube config file to tell it where to talk to the webhook authorizer. It's a lot of flags, not super intuitive. And a lot of people came to us and said, I've got one webhook authorization integration off to some IAM system. But I actually have a second one I wanna run locally to maybe do some deny rules or something. I wanna run two webhooks. And the system we had only lets you configure one. You start trying to add multiple versions of webhooks via command line flags. It just kind of becomes a mess. And so we added the ability in 129 to have a config file that lays out exactly what authorizers you want. And you can have more than one webhook authorizer. You can order them the way you want. When you do that, some questions come up, right? Webhooks tend to be slow. They tend to be less reliable than the authorizers that are built in. And so as soon as we start talking about running more than one of these, we have some questions, some goals. Can we limit the blaster radius of these webhooks? Can we, if we know that the local one we're gonna run that has some deny opinions, if we know that it only cares about requests shaped a certain way, can we avoid sending other requests to that webhook? Avoid paying the latency cost. Avoid paying the reliability cost. And then finally, some users want to host authorization webhooks on the cluster. And so right away, you've got a cycle problem. If the authorization webhook is gonna maybe make certain requests not be able to be authorized, what about the requests to make the pods to run the webhook that's making the authorization decisions, right? It's easy to get into cycles and problems like that. So in addition to just being able to express having multiple webhooks, we added the ability to exclude certain requests from going to webhooks based on something called match conditions. For those of you who have seen match conditions on admission webhooks that API machinery did, this is a very similar concept. We followed the same approach, but this is in a file-based config. So we're gonna demo that now. Let's see if the demo works. And while Jordan gets that going, just as a reminder, admission webhooks run on mutating requests, authorization webhooks run on every request. So it's an excellent way to break your cluster if you do it wrong. All right. So what I have here, I've got a little status dashboard showing three types of clients. On the top here, I've got a load balancer that's poking at my healthy endpoint. This thing may or may not be authenticated. It just wants to know, is this server good so I can include it in the load balancer? Then I've got a system controller. This is a CUBE system service account. It does really important things. If we disrupt it, bad things are gonna happen. This is sort of a stand-in for every system controller that exists in Kubernetes. And then down here, I've got my demo client. This is the thing that maybe I want to authorize in a different way or put some safeguards around. And so this is what the authorization configuration looks like in 129. So to start, I just have the node authorizer and the RBAC authorizer. Pretty standard setup. If I enable a webhook authorizer at the end, first of all, dynamic reload. Not in 129, but coming in 130. So that's pretty great. You can change your config without restarting your API server. Now I've got a webhook at the end. And so you notice a few things. First of all, it is now authorizing my demo client. So that's cool. You'll notice the latency is higher for the demo client, right? But not for the system client. So it's because these are being authorized by the RBAC rules. And so they don't even make it to the webhook. But I am paying a little bit of a latency cost on my demo client. That's okay. So other things that you see here, you can actually customize the timeout and customize the failure policy. Like if I hit an error with this webhook, what should happen? Previously, the timeout defaulted to 30 seconds, which is really bad. And the failure policy just continued on in the authorizer chain, which might not be so bad. But if you're wanting to put deny rules in, you actually probably want it to hard fail. All right, so now we've got a setup where, I've got my webhook at the end. That's fine. But what if we wanted to have some deny stuff? Maybe we don't want anything outside the system controllers messing with CubeSystem namespace. Well to do that, we have to put maybe another webhook, a local webhook that knows about deny stuff up at the top. So let's do that real quick. So I'm just gonna do this part to start. All right, so now I've got a deny webhook up at the top and you see right away, like I'm paying latency cost on everything now, okay? I'm actually double paying it on my demo client. So everything just got slower across the system. So that's not ideal. But that's not even the worst thing that could happen, like this is when the webhook is working well. If it's, what if I get errors from it, right? Okay, my deny webhook just went down and I've just completely toasted my cluster. The load balancer thinks it's unhealthy, the system controllers are broken, everything is broken. So what we really wanna do is scope down this webhook to only requests that we know we might want to deny. And so we're gonna add a few match conditions. So the first thing we're gonna do, we're only gonna pay attention to resource requests. We don't care about, this deny webhook doesn't care about non-resource requests like discovery or health Z or metrics, okay? So we made that decision. So right away, our load balancer isn't impacted at all if this webhook goes away. The second thing we wanna do, this is only, in our use case, we're only caring about protecting cube system, okay? So we're only gonna pay attention to requests inside cube system. Everything else doesn't even touch this webhook, okay? So now my system controllers are happy in other namespaces, that's good. We're not paying latency costs there, that's good. But we still have cube system broken, so that's obviously not good. And so the last thing we're gonna do is we're going to exempt system service accounts from even being intercepted by this webhook. And so once we do that, my load balancer is happy, my system service accounts are happy, my non-system service accounts are totally fine in other namespaces, right? The only thing that would be impacted by an error if this webhook was completely unavailable is the clients I was trying to deny anyway inside the namespace I was trying to deny anyway. And so that one's still hitting an error, but when we fix the webhook, and that is actually working properly, now it can get the forbidden error we want. But we've really, really scoped down the worst case scenario. That makes me so happy. When people say, I wanna run multiple webhooks, and I say, why would you put yourself in that position? With this, we actually have a tool to make that a reasonable thing to do. All right, thank you, demo gods. A couple notes on that. Automatic reload is not in 129. For 129, you just start up your server and just config you start with, that's what you get. But automatic reload will be in 130 before it goes to beta. All right. If you have questions about any of the cell stuff or any of the other stuff that we talked about or did this release or stuff that's coming up, I wanna let you know how to get in touch with us. That's a link to the community page for SIGAuth. We have bi-weekly meetings, we have mailing lists, we have Slack channel. We're very friendly, we promise. If you have questions or have ideas or want to help with progression of some of these things, please come join us, participate, give your feedback, give your thoughts. Yeah, so with that, we have some time. I wanted to open it up for questions about this or any of the other stuff we talked about. There's a microphone there, I think, if you wanna line up if you have questions. Thank you. Also, a lot of the stuff we talked about is alpha, so please go play with it and give us feedback because we can still fix it. Once it's beta, then we can fix it with permanent tech. The question was about what's the motivation for the restrictions on cell execution and resource consumption? It wasn't nearly as specific as specific hardware, it's just the knowledge that the person operating the Cube API server, providing the hardware, the resources for that, is not always the same person that's populating cell expressions via the API. And so, we needed to be able to prevent abuse, essentially. For something like what we talked about here, where it's actually config files for the API server, abuse is not really as much of a concern, but it is still a critical path execution. And so, where it might be acceptable to pay milliseconds of latency going out to a webhook, when you're talking about the conditions that are gonna be evaluated to avoid going down that path, we really want nanoseconds to microseconds. And so, that's where cell fits, for authenticating out to an authorization webhook. We don't have that planned right now, but we structured the config file so that there's a spot for it if we get there. So, the initial connection type, there are actually two connection types. There's run in cluster, so this is if you're running inside a pod and you're delegating back to the main Cube API server for people running extension API servers, the second connection type is here's my cube config file. So, the first goal was really to get parity with what we already had from the command lines, but we have a spot for additional connection information styles in the future. Yeah, I really wanted what you asked for, but there was a separate cap that had tried to do it a couple years back that didn't really make progress. And it was probably gonna prevent this cap from progressing if we tried to solve that problem at the same time. So, I listened to Jordan basically and said just... You try to be incremental and actually deliver some things and then build those out, yeah. Yes, whether you use HTTP one or HTTP two is not currently under the control of this config. The client that it builds will actually try to use HTTP two. So, it will try to keep connections open to it, but that's not part of the control there. Yeah, it? No, not at all. Yeah, the client that's constructed, when you load the config and say, I'm gonna be talking out to a webhook, it creates a client, that same client gets reused. This is just sort of putting conditions in front, so sometimes we don't even call the client. Yeah, now, the one caveat there is like when your config will change, then your clients are gonna get recycled. Yeah. But that doesn't happen. Thank you for going to the mic. I've got kind of a beginner cell question for you. Can you talk a little bit about how you guys like bind, it seems like cell fields are sort of dynamically generated on the context in which the cell is running. Can you talk a little bit about how you bind or like validate that like the field that you're operating on actually exists and like, or just maybe give me a guide on where to find resources for that. One, let me take the start of that. So I'm gonna back up, right? So it's situational specific, right? So if you look at, for example, the username expression where it's doing claims.username, we have absolutely no idea what's in your claims because your jot has no known schema to Kubernetes and there's no way for you to tell us what the schema is. So that is a runtime check and the runtime check will basically make sure that that cell expression returns a string, right? And if it doesn't, it's gonna error out. So if for some reason returned an integer or the claims.username wasn't there, it's gonna fail. But for example, in the user info validation on the right side, that user info object is a Kubernetes internal well-known schema object. So if you tried to type in user info.foo, your API server is not gonna start because it's like there's no foo field on that. So we basically try to do the absolute best we can with the schema that we know, but on things like arbitrary payloads, we can't help you there. Now, cell does have the question mark operator, so it has like optionals. So you can write cell expressions that can optionally pull in information depending on if you think it's gonna be there or not. So you can be as restrictive or as open as you want in those expressions, but go ahead, sorry. No, no, I was gonna say, thank you. There's two steps to using cell internally. There's a compile step, type checking step, and then there's an evaluation step. And so for things where we know the schema, the compile step will actually catch any errors where you're using a string where it's actually an end to using a field that doesn't exist. For dynamic things like the claims, like Mo mentioned, it's all run time. Thanks. Ready to go? Yes, yeah, that's correct. So the things that you have access to are the token, if you're doing token authentication, and then request attributes. So source IP is not included in request attributes yet, but if it's included in the token itself, then you could map that into extra information, and that would be visible to the authorizer. Yeah, so what you might want to do instead of saying that I'm gonna encode source IP into the token, you could instead encode the concept of internal network, or as I mentioned, something like Tuofay. Some claim that gives you the sense of okay, I trust this client in a different way than I normally do, so when they're gonna do, maybe they're gonna exec into a production pod. Sometimes you have to do that, but you still might want to, I see you laughing Jordan, sometimes you do. And so you could constrain it that way, but yes, that is totally possible. I saw a hand over here, yeah. No, no, no, no. Cell just evaluates expressions, and the variables that the expression has access to are entirely up to the integrator. So in this case, for the claim mappings, the inputs, we provide it a map of string to anything based on the claims in the token. For the validation rules, the input variable is a user object that has really well-known fields. So it's entirely up to us as we integrate cell into different aspects of the server what the variables are that it has access to, what the schema of those variables are. And we try really hard to make the schema match the same types you would be dealing with if you were using a webhook. So for the authorization rules, the object that you have access to is a subject access review schema. It's the same object you would be dealing with if we actually sent it out to your webhook for admission webhook conditions. The object that the cell expressions have access to is an admission review object. So it's the same object your webhook would see if we ended up sending it out to your webhook. We tried to make the schemas sort of a line between the cell expressions and the integration of that area. APM machinery has the ability to run cell expressions in validating a commission. So this is the 3488, and it's very generic. It basically is running the cell on an admission review. So you can do anything you want to do from the information in an admission object. So if you wanna talk meta, so cell for admission control has access to a function which lets you invoke the authorizer. And that authorizer you can ask it any question you want which then might flow into your webhook that we just configured, right? So yes, you can go as far as you want. Turtles all the way down. Yeah, just be careful, right? If you end up making like seven authorization calls. You can't, we don't let you. So this goes back to the cost. All right, right, right. The cost aspect. One of the great things about cell is we can be very specific in attributing cost to different parts of the expression. So when you're working with data, the cost is based on the function you're calling or the macro you're calling and the size of the data for something like the authorizer function that we make available to admissions cell expressions. We actually give that a really high cost. So you, I think you can basically only do two calls. Just because that could be going out into webhooks, like it's a pretty expensive call. So the cost is actually done at compile time. So when you are setting up your admission rules, if you tried to create a validating admission policy and your rules were too expensive, you wouldn't be allowed to create that API object. Yeah, so it'll just fail before you can create it. And then there are additional protections at runtime. So if something unexpected happens, we thought you fit within the cost parameters, but then at runtime you're actually going along. There's also like a context cancellation mechanism so we can keep the server from running away with cost. All right, thank you for coming. We'll hang out here for a few minutes. Appreciate it.