 So my name is Roman, this is James, and we both work at Cruise. Cruise is a self-driving car technology company that works on a lot of different things, but one of the main ones is trying to make a car drive by itself. So today we're going to talk to you about our journey through authentication and authorization at Cruise. So James. All right, thanks Roman. So just to sort of set the stage for the problem space that we'll be talking about here, I'll just start by saying that Cruise currently has about 2,000 employees. Of those employees, approximately 1,500 of them are engineers, and we've got about 150 services running at any given time. The majority of our services are running in Kubernetes with a mix of both microservice and monolith architectures that span multiple clusters, and most of the traffic that's passed around is HTTP-based. As for storing and managing secrets, we use HashiCorp Vault, and we use a typical cloud-based identity provider for managing employee and contractor identity. And so when talking about Authent and AuthZ at Cruise, we really have three different types of callers or clients that we have to take into consideration, and of those callers are browser clients, CLI clients, and service to service or application clients. And so I'll just start at the beginning. This is more of handling Authent and AuthZ at Cruise, to be honest. In the early days, it was sort of like the wild west, and this was really more, our talk is called crawl, walk, run. I'll say this is more of the embryonic stage, but it's worth mentioning, nonetheless. So in this early stage, we had, for example, engineers that are building services, for example, on top of our PaaS GKE clusters, and then in terms of how they handled authentication and authorization, they're basically on their own. The security team may have implemented policies or standards, which required service owners to implement these controls, but we didn't really have a whole lot in the way of tooling or standardization. So I'll affectionately refer to these as the bad old days. And despite my initial impression, the span sounds absolutely nothing like kiss, but nonetheless. I'll say that at this early stage, we had authentication and authorization sprawl. Teams were handling Authent and AuthZ entirely differently. For example, one might be doing OIDC with IDP1, and another might be doing OIDC or SAML with a different identity provider altogether. We also had naive implementations using static or shared credentials, and there may have been one or two services that had no authentication at all. And so we realized that that was problematic, and we had to start somewhere to fix it, and that's where we reached the crawl phase. In the crawl phase, the first thing we did was we just built a SAML authenticating proxy sidecar and just sort of hoped that people would use it. In this case, the service owner would add the sidecar to their pod, and then every HTTP request that hits the service will eventually, or we would enforce authentication via a SAML flow before proxying the request. And so to give you an idea of what that might look like, we'll take a look at this diagram. And we'll actually use a similar diagram throughout the rest of the talk as we introduce new systems and concepts. And by the way, this person isn't praying, they're crawling, but it was the best crawl icon I could find. But in this case, we have these different types of clients that I mentioned earlier. They're making requests from outside of our cluster, and they come in and pass through this nginx ingress gateway, and eventually we'll end up hitting the service. In the case of this early stage where we're using a SAML authenticating proxy, once the request would reach the service, the authenticating proxy would check to see if the user is authenticated, and if not, just bounce them over to the identity provider to complete authentication. And then once the user is logged in, the SAML sidecar would set an HTTP header that contained the subject, which was a user application, and then it would be up to the service owner to figure out what to do with that in terms of authorization. This also required a fair deal of setup for our users. First of all, they needed to set up the sidecar. They also needed to talk to IT to get a SAML integration setup with our identity provider or to have groups created, for example. And another problem here is that SAML as a protocol for the most part isn't really friendly to non-browser clients. So when people needed to provide API access, for example, they just passed a bypass header to circumvent authentication entirely, which obviously isn't good. And so obviously there were a lot of problems at this crawl stage. As I mentioned earlier, creating these authentication integrations was still a request-driven process that involved users reaching out to IT or an opening a ticket with them to have that done. Also, creating and modifying groups in our identity provider was still like a request-driven process, as well as assigning access to these authentication integrations. So now I'll talk about the next stage, which I'll say is the walk phase. And since I mentioned that we realized there were a lot of problems in the crawl stage, to solve them, we set out to do a couple of things. And starting by choosing an authentication and authorization standard. So for humans, like human to service authentication, we kind of settled on OIDC, which is the open ID connect, which is like an authentication extension to OAuth 2. We also built a custom entitlement-based authorization broker that we'll call C-Auth-Z. We also set out to build self-service tools to promote use of this standard. One of those such tools is something called identity orchestration platform, or IOP. And IOP has a lot of functionality, but one thing in particular that's worth noting here is that rather than users having to create a SAML or OIDC integration directly with an identity provider or ask IT to do it, we built a web app that allowed them to do that as a self-service option. We also wanted to build frameworks and libraries to foster implementation of these standards. This included things like opinionated auth-in and auth-Z libraries and commonly used languages at cruise. Another auth proxy that used OIDC rather than SAML and also was able to validate tokens coming from C-Auth-Z, as well as an authentication CLI tool that was used for retrieving and storing non-browser client credentials for humans. We also sort of settled on offline tokens or JWTs that were minted by C-Auth-Z. And for those that aren't already familiar with JWTs or Jots, they're basically usually the payload of a JSON web signature, which is like a JSON blob containing structured data or claims that can be cryptographically verified. Okay, so, oh boy, now we're at this diagram. There's a lot going on, and I can also tell that Roman made it in my case because he worked for Microsoft for a long time and he put the DLL icon on here, so none less. Moving on, so yeah, we'll go ahead and break it down. So we'll start with C-Auth-Z, which I mentioned earlier, and I also said that it has to mint tokens, but how does this authorization broker mint these tokens and what do they contain? So firstly, C-Auth-Z would need to know about or be able to reason about what users and groups exist at crews and what groups users are members of. Once the C-Auth-Z system is able to talk to the IDP in order to figure out what users and groups are present, it's then able, or service owners at that point are able to create applications in this system, and that allows this authorization broker to assign or allow service owners, rather, the ability to assign various permissions, I mentioned entitlements earlier. They can assign permissions to various users groups or even other applications, so it's sort of like a record of internal applications at crews that is used for authorization. With this information, C-Auth-Z is able to create these tokens, which contain claims that can be verified. Speaking of verification, the tokens themselves are cryptographically signed, like I mentioned, but how do you go about verifying them? In this case, we gave service owners two options, basically. In one case, they could use a library, like the opinionated library that I mentioned earlier, to validate tokens as well as to handle the OIDC authentication with our identity provider. The other option is the sidecar, which also does OIDC authentication and is able to validate these tokens directly from C-Auth-Z. The reason why some service owners may choose to use a library over a sidecar is, for example, if they needed more finely-grained authorization controls, for example, to be able to authorize based on application state, which is something that the sidecar would have no context of and only be able to authorize things based on the HTTP request or the path. Also, astute observers may notice that, hey, what about OIDC? How do you get a C-Auth-Z token then? In this case, we don't actually get a C-Auth-Z, like a JWT token. What happens in that case is, after the user authenticates, we do a real-time API call with C-Auth-Z to see what actions or operations are entitled to perform. Now, I'm going to pass it off to Roman to actually talk about some of the problems with this approach and then how we built the next stage of it to address them. All right. So, James talked about the walk stage, which was definitely an improvement over the crawl stage. No doubt about that, but it's not some problems that we didn't like so much, especially in security. So, the first most obvious problem is there's no enforcement of any of this stuff. Like, yeah, we're providing some libraries and some sidecars and people can use them, but they don't have to. We recommend that they do. But coming from kind of a security side, we definitely don't want to be in a position where this is an optional thing. So, if you're going to choose the library that, frankly, a lot of folks have, you need to change your code. You need to pull in this library. You need to change a bunch of code. You need to call it. You need to put in the right path. You need to also, as this library is changing, as we fix it up, if I'm bugged, fix them, you have to go and update it. So, there's a whole cycle there. And James mentioned that your permissions or entitlements are actually encoded into this JWT, which is great. But as these applications evolve, they become bigger. And as they become bigger, a user has more permissions to various aspects of this application. And all those permissions need to be shoehorned into the token. So, now this token is growing and growing. And these tokens, JWTs, are sent as headers, right? And then that means the header size is growing. And as that grows more and more, right, then you'll hit the dreaded, I think it's HTTP 431 or something like that. For 29, sorry. And, you know, that's really hard to debug, right? Because depending on the state of your headers at any given time, this will fail or succeed. So, it's just not great. You know, for us in security, right, that's great that we're providing all these libraries and sidecars for folks. But, you know, maintaining them is really hard. You know, at the end of the day, we're kind of a security team. We're not just like a development team. But we have to not only support these libraries, but support them in like four or five different languages, right? With their own, like weird gotchas, without really a common core. And, you know, as these languages change, you know, the libraries that they require change, and we have to like figure all that out. And then, you know, build them for every different, like every language has its own package manager. So, it's pretty difficult. But even then, you know, when we build it, let's say we figure all that out, and we kind of did for a while. To get folks to use the latest version is very difficult, as, you know, you might imagine. So, you know, what ends up happening to us a lot of the time is, you know, we'll release a new version. And, you know, maybe it'll have a bug in it. But nobody uses it until like four or five months pass. And then somebody starts using it, and they're like, oh my god, everything's broken. And we have to remember what we did five months ago, which, you know, we're really bad at. And, you know, lastly, one of the other big problems of this whole solution is that all the policies, all the, you know, this user has access to these things, that all lives like deep, deep inside the application, you know. So, in order to like figure that out, you would have to go to the repo of the application, understand the code, figure out the language they're using, you know, dig around, figure out the call paths. Super difficult to do, especially in a scenario where security team maybe needs to audit, you know, hundreds of these, right, not really scalable. And lastly, you know, kind of a high level problem here is that we, we be in the security team in this case, you know, we don't really control both ends of this equation. What I mean by that is we control, you know, we have the broker that's minting the tokens, but we don't control the code that's validating them. And the reason that's a problem is that it doesn't really allow us to iterate and make this end-to-end experience better by using, you know, maybe different tokens, using maybe tokens that don't have some of these limitations. The only way we could do that is then we have to go to every service and say, hey, you need a bunch of code changes. You need a bunch of code changes, right? That's, again, not scalable. So, to move on to the run phase, well, I know we initially said run phase, but it's more of a speed walking phase for us. And we'll presumptuous to say that we're at the run phase. But there's a bunch of things that we kind of want to improve here, right? So, the first thing is we want to be able to move authentication and authorization to an opt-out as opposed to an opt-in. What that means, kind of in simple terms, is that we want to just make this secure by default, so that if you have a service, you deploy a service as a developer, you know, it has authentication, it has authorization, right? Just from the get-go. And then for some reason, you want to do something custom that maybe you don't want our authentication authorization. That's cool too, but then you have to opt out of that, right? Whereas, you know, kind of in this state, you have to specifically add authentication authorization to your application. So, the next thing is, you know, we want to, you notice we built a lot of libraries and sidecars and stuff, you know, which is great, you know, give us something to do for a while. But ideally, you know, we don't really want to support all that stuff going forward, as I mentioned, all the problems around it. So, we want to, you know, participate in this amazing open-source ecosystem and, you know, both use the fruits of other people's labor, obviously, but also, you know, contribute our own, right? So that we submit, when we solve solutions for our company, we're actually solving it for others as well. And, you know, also we want to move this policy code from just being very deep within the application. We want to move it out. Move it somewhere where it's a standardized place, where security teams can review it, other people can review it, and, you know, verify it and make sure it's okay without having to dive into the application itself. And, you know, obviously, right, as if that wasn't tough enough, we're going to do all this without going to every service and saying, you've got to change a bunch of code, right? Because, you know, frankly, anybody got time for that. So, next diagram, right? So, we have a bunch of new components here. I'll go through them. Don't worry. So, let's start here, right? So, the first move that we wanted to make is from using the NGINX ingress to using an Istir ingress. Now, why would we want to do that? Well, the Istir ingress, it is actually powered by Envoy. And Envoy has a lot of cool things outside of this. But one cool thing that we wanted to leverage was its extensibility mechanisms. It's got a bunch of those. One of them is called external authorization. And what external authorization does is it basically lets you define another service that Envoy will call for every request. And it will basically let that service either approve the request or deny the request. So, that's very helpful. But in addition, if you deny the request, you can actually mutate it as well, right? So, we leverage that by having another service live alongside the ingress and hooking into the ingress via this external authorization mechanism that I described. And for every request, this other service called CAPI basically takes a request and says, hey, is this request authenticated? And if it's not, it denies the request, but it actually denies it by overriding the response and sends the user a 302 to the IDP. Now, the user completes the IDP flow, logs in. And because when we do that 302, we actually set a redirect URL for the IDP. The user gets bounced all the way back to where they came from to their original request. This time, CAPI looks at the request and says, okay, you're already authenticated with the IDP. I'm happy with that. I'm going to actually just set a cookie just so you don't have to go through this flow again. And then it lets the request through, right? So what this allows us to ensure is that any request that is coming from outside the cluster to inside the cluster has been authenticated, right? Just there's no way to do it otherwise, right? Doesn't matter which service you deploy. So the request has been authenticated, that's great. But, you know, we also want authorization, obviously. So how do we do that? Well, we decided to move to a different sidecar. You'll notice they're different from the one on the right. And this is an Istio proxy sidecar, right? Because now we're living in the Istio ecosystem. Now, Istio proxy, what's really cool about the Istio proxy sidecar is, guess what, it's also envoy. It's the same envoy. But this time, instead of living kind of globally in the cluster, it's living inside of the context of the service itself, basically, but doing the same thing, right? So it takes every request and proxies it through. Now, because it's envoy, guess what extensibility mechanism it has, external authorization, right? So let's use that again. But this time, slightly more appropriately by actually allowing or disallowing the request, right? So we need something else to hook into this envoy, right? To allow or disallow the request. We have CAPI globally, right? So what are we going to do here? Well, there's another really cool project called Open Policy Agent, right? So Open Policy Agent, what it allows you to do is it allows you to hook it up to your sidecar via external authorization. And then for every request, Open Policy Agent basically lets you write a policy in this language called Rego. And basically, the policy, if you think about it as implementing a function that always returns true or false, right? So you can have kind of a big flow of things that you can do. But at the end of the day, you're returning true or false. You're either letting the request through or you're not. And in our case, what we did was we actually used the same exact policy for every service in our cluster. We were able to do that because Open Policy Agent also lets this policy be augmented with additional data. You can think of it as like parameter store function, right? So the actual implementation of the Rego policy, really all it does is it says, okay, like, give me the routes that, give me the route that the request is trying to access, like the path, right? Let's say the request is trying to access my application slash admin. So it says, okay, you're trying to access slash admin. So it gets stuff from the request. And then it says, okay, this is additional data that it has specific to the application says to access slash admin, which permissions do you need? So it looks at that information. And then it looks at the request again to look at the incoming token, the JWT, right? That's attached as a header, as we said. And says, okay, does this token have those permissions, right? And that operation of matching the routes and their permissions to the token permissions is the same across all applications, right? The only thing that's different is that those two lists, right? The lists of, you know, which routes need which permissions and, you know, the actual token itself, right? So you might be wondering, okay, that's cool and everything. But, you know, as I mentioned, right, like the open policy agent actually needs to have some data that's specific to every application, like these extra files, right? So where does that come from? Well, the other cool thing that it has is it supports something called a bundle API. So bundle API is really just a polling mechanism using HTTP that it can pull various cloud providers. So it already supports polling, like a three buckets, you know, GCS buckets, like all that kind of stuff. So we leverage that. So we basically tell it to pull a bucket that's specific to the application. And in its polling mechanism, it already supports caching as well, right? So it knows just uses basic HTTP caching using intags. So it knows if there's a change. And if there is, it downloads it, loads the new policy, the new Brego, in our case, it's always the same, just loads the additional files, and it's good to go. Now, how does this bucket get populated? Well, remember, James talked about CODZ having all this context, right? Having the users, their groups, the various permissions that users are allowed to, you know, action on a given application. Well, all we did was we took CODZ, right? We serialize all that data on a regular cadence, and we write it to each application's bucket, basically. So, you know, that's great and all, but, you know, it can't possibly be that simple. So it wasn't, and it isn't. So, you know, let's kind of go step by step to see what the challenges were. So kind of how we dealt with some of them. So basically, moving from NGINX to Envoy, you know, that pretty much involves, for the most part, taking an ingress resource as defined in each service and converting it to kind of a matching virtual service resource. They're kind of analogous, not exactly, but kind of. And Istio has a utility that lets you kind of convert from one to the other, but it doesn't always work, you know, depending on the complexity of your ingress resource and the routing rules that you set up. So, you know, that required some hand-holding and kind of doing some things by hand. One thing that I want to mention that we did there, which was a good idea, and I think it helped folks, is we actually ran both side-by-side for every cluster. The way we decided, you know, which, like when you do a request to a service, which ingress to use is just simply by the DNS. So the DNS of every application was mapped with CNAME to the DNS of whichever ingress that we're using. So in the beginning, they were all CNAME to the NGINX ingress. And as folks, you know, made changes, right, they could actually create a new DNS that CNAME to the Envoy ingress, to the Istio ingress, and then just test that out on their own, you know, mucking around with a virtual service. But while they're doing all that, nobody, you know, the ingress, the NGINX ingress isn't looking at that data, so it doesn't break anything. So they can iterate, they can make sure it works. And then when they're happy with it, they just switch over the DNS to the Envoy ingress, and off they go. So that's kind of how we are handling the NGINX Envoy ingress migration. So with CAPI, you know, we obviously want to enable it globally in the whole cluster, but you know, we got to start with still an opt-in model, make sure, you know, things don't break. And the way we did that is we basically had people add an annotation in the virtual service. And then we have CAPI basically scanning the cluster and saying, hey, if I see a service with this annotation, I'm going to authenticate traffic routing to it. If I don't see that annotation, I'm just going to step out of the way. And this is kind of like a solution for the migration. Now that code is going to get completely removed, you know, once everybody's migrated. So you might be wondering, like a lot of these applications, you know, as services, as James said, you know, are using these libraries, these legacy site cars, legacy libraries, maybe doing things on their own. So how do we handle that? Well, the good news is that's totally cool. Like we can have multiple, we can do authentication multiple times, and that still achieves our goals. So for example, what can happen is you'll get a request into the cluster. CAPI will do that whole magic of authenticating the request. The request will go to the actual service, and the service will authenticate it again, right? Sorry. Authenticate it again. Security people don't install their updates. Anyways, but the good news is that our IDP already has a session for that user. And so really, it'll just be an additional redirect. So from a user perspective, it's not a perceivable difference, even though authentication is technically happening twice, right? And that like kind of latency of that additional redirect is only present while you have this two authentication mechanism. So once that's removed, you know, you don't have to do that. But the cool thing is, you know, we can still enable it and say the entire cluster has authentication. Some folks have it twice, but none have it zero times, right? So that's the goal. And then we can also integrate with some of these libraries and site cars that we provided previously because, you know, we do own them. So we can, you know, add some code there to make CAPI kind of more intelligently reason about some of these services. Yes, it requires folks to upgrade to the latest versions, and I mentioned some of the problems with that, but that's kind of another option. So moving on to enable the STO proxy. STO makes it relatively simple. You just need to set this annotation and STO proxy will be injected into the pod. You can set this annotation at, you know, the namespace level, the pod level, a bunch of different levels. Once you have that, you can actually start working with the OPA agent, the open policy agent, and that's again opt-in. So you set this OPA agent enabled annotation, and then you get this additional site car. So now we're up to two. I'm going to see how many site cars we can shove into every pod. And as I mentioned, you know, there's a single Rego implementation of the policy in every, for every service. And really that's doing what we'd like to call stateless authorization. So basically it's just looking at the request, just looking at the token, just looking at the local data that it has to decide if that request should be authorized or not. Now, if a service needs to do finer grained authorization, so for example, it's maybe has a matching database that it needs to look up a user and kind of augment this authorization data, it can still do that, right? So the idea is that, you know, the OPA agent does one authorization step, right? And then it can be authorized further by the application itself if they so choose to do that, right? So again, you're doing kind of authorization twice, but one is coarse grain, one is fine grain. And then lastly, our CLZ broker, right, needs some new UI so that you can go and define some of these policies. You can define, you know, different routes, as I mentioned, like you have an admin route, and these are the permissions that are required to access it. And, you know, obviously, folks don't just want to use the UI, they want to use the CI, you know, the current CI pipeline to deploy some of this stuff. So we also have that as well for them. And then lastly, you know, as I mentioned, we want to in the background serialize all this information, including the group memberships and push all that to those buckets that then the OPA agents can read. So kind of, what have we achieved here at the end of the day? Well, I think we can finally make this statement that all services in our cluster have authentication authorization. That's not something we've been able to say before. Before it was kind of like, maybe they do, maybe they don't. Authorization policy now does not live in the depths of every service. It's actually lives in our broker, right? That can be reviewed, can be exported, or it can be audited centrally, right? We're no longer using, you know, custom tools that we've built and have to support, right? We're using open source tooling. And the best part of this is actually, you know, we're not proxying the traffic ourselves, right? We're using Envoy, which is, you know, kind of a battle tested piece of software to do all the proxying and traffic management and all the cool stuff that it does. We're just kind of hooking into it to provide, to augment some of its capabilities a little bit, right? But we're not doing it ourselves. And best of all, you know, now we kind of control both the minting part of this equation, right? The creation of these tokens and also the validation, right? And all the way through the entire channel. This means that, you know, if tomorrow we like say, hey, some new standard comes out, we don't want to use JWTs anymore, or we want to sign them differently or something, right? We can actually change all that without every service having to modify a bunch of code. So that's pretty cool. But there's more, right? There's more, that's the image I should use. So there's more that we can do here. We, because we have these Istio sidecars, we can actually do transparent MTLS between services, right? Why can we do that all of a sudden? Well, remember that those Istio proxy sidecars are doing both ingress and egress filtering for each service. So now, really, you can have these two sidecars establish an MTLS channel between themselves. And so whenever an application wants to talk, a service wants to talk to another service, it actually funnels through those sidecars, right? So it goes egresses out of its internal sidecar and ingresses into the destination services sidecar. And that's even cooler because now you can actually have MTLS between the ingress itself and the service, right? So that channel is usually not secured, right? So like, for example, if the ingress were to add any headers, right? As part of that request, technically speaking, you can't trust those headers, right? Because that channel is not secured. But if you have MTLS between those two endpoints, those headers can now be trusted. Also, MTLS, you know, brings you certificates, certificates usually bringing you some form of identity that can be used here. And, you know, we have that now. And so now instead of using something like API keys for services to authenticate and authorize themselves to, like, say the COG Z broker or other other systems, they actually can use their own service identity that is actually rooted in the cloud provider's identity, right? In terms of the rego, you know, we've had, I'll give you one example of a request that we had. So, you know, somebody wanted to authorize based on the method of the HTTP request. So as I mentioned, you know, you can only right now authorize based on the route, right? So if you slash admin, right? Sorry. But if you wanted to say, hey, you can access this route, but only as a get and this other route only if it's a post, you know, that's not possible in our the rego that everybody uses today, right? But it'd be really cool. It'd be a trivial to add this functionality, right? So we want to move to a world where this rego is actually can be modified by services service owners themselves to fit their needs. But we don't really want to just unleash rego on them, as is very complicated. So what we're thinking is we want to provide some libraries, as well as kind of like a CI pipeline with some linting validation and various, you know, approvals and various other things, so that when they make a change, you know, we can have some confidence that it is doing what they're intending that it would do. So, you know, if this is interesting to you, come join us. We're hiring. Yeah, we also work on self driving cars. Okay. And so I'd like to a special thanks to Natesh, the top there that he couldn't be here today. And it's also actually his birthday today. So happy birthday, Natesh. Cool. All right. Yeah. So the question was, how do you do service-to-service authentication and authorization, I guess, and do we use JWTs or MTLS? So right now, we're still using JWTs, because we're not fully in the STO model of having MTLS available. Now, we do when we do have MTLS available, what we're thinking is we would use MTLS is again, use the same core screen fine grained method, right? So use core screened authorization to say like, Hey, this service can maybe talk to these three other services, but nothing else. But then layer the JWT on top as the user identity that's actually making that request. So kind of use both. Yeah. Yeah, that's a really good point. You know, right now, we're just simply scale it and use Kubernetes kind of to handle that, you know, if it blows up, we have more instances of it available. And we also have multiple clusters. So we have CAPI kind of one per cluster. So we don't have one like globally, right? So one is kind of matched up within Istio ingress and you have, you know, one Istio ingress per cluster. So I think we just have maybe more clusters in that scenario. Maybe you folks have less. But yeah, that's possibly something that we'll have to consider in the future for sure. That's a good question. It was sort of the predate in both of us, but yeah, I think sort of like the idea was like to Oh, sorry. Yeah. So the question was basically that CLZ broker, you know, why did we decide to build it and why, you know, we didn't use something off the shelf? Yeah. So yeah, I think it did like predate both Roman and I at cruise, but I think like it was originally something that was built by like a product team for things to do with with like our product and the car and sort of like we inherited it because it was responsible for like dealing with, you know, making authorization decisions. I think it also like provided, you know, quite a bit of flexibility in terms of giving these like service owners that are like registering their applications with this CLZ system, the ability to like define like, I mentioned the entitlements. So basically like being able to express like what operations are able to be performed by different human and non human subjects in like a really flexible way. I don't really know of anything like there are a handful of other, like I guess maybe my follow up question would be like, like what alternatives are there? Like we did also like at a later point kind of an investigate using like OAuth for authorization, but it really isn't the same type of, you know, it didn't really work very well for us in terms of, you know, it's like handling like a delegated authorization. But yeah, I don't know, we'll go, I think you in the back or first, yeah. So the question was if you have like automated testing, is that so like if you have like you're trying to do like load testing or something like that? Yeah, yeah. So we right now, honestly, we don't have a great solution for that right now. We use, but that's a problem that's existed in the past with basically all our other solutions as well. And so, you know, sometimes people will have like test accounts that they'll use in a different dev environment as opposed to using the production environment. And then have a cluster that's specific for that use case. If they need complete end to end, right? There's nothing like if they're already in the cluster, right, then they can do that without ever interacting with Cappy. We're at the end of this session. So if anyone has questions, you could bring them to the speakers. The live stream is going to end now. There are additional questions from our virtual attendees that will be placed in the Slack channel for y'all to answer. Thank you.