 So let's go ahead and get started. So I'm gonna be talking to you about that the start is just the scale that we deploy to at Uber, including with Spire, the environments that we work with, and then how we integrate with Spire server on various levels. After that, I'm gonna be going into how did we actually onboard our consumers, make that experience nice for them, make sure as much as possible, everyone's getting the identities that they need. So for the scale, we have about a dozen independent data centers with less than 10 Spire servers per data center. Each one has tens of thousands of hosts in it, and then a dedicated Spire agent on each. Every data center every day is performing tens of millions of X5 and 9 signings from Spire. And then today we have over 500 services onboarded to Spire, which is less than 10% of the internal services at Uber. For our environments, we have several in-house orchestrators for better or worse. We build a lot of things in-house. Our hosts are both on-prem and provided by multiple cloud providers. And then nearly all of our services are containerized and written in either Go or Java or Python. So to go over our server deployment and integration, it's already been gone over before by Andrew Harding and Evan Gilman, how Spire works in general. So we have the Spire server cluster in a data center. We have the Spire agents deployed on every single host, a testing with Spire server, syncing on what registrations they're responsible for. The Spire agents are baked into an enforced image fleet-wide to ensure that it's present. We have next to the Spire server a low-level registrant that we own. This is a service that creates Spire registrations for things that are very low in the stack that otherwise would not be able to get identity. For instance, that would include orchestrators of workloads. Somewhere there's that bottom turtle orchestrator that's deploying other orchestrators. That bottom level thing needs to get an identity from Spire. So low-level registrant will usually make the registrations for those. After that, it's the goal state that these orchestrators would be managing registrations for the workloads that they actually own. We also have things like health checkers on the infra level. For instance, when a host comes up, make sure that the Spire agent on it is actually a tested Spire server. When they're bringing a host down for repair or just normal host lifecycle, make sure that the agent gets evicted out of the Spire server. And then there are other just special interest services that just want to know various things about what's registered in Spire for their own purposes. So getting on to our consumers. So how did we get people start onboarding to Spire? Well, there needs to be as little work as possible for them. They also need to not have to be experts in Spiffy or Spire or security or any of that. They shouldn't have to do major refactoring of their service. They shouldn't have to do a bunch of reading to figure out what it is they need to do. We shouldn't change their service behavior unless they want it to. We have to allow a lot of customization because every service has different needs. And then we have to interoperate with the existing authentication and authorization at the company. So keeping in mind that most services are containerized and we're limited to just a few languages, our solution for this was to create a library that wraps off for services. So of course, wraps authentication with Spire agent or whatever authentication the service happens to have configured, exposes functionality for using whatever identity has been assigned for say like signing various things, performing NTLS or anything else where it's just useful for a service to have in X519 with an associated private key. This library also uses middleware to inject short-lived tokens into outbound requests and then those tokens have the SPIFI ID of the calling service and are signed by the X519S FIV. Then also we wrap all logic for authorization and then authorization decisions are made based on the calling SPIFI ID and the called endpoint. How this actually works. So let's say we have some workload A. It's onboarded to the security library that we made. So on startup, on behalf of workload A that security library is going to call Spire agent and get the X519S FIV for it. Now let's say that workload A wants to talk to workload B. For simplicity we're gonna say workload B is also onboarded to the security library. It doesn't have to be, it just makes things more convenient. So what workload A is gonna do, it's gonna send its request as per its normal business logic off to workload B. What the security library is gonna do on behalf of workload A is inject that token I talked about where it's gonna have the SPIFI ID, it's gonna have the intended audience, it's gonna have the search chain needed to validate that this is valid token and it's gonna be signed. Then there's other custom details that we throw into these tokens just based on our own needs. Workload B side, we're not touching the workload B business logic at all at this point. This is the security library on workload B side. It's gonna check, does the signature on the token match? Am I actually workload B? Did I somehow wind up getting a request or a token intended for some other destination? Do I trust the presented certificate? And then that's just another configurable trust verification within the library. It can include if workload B is onboarded to Spire, it could include just verifying against the distributed Spire trust bundle. Is this SPIFI ID allowed to call me on this endpoint? And then that's just a configurable authorization policies as part of the library. And finally, assuming just all of that passes, we finally make it to workload B's actual business logic. The code that the owners of workload B are actually worried about and just process the request as normal. So the goal here was take as much out of the hands of service owners possible so that we could do like it's generally said nowadays is let owners focus on their own code. There might be a question of, well, we're injecting these tokens, why are we not just using JWTSFIDS? So for one thing, X509s just work better with our existing services today. If everything's receiving the same standard, just makes it easier to create a library. There's also the issue of resiliency. If for some reason, a Spire agent were to become even temporarily unhealthy, then callers would not be able to get their tokens anymore which are much shorter lived than the JWT generally. And then receivers of requests would also not be able to verify if their Spire agent is unhealthy, wouldn't be able to call verify JWTSFIDS anymore. So that's just a resiliency problem and then also the compatibility with existing technology in the company. The results of this is that it's pretty trivial work in nearly all cases for a service add Uber to onboard to Spire, they just need to import our library, do a config change which is usually less than 10 lines. They can piecemeal import the library if they need to if they have some special use cases. If they outright can't import due to some language constraints, then we've also created a binary that will run inside the container and auto fetch the estimate on the workloads behalf. All new authentication authorization uses the SpifiID standard and then existing ones are either planned for decommissioning or have been moved over to the SpifiID standard. And then this library is extendable for like future environments that we might have and also extendable for any future auth solutions we might have if like say Spire is not sufficient for a particular case, for instance, with personnel identity. All right, then is that Q&A? I think I don't know if you can see the chat window there's a couple of questions there. I can read for you or I think is that I can try to... Yes, I'm not on Kubernetes. Yeah, we don't use Kubernetes yet. And then as far as I see also the questions about the sidecar approach, we are also bringing up a sidecar approach is just not yet productionized. So there will be a case of probably a bunch of off-boarding in the future. Things will still be able to inject secure tokens based off secure identity, but they won't necessarily need to be integrated directly with Spire. Yeah, Andrew, can we read the questions out of the floor? Yeah. So there's curious why each of us use library architecture versus sidecar. Just answered that. And then someone just had this string of things asking why we're not using other projects that might be available to like Kubernetes. We're not on Kubernetes at Uber today. So we're not using something like Envoy. We're not using other solutions like that. So library architecture just worked out better. What sidecar we are using, the sidecar plan I'm using is an in-house solution today. But we might move on later. For instance, we do migrate over to Kubernetes. Now we might use something that's more compatible there rather than an in-house solution. There's roughly how many different trust domains are you operating and how much traffic do you have crossing trust domain boundaries? So our team is responsible for just two trust domains. Only one of them is a real trust domain because it's not used for like testing your development. And then we do have some cross traffic with the team that is in charge of the personnel identity trust domain. So we do have to handle verification of personnel identities crossing over and contacting things on board to the spire trust domain. I don't think we have any other major trust domains today that doesn't mean we won't in the future. And then Uber is also very enthusiastic about partnering with an acquiring company. So we might have like Federation in the future, things like that, who knows, but not today. Does the receiving and validate the specific ID and the JWT matches the specific ID in the signing X79SVID? Yes. I don't see any new questions. Most I've missed something. We do have a couple of minutes if anyone has any more questions, one more. I think one just came up from Dave. Sure. So these thousands of workloads are mostly managed in the single domain. Has that caused any challenges? No, that hasn't caused any issues at all, really. The only thing that's come up was our initial work to split a test domain versus a real production domain. Beyond that, no, having all these workloads being the same domain has worked out just fine. There hasn't been any issues related to that. Is the security library open source? No, it is not. There's a lot of custom stuff related to Uber in it. And if we were to say, I would theorize that if we were to strip out everything custom to Uber in there, it would be a pretty trivial library because you ultimately are just contacting spy or getting your SVID creating tokens to outbound requests. So that'd be more an exercise for the audience, I think. Not something I think that we're gonna open source anytime soon. Is SIFI the only identity management framework you're using at Uber? If not, is there a transition to SIFI? Right, so we do have some services playing for deprecation or decommissioning, which didn't start out with SIFI standard because it didn't exist yet. And we have some special logic in our security library that handles calls related to services onboarded to that that kind of takes them over to being on the SIFI ID standard. Use any service mesh, what kind of topology do you use for multi-cluster federation? So right now we're using Spire as the basis to start up a secure service mesh. We do have other things currently implemented that are entirely in the house, but we're in the middle of transitioning over to more secure things based on the SIFI framework. And we don't use federation today, but we could in the future. What were some of the bigger challenges you had in uptake of SIFI Spire? I'm gonna assume you mean for uptake on the onboarding of customers, Don Ross. So challenges were, of course, everyone that's currently using the legacy software for authentication wants to just continue using that forever. So that's probably the biggest challenge is always just, hey, this already works for me. Why are you changing it all? Beyond that, we knew that there would be a lot of uphill battles if we made onboarding difficult. So that's why we've made a lot of the upfront effort to make this security library as easy to onboard, onto and configure as possible to make sure all our internal guides related to it are also like a straightforward and easy to understand as possible. So we tried to address that, the potential of those challenges head on.