 Awesome. Well, for me it's been, for me it's been a great learning opportunity. You know, I am not an expert in this particular space or this particular domain even. Like most of you, I have to, you know, security is just a part of the job. There's no way to escape it. So I take every opportunity to dive a little bit further. So what I'm going to try to do today is just take a beginner's look at this whole spiffy thing. I remember when the project was announced, I didn't know what the hell they were talking about, but somehow it's found its way into things like Istio and I see it popping up in other systems. So like you all, I'm curious to why. So one thing that I did, I'm going to share my laptop in, there probably won't be a lot of slides. I'm just going to walk you through kind of a similar approach that I took for understanding things. Like most people, I just went to the spiffy website and it's a universal control plane for distributed systems or identity control plane for distributed systems. So in order to really understand that you have to kind of look at the whole package, like Spire is one good implementation where, you know, it gives you the full suite of things for trusting workloads, assigning their identity and doing all the things to give you a complete system. But I was always curious about like, how does it all work? So for me, I kept digging a little deeper. And the documentation is fabulous, by the way. So whoever is on the documentation team here, someone should give you a virtual high five. You've earned it. So clicking through here, one thing that has been a challenge, I've been doing this for about 15 years. And whether you're talking about minting SSL certificate and doing TLS mutual auth or some other form of authentication or credentials, it's been real hard to identify services in a way that scales well across multiple systems. And we've all tried it throughout our careers. But when spiffy came out, it seems very complex. And I'm pretty sure I may be oversimplifying things. But here's the thing that made it click for me. Once I really took the time to understand what is a spiffy ID, naming things as hard, but the way they've chosen to go about identifying the service. So we're going to get to a couple of things. One is, what is a service identity? How do you give one to a service? And then how do you use it throughout the system? Those are the three things that I'm most interested in as a beginner. So the spiffy ID is the thing that kind of made it click for me. One is I asked myself, can it really be this simple? Can it really be as simple as this string? Spiffy is the kind of URL, then there's this trust domain like example.org. And then the workload you're talking about, it could be your payments workload, it could be your API, or it could be a calculator app that does add functions. But either way, they've simplified this whole thing down to string. And this string has a lot of information that it communicates very succinctly. This can actually help you make work with different applications and different domains. You can have different service polls. You get the idea. So looking at this simple string is, I asked myself, like, what do I do with it? Wouldn't it be crazy if just any app could take this string and give it to another app and say, hey, trust me. This is who I want to be. That definitely doesn't work. You need to kind of earn your credentials. So how do we do that? How do we trust? And I may be getting some of the terms wrong, but it seems like bootstrapping, trust, and security is one of the hardest things to do because at some layer, you have to trust something or someone. And for us, we're going to choose to trust something. So we understand this string, but now we've got to figure out how to create them and hand them out to applications or workloads that deserve it. Now, if you're using Spire, and there's going to be plenty of talk throughout the rest of the day about Spire, they have a really good implementation of doing this by making sure only the right workload gets the right spiffy ID. But we're going to roll things by hand just so we can get a little bit deeper in details of how it would work. So just like most good projects, there's a lot of documentation about the details. So what most people do is when they think about Spiffy, you can embed them into a TLS certificate. If you use something like Istio and you're doing TLS mutual auth where app A wants to talk to app B, well app A can present its SSL certificate and have a Spiffy ID embedded in one of the subject alternative names. And that's cool because then proxies and other tools can extract that and decide to trust that you have that identity and let you do things. In this case though, I'm actually sitting behind a proxy that doesn't allow TLS mutual auth. Actually, I've made things a little bit harder for myself. I'm actually running inside of a serverless platform called Cloud Run. No Kubernetes, no VMs, none of the environmental things that I'm used to. If I was doing on some VMs, this would be a whole lot easier to solve, but I'm not. So I have to solve this problem in a slightly different way. I'm not going to be able to use X509 or TLS certificates for this. I'm actually going to have to use a Jot token. The nice thing is they actually have a specification of how to actually embed a Spiffy ID, which will remind people. We're talking about a string that's your identity, but we need to mint one only for the workload that's supposed to have it in a way that other workloads can trust. So with that, I'm going to use what they call a Svid or Secure Verifiable Identity Document, and you can use certificates or, in this case, a Jot token. We've all used them before. I'm going to show you how that works underneath the covers, but basically what we want to do is we want to create a Jot token, and if I go through their documentation, they have things about certain claims, and if you're not familiar with Jot tokens, basically I can put in who this particular token is for, who should be able to receive it, and also who's holding it, and that's going to be that Spiffy ID, and that Spiffy ID will be actually what we call the subject. So we're going to embed a Spiffy ID into one of these Jot tokens that we just can't go and give them to everyone. So who do we trust? I can't trust the workload. Damn sure not, I'm going to trust you, but maybe I can trust the cloud provider, and that's what I've done. I'm going to speed things up quite a bit here because we only have so much time, but basically I have a container running as an off service, but I've named it JWTSvid. That's the service, and the way it works is if I take a workload and give it a Google Jot token, what I can then do is map a set of Spiffy IDs to Google Cloud's service accounts. I'm going to show you how that works real quick. So first thing I'm going to do is I've deployed this service into the cloud. The thing I want to do now is I want to get a token from it, but in order to get a token, I have to prove my identity by getting something that this off service trusts. So I have a little command called get token, and what it does is it gets my GCP project. I get the URL of the cloud run service you can see here. So I know who to exchange this other token with, and then I'm just going to use the Google Cloud command line tool to generate an ID token. I'm going to impersonate this service called calculator. So basically what I'm doing here is I have a calculator app that can call a backend to add some numbers for me, but the ad service only wants to allow only a calculator to do it. Now what I don't want to do is rely on Google service accounts. The thing is those only work for things that understand how to verify them with Google, but you might want to run things on VM or on-prem. So this is where this whole Spiffy thing becomes super important. It's pretty portable across all the cloud providers. So what I'm going to do is I'm going to generate an ID token by calling the Google Cloud IAM service, and I'm mimicking what I would do in a deployed application. So once I get an ID token from there, I'm going to pass that ID token to the serverless off service and see what comes back. So we're just going to run this really quick called get token. And again, I'm getting the IAM token, ID token, and I'm going to use it to connect to the serverless application. And what that serverless application is going to do is going to mint me a token. Now when I call this command, everything is working, come on internet gods, move a little faster. You can see I'm impersonating this account and I'm getting back a token. So let's look at this token. Now what I'm about to show you is dangerous. So don't be doing this in production with your real tokens. But for this example token, it's okay. I'm going to go to JWT.io and I'm going to post this token. Now be careful. There's a header, the actual drop token and the signature. These can be replayed if you do this with the live token to get access to resources. But this is for testing. So I'm not too worried. Here you can see the contents of the payload. I could verify this using the private key from this particular service. But you're seeing here that when I authenticate it, I got back this audience. That means I can actually pass it to the addition service. And then the issuer is my off-service running in my domain. And look, it gave it a subject back of Spiffy Hightower Labs calculator. Now one thing we're going to do is I'm going to show you how I actually made this work. Well, there's a couple of things. So then I'm going to run a command and I'm going to show you my registry. So in my custom implementation that's minting Jot tokens, I also implemented a in-memory registry. And the way it works is I can actually send registry entries up to a in-memory database. So basically the registry basically says, for this given ID, this is a Google IAM service account ID, the audience should be x and the Spiffy ID should be y. That's how it works. So all I'm doing is saying, give me your service account. I have a mapping that maps this calculator IAM service account. You can see it's a unique ID. And I will find that unique ID in the registry. And if I don't, no tokens for you. So this is an easy way for me to map Jot tokens using something that I trust. So I verify the Jot token from Google, make sure it was issued by Google, verify the public key, and then extract the subject from the Google service account and mint a new token with those attributes like we see here. So great. We have that first part of the puzzle solved. We have a third party we trust, Google Cloud's IAM service. Now you have a token in your hand. And now you have an identity that others can trust as long as they trust my certificate. So quickly, we're going to move up really quickly here. We have a certificate. And how do I get the certificates? Well, if you know anything about Jot tokens, you know that that all service can also serve up the public keys that can be used to verify things, or you can embed that public key. So here I'm following the spiffy documentation saying that the use of this public key is for validating Jot Svids. And here's the public key that you can verify that this was signed by this off provider. Now one last thing you got to do here. So we set up the first two stages. Next thing we got to do here is figure out how do you use it? What's the purpose of having this identity if you're not going to use it for anything? When you think about service mission, all of the features you want to do and most security models, they're all hinged on IDs. Who are you? Then what can you do? And this is when we bring in something like Open Policy Agent. And this is the last component of the demo. So what we can do with Open Policy Agent, let's see if I can screen my screen down here. And what OPA allows us to do is basically do Auth Z for what we can do with this identity. So I know who you are, but what can you do? So if you look at this, I have a package for, this is a Rego file. Don't worry about the syntax now. Just know it's a way for me to communicate policies. I also having Open Policy Agent running as a serverless app. So here it is. It's running in a container in Cloud Run serverlessly. And I populated it with this policy that I'm going to walk you through. So by default, I'm going to deny everything. But when you make a request, you can say, hey, from your application, so let's say you are the ad service, you would go off to the back end and say, hey, Open Policy Agent, here's some metadata about my request. I'm going to do something interesting here. I'm also going to pass the token. When I put the token into the request to Open Policy Agent, Open Policy Agent can verify that token. It can also decode that token, extract the header, the payload and the signature. The nice thing about this is I can use that information in the token, in that dot token with the spiff ID to make policy decisions. Do I allow you to add the numbers that you're requesting? So from my service, I need to put in some data. And what I'm going to do with my input data, I'm going to check for some matches. Is this a post request? Is it at the ad path? Is the audience for me, the addition service? Was it issued by the off service? And again, I should validate the private key here. And then also down here is, is this the calculator app for the Hightower Labs domain? If all of those things are true, then I'll respond either yes or no. And we're going to do the last piece here. I'm going to show you what it looks like. I'm going to use curl to do this so we can get a clear picture. Now I could have deployed all the apps to do this, but I want to give you a peek at how this works underneath the covers. So one thing we can do here is validate a request. So assuming that your app was presented with that dot token we saw earlier, what you would have to do inside of your application is you would have to craft the input data. Again, the method, the path, and the token. And then what I'm doing here is basically getting the URL to the open policy agent service. And then what I'm doing is I'm authenticating to the open policy agent service. I know that can be a bit confusing, but I'm not going to keep that thing wide open for various reasons. And then the next thing I'm going to do is pass in this request. So the way that request looks like on my file system is basically a post, the method, the path, and the token. I'm going to submit this and let's see what response we get. Then and we'll do a validate request. Again, I'm simulating the addition app, assuming that the calculator front end is calling it. So we'll run this now. And what should happen is I should get a response from the back end. So it's saying, hey, method is get or allow equals true. Method was get, path was add. And here's this token. So I know we're getting low on time, so I'm going to stop here. But basically this shows you full circle. One, we have a root of trust, in this case, Google Cloud IAM. We have a custom service that's minting Jot tokens by trusting service account tokens and mapping them to a registry ID and yielding back a Jot token that meets the Spiffy requirements. That token now can then be used with other systems that can also delegate off Z to back ends like open policy agent. I'm going to stop there, see if we have any questions. And hopefully that kind of explains this thing at a high level for you all. Hi, everyone. If you have any questions at this point, please feel free to put them in the chat window or raise your hand for Kelsey. Thank you so much, Kelsey, for doing this. There's one or two takeaways you'd like to leave folks with, personal learnings. Yeah. So I think for me, I can see, you know, I work at a cloud provider. I'm working at Google Cloud. And I think about the Spiffy ID actually taking off in a big way. We already supported in tools like Istio and our service mesh technologies. But I can imagine us adding maybe one day a Spiffy ID to go with those ID tokens. And maybe you won't have to exchange them anymore. Or the other way would be if you're in a serverless environment where you can't necessarily control ahead of time and maybe you don't want a pull model where your app is exchanging one token for another, you might be able to push things like into a secret. So if you're in the Kubernetes space, I would encourage you just to look at tools like Spire that does all of this for you with way more robustness. But I think there's going to be a world where if this thing takes off, I can see almost every tool being able to trust this identity. And it's actually a lot simpler than I thought before. So I'm hoping more systems integrate natively so we can all standardize on what a service ID actually means. Fantastic. Well, go ahead. Yeah, I was going to just say, I think people message you directly, privately. So I don't see those questions. I'll read one. So here's one. How do you know that I am token can receive the token for the off calculator? So when I deployed, I'm going to, I think I'm still sharing my screen, right? Yeah. So when I deployed this JWT off service, I had to do a few things. One is I said that when I deployed it, that it must not allow wide open authentication. So services off been deployed. So the first thing is I don't allow authentication by default. The next thing is that means you have to present me a Google Cloud IM ID token, right? You have to get that from the metadata service. Once you do that, I also have another thing that says I granted permissions to the calculator service account to be able to even invoke this JWT Svid service. So this is an IM policy where I have a binding that says if a caller were to present a job token signed by this service account, it can invoke this function. So if you don't have one of those IDs or service accounts or job tokens, then you're going to get rejected at the Google Cloud low balancer. It won't even let you make a request. So then when it flows to the auth application, I can also validate there. I can take that job token and use Google's public certificates to also validate it. And then once it's validated, I can then extract the ID because that's going to be in the subject field of the ID token. And then I have my own internal registry, right? And all I'm doing to add things to the registry are just all I'm doing here is just basically collecting enough data about the service account ID. And I'm just posting it again by authenticating to my off service. So checks and balances end to end. I think we've got time for maybe one more or two. Yeah, I think so. I don't see anything live here. Do you have, I think you had one more question there about architecture diagram? So let me go here. There's a couple. So service, so the architecture diagram, I think this turned out to be a pretty good demo. So I am going to publish this whole thing on GitHub. It's in a private repo now, but I'll add the architecture diagram and some of the trade-offs I'm making. And what you'll find in that GitHub repository, most of the code is already there. There's a custom Jose implementation of JWS and also all the keys. One thing I try to make it easy to do also is to make sure that this works in a real world. So if you ever have to validate these tokens across other systems, I do implement most of the standards so that you can actually verify these using the public keys that you can get from the off back in. We'll spit this one out here in a second. And so that'll be on GitHub soon and maybe I'll tweet out when it's done. So I'll just say, hey, my spiffy demos now on GitHub, you can go run it yourself. The next thing is the token issue is something we have to build ourselves. And if we're not on a managed cloud or on-prem Kubernetes offering. So if you're using Spire, it's going to do a good job of dealing with probably X509 certificate is TLS Mutual Office, probably the path you want to go. If you're on-prem with Kubernetes, just go get Spire and make certs that way. That would probably be much easier than what I'm doing here. Now I'm not sure Spire supports Jot tokens at the moment. So if you have a use case similar to mine, we're going to be going through a load balancer that won't be able to handle TLS Mutual off. The way to think about this is maybe something like ping identity or off zero, if they allow you to have some metadata where when you authenticate, typically what you can do is you can set custom fields using some kind of template that when you return back a response, they'll have a Jot token. So ideally, most of the standard tools for minting Jot tokens should work. You just got to make sure that those tools are configurable. Based on the original identity, whether it's basic auth or some other factor, then you can actually say, hey, based on that identity, here's what should be in these fields when I return a Jot token. So my guess is standard things will work. But if you're not using Jot, go check out Spire and do it. Oh, it turns out that Spire has support for both X509 and Jot. So I actually spent the whole night when I can adjust to use that. But look, I learned a whole bunch in the process. Spiffy the hard way. And so that would be along with you. Thank you very much, Kelsey. Yeah, we have time for one more question. Kelsey, which one in your answer first? All right. So, well, so the next one, I'll end with this one. How is multi cluster and Federation supported in this kind of use case? So I think this is what this whole thing is about. One thing that's nice about X509 in general, it does allow each of the servers to have a certificate signed by authority we trust. And we can trust lots of authorities. If you look at your, you know, your TLA certificate bundle for CAs on your file system, you'll see that there's a lot of bundles in it that you trust. What we're doing here with Spiffy is giving some, you know, very similar technique of using these federated trust models. This is what Jot tokens are already good at. This is what X509 certificates are already good at. So when you start thinking about multi cluster and Federation, what we're doing is we have this detached way of saying, here's a bunch of public keys inside of our clusters, give them to your app so they can actually trust things. And then when that Jot token shows up, you may not necessarily need to call out to another service. You can actually use your public key or something like open policy agent that is configured to be able to decode and verify these tokens. And that's going to give you that decentralized federated method. But again, the smart thing that they've done is just leverage existing things that already work in a federated and distributed manner. Great. Thank you, Kelsey. I think we're right on time, but we could take a short question if you want, or we can just send you these questions. Maybe you can give your responses and we can post in a doc in the community. Yes, I'll do that and I'll answer one more since we have one more minute or so. So can we use an app in one cloud to talk to an app in another cloud using Spiffy as a single trust provider? And I think that's where things get super interesting because in my off service, I could trust multiple identities, multiple back ends. This is something that Spire does really well. It gives you a universal way of saying, I trust all these things. So imagine if you're using Spire, Spire can be configured to trust Amazon's metadata service. So when you get your identity document from Amazon's metadata service, say you're running on a VM, you can exchange that or use that to get a different token or X509 certificate. And I can use a whole different trust model inside of GCP. But at the end of the day, you end up with these identity documents that are standardized that keep a common Spiffy ID. So maybe you're all in the same domain, even across multiple cloud providers. You can look at that domain to say, hey, we're all in the same app domain slash this is your app. I can trust you to do these things. So it really helps us to kind of normalize the differences between the different cloud providers and then have a standard thing that we can all rely on whether we're in multiple clouds or on-prem. I think it started to take off, hasn't it? Yeah, I mean, yeah, there's people that are already using this. You see this, it's a hidden thing. The bad thing about it, it just works. So this is why there's no hype train. That's why there's probably not a bunch of 10,000 people conferences. It just works so much that most people kind of take it for granted that this kind of complex problems being solved in a very clever and simple way. I think that's why most people may not talk about it, but guarantee you that it's there and I think it's actually taken off in a way that will become, I don't know, you may not ever realize it, but you should learn about Spiffy IDs. Great. Awesome, Kelsey. Definitely, thank you so much for your effort and time.