 Okay, today we'll take a deep dive into the complexity surrounding identity propagation and API security in microservices environment. We'll explore our current workarounds and their issues. Then we'll see how a novel relatively unknown and under implemented extension of OS2 might help us to solve our problems. At the end, we'll also demo the solution. So my name is Armit. And I'm Yara. And we're both from Tyre, an open source API gateway management platform. I've been here for coming on seven years now, starting as a consulting engineer for our prospects and customers, and then moved into product leadership. And now my focus is on R&D, where my team and I are looking a little further ahead strategically to help shape the future of our product offering. I was a C++ developer most of my career, but I needed a change and decided to cross the lines and work with the users directly. I joined Tyre three weeks before Armit. And also as a consulting engineer. Later on, I moved into product leadership as well. As a head of developer experience, we're focusing on creating SDK, CLIs, and other dev tools to enhance our users' experience. Okay, let's get started. When I visit an online store and wish to place an order, as far as I'm concerned, I'm just placing an order via some web app. I'm completely oblivious to the different chains of offends and communication between services or different providers that are involved in order to fulfill my transaction. If we zoom in a little bit, this slide shows what the world looks like from the perspective of a service, which is responsible for processing an order. A service, such as a micro-service, can use and consume many other services. Some internal ones, like inventory, pricing, and user account. And other third-party APIs, such as fraud checker, shipping, CRM, and payments. Some services, sorry, but there are other services that might benefit, sorry, I'm a bit nervous. Some services, such as shipping services, are happy to just accept a simple API key in order to integrate with that service. They don't need to know anything about the user, apart from perhaps the delivery postcode. But other services might benefit from knowing who the user is. Perhaps this would include a CRM or a fraud checker. Some services, such as shipping, need the user's address to book a delivery or make a payment, while others, third parties, like a fraud checker, that are on less trusted domain would need to be downgraded to read only access rights. We can see that the micro-service environment is complicated and convoluted with diverse security and user's needs. Let's dig in a little deeper into this. So identity propagation refers to the process of maintaining consistent user identity across various systems, services, or components within a software application or a networked environment. In distributed systems or micro-service architectures where different, sorry, consistent user identities across various systems, services, or components within a software application or a networked environment. In distributed systems or micro- service architectures where different services might be responsible for handling different various parts of a user's interaction. It's crucial to ensure that the user's identity is accurately propagated throughout the system, and this allows each service to properly authenticate and authorize the user without repeatedly prompting for credentials. Identity propagation typically involves mechanisms such as auth tokens, where users are authenticated by some central authority, which issues tokens like a JSON web token or an OAuth token representing their identity, and then these tokens are typically passed between different services to prove that user's identity. We also have single sign-on systems, which enable users to log in once and access multiple related systems without having to re-authenticate. The user's identity is propagated seamlessly across the various services indicated with that single sign-on solution. Then finally, sessions are established upon user authentication, and a session identifier is used to maintain that user's identity as they interact with the different parts of the system. So session management ensures that the user's identity remains consistent within that same session. Now, middleware components like API gateways and service mesh data planes, they intercepts, requests, and add identity-related information before forwarding them to upstream services, and this ensures that each service has access to the user's identity without explicitly passing it in every request. So, along with identity, there's conceptual information related to the user's session or request, like roles, permissions, preferences, etc. They may also need to be propagated across the system to ensure consistent behavior and proper authorization decisions. Identity propagation is essential for maintaining security, providing a seamless user experience and enabling effective authorization and access control mechanisms in distributed systems. So, how do we go about propagating identity across our systems? We've gone back. Yeah, next. Oh. Back. Okay. Okay. Now, in this slide, we'll discuss and title workarounds and hacks. Many of these approaches could actually be okay, depending on the situation and use case, but there are a few approaches which we have seen out in the wild. First one is user ID token as access token. Amit, what's wrong with that? So, we've all seen many tutorials and implementations on the Internet which have recommended this approach. This approach is probably one of the simplest to implement, because once the user has authenticated and a client has obtained an ID token, then they're able to use that token in order to access the protected resource. But there are several reasons why it's not best practice to use an ID token in this way. Like, the first one is the purpose. ID tokens and access tokens serve different purposes in the OpenID Connect and OAuth 2 protocols. ID tokens are intended to provide information about the authenticated user, while access tokens are used to protect, are used to access protected resources on behalf of the user. So, mixing their purposes can lead to confusion and potential security vulnerabilities. Access tokens also, they typically have a shorter lifespan than ID tokens, and they may have different security requirements. For example, access tokens, they might need to be refreshed. They might need to be revoked in the case of the user logging out, or maybe a security incident, or have specific scopes and permissions associated with them. Access tokens are used to authorize access to specific resources or APIs based on the scopes granted during the authentication process. And ID tokens don't provide the same level of authorization information that are primarily focused on authentication. Using ID tokens as access tokens could result in improper authorization decisions. Okay, so now I'm convinced we will use access tokens in order to, in order to, for client to access protected resources. But why can't we just embed all the, all the, all the required identity claims into an access token and share that, and share that same access token between all the microservices? So the JWT specification specifies AZP and AUD claims, which enable us to control and understand who the authorized party is and who the intended recipients are. Sharing access tokens is like sharing your password with everybody you talk to and then trusting that the recipients will be good citizens. From the recipient's perspective, the access token is private to that intended recipient or audience. And as a recipient, you should be checking that the token was in fact intended for you. Because if it's not, then that token shouldn't be authorized to access the protected resources that you're serving. So by sharing access tokens, even between trusted services, you could be granting overly permissive rights. In our e-commerce store, should a random service like, I don't know, let's say a product inventory service be able to access or even perhaps modify user resource, should it be pay, capable of initiating payments? Okay, so we won't share the same access token with other microservices like a hot potato. Fine. So if every microservice has the ability to authorize itself, then what's wrong with propagating identity and other contextual information as request headers? Well, this is better. And it's not necessarily bad per se, but then how do you go about verifying that a client didn't self-declare their administrative permissions? How do you verify that headers haven't been manipulated? Or there's a lot of manual checks and balances like header request signing. And these are complex and they're non-standard approaches, which might just about work, but they're difficult to scale. A small change could open up a completely new attack vector. These approaches might have worked when we cared about the perimeter security when we took monolithic approach. But when we, with the introduction of microservices and consumption of SaaS services, the perimeter can no longer be well-defined. We have to aim to achieve zero trust architecture in a consistent and scalable way. So we think that the token exchange can solve this for us. And this specification extends the OAuth 2 protocol to enable clients to exchange one access token with another, via an authorization server. And why would we want to do that? And what does token exchange enable for us? So we have a few use cases. First one, enabling user impersonation. A client can impersonate as a user. This is useful when an administrator needs to rerun a task as if they were a user. Next one is internal to internal token exchange. A client can exchange an existing token created for a specific client for a new token targeted for a different client. This could be useful in case you want less permissive access rights when the service is calling another service. Internal to external token exchange. A client can exchange an existing token for an external token such as a linked Google account. And the last one, external token to internal token exchange. A client can exchange an externally minted token for internally minted token. This is very useful when you're a resource server, does not understand the issue of your access token or when you need to convert a SAML to a JWT and even enrich the access token with new user claims. Okay. So now let's go through the sequence diagram to see how the token exchange works. Okay. So before the exchange, an old client has already received an initial token that represent the user authorization. This may have been via client credential flow or an authorization code or another mechanism. This is written in green. Then the client request token exchange from the authorization server. This request includes that initial token and may specify desire scopes and intended audience or other properties for the new token. Then the authorization server validates the token exchange, the token exchange request. This involves authenticating the user if required and then authorizing the token exchange based on the initial token and the requested parameters. If the request is valid and authorized, the authorization server will issue a new token for the client. This new token might have different scopes, different audience and other properties based on the exchange parameters. If the request is invalid or unauthorized, the authorization server will respond with an error. At the end, the client now can use the new exchange token to access the protected resource from the resource server. So now, if the demo gods are willing, Ahmed will show us how to manually perform the token exchange. Okay. So I'm going to show, can you guys see my mouse? That might be easier. Yeah. So I'm going to show you an example, a token exchange request. The request gets sent to the token endpoint of the authorization server. In this case, because we're at Kubecom, we're using KeyClub. The request contains the client ID and the client secret of the client performing the exchange. You'll see that we have a new grant type. I'm just going to call it token exchange for short, but it's grant type URN, IETF, PRAMS, AWARF grant type token exchange. And finally, we've included the subject token, which is the access token that we initially received in order to access this service. Now, let's go ahead and see this working in action. We're going to do it manually. Hopefully, the demo gods are on my side. Can you tell me if this is large enough for you? Make it larger. Right. Let's hope I can, is that better? Okay. And is this okay? Or is that too big? Or is that, that's fine. Perfect. Okay. So before we do the token exchange, we have to authenticate the, you know, the very first request needs to be authenticated and authorized. So we have to, I've already set up key cloak here with my own users. It's logged back in. Then where I've got super insecure credentials, but I'm on local hosts. So I've already got a user set up here. And so I've configured a client inside key cloak and it's just called web app. And this is the, I suppose the browser application and we're going to be using, or postman in this use case. And this is an authorization code. We're going to be doing the authorization code flow. And the client, which is going to be performing the token exchange, is what we call tight gateway. And the way that I set it up like this is because of the following demos after. So, yeah, so that's basically it. If we head over, we're going to just log in via the web app. So this is just the authorization code flow. And I'm going to make that a little bit bigger so we can see what's going on here. And under auth, we can see that I'm configuring, I can, if I scroll up, you can see it's just a regular authorization code flow with postman as the callback URL. And then we have the regular authorization URL and the token URL there. And it's the client ID, so it's the web app that's that's authenticating. So this is just a regular authorization code flow. I go ahead and get a new access token and then I master sign in. So let's go ahead and do that. And go ahead and log in. Woohoo, it works. And proceed. And I now have an access token. So let's go ahead and have a look at what this access token looks like inside. And copy that. And we can paste that into jwt.io. And we can see in this access token, does that need zooming in as well? Is that okay? One more. One more. Cool. And what we can see is that when we generated this access token, this is because of the way that we've configured key cloak already, that the authorized party is the web app. We've got my identity in here in this access token. And the intended audience for this token is tight gateway. So that means that tight gateway is allowed to accept this client, is allowed to accept this access token. And now what we want to do is perform a token exchange. And we're going to do that manually. So let's go ahead and copy this token. And we're going to do an exchange of this access token with the body. Bear with me for a moment. I might have to just zoom out a little bit because I don't have much real estate here. And I'm going to replace that sub the token in here with the new access token that we received. And we're using in here the credentials of the different credentials of the client. And we can go ahead and send that request. And you'll see that now the client has performed the token exchange and obtained a completely new access token. So if we copy this access token, let's hope it copied or not. I'll copy it again. No, I'm suffering here. Copy that. And we paste that in there. And you can see that now the authorized party is the tight gateway. So this is a completely different client that's authorized. So but we've managed to propagate the identity of my user when I authenticated. So that's the most basic use case of the token exchange. Do you want to jump ahead now? And we can now build on this a little bit more. Okay. So we've seen how the mechanism works. For this talk, Ahmed and I also created a native middleware for tight gateway, which can perform for us this token exchange automatically. Rather than passing around the web apps access token, the gateway as the API gate, the gateway as the API gateway performs token exchange using its own credentials. And because of the way in which it has been set up in Kiklog, we're able to automatically secure and propagate the user identity into the gateway's access token. Using this impersonation semantics, it is almost as if the user who originally gave permission to the web app has actually given permissions to the gateway. So this JSON document represents a configuration for tight API gateway, which tells tight to validate an inbound access token, and then exchange that for a new token. You can see here as part of the API definition, the token exchange object, and we can see minimal setting required for performing token exchange against Kiklog, the authorization server. The target URL in this example is HTTP bin service. This service is an echo server which helps us with this demo, as it allows us to see the content of the request when it is received from the gateway. Okay. And okay. Perfect. So we're going to go ahead and show us how this is working now. So let's open up my next collection, which is login and request, and I'm going to close that collection to give us a bit more space, and make that a little bit smaller. Authorization. This is really difficult now. Okay. So let's go ahead and do that. I'll get new access token to authenticate. Oh, hold on a minute. I've done it wrong already. Bear with me for a minute. Let's log out my user. Do I need to log out for this example? No, I don't. Do I? Sorry. And we can go ahead and use that token in our request. And what that's done is pasted the token here, and we're going to access the HTTP bin service via the gateway. So now, when we send the request, and the gateway has now received the request, it's validated our initial access token that was issued by Kiklog, exchanged that token for a new token, and then proxied that request on to HTTP bin, and we can now see that we've got a different access token here, and this access token that was received by HTTP bin, if we paste that into jw2.io. So we can see that the gateway, in fact, was the authorized party. So using this, it's really, really seamless, and we recommend, or we would like to see more gateways and more service mesh services implementing these kinds of capabilities. Okay. So in our previous example, we used the gateway to protect the perimeter and exchange an inbound access token for a new token which could be used internally. It's also possible to exchange a Kiklog token for an external token minted by an externally identity provider. Consider a situation where our web app application possess an access token issued by a default identity provider, or Kiklog, which is in a trusted domain, but now our application needs to access Google Calendar APIs, which as we all know is residing in Google's domain and secured by Google. Our application must obtain a secure token from Google to access the Google API. This is another great use case for token exchange grant type. Just that this one involves two identity providers, which requires existence of a trust relationship between the two of them. So let's quickly go through this flow. First, the user authenticate with Kiklog. Then Kiklog issues an access token to the web app. And you can see the issuer is Kiklog. Then the web app is using the Kiklog and making an API call to the API that sits behind TIE gateway using the access token it got from Kiklog. Then in number four, the gateway is doing a token exchange against Kiklog, which is its own its identity provider. And it's using that original Kiklog token. Kiklog in its turn exchange does a token exchange again against Google. And Google returns the Google token and then propagates it to Kiklog and Kiklog to the gateway. With that, the gateway can reverse proxy to the service that is now using the Google token and can access the Google calendar. And now we can see this in action. So here, we have an API definition. It looks almost like the previous one, but with the exception of the target of the URL pointing to Google. No, it doesn't work here. Okay. And then you will see from the screenshot here that we have a configured Google as an external identity provider inside Kiklog. And here on the right bottom, we set up Kiklog as an oath client in Google. Also note that we have introduced a new parameter for this to in order for this to work. We're basically telling Kiklog that during the token exchange it needs to use an external identity provider Google as the external issuer. Okay. So this is the one where I have to actually make sure I'm logged out. Bear with me for a moment. So as my user, I'm going to kill my session. Okay. So there's no users for this session. But what you will see is for this to work, you have to ensure that under identity provider links, you've got Google as the, that Google has been associated with the user here. Now in this example, we're going to open up the collection, the collections, and this is demo three. And I'm going to close that to give a bit more room. I think I'm showing the end before I should. So what we're going to do is obtain a new access token, just like before. And now I'm going to be signing in with Google, signing into Kiklog with Google. See if it works. And I'm going to choose my Google account. And now Google needs, now Google wants to know that it's allowed to access my calendar. So I'm going to allow that and proceed. And now we have a new access token here. So despite the, you will have seen that Google has authorized, so I've allowed my app to have access, sorry, Google, I've granted permission for my app to have access to my calendar. And now I've got an access token. So let's have a look at what this access token looks like. Okay. You'll see, just like before, it's a regular Kiklog access token. There's nothing, nothing different about it. So because my services within my infrastructure doesn't understand Google tokens, they only understand Kiklog tokens. And now I can go ahead and use that token. And because the gateway has been configured to talk to Google's APIs, I can now browse the body of the request. And maybe I want to have access to my calendar, for example. So I can now send a request. Let's see if that works. Fingers crossed. And now I'm able to see, using the free busy endpoint of the Google API via the gateway, I'm able to have access to my calendar. Okay. So token exchange within Kiklog is in technology preview. Kiklog's token exchange implementation is loosely based on the OAuth token exchange specification at the IETF. And whilst it extends some aspects, it also ignores and loosely interprets other parts of the specification. And we do have some observations for the implementers. So developer experience, we're not Kiklog experts but neither are we complete new business. Configuring Kiklog for token exchange is far from a simple task. Creating policies, client scopes and mappers and enabling token exchange for a client is very time-consuming, error-prone and could greatly benefit from a simpler workflow. From building out our demo for this talk today, we found out that for internal to internal tokens, we've not yet been able to control the desired audience of the exchange token. We've gone through the documentation. We've tried to run through examples and there's not much, there aren't any tutorials online that are comprehensive. And even the documentation, it does have warnings but I think there's something that we're doing quite, you know, a little bit wrong. It's really difficult to configure it. Properly, you know, we're not able to control that audience and we'd love to be able to do that. We just failed. We just failed miserably. In addition, scope support is on the roadmap but not yet implemented. Scope support is important because it enables us to define fine grant permissions to an access token. And last one. Yeah, lastly from what we see with the Kiklog implementation, tight gateway by exchanging the token has been able to impersonate the user but it would also be useful if the web app would be able to delegate rights to the gateway. In other words, tight gateway would like to be able to act on behalf of the web app and we'd like to be able to see this as part of maybe a composite, as part of maybe a composite token support as part as per the specifications. This is because the user delegated rights to the web app and has no knowledge of the tight gateway. Okay, so first of all, thank you very much for listening to our talk. You're welcome to come and see us in L14 booth and say hi, but first of all, if you have some questions, you can go to the mics on the aisles and ask us something. We still have time. Does anyone have any questions? Yeah, to the mic. Yeah, just in the middle. Yeah. Don't fight. Hi. Okay, so hopefully this is not too difficult of a question but I was thinking to myself, as you were talking about all these mechanisms to refresh tokens and exchange tokens, what if my system is simple enough that I only really have one service? Does all of this still apply or is there like a simpler setup that makes sense? So I think I can answer that. If you've only got one service then you're not needing to propagate the identity anywhere, but that one service, does it need to talk to third-party SaaS services or anything like that? Or if it has an API gateway in front of it? Yeah, should that gateway have the be able to pass that access token around? How does the gateway know what's going to happen to it after it's left, after it's proxied the request elsewhere? And I'm struggling to believe that there's only one service because you are using other third parties like Ahmed said for sure and you're checking other stuff, even if it's just an API call, it's it's in a different trusted domain, so potentially you will improve your service, you'll add just something, you know, like just another thing and you will find yourself passing on user information. Thank you. Hi, nice talk, that was super super good. So my question regarding kind of performance and latency increases and things, so with the current authentication authorization flows that we might already have, if you've got multiple chains, like be on the gateway and a single service and then that service calls the other services and it goes back, you probably want to do these sort of token exchanges on each call between each services to make sure that maybe like service C in the chain then needs to call service D and it doesn't want to pass on and do privilege sort of passing the privilege of the token. What sort of measurable, have you looked into performance issues of like doing all of these calls plus your own maybe authorization calls to your own authorization system? So we haven't measured the latency or anything like that but from for our for our proof of concept and this this middle web that we created we implemented caching so we're caching the signature of the initial access token and we're looking up that we're using it as the cache key so that way you're not having to re-perform the exchange when you receive the same initial access token over and over again so then it's much much faster. Okay yeah thanks okay from that side Hi thanks for the presentations and for the demos I was wondering if you have any news about you know token exchange being still in preview it's a preview feature in Keycloak since long time do you have any news when this could become a supported feature? We're not in the Keycloak team so but they are here so well hope to hear you just behind you but I was wondering if you know anything about it like yeah we're not sure we're not sure yet so we're keen to see it move on and you know I'd love to help the Keycloak team to to make it you know to make it GA and raise more awareness about the about the protocol as well so yeah no thanks yeah Hi my name is Thomas thank you very much for your talk I'm one of the Keycloak maintainers and we are currently working on re-wising the token exchange support and getting it out of preview and solving a lot of the issues that you mentioned there so I'd love to talk to you I'm going to come and see you I'd love to help so I've got a question about your middleware how closely does it follow the RFC because obviously Keycloak is very handy but what if you want to write your own token exchange server sorry did you say how how closely does Keycloak follow the RFC no how closely how closely does your middleware follow the RFC okay we literally just built the middleware for the purpose of this talk right okay so it does follow and it works with Keycloak so we haven't tried it with any other identity providers or anything like that it works well with Keycloak it's a poor request in draft mode so it's not it's not bulletproof yet but we're you know we want to move it forward and get the community playing with it using it and seeing if it works for them so come and see us BoothL14 and we can kind of talk about that and maybe give you a demo as well about it you know and discuss your use case as well I have I have a question do you happen to have any scale scalability data on this how well it works at what scale have you seen it work so well it's well I think we were asked that before we built that as a POC we do have caching I hope I understood your question correctly but we have caching for for that token exchange and for example if you implement if you think about for example service mesh where you have a gateway front in an API so every API call will will go through the gateway and the gateway is implementing the caching so it is an extra hop but then it depends on fine tuning the caching right I was looking for like how many services have you tried with this like in the number of services microservices you mean in a chain of chain of calls yeah yes it will add up it depends on the connection to the authorization server however there is value for that in terms of security so it really depends on what you're doing and you know if you need to to make an API call to a less trusted domain then it might be worth it right okay thank you and it will be faster than calling out to Google so because obviously key cloak will be in your own infrastructure so that will be you know local network latency as opposed to going you know out over the internet into the cloud somewhere thank you for the great talk and it was very useful because it also directly relates to what we are facing right now and figuring out that this option exists was an amazing thing thank you but I have a question regarding long lasting operations or whenever your gateway would need to schedule something happening in a long time away from the point that the request was made scheduled in it it might exchange the token right away and get a access token and a refresh token but it will still need to push it somewhere for the duration and the refresh token or the access token duration might expire during that time how would you advise handling a situation like this okay we just we just got the message that we have to finish but it's a really good question I don't think if I remember correctly it's not part of the RFC so whatever you'll do depends on you know specific implementation it is really good idea to think about that I agree and yeah maybe we can think about it together I'll drop by the booth yeah yeah cool so in the demo that you gave you signed in as Google in order to exchange for a Google token is it possible to have the identity provider that you sign in and the third-party service that you're exchanging for be different or they have to be the same identity I think that's dependent on the identity providers or that's dependent on key cloak and I think it's worth speaking to the guys that are here to understand how that the mechanics of that works exactly because we have to ensure that the Google accounts are linked in order to be able to access the to access that calendar right so this is specific to key cloak and not according you know it's not a standard and different providers will implement it in a different way yeah and that part isn't and the way that it's been implemented within key cloak isn't actually part of that token exchange spec it's just how they've leveraged those capabilities to enable this this kind of feature right okay thank you so thanks very much slides are on the QR code if you're interested