 Afternoon, how's everybody doing? Wow, that's exciting, huh? Well, if we can't be informative at least we'll try to be somewhat entertaining keep you awake I'm Adam young For those of you who don't know me, and I think that's a very small fragment here. I see all my friends in the audience. I Am from Red Hat and I'm Christy Nicola. I'm from the Massachusetts open cloud Yeah, we're here to pump you up. No, we're here to talk about per API roll-based access control which I know is just the Burning thing you all want to hear about so we're gonna talk a little bit about how access is Done in Or how people want to be able to do access control within open stack deployments the current Implementation of how our back a roll-based access control short into our back is done And the things that we're proposing to make things better of things that we're we have in flight to make things better Okay, so the the analogy I often give when talking about access control is This is this is my car and if I were to try to lend you my car I'd have to give you the key to my car Which you can also see in the picture here and this key is what's called a proximity key I don't actually have to put it in the car in order to start or even to open the door I can just touch it and You can imagine that you know this this is very convenient But it also means that if I have two of these I don't really know which one's running the car I don't know which one's actually opening the door to the car Which is probably not a problem So they only have one car But if I had more than one car if I were say a rental car company or something like that Then I would need some way of being able to figure out You know of these many keys that I have Which one do I use to get in the car and imagine if you had to lend out your entire key ring? To anybody who wanted to borrow any one of your cars and you were a rental car company This would probably be a suboptimal solution So that's kind of what we're at with Keystone nowadays right now The key to the car is the role you have a role on a project and you use that to get access to things in Open stack and you don't really know which one of them you're using to do the actual operation You just give them all Including if you're an admin and you're just trying to do a really small thing now Another thing that's come up time and time again is that we have new types of resources or even for the different types of resources We need different roles and I kind of liken this to the different permissions You have on your driver's license for being able to drive different types of vehicles I actually was at one point qualified to drive a motorcycle but that does not mean I should be driving either an 18-wheeler an aircraft or Strapping something to me and jumping out of said aircraft. Those are all things that require their own qualifications to get out those are Comparable to separate roles for separate resource types and then so I don't have a funky Metaphor for this here, but something and the thing that kicked us all off was many many years ago I was asked by the horizon team. How can we figure out based on the roles that a user has? What to show them and so I've circled on the screen here the The toolbar was just like the top level set of operations that somebody would do if they're going into horizon and trying to manage their Open-stack cluster and so the three use cases as we've deduced from this is first of all on delegations What role do I need to give out to somebody in order to perform this action? How do I add additional roles to restrict a user to only a subset of? the overall actions not everything and How do I modify the user interface? Not just for horizon mind you but for all of the tools that are out there to show them only the ones that they can And give them directed use cases and directed user experience on their interface This is a simplified model of the access to resources type of objects that we have in Keystone this used for the rest of OpenStack a user or a group is given an assignment on a project and that assignment is tied to a role so it's a it's a it's a tuple with three values in there user or group project or domain, but we're not going to talk about that and the role object itself and then when we get to the actual service there's going to be some policy rule That's tied to the API that they're trying to call that gives access to the resource Again tied to that project. And that's what you see here. Let's see if I can do this well That's why you can see here. There's two different things coming into project There's the user side and there's a resource side and that's what we're trying to figure out That's that access control who can who can get in there so user goes to Keystone and they send Something I'm using a picture here kind of like their picture ID some credential some way of identifying themselves remotely To the Keystone server and they get back a token that token is then going to be used when they go to talk to Cinder yes, that's a cinder block talking to Keystone And they're going to pass that that token on to Keystone is or cinder is going to pass that token on to Keystone and request a validation and assuming that it's okay. It's going to come back. You see our Person has a hat on to apply. They're wearing a different hat. That's their role and Cinder says this is okay. Does it's work and returns a success message to our end-user and What's in that validation response? Well, there's three basic things. Who is the user requesting this? What roles do they have and on what project is is it there? I'm trying to give you some visuals again to go along with that Okay, once cinder or whatever the remote service is gets the The request and you can see that's going to probably start off with something like ha proxy something like that It's going to pass through a pipeline The first thing that happens is it hits the Keystone manage code called off token off token is a middleware piece And one of the biggest things it does is does this remote call the Keystone and assuming the Keystone returns back a Successful validation including that data. We saw on the earlier slides. It's going to validate the parameters It's going to read from the database and then it's going to force policy on that and assuming that again the policy checks it's going to do a state change and This is not this is my view of trying to show what's happening logically within The the scope of policy enforcement. Nobody else would present things this way most people are far more sane That policy check is Implemented by a library called Oslo policy and which is a rules engine and it answers a yes or no question Do you have access to what you're trying to do based on the token data and the resource data? And I use the term resources a generic term to imply virtual machine Network if you're talking to neutron image if you're talking to glance and so forth It's Keystone doesn't know about the different resource types out there Keystone middleware doesn't know about the resource types going out there and keep that point of mind because it's going to come up later It's just going to say based on this rule and the values you've given me Do you have access to it or not? It does a roll check to see what are the set of roles on the token and do those match the ones that I say are required to get access to this API or This object whatever is and it also does a scope check does the project on the token match And while you might think of those all as one things, they're really two distinct things The scope check requires a resource from the database You've got the token and the token has associated with a project on it In order to check that that matches on the resource Locally you have to fetch it from the database the roles check doesn't have to the role is just on the API that you're calling itself So this means that it has to be performed a deep in the code not in middleware This is code the Nova team has to decide where to call from within the Nova API the Cinder team has to decide where to call that from within their code and thus it's very very service specific Some services have additional logic beyond these two checks the role check and the scope check But we're not going to go too deeply into those and in fact, I pulled out a couple rules from I believe this from the neutron Policy.json file they give you a sense of the kind of stuff that they're doing there They have a top-level rule context is admin and that just just checks to see if the token has the admin role on it They also have a rule that they call owner and that checks to see that the project ID matches and then later on a rule the checks that these two things are together or for This one other case they have another role that they check and roll ADV SVC which to be honest with you I have absolutely no idea what it does Okay, if there's no explicit scope check Every project has access to every project's resources a user with a token on any project has access to Any project that's why this is a big stop sign. This is a big warning sign the it has to be an explicit scope check if there isn't an explicit role check then having a Role on any AP a Role on a project gives access to everything there It doesn't matter the role you have once you have that role you get access to all the apis for that project and Here's some of the problems with it People have stumbled across we're trying to move forward with the scope check should not be edited while you can crank it down You can't change this. This is hard code This is an engineering decision that should be made by people checking in Python code for the various remote services And yet we have it in the policy dot JSON file and are telling people go customize policy This is Again suboptimal and again the scope check requires a resource from the database which is deep within Python code Which means there's no way to map it back to the URL that you actually called all in there Is it possible to write a tool to do that kind of stuff? Yeah, but then you have to enforce that everybody uses it and all that kind of stuff practically speaking there is no way to map from the policy rule in effect to the URL that you're trying to do and That you know we I think you can hopefully you can see how that connects to the use cases I talked about earlier So once we dug through all this stuff we tried to come up with you know What are the what are we trying to do to fix things? What are the inferred requirements that we can pull out of here? And these are the requirements we've inferred How many people would be ready to string me up if I broke your deployment by changing policy, okay? Thank those of you who didn't raise your hands. I appreciate your restraint But I know that that didn't I know there's people who are very willing to tell me they'll string me up Actually, they do in any way We want to clear map from the API you're calling to the role required to execute that API We want to explicitly split the control into a roll check stage and a scope check stage We want to perform the roll check in middleware So it's code that can be applied to all services without them having to go and do additional engineering But we want to enable it by a configuration again We don't want to force us on you up front because we don't want to break your existing stuff But we want to make it available so that you can use it when it's ready for you And finally we want to enforce a default role to get around that whole You know any role is every role we want to enforce a default role on the API's Until a more specific role is actually explicitly requested or an explicit rule is requested for a given API And so that that gets us down to the question. I've been asked time and time again Why can't we just do this with policy dot chase on? We need an inventory of policy enforcement points and we don't really have a way of Mapping from the existing step that we have to the URLs the policy cut points are deep in the code We can't map there the roles and the role assignment data are Managed in Keystone and yet policy.json is config file if we want to start making changes We're gonna tell you go to keystone make these API calls and then deploy a new configuration file that's a really strange workflow and Redeploy that may require restarting services Deploying new files And again, it's it's not the same when you start talking scalable multiple multi site deployments It's it's not a trivial thing to do and dynamically updating the policy.json file on the remote service side If we should tell Nova here's how you update it That's also really untenable because then every service needs that API to update its own policy So once you start pulling all these things together you realize the policy.json has some Shortcomings especially and remember. I'm only trying to talk about the the the role-based access control side of this making that more dynamic So once you go through all these stages and you say try this and try that and believe I've been trying this for years I've been battling dynamic policy for years Some of you might have said in my talks back in Vancouver and stuff like that and go Yeah, you've been talking about that and I'm talking about it now Whatever's left must be the truth. So I'm hoping that's what we've gotten to here We've gone through a bunch of iterations and what we have and we're about to present is I believe the right approach to go about doing it And Chrissy is going to walk you through what we've got for an implementation in route say there you go so Every operation in open stock is performed through API calls through HTTP or HTTPS Adam really strictly argues for So each HTTP request has headers and in the others There is the token in the form of the X of token header. This is the user's token There's also accepts and content type, but I'm not gonna talk about those here There's also the verb of the operation which can be get put post delete or patch or whatever There's the URL Which includes the endpoint ID and the path on the server and For some operation, there's also a body for post food patch and Jason and Nova relies heavily on This body for for performing actions and Adam is going to talk about that after So as we said, we want to do a roll check in the middleware such that an HTTP request is Allowed in only if the user and in this case the token that the user presents to the middleware has that role we also want to Have defaults such that if There is no clear rule for this API than to have a catch all default for this API call Yeah So as we said, this will be done in the old token middleware right now the old token middleware just done It's the keystone server token validation and during this tokens token validation the Middleware gets back the list of roles that this token has on a project so the middleware knows the roles that the user has and It can therefore perform a role check for this HTTP request But we need a way to define which API requires which roles For that we have defined an entity called a route. So a route is composed of a service Which is a string it can be the endpoint type like compute image But it can also be any type of string. This allows for endpoint specific values a verb which is the Normal HTTP verb get post-food, but it also allows for a wild card so you can catch all the verbs and A path which is prior to the variable substitution This will include the version so that you can have a different role requirements for different versions of the API And it will also include a wild card default So you can have a catch all or you can catch only some part of the the API calls and of course the role requirement for this API one role and Why should we be restricted to one explicit role per mapping this is because You can build more sophisticated structures with implied roles and only really one role is required to To be defined so for example with implied roles if So implied roles are a relationship between two roles such that if you have the first role Then you also implicitly have the second role. So if a user has admin then They implicitly also have member and they also implicitly have reader as defined in this diagram here Roll here are kids implied role here are kids should be Directed icicle graphs, of course and the Effective list of roles including the implicit roles is Gotten back to the middle where from the from the token validation step from the server. So this is built in Keystone this is an entity a root definition so For getting information about the servers you would call the v3 servers API the v3 service path with the get and The service type the service type is compute here the role required for this is reader But by doing the expansion of the implicit roles admin and member can also do this operation now How can you find which path and which verb you need how do you construct your routes? And this is all well documented in the API references for all the projects and you can even programmatically fetch it and Update your your routes in bulk and we have created a bulk upload interface for For updating your routes. There is also a single Create read update delete interface for single routes So this is the example for a catch all exam for catch all route which catches all the API operations for Keystone and adds the member requirement to them. This is as close as you can get to the current defaults and This is not enabled by default in the middle where there are configuration options for you to enable it as you can see there checker all in middle where equals true and you also define in the Nova.conf the service type so it doesn't have to be Compute it can be anything and point specific but yeah, this isn't enabled by default you have to explicitly enable it and The spec status was a so the spec for this was approved and it's currently in backlog It's been reproposed for pike and there's discussion going on in that there's Working progress code which is at a high level of completion for this and it's waiting mostly for reviews and That's my part of the talk going back to Adam. Thank you, Christy Okay, so And I know there's gonna be a lot of questions about this kind of stuff later on And that's hopefully we can have time for that here one of the things I know that a lot of people are wondering It's like what is this gonna do to my workflow? How am I gonna be able to take advantage of this? How is this gonna solve those questions that you brought up in the use case up at the front? So one of the big ones I want to walk through is what's gonna be like for creating a new role that third use case So the first thing you're gonna do so let's say we right now We just have admin and member which is the standard for a deployment. I want to add a reader role Okay, this has been a request probably the single largest request for new roles first. I'm gonna create that role Yeah, I see I see some thumbs up good. I'm getting some some positive feedback So I create the role I create a role inference rule that says member implies Reader so now everybody that was a member is also a reader and This means that I can now add a route. I can add a new API mapping to say Some operation such as you know get information about a server That requires the reader role and everybody that was able to do it before because this was under the catch-all as member They still have The the member one explicitly assigned they also have reader implicitly assigned they can do it But that means that we can assign this new reader role to a new user and those users can now do just this one route And then you can remove member from them if they were there before or from other users that should only be readers Remember if you remove remember, but you don't assign them reader explicitly You've just taken away everything from them, but that's a good thing. That's how we wanted to work So you remove member from the unprivileged users and now you've just rolled a new Roll the new role into your system So we have workflow in place and this is You know this is before us coming up and saying here's the set of roles that we're going to provide for you This is a way for us to be able to get a tool in place to move it forward without breaking anything Remember that first requirement? We talk about roles one way that I've often described Thinking about it in way that you could then use this is a three-tiered approach. And again, this is an atomism This is the way that I talk about it. I think it's saying I Can't really claim that I came up with this I stole it from what we're doing in free IPA and so it does map to pre-existing art But at the top level you have your position in the in the organization, you know I'm you know to make the distinction between a program manager and a product manager and a professional manager or whatever it is I'm a database administrator, whatever it is But just because I have the same role as somebody else doesn't mean I'm doing a Or I'm having just because I have different roles from somebody else doesn't mean I'm not doing the same operations as they are So I want to be able to share workflow Across those users and in fact certain workflows share operations. So if we have a three-tiered level Structure like this we can build the structure that maps to how our organization wants to provide access control So I'm gonna keep that up there as a key and walk through an example where first I create an accountant role and At the year-end they have to do a roll-up, which let's just say it's a Hadoop job Okay, so we grant them the an implied role between accountant and Hadoop job and yeah underscores and all that kind of stuff Let's just whatever and then no this is our discourse So we also have a the create server role which maps one-to-one with an operation To actually create a server well in order to do that We have to make sure the Hadoop job has the implied role create server and anytime you create a server You always need to connect it to a network. So We're gonna make a role inference rule between that so just by giving them the accountant Organizational role they're gonna have three additional roles as well and can perform their job This Hadoop job to do their year-end roundup Now our database administrator has to do civil kind of stuff They want to create a Galera cluster So we give them the role that allows them to do that and creating a Galera cluster of course has to create server and look by having that They also have the role to be able to connect to the network Now QA engineer has to do the same kind of thing because he has to test what the database administrator is doing But he also has the additional job the additional workflow being able to push to production Which also needs to be able to create a server So we provide them with that role and it turns out that they also need to be able to create an image So we give them additional role off the purge to production Workflow role that gives them that all this means that we can only specify one role and yet Have everything that they need to be able to do their job and delegate on down and why delegation Well, who does that DBA and who does a QA engineer get their power from in theory? They get it from their manager and their manager should also be able to perform these operations So if we give the manager the DBA role or the manager of the QA role They can now perform all the operations of the people underneath them So these are some of the workflow roles that we've we've discussed again the read-only the auditor role Perhaps you need a special role for high availability, which I believe Heat was the one who asked me for and what's another thing that drove the stuff from many years ago Perhaps I want to be able to Separate the people who can create project networks from the people who can't most of my people should just be connecting to the Networks because they're gonna screw up neutron neutrons the hardest thing to get right when you're doing a lab as I found doing a lab here X number years ago Let's keep that for the people who are smart and then have a distinction between the members and the the people who are network admins and members for a given project and That's kind of like how dream host actually does their stuff by default Or perhaps I want to do the opposite side of that which is a limited member who can only do Some of the subset of stuff and in the extreme as you saw we might want to be able to say one role per operation So anything we want to be able to delegate sans any other power We give its own role and you can see we can how we can do that on a on the fly basis So what about the actions API as Chrissy alluded to before there is an API in? Nova that's a little bit It's a little soap-like including the bad taste that leaves in your mouth In this case it is called the actions API and this you can see the URL there is Next to the the post path colon service server ID and then whatever the action is So we're not gonna get this in the first go around In these the method that you're trying to do is actually in the body of the request We can't do that yet, but we do have a plan that's gonna come as a phase two The idea of being able to specify a body key being the search through a JSON document and do a match And that will be in addition to the rest of the role stuff But getting that right is tricky and I don't want that holding up the rollout of the rest of the stuff So we're gonna phase to that there's there's Things hidden in the weeds there We'll do it once we get verb plus path working alone Okay, so a little bit of Talmud for you You know this is a famous quote if I am only for my if I'm not for myself who will be for me If I'm only for myself who am I and if not now when which as you know rabbi Hillel and you can see our wise men They're discussing it. Well, here's the keystone equivalent of the wise men arguing. This is the keystone middle mid-cycle from a few years back and how I apply Hillel the keystone Keystone has to support open stack first and foremost or we have a vacuum We have stuff that just doesn't work But if we only support keys open stack in its definition today Other services can't just integrate with open stack and we can't integrate with other services We have a problem there as well. And if we don't fix this now, I've been on this project a lot in damn time When are we gonna fix it so With all that I'd like to open it up to questions. I hope you have some I hope I haven't borne you all to sleep and What do you got and yeah, please go to the mics because the recording if you're not speaking on the mic They won't pick you up So I guess you said it but just to reiterate What if there are two roots and one is more specific so to say for you like if you have two matches basically Okay, so you're saying there's two different routes which affected change but basically have the same meaning That's like v1 versus v2 or v2 versus v3 in keystone. You need explicit routes for those You would need to make sure that the roles match you can screw it up by saying you have to be admin Oh, but I can make a member over here. So that's on you I don't have a way to fix that right now That's beyond the scope of keystone to be able to do and if you do want to have them have different behavior But one implies the other then you wouldn't use implied roles to do those those matching between that Does that answer your question? Okay Where are these policies actually evaluated and do you have an idea how it affects performance of the API part? So obviously since we're still writing the code we haven't done much performance testing yet. It's you know, there's this kind of An inference of an implication of ordering there We did a little bit of you know back of the envelope calculation and if you look at how the role the routes are matched It's the same thing that happens in Nova when it has to match the route to figure out what code to call So if that API if that processing is fast enough now, we figure we're gonna be fast enough now there It won't be it's it's a reg edge X match. So it's not trivial, but it's not horrible and it really depends on how many Explicit role roles you have so if you only have one catch all it's gonna be fast If you have a lot of little ones, we may start to see something there, but we won't know until we put in Put it, you know put it under test How much do you need for the projects for this to actually get Working implemented in the individual projects and it will be a keystone only thing for a long time or will actually be Okay, so it's gonna be a keystone middleware implementation Which means that anything that's running keystone middleware will have this by default Which means that all of the other Remote services the Nova's and the glances of the world will get it when it rolls into keystone middleware That's one of the reasons why we want to do this I've tried on other bug fixes to get change into absolutely every other project And it's one of the reasons why there's so much gray in my beard right now. It's Nothing against those things everybody does things a little bit differently It's a whole different process So the goal is to do this within a single project because it is a single idea and so it's going to be in keystone middleware It's not going to be a new component that's rolled in the configuration changes that you saw there There's already an auth token section in their config files This will be adding a new set of values into that section. So there's nothing new that's happening there Do they have to proceed somehow in code in the actual component? Do they have to deprecate their old? No, no, I didn't go too deeply into what's going on right now but very few projects check any role other than admin and One of the assumptions I have going forward is that we're gonna want to leave those admin checks in place Rolling if you have a reference which says Admin implies member Then it will pass this check and go all the way on through anyway So you don't have to do any changes there to keep admins from being able to get on through but If you want to then say, you know, these these apis we want to vary independently. We'll be able to do those as well. So Over time I foresee Stuff being pulled out of the policy files I see the policy files getting really small over time the scope checks are more and more heading into Python code as it is And we've kind of identified on the keystone side We don't want to do those in policy.json. Nova's already said that and in fact Nova already enforces policy in code They just don't tell you that they're doing it. So there should be no changes there until we can start removing stuff It's not a magic bullet, but it should simplify Pretty much for your presentation. I really like your approach in the sentences. It looks like it offers a lot of flexibility and in our implementation, we've actually Gone ahead and implemented a custom read-only role in JSON policy files So it has a lot of shortcomings and one of the shortcomings is that it's hard to see exactly what it's doing It's kind of opaque So I'm just wondering that once this is available. Will there be like a test suite that's available to run along with it? So that we can see, you know, exactly like if you feed in, you know, a sentence of a service and a role and Yeah, what I would like to see would be, you know, what commands can you run? What's the result and does it fail for the right reason when it fails? So if you look at what you're just asking for there, that's a lot like what Horizon was asking for. Given this role What can I do with it? And there actually is a tool right now with policy that might help you out Which is I create a token format, the token response And it can be a canned one and run against the policy file and the rules will tell me true or false Will this pass or fail? So that might be a tool that today will help you In in existing policy But being able to build tooling on top of this to do that kind of stuff is very possible In fact, what we want to be able to do is build a writing to open stack client to say don't actually run this Just tell me which role I need to do it and we should be able to do those types of queries Horizon should be able to use this to dynamically Generate the UI based on the user that's coming on in so then you can come up with a bunch of scenarios and say I want to give A user with this token. Let me see what would happen there So we should be able to build that kind of tool and that's one of the major use cases. We're trying to support question about Only being able to specify a single role and Yeah, in each role and it figures out through implied role in print rows and all that stuff Does that doesn't that mean that you're going to need to specify? Create roles for literally every endpoint if you want to say give an application Access to only one particular endpoint or something like that Potentially we will have an explosion of roles if we get to that point. I will declare victory Seriously, I I think that what we can do when we start seeing that we're having too many roles and start saying On the role API itself, how can we classify these? How can we filter them? We already have a concept called domain specific roles Which are a class of roles that are never enforced in policy. Those are those organizational roles I didn't want to talk to about them too much here. I didn't want to muddy the water In certain cases when you list roles you see them in certain cases you don't and what I think we're gonna start Seeing is a way of saying that three-tiered approach I did there I want to be able to tag things at one of those three tiers and when I do list roles I only want to see at one of those but we're the somebody's gonna need to be able to see all the way on down if we really get Do get to the point where we have too many roles? We've seen the success of the system if it's more like what we have now, which is we have two roles one of which is Treated as a global role and one of which is never checked. I would say we have a broken system So potentially I do want to address one other thing on the one role per Per API as well, which is this is a requirement for something else Which is the fact that we use for net and we want to keep the for net payload from getting Randomly sized and so in order to be able to have a for net token with a subset of your roles specified in there We need to know a fixed size there. Well, the best fixed size we could go with with is one And so if we say there's one role in a for net token Specified as a subset of your total roles We can spend expand the rest of those out with with the the role inference rules So there is a other work that we're trying to get to tie in with here that it works better If it's only one role if it really turns out to be a problem I don't know how having more roles specified explicitly Will get there but we might have we might have problem with the in the number of interim roles in there We might need to do some sort of scoping on those as well But I again big bang up front trying to get a perfect solution. I Think my concern is yeah, every Speaking from an application perspective, right if you want to give give limited Permissions to an application to go, you know auto scale itself or whatever. Yes It sounds like we're going to need to then Go through every project and say hey you need to break down your roles into smaller things so that we can Give an application a limited Thanks, so then it becomes a whole of open-stack scarf Yeah, I think that that's true, but I also think we'll get that information From many different channels. We can't even do it now. We can't even get that information now We might get it by some people doing trial and error and breaking things. We might get it by No, the engineers who just smart enough to say whenever you're doing this You're always need to make this remote call the novel one in the neutron one always need to be in sync And it might be that we say that this role is not an off You know like to a single API we know that these three APIs are always called together These three four APIs are always called together. We'll do it that way We don't even have the tool in place to be able to do that yet So I think if we have some proliferation of roles and then we standardize and we have to help people migrate from one To the other again that to me is a success criteria. We can't even do this stuff now Yeah, this is definitely not great This is sort of related to the one role in the fernet sure Can you say a little more about where the implied role expansion happens and how the middle where Sure, so there actually is a couple options here that we could go with and I went with what we have working right now implied roles been in Keystone for a while and I finally got a commitment from Dean to look at my open-stack client finally getting into the CLI but the other parts of it have been in for like a year and there's a Flag in Keystone that can say expand in the token validation or not and the assumption is at the start of this is that you Can have that on so that when you validate the token if you only have one role assigned but that implies 15 roles you're going to see 15 roles in the token and the Assum the secondary assumption is that eventually that's going to be a problem and we're going to want to reverse things and so when we generate the The the file that Nova calls to fetch its roles its routes from Keystone We'll want to expand them there instead and we can do that as a second rev But right now since we don't have those and we're not hitting that barrier We're going with simple which is it's going to be when you validate the token Assume you have the flags the flag set do it at that time Thank you Yeah, there's been a lot of practical considerations going in here to try to make this something we can actually achieve so Maybe this is like too specific implementation wise But when you do the validation in the middle where the oath medieval when will we do the validation yet? Check policy. It's going to be the last stage. It's going to be after well actually. That's a good question But so that my real question is yeah, is is is there any caching of these things living? Yes. Oh, absolutely. So well first of all Okay, so a little bit of history on this one of the things that we toyed with is saying Why don't you just do it when you validate the token right? Is it valid the token do the route check just pass the round information to Keystone and you get a true or false back? That doesn't work if you cash token validations and they're coming in for different routes So we know that we need to cash a token validation and be able to reuse that cash validation for a secondary step Doing the check Between what you're asking for and the route that I don't think we will cash Because it's it's going to be just as fast to do a lookup It's a it's a hash table lookup in either way and this means there's no persistent storage behind it So there's no difference in in storage requirements So I don't think we would cash that the routes themselves being fetched from Keystone absolutely Those are those are good stored in memcache as well probably use the same memcache instance that we do for the tokens and if it gets You know expired once a minute and has to do one minute per fetch and that turns to be a problem We'll tune that too Okay, but that right now is what we're thinking get something practical that people can use See if we start hitting real limits on it I'm not going to try to guess what those will be until we have something that we can actually make work all That stuff's really tunable Any other questions Any other answers How many people out there think I'm stark raving mad of those people how many people didn't think I was stark raving mad Before I made this proposal So I've convinced you now that I am stark raving now, but you haven't seen me talk before so that's not really definitive Yeah, you guys knew me so you most of you know that I'm stark raving now. Um, thank you for coming for this. Thank you Christy for Helping out with this effort because I think it really would have stalled And the the Massachusetts Open Cloud for supporting him contributing to this as well And with that I'm done go away