 Good morning. Welcome to the OpenStack Newton Summit. I hope you enjoy this event as much as we do. I'm going to present our talk called Flat No More, Harakumotu, Tennessee, and Project Zecnia as Domains in OpenStack. Just to introduce us, I'm a Hickey, he's Andre, he's Hayudo. We're all from the Federal University of Campina Grande, and we work at the Distributed Systems Laboratory, which you may have seen at the keynotes, so Hayudo and I are software engineers at the university, and Andre is a professor. We'd like to go through this agenda at this talk. First of all, we'll introduce Keystone, and then Hayudo will talk about Harakumotu, Tennessee, and asset quotas, and then I'll come back to talk about projects acting as domains, and finally, Andre will summarize this and talk about the next steps of this work. Just a quick overview of Keystone, which is the OpenStack component responsible for identity management. Keystone deals with authorization, authentication, and audit, and among its features, there are support for multiple identity providers, which he enables throughout Federation. And Keystone also have support for very known tools at the market, very off-backends and framework, such as LDAP and WALTH, and the very big feature of Keystone, we are interested here on this talk, is multi-tenancy, which we define multi-tenancy as a single instance of a software that runs on a server and serves multiple tenants, and here we define a tenant, a gender term, as a group of users that share common access with specific privileges to some parts of this software instance. And just to start here, just a little bit of history of how multi-tenancy has been evolving in OpenStack, at the beginning of it, we had from Austin to Cactus a very simple multi-tenancy model. There was one user, one tenant, and at this moment a user could not make an operation in a different tenant than the one he belongs to, so it was very simple at the beginning, and here Nova was still the responsible component for authentication OpenStack. Then at the Diablo release, we have introduced Keystone, which together within, we have made the API v2.0, which in fact was the first Keystone API, and this beginning of Keystone still hold the same model of one user, one tenant, which still was pretty simple. At this beginning, we still had a very simple role-based access control model, which were hard-coded to admin and member operations, so depending on the role your user had at the beginning, you could make an operation on each of those operations sets. And the biggest step on multi-tenancy in OpenStack was made at the Grizzly, so in Keystone, we have introduced the v3 API, which introduced the concept of domains, which are now the containers of projects and other OpenStack resources, such as users, and at this point, tenants are no longer called tenants and we call them projects, so, and these users are now part of the domain and not anymore of the tenants. And with these new introductions, here we have a more complex multi-tenancy model that we define it as one user, many projects that is possible due to our new role-based access control, made via a policy file, which you can do an operation in a given project depending on the role you have on it, so that's a very big step for OpenStack multi-tenancy. And just a known issue that we had at this beginning is that the admin role in Keystone in OpenStack is global, which means that if you have the admin role on a given project, you are able to administrate the whole cloud, this was a very known issue at this point. And in NiceHouse in June the releases, we had the first efforts to eliminate this global admin role, so, and together with that, we had also great improvements on the domain usage with introduction of features like domain-specific backhands and better domain policy enforcement. And finally, in Kilo, we had another big step in multi-tenancy, which was the introduction of hierarchical multi-tenancy, which I will explain right now. Hello, everyone. What was our motivation to implement hierarchical multi-tenancy? The organization are naturally hierarchical, so in every company, we have departments, sub-departments, sectors. So how to represent our hierarchy? On this sample, we have a sample of our cloud in your anniversary. So we have the UFCG cloud, and we have a couple of labs for LSD, SP Lab, and Analytics, and so on. And inside LSD, we have the Fogbowl project on our lab when we have another project for big data. And more specifically on OpenStack, we have a Keystone team, Iron Team, Monask, and so on. So how can we represent that on inside OpenStack? We need to make our work around in a flat way today, so we need to create LSD domain. And inside this domain, we'll create a list of projects. For every project that we have on our lab, we need to create a list. So we will have the LSD Fogbowl, LSD Big Data, and so on for OpenStack, Keystone, Iron, Monask. And we had a couple of problems with that work around. For example, how can we manage the access control for a lot of users when you have a big list of projects inside a domain? And how can we organize our results? How can we find some instance after that we have a big list of projects? So we implement HarkMood Tennessee when we provide the ability to create sub-projects. So now we can create a project OpenStack inside our LSD domain. And inside this project, we can create sub-projects for Keystone, Monask, and so on. So now we can represent our departments and sub-departments in a Hark way. And you can do the same thing for the projects that we have in our lab, for Fogbowl and so on. And to make this, you just need to add a parent option on the OpenStack call for create a project. So you can say, I will create a sub-project and this project will for this parent using the parent name or you can retrieve the list of the parents using the parent option. And you can use the project show to retrieve the sub-tree and see the sub-project for some project using the sub-tree option. So it's a base cooperation is related to HarkMood Tennessee. And how can we improve the access control now that we have a hierarchy? So imagine that A-Hick is our project manager for every OpenStack project that we have in our lab. So if A-Hick is the project manager, we need to repeat this grant for every project that we have on the sub-tree. So we need to make the grant on the OpenStack and for Keystone, for Monask for every project that we have. And if someone create a new project, we need to grant this role again. And if someday A-Hick left our project, our lab, we need to delete all the grants for these users. And if we forgot to delete some grants, we will have a security issue when A-Hick keep having access for our cloud and this is a real problem. And another way, Andre is our professor, so they need to have access for all the projects in our lab so we can grant the role we need to grant the role today for every project that we have in our cloud. So this is a lot of entry for the same user. And our deal was implemented hard to inherited role assignment, sorry. And on inherited role assignment, we can grant the role for A-Hick on the OpenStack level and you'll say, hey, this role will be inherited for every sub-project that we have on the sub-tree. So A-Hick will have automatically access for Keystone, for Monask. And if someone now creates a new project, A-Hick will have automatically the access for this project. And you can do the same thing for Andre. So if you grant a inherited role in a domain level, these users will have access for every project on this domain. So we reduce the entries for these users and make easy the control for the operators. And to make this happen, we have a flag inherited. When you create an now assignment, you can say, I will create this assignment for this user on this project or domain, and this role will be inherited for every sub-project. And to retrieve these informations, we can use the inherited option on the role assignment list. When you can retrieve, you can see the inherited role assignments and you can have the option to see only the effective assignments. And to delete the assignments inherited, now you have the option to use the inherited flag on the delete operation and you can delete the inherited assignment. So imagine that you want to set up the hierarchy that we use as our example. So first of all, you create the LSD domain and after that you can create the OpenStack project inside this domain. And you can use now the parent option for create Keystone and Monaska project and you say that you pass the parent option using the parent name on this case, OpenStack. And after that, you can create the user and Hick and we will grant the project manager role to Hick and this role will be inherited for every project inside this sub-tree. And how can we handle with the resource? Now that we know how to control better the access control, how can we handle with to enforce quota for these users? In the current quota implementation, there are existing drivers handled with quota when projects are independent. So a quota for a sub-project can exceed the parent quota and their project manager cannot control the sub-project quotas. So this is a problem. And the other thing is that other servers do not support domains. So we can't, there are no quotas for domains. And if a project manager wants to handle with their own users, in other words, we need to create a domain for this project manager, there cannot control their quotas since we don't have quotas for domains. So with the current implementation, we can have the situation when the sub-project Keystone can have more quota than their parent. And on this case, every project are independent. There are no relation between us. And we don't have quota on the domain level. So we can set the quota for LSD on our cloud. What's the idea? The idea is create a new driver to represent nested quota and works with nested projects that will allocate parts of the parent's quota to the sub-projects. And now the project manager will share his quotas. In other words, there is plenty there resource among the subtree. And a quota for a subtree will always be lower than their parent. So the previous behaviors that we had when Keystone have more quota than their parent, now we will not be allowed anymore. So Keystone needs have lower quota than their parent and we are allocating the quota for the OpenStack level to the sub-projects. So the OpenStack quota will be splitted for the sub-projects. And now Eric will explain about the visibility. So as Hilde said, we're not able to set quota for the domain which in this, in our case, we cannot set the quota of the LSD domain. And that raises some issues regarding the, what's the visibility of the cloud admin should have in resources it creates. And the first scenario of it is where the cloud admin delegates the control for the project manager, which may be me in our example. So here the cloud admin creates a domain which is in this situation a black box and then he won't be access to the domain itself because he delegates all responsibilities to the project manager. So in this example here, the project manager is responsible for creating users, projects, hierarchies and finally to set the quotas inside these projects. So the bad side of it is that although the cloud admin delegates the full responsibility to the project manager, he cannot control how the project manager distributes those resources. So here in our example, the project manager creates two root projects that are independent on each other and then he can set a huge quota for each sub projects of that. So this is not a scenario we want. So in the next case here, we have the cloud admin controlling all of those resources inside the delegated domain. However, this domain will not be a black box anymore here. So the cloud admin is able to see all resources there are in the domain. So here, although the cloud admin is able to have a good control of those resources, here the project manager will have be very dependent on him since you need to contact the cloud admin on every single operation, such as the project creation quota setting that the project manager may need. And to gather with that, the domain, the cloud admin is very strong here. So this is not a scalable approach which in a, especially in a public cloud environment where there are lots of clients, lots of domains, the cloud admin may have an unfeasible task. And we had an approach to solve those problems. So just the statement of this problem is, how can we give project manager the control of their resources without giving them all resources of the cloud? And to solve this small problem here, we have implemented projects acting as domains, which, as Haile said before, this is our representation of our cloud. At the top here, we have the LSD domain which has two root projects called OpenStack, called Fogbol, which have their own children. And with our new implementation on projects acting as domains, we change that here now. LSD is a project that acts as a domain at the top of those hierarchies. And that's also happy with analytics here. And the OpenStack that was once the root here is not anymore, so it will have LSD as the parent. So just better explain here what has exactly changed is that LSD is now a project, but also a domain. And this new kind of project is internally represented with the EastDomain flag. In this case, it's set to true. And this LSD domain slash project is still the container of users and other projects. And the big advantage, the big win of the situation is that the cloud admin is able to set the quarter for LSD, which is the project. So other services such as Nova and Cinder cannot handle with domains currently. It looks like it's not something they want to do. So now they only need to deal on the nested quarter example with a project, which is the root is LSD. So that change will be totally transparent for other services and will be able to set the quota for a domain. And with that, the project manager will be able to distribute this domain quota across history and will create subprojects and split this quarter for subprojects. And just here, I want to show how can we create a hierarchy, as Hyde explained, in the projects acting as domains approach. First of all, we just use the project's API, which the only difference here is that we need to pass the domain flex set to true, which will generate a new project called LSD, as we stated here, which will have the domain flex set to true. And this new kind of project access domain have no parent ID and no domain ID, as it's indeed the domain. And an alternative step to do this operation here is using the domain API, which you can do a simple post to the domain, or either you can also use the CLI, creating a simple domain here. And these operations in projects acting as domains will create the same entity as you have just created, which is the project with no parent and no domain ID. For a while, here we have the domain and projects API, projects acting as domains mirrored. So if you do a GET project on the domain API, we will have this project. And this also works for the update and delete operations. So this is what we are keeping for a while, although we recommend that in the short future, you start using only the project API because due to a better user experience, and there are also some thoughts of the Keystone team that we intend to deprecate the domain API. So we strongly recommend you that you focus on the project API. And just going on with our structure here, we are creating here the OpenStack project, which is now a children of the project that acts as a domain LSD. And we have here the flag is domain set to false, which is optional since this domain is false by default. And we pass here the parent ID, which is the LSD's ID. So here, this new project is created with LSD as a parent, and LSD is also the domain of it. So we can also do this operation at the old-fashioned CLI. Here, we can have a project created OpenStack. And just a reminder is that the Keystone client in OpenStack CLI does not support this domain flag attribute yet. So that's something we intend to do at Newton release. And then we can create this that's all, the doll have changed, which was this project before was a root project, but now it has a parent, which is the new project that acts as a domain. And finally here, is creating another child project here. We created the Keystone project, which is a child of OpenStack and is also inside the domain LSD by just passing the parent ID. It's good to know that at this point here, the domain ID is optional and is even redundant as we can always imply the domain ID from the parent. So that's a more comfortable way for dealing with hierarchies or for dealing with trees at a whole. And finally, can also create using the CLI as I said before. So now Andrea will give the summary and the next steps for this work. Thank you, Hiki. So Yorakamuti Tendency is something that helps you manage your cloud because it helps you to reflect your structure also in the cloud. And it's basically made possible because of a combination of several features. The first one being the ability to create projects that have relationship with other projects like parent-child relationships. And using that relationship, you can build hierarchies and then the next point is that you want to assign roles to users using that hierarchy. And with that, you cannot, you can directly assign roles to users that consider the hierarchy and you don't need to pick individual project user assignments, as I said. So if you have a huge cloud, you don't need to assign the role in all the projects, you can just use inherited roles to assign the roles in the subtree. The next important feature is the managing the resources using this hierarchy. And that's also a very important feature. And the final one is the ability to delegate the control of a subtree. So you want not only that the project manager can handle the resources and the sub-projects but also the users. And this is what we call the reseller use case is the ability that sub-project will also manage the users. For example, in our hierarchy here, now we could have that the Fogbo is a project acting as a domain, but even could have a different backend, a different user backend. And this is not yet possible for the sub-projects, it's something that is coming up next. And there is where feedback, use cases and contributions are always welcome. Okay, and this reseller use case also enables you to completely delegate the control of a part of your project, right? So if you can manage your users and you cannot look inside, then you could even resell this part of the cloud. There will be some discussions about the reseller use case in this session here. So the new features in Keystone and it will be in the Hilton Saloon E on Wednesday. Okay, and some of the topics will be the reseller use case and the project three operations. The next thing that is also very important that there is still work to do is the nested quota. So you can do a lot with your project today, especially regarding the organization of the resources in the hierarchy, but the quota is also very important and it's working already in Cinder and it's under reviewing Nova, but there is also some things that could be done to easy the implementation of this in other projects so that the projects could use a single implementation instead of each one implementing their own version. And the goal is then to build a quota library for all the services. There will be a cross project workshop around that and it will be tomorrow at the end of the afternoon, also in the Houston Saloon D. So as this is, there is some work to do here. You are welcome to contribute with the code, share use case, to review code. There is even a channel for this specific topic. CERN and Yahoo are the leading partners on this one. Finally, there are some forts on the user experience front. So the expected is that when you create a project, you have something like this parent project dropbox where you could select the place, the new project in your project hierarchy. And then it will also be expected that you could select the project using hierarchies from the dropdown on the top of your horizon screen. These are working progress things. There's a lot of activity going on in the envision. Again, feedback and contributions are welcome. So finally, this work, although Enhiki and Hayudo and other guys from the lab made a huge work to make this happen, of course, this would not be possible without the heavy collaboration with the Keystone community and with the companies they belong to. So Red Hat, Hewlett Packard, Yahoo, IBM and CERN have collaborated a lot with us in these features. We discussed it today. And then I would like to thank you all for your time and thank also to the members of the team that could not come to Austin and that left the team recently. So if you have questions, you would be very welcome. So I'm Adam Young, Keystone and these guys have been absolutely fantastic. I guess I should say Adam Young from Red Hat works on Keystone. So there's a couple of details that are probably worth pulling out here. On the delegation front, the ability to do reseller where you say you can manage your own thing, there's a long outstanding bug where admin somewhere is admin everywhere. And we'll be just talking more about how we're tackling that as well as some of the other roles stuff in a talk. It's me and Henry Nash from IBM tomorrow around for something-ish. I should have this more as now, but I'll be there, so hopefully you'll too. But that very much ties in with what they're doing here and actually we'll tie in with the other role work that we're doing with the HMT and stuff along these lines. Yeah, thanks. Question. Yeah, so when we have integration with let's say a third party identity manager, in many cases we have centralized identity managers and we have the backend integration to that. How do you see this multi-level hierarchy and the excess control in that hierarchy for user identities that don't leave within Keystone because those user identities are coming from some central item system. So basically the backend of the users will stay the same that has been handed today. So the ability to put these identity management systems in the sub-project will be possible soon. Today this is only possible for the root project, which is the same way that you do- The domains. We can have these different kinds of backends we use in domain-specific backend. So we can have a backend for domain. What we want to do as our next step is have one backend for level, for sub-project that act as a domain. We are just extending these domain-specific backends for the sub-tree. So today we have this ability, but for domains. So if I have more than one hierarchy in a domain, I can just have one backend. Oh, sorry if I misunderstood your question. No, I think you've kind of answered that. It's basically what I'm hearing is that you have to configure one backend per level of the hierarchy. Per-project, yes. Per-project, yes. So I'm gonna go with the assumption though. What we're seeing is that they're not actually wanting to expose these as LDAP. Is that correct? It's more likely through Federation Protocols? Yes, yes. So with Federation, what you do is you map a user from whatever remote ID, the protocol, and the big two protocols really well supported right now, SAML and OpenID Connect, to a user. And there's other talks that we talk about shadow user and advances in that that HMT will be able to make use of. So the idea of Federation as a part of Keystone is really becoming a first-class citizen. And so that's what you would use there. What you get from the identity backend, which is what you get from Federation, is the users and the group abstraction. So if you can represent it as a group, and it doesn't necessarily have to be like an LDAP group, it just has to be something that we can map as a group once it hits. Keystone has to be in that assertion. Then you can use that to do assignments within whatever domain. Just because a user is in one domain. Say Red Hat had a domain in their cloud there, but they had a projector in the LSD domain that they wanted me to have access to, they can assign my user a role in that one. So assigning a role across a domain is different from what domain owns the user database. Okay, so that stuff is all very well-supported Keystone. Okay, so it puts some requirement to put some mapping of the group on the central identity. Exactly, and that is not a light touch. Doing, adding an additional identity provider, adding additional protocol, because it's done at the Apache layer, it's something that requires a little bit of, it's not something you can just do using a Keystone API, but it can be done. Okay, thank you. Hi there, I have a quick question. It seems like once you add the ability to assign hierarchy to projects, and given that users have visibility at the project level, the idea of domains kind of goes away because a domain is really a top-level project. Is that anything that's envisioned in the future, or do we need to keep the domain concept? That's related to what you said. So you want to take this one? I don't see it like that. I see that the concept of domain will be there, I guess, forever, but will be mapped into a project. So this project will keep doing the same work as the domain does, if that answers your questions. Oh, I understand, but now that you have the hierarchy, it almost seems like you could just have one kind of container and it doesn't necessarily have to be called something else. For sure. Sorry, I'm not trying to get you there. You're 100% correct. When somebody first loaded the idea of domains, I said, now, why don't we just make projects hierarchical? And I lost. So I'm going to win in a lot of terms, but I would have a lot of pain. I think you're right. If we continue with domain-specific backends, which we need for LDAP, we'll probably have the concept of domains around for that because it is a top-level bucket for users as managed by Keystone. With Federation, you need that last, especially with Shadow users. And with Federation, we tend to map all users into a single domain, just a Federation domain. If you don't do that carefully, then you can get them to step on each other. That's why the whole Shadow users, I keep bringing that up again. And in that case, then we'll have one big bucket for Federation. But even then, you might decide that you want to divide things up. So domains will be a namespace for users, mostly. And on the project side, you're right. It doesn't need to be there, which is why projects as domains or domains as projects is so powerful. Okay, thank you. Thank you very much as well. Mm-hmm. So you've restricted the focus on Keystone-ish objects like RBAC and quotas. And that's cool, I get that. But we have other things that we think of sharing. For example, security groups. I would really like to be able to share security groups. Have you thought about whether the structure you've created here? I understand that from a code point of view, there's a whole lot of mess to make that happen. But structurally projects and sub-projects, do you see them expanding out potentially into other entities and resources? I think that first of all, these projects keeps isolated to the other servers. So the hierarchy see that we are showing here, it's only see by Keystone. And now on seeing there, using the nested quota, but for the other services, they are isolated. And we are just, we start doing this, but we are a little afraid to create a Pandora box on everything will be nested in hierarchy. So it's something that I agree. We are looking for new use case, but it will need to be a little aware to don't open a Pandora box for a nested set. That applies for many things. So do you want the images to be accessible from children projects and all these things? So this is something that should be evolved with careful so that you don't mess things. So just as an example, domains there are since Grizzly and other services do not know yet. So imagine hierarchy multi-tentancy, which there are since kilo and we're starting out to implement it cold. So that might be a good next step for sure. Yeah, I just have question regarding this role management. So first question is just want to make sure my understanding is correct about the role management and how we are going to assign permission to the role. So it looks like we have API allows us to dynamically create a role. But in the meantime, the real permission for each role is managed by this policy file, which means I have to manually change policy file to give the real permission to a role. So my understanding is correctly. So my second question is for me, this is halfway down for the robust API. What's our plan to address this issue? Come to me. Yeah, you're correct. So if you come to my talk, I will continue this conversation. It's a tough problem. We looked into doing a dynamic policy distribution and kind of this very problem that we just discussed about the lack between a feature going into Keystone and other products being able to consume it kind of made that untenable. And the other response that we got is that policy being perceived as a configuration type thing, they want to have the configuration management tools manage it. So I personally have been focusing on triple O's approach to this. And talking with the keystone poppin and another poppin module team to come up with a way that we can do exactly that. Distribute policy in a lightweight way across. And if we can solve it for triple O, I'm pretty sure that we can say, this is what it looks like with this content management system, here's what the deltas are going to be. We know that we're doing it with Puppet here and we can do it with Ansible or whatever. But it's the fact that it's the cloud management integrated mechanism, that it's not Keystone's ability to affect change but on these files, that really drives that conversation. Thanks. With the subprojects and being able to have project managers assign quotas and stuff, if I am a user who gets added to multiple groups, do I have to source my credentials for each different group for those quota assignments or do I just get them all for me? So if I'm a member of three subprojects, is the quota that the project manager assigns to me just mine or is it, like, do I have to source three different Novars? There's two kinds of quota. There's a quota for user and quota for project. But what we are handling here is quota for project. It's more, the quota for a project will be shared for all the users inside a project. I have just questions to clarify for myself. So the hierarchical multitenancy is not upstream, right? Yeah, it is upstream. It is upstream in Mitaka. Kilo. Kilo. So there are a few changes, but yeah, the basics part is there. So the things that you described, it is available in Kilo already. So the CLI, the CLI examples that Hayudo and Hiki showed are OpenStack CLI. So you can go there, OpenStack, project and create a hierarchy today. It's not available on Horizon yet, but those are our next steps. So using multitenancy makes Horizon useless? No, it's just that you cannot see the hierarchy. You cannot see the hierarchy. Still, I can't walk into my project. Yeah, the same thing. You just cannot assign roles, inherit it, and it cannot create sub-projects in Horizon. So I just asked. Last time. OK, so that's a tricky question here. If you think that through, it's like, well, how do I find? Or if I have two projects that are named the same thing, because they're in different hierarchies, how do I know which is which? And there are a lot of issues with naming. There is one config option in Keystone, which I would recommend people thinking about doing HMT set, which is that it's strict checking on the name of a project. And that will allow you to build that project as a sub-url. I don't know that it will actually do anything today, but it means that as you move forward, you're not going to be putting non-URL safe characters into your project names. And thus, when we get to the point where we want to be able to say, lsd.keystone versus redhat.keystone or slash, to define the two different projects, we'll have a way of being able to build that user interface on out. So definitely make sure you set that. It will check back any projects. If you try to do a create project where that project name has a non-URL safe character in it, the create will fail. So I would recommend that you do that. And the last question, is there any easy integration with Cilometer? I mean, how can I, for example, ask Cilometer about? I think that you can use the case that we showed when you can retrieve the sub-tree or see the parent. And I can, for example, see all the sub-tree that I have. And I will have a list of these projects. And I can use on Cilometer to see how the hierarchy I use in these resources. I think that is a simple integration with the API that we already provided. Cilometer does not know how to multi-tenancy yet. So you cannot say, give me the amount of CPUs used by this hierarchy. So that's not something we have so far. You can do it by some simple hacks, but Cilometer does not natively support it. OK, thank you. Good question about the project admin privileges. So now I notice that in Kilo, for example, if you are an admin user from project one, you can always see the resources in other projects. If there's some problem we are trying to solve in Keystone side, or it has to be done by other projects like Nova, Singular, or something like that. It requires a magic keyboard because of that. Maybe you want to fix it. But yes, it's been handled here. It's already been handled in Mataka, or it will be? No, it's been previously handled. So there are some steps to it. OK, is there any place I can track those progress? Yes, there are. Adam? Your father, he will explain everything. He probably has a talk for that. Thank you. Thank you. So thank you, everyone.