 Hello. OK, thank you very much for coming here today. My name is Mikhail Fedosin, and I'm a senior software engineer at Red Hat. And today's session will be dedicated to identity integration between Kubernetes and OpenStack. In this talk, I'm going to cover new advanced features we have added to Kubernetes to improve current integration with Kistone service. And first of all, I want to define what we mean by identity integration, because, for example, authentication is available for a long time. But it's not enough. I will come back to a detailed list of features a little bit later. But now I want to say that the idea was to make Kistone a native identity provider for Kubernetes, like it's done with Nova, or Neutron, or any other OpenStack service. That was our goal. And in the end, I'm going to present a demo where I, as a system administrator, create new users, projects, role assignments in Kistone, and they are immediately applied in Kubernetes. But first, I want to talk about terminology, because in both systems, some entities are called differently. But we should keep in mind that, in general, they mean the same. And in this talk, I'm going to use OpenStack terminology where it's possible, because there are a lot of different ways to start OpenStack Summit. OK, so first, the base unit of ownership in the cloud, where all resources should be owned by a specific unit. In Kubernetes, it is called namespace. In OpenStack, we call it project, and previously, it was called tenant. Then we have several authorization-related entities for our bug, role-based access control, which is the only one available mechanism of authorization in OpenStack, and one of several Kubernetes. First, role that is applicable across all projects, which means there is no specific relations between the project and the role. So in Kubernetes, it is cluster role, in OpenStack, system role. On the opposite, if there is some relation between the project and the role, then we call it local role in Kubernetes and project role in OpenStack. And finally, associations between users and or groups with a role. It is role-binding in Kubernetes and role assignment in OpenStack. If you are familiar with CLI, more precisely with OpenStack Client and Qubectl, then you should remember such comments like OpenStack role assignment list and Qubectl get role bindings. And they return more or less the same information from both systems. OK. I think it's all we need to know, and we are ready to begin. In the beginning, I said that some integration had existed, and that was mainly authentication capabilities. So let's cover what we had before. On the server side, there was built-in house provider that sends tokens to Kistone and returns yes or no. So whether Kistone knows about the user or not. No information like user projects, his role assignments was used by Kubernetes. And now this provider is deprecated in Kubernetes. Then on the client side, there was built-in house plugin for Kistone. And it needs manual configuration in the QubeConfig file, where you need to enter your OpenStack credentials and additional information like Kistone house URL. And of course, it doesn't support environment variables. So you can't source your RC file like you do in OpenStack. And there is another issue with that house plugin that its code was included in Client Go library, which is a backend for Qube CTL. And this is not a good design pattern. So now this plugin is also deprecated in Kubernetes. And finally, the most serious issue, a disadvantage from my point of view that all namespaces, groups, role bindings had to be configured manually by the administrator. So you can imagine a system administrator who needs to create new role assignments in Kistone and then repeat his actions in Kubernetes to create new role bindings. And he needs to keep it consistent. And no need to say that it's not really convenient and highly error prone. And let's admit it was hard to manage such clouds. Now I want to say what we have done to improve the situation. And by the way, all work was done in Cloud Provider OpenStack. This is a subproject under Kubernetes governance, which contains a lot of useful things to help integrate in both systems together. And it's not just about identity. It also helps projects like Cinder or Octavia and many, many other. So take a look when you have time. I will provide all the links later. OK, to solve this, first we worked authentication. And yeah, it's obvious because without authentication we can't talk about identity at all. But it's not the end. Then we added project namespaces synchronization, which means that if user belongs to a project in Keystone there, then there should be a namespace in Kubernetes for him. And of course, it should be done automatically. No manual configuration is required from the administrator. We did the same with role assignments. So if user has some role assignments in his Keystone project, then he also has the same role bindings in his Kubernetes namespace. And of course, it's also done automatically. And finally, we worked client password authentication. Now it is done with a standalone binary application that is used by Cube CTL. And this application supports environment variables. So you don't need to enter your login and password in Cube config file anymore. You can use your OpenRC file if you want. OK, now let's see how we did that. In general, it was done using authentication webhook. Kubernetes provides several mechanisms of authentication. And one of them is called webhook. Webhook is just a callback when Kubernetes delegates its authentication responsibility to a standalone application. And it's done by HTTP. So when Kubernetes API receives a request from a user, it asks the application to authenticate him. That application, in turn, sends his token further to Keystone. And depending on the results, the application either rejects the request or provides information about the user to Kubernetes. But user authentication is not the only one feature of that application. It also supports AusData synchronization. Now we support project name space slash name space synchronization and role assignment slash role binding synchronization. So when user sends a project scope token, and this one is correct, Keystone responds with information about the user. It includes his project ID, project name, overall assignments in that project, information about domain, and so on. Then that application, before responding to the API, creates new entities in Kubernetes. It is configurable. So administrator can define what actually he wants to synchronize. So he can synchronize only projects with name spaces or include role synchronizations as well. So that application uses the received data to create or remove new instances in Kubernetes. And now to see the full picture, I created a small sequence diagram that represents the workflow, how it's done. The workflow shows a user who wants to create a pod in user project name space using his Keystone credentials. And there are several prerequisites. So user belongs to the project, user project in Keystone. So administrator has already done that. Then user has a role assignment member in that project. Also, administrator needs to create a cluster role in Kubernetes, because we know Keystone doesn't contain information about roles, just about associations between roles and users. And yes, there is no name space user project or any role bindings in Kubernetes yet. So here is the diagram. The workflow begins when user sources his Rc file that contains OpenStack environment variables with his credentials and additional information. When it's done, he enters a command to create a new pod in his name space. And again, by this time, name space doesn't exist. But before sending a request to Kubernetes API, KubeCTL tries to get a token from Keystone. If credentials are correct, Keystone responds with a token. And only after that, KubeCTL sends a request to Kubernetes API and includes the token there. Since the API is configured to use authentication webhook, when it receives that request, it takes a token and asks webhook application to authenticate the user. Webhook application, in turn, sends it further to Keystone. And since the token is valid, Keystone responds with information about the user. And now we have this interesting part. This one is optional. But here, webhook application creates new entities in Kubernetes API. First, it creates new namespace based on project ID. Then it creates new role bindings based on role assignments. And when it's done, it responds that user is authenticated. Next step for the API is to authorize the user. So it asks the airbug module whether user has rights to create a new pod. And now, by this time, the namespace and role bindings are there. Airbug, using his own logic, validates that user has rights to create a pod and says, yes, to Kubernetes API. And then Kubernetes creates a new pod and returns information about this pod to the user. So it seems easy. There is no magic here, but it works. And now, I want to present how this actually works in reality. I want to show you a demo. In the demo, I want to present a workflow when user wants to create a pod in his namespace, app using his Keystone credentials, which is more or less the same as I showed in the sequence diagram. But this one is reality. It's not a picture. There are several prerequisites. So I don't think that we need some huge deployment. So I created a small deployment with Minikube. I deployed a Keystone server there in a container. We don't need full OpenStack deployment for this demo, just Keystone. I started Webhook application in a pod, which is pretty easy. And then, in Kubernetes API, I enabled Webhook authentication. And also, I created two cluster roles. First one is user, which allows users to read information about pods, deployments, but do not create, modify, or modify or delete them. So just read only. And then developer. Developer is allowed to create new pods, deployments, and see sensitive information like secrets. OK. And now, user wants to create a pod in the namespace app using his Keystone credentials. OK. This is Minikube status that it works. By the way, so far, I haven't created new roles. So let's do this. kubectl create user, OK, and developer. OK. If you want to see how this role looks like, we can actually do that. So this one allows to get, watch, and list deployments, replicasets, and pods. This is user. And developer. Developer allows to do almost everything here. All right. Here, I have a Keystone managed console. I want to create new user here. OpenStack. Let's create project first. Create project. Let's call it app. Oops. Project create, of course. I need to source my RC file. It may take some time. OK. Now we created new project. And here, we see the ID of the project. This will be the idea of our namespace, new namespace that we will create in Kubernetes. So let's keep it. Then I'm going to create several roles. OpenStack, role, create, user, developer. OK, and now I want to create a new user. Let's call him John probably. OpenStack, user, create, dash, dash, password prompt. I hope I did mistake. Password will be password. OK, we have a new user. And now I want to assign a role user to this user. OpenStack, role, add, dash, dash, project. App, dash, dash, user, and user. That's it. We have done. Let's come back to the kubectl. Right now I'm using administrator context. So let's make sure that no namespaces are here. kubectl get namespaces. Oops, we have one. Anyway, it's different from this. This starts with 0, 8. Next step for me to switch context, to use new context. And I'm going to show you what we have here, kubectl config. In this config, I call it user mini kubectl. We have information about the user. And this one has two obligatory parameters. First one, plugin is called exec. Then you need to specify API version. And then you need to specify the pass to external binary application that will be used to get tokens from Keystone. So you don't need to enter sensitive information like password here. You may, but you may not. You shouldn't. OK, and for namespace. Here is the namespace we will use. To switch context, I'm going to use this command. Now this is called user mini kubectl. I think I did everything. And now we need to source our RC file. Let me show you how it looks like at user RC. So we have our URL. We have project. We have our username. We have our password here. And again, it's not necessary to keep it there. Because if you haven't entered before sending request, that application, standalone binary application, will ask you to enter it from comment line. OK, so let's source it. OK, as you know, as you remember, I created just one role assignment that allows users to list pods but not to create them. So now let's try to create a new pod. I hope everything will be OK. Probably I have a comment here. This is one. You see, here are from server. Deployment is forbidden. So we do not have permission. User John cannot create deployment apps in the namespace. But we haven't created that namespace. It was created automatically. And that's the magic. OK, now let's go further and create new role assignment for the user developer. Now user John has two role assignments. Now let's try again. Create new deployment. OK, let's repeat. Yeah, it may take a couple of milliseconds to do that. But anyway, you see, we created new role assignment in Keystone and we use kubectl and we don't need nothing else. To make sure that everything works fine, I'm going to list secrets in our namespace. Get secrets. Yeah, because we have this role binding developer, we are able to list secrets. Now let's remove that role in OpenStack. Let's try it again. Nope, you can't. And it's done almost simultaneously. So that's it. You do not need to synchronize all those entities manually. You just need to create them in OpenStack and they are almost immediately available in Kubernetes. Now I want to show you the links. First one, here is the slides so you can watch them. And all the documentation, how to deploy that web application, how to enable our data synchronization, and how to do it on the client side. OK, I think that's all I want to tell you today. And if you have questions, you're welcome. To be honest, I don't know, but I think it should work at 1.8. So when WebHook is available, I don't remember the exact version. But actually, yeah, we are working on that. And of course, we are going to add kind of additional mapping. So for example, right now, you can disable some projects from synchronization, kind of blacklist them. Yeah, and if we are talking about direct mapping, we are going to add that possibility to create different roles, different role assignments with different IDs than role bindings in Keystone. So there is no need to have the exact match. Right now, it's not implemented, but I think it will be available in a month or so. In that case, user won't be authenticated. Yeah, it still will be available in Kubernetes. Currently, yes. Please. I was asking about why we need that WebHook application. Because Kubernetes requires a special API that Keystone doesn't provide, so we need some middleware for that. It can be done either on Keystone's side, and implemented in Python, or on Kubernetes API side. But you should understand that there is no actual network connection between them. They are running on the same host, so it's just like inter-process communication there. But yeah. So it shouldn't take a lot of time. But answering your question, it happens because Kubernetes requires to implement the API they need. OK. There are no more questions. Thank you very much.