 here. I'd like to thank everyone who is joining us today. Welcome to today's CNCF webinar, Cooper-Meddy's RBAC 101. I'm Kristie Tan, Marketing Communications Manager at CNCF. I'll be moderating today's webinar. We would like to welcome our presenter today, Oleg Chinkin, CTO at Kubler. A few housekeeping items before we get started. During the webinar, you are not able to talk as an attendee. There is a Q&A box at the bottom of your screen. Please feel free to drop your questions in there, and we'll get to as many as we can at the end. This is an official webinar of the CNCF, and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of the Code of Conduct. Basically, please be respectful of all your fellow participants and presenter. Please also note that the recording and slides will be posted later today to the CNCF page at cncf.io slash webinars. With that, I'll hand it over to Oleg to get started on today's presentation. Oh, thank you. Thank you, Kristie. And thank you for the introduction. I'm going to talk about Kubernetes RBAC today, and so why I am talking today about that. I'm a CTO of Kubler, and Kubler is a company who builds a Kubernetes Enterprise Kubernetes Management product called Kubler. So intended to essentially bridge the gap between enterprises and cloud native technologies tech to make sure that this historically natively open set of technologies, relying on open technologies, can work within more restricted, again, historically, sometimes more limited enterprise environments and bringing the product which caters both to developers and operations teams within an enterprise. And so that involves essentially providing a centralized Kubernetes Management panel at Flame and centralized monitoring, logging, various security and governance requirements. And on our, so through our history of developing this product, what we found is that in many companies, well, having a reliable enterprise Kubernetes Management product is just half of the way, if not a smaller part. Many companies after they start working with Kubernetes also struggle with defining new practices around this new technology stack, and adjusting their practices and processes to the technology and learning essentially how to manage how to manage these new modern applications within their policies and requirements. So, and so what we found is that RBAC is one of the areas which causes some confusion. So not that it's so much complicated, but people look for guidance on where to start, what's available in terms of capabilities, and how these capabilities can be used in the real world. So, and oh, let's try to interrupt. Sorry to interrupt you. Could you just speak a little bit louder since folks are having a hard time hearing you? Okay, I'll try. So, okay, thank you. Is it better? Yes. Okay. That's great. Thank you. So what we'll do today, so we'll essentially go through all capabilities of Kubernetes RBAC, focusing on the ones that are probably most useful from practical standpoint, and that includes what RBAC is, various ways to authenticate a user within Kubernetes clusters, and several different methods Kubernetes can use to asurize different actions within Kubernetes clusters of which RBAC is probably the main one. So we'll, you'll see that we'll spend probably a bigger part of our time on authentication options, a little bit less time on actually RBAC management, because surprisingly it's it's a simpler topic. And we'll talk about a couple of practical use cases and how you can start managing RBAC in Kubernetes clusters. So RBAC, essentially RBAC or access control within a Kubernetes cluster is a way to define either through extensions or declaratively. So which users can do what with which objects. So for many of you who are familiar with Kubernetes, so you understand that there are different resources and subjects. So for those of you who are not, I'll give a quick introduction. Essentially, Kubernetes is a container registration platform, a tool which on a high level from the outside gives you a single API endpoint through which you can manage your containers across multiple distributed physical virtual nodes, virtual machines, physical machines, etc. So that API is follows standard rest conventions. So everything you manage within Kubernetes is managed as a number of resources, objects that exist inside of Kubernetes master server, and that are available to you through API objects like pods, nodes, config maps, secrets, deployments, etc. So I'm so sorry to interrupt you again. I think your slide deck might be frozen. We're just seeing the first title slide right now. Okay, that might be the case. Sorry about that. No worries. I apologize. So it looks like it was really paused. So yeah, just quickly scan through the slides I talked about. Sure, no problem. Enterprise requirements that we identify as once people follow in larger companies. Kubler high level structures, I mentioned what Kubler does and agenda of this demo. So I talked about that a minute ago. So and now we are on this slide talking about how API of Kubernetes is structured and what are objects and various operations on those objects. So as it is a standard rest API, so you have resources like pods, nodes, config maps, various operations on those resources, which, well, are expressed in form of HTTP verbs that you send to the API. But when Kubernetes maps them onto actions on those resources, they sometimes may map into a wider set of operations. And I'll have a slide talking about that. So and then there are actors, so subjects, objects that represent or Kubernetes credentials of processes which work with various Kubernetes objects. And those fall into three categories. Those are users, groups, and service accounts. While only service account actually exists within a Kubernetes cluster, within a Kubernetes cluster API as an object, users and groups are a sort of virtual. So they don't exist in Kubernetes database. Kubernetes just identifies them by a string ID. So when you send an API request to Kubernetes API, first of all, Kubernetes authenticates the request and identifies which user is sending that request, what groups that user belongs to, and it also can extract a map of extra information depending on authentication and method use. Another attribute Kubernetes API server adds to the request for access control framework to work on is whether, is a flag essentially talking whether it's a resource flag, resource request or non-resource request. Essentially, in addition to resources or objects with which you can operate under API, under Kubernetes API, you can also send requests to non-resource API endpoints like version, for example, or list of APIs available, meta information Kubernetes API server provides. So then for API resource requests, Kubernetes decodes essentially API request verb, namespace, if it's a namespace resource, API group, resource name, and if available a sub-resource identification. For non-resource request, this set of attributes is smaller. It's just an HTTP request verb and request path. So those attributes are available to access control framework to analyze and provide decision on whether this request will be allowed or not. And as I said, there are a number of... So with non-resource request, HTTP request verb is pretty obvious. For resource request, this verb gets mapped onto an API resource. Action and most common are, well, get list, create, update, delete, but there are some less common things like watch, patch, bind, escalate use. So we won't stop on those in much detail in this presentation. So for the next few slides, I'm going to talk about different ways you can authenticate with Kubernetes to identify who you are. But before going there, just a mention of tools that can be used when you learn this for yourself or analyze or try to debug how your application connects to Kubernetes API. So CURL is always great tool for experiments. And as Kubernetes API is standard trust, API CURL works perfect with it. A kubectl, of course, Kubernetes command line tool, command line utility. And I want to mention JQ is a command line tool that helps generalize JSON and you don't work a lot with JSON and YAML. And here's just an example of how different clients can be used with Kubernetes. So I have here a console already configured to work with kubectl and it has a number of various secrets set in different environment variables. So I can demo different commands. So the simplest command you can issue against Kubernetes API, kubectl get nodes. Kubectl shows you the list of nodes you have in the cluster. Very convenient switch that you can use with kubectl is verbosity. Setting up to eight, you will be able to see which HTTP request kubectl sends. And it's very useful for debugging. And if you up it to verbosity nine, kubectl will just show you a KURL command that you should issue to repeat that request. And here's an example of how this works with KURL. So I have exactly the same command listing nodes in Kubernetes API. I'm just sending through JQ and less to have a colored convenient output. So you see what Kubernetes API server gives you back in response to the list of nodes. Okay, going back to our presentation. There are a number of authentication mechanisms available to clients, starting from client certificates, bearer tokens, HTTP basic us is an option. And then there are two more, which I will talk about authentication proxy and impersonation, which is sort of not a impersonation is not a way to authenticate yourself, but a way to use already authenticated connection to pose yourself as someone else, which can also be useful for debugging or in some real life use cases. So when we are talking about client certificates, we'll talk about two different ways of signing those certificates. You can either have an enterprise certificate authority or PKI infrastructure, in which case, you may use that external infrastructure to sign client certificates so that they can be used to authenticate this cluster, or an easier way that does not require this external infrastructure, but may not be suitable for large scale use is through Kubernetes itself, Kubernetes can sign those client certificates by itself. With bearer token, we have a number of ways to get that token. So there is bootstrap token capability and all authentication token, we will not talk about them because they are mostly used for internally in Kubernetes, for Kubernetes initialization and bootstrapping. It's possible to use static token file, which we won't talk about again, because it's considered a bad practice and secure way of providing tokens. And then two ways of getting tokens from service accounts and OIDC. So we will talk about today, because they are practically useful. HTTP basic calls, again considered and secure, can only be done through static password file in Kubernetes. I want to mention it, but won't talk about that much. And then two other options, authentication proxy and impersonation, we'll talk about that. So starting with client certificate. So the process with client certificate with externally signed certificate looks as follows. So client needs to get a client certificate, which a client can provide to Kubernetes API together with, not together, but a client will have some private key. That certificate will be paired with that private key. So this private key and certificate will be used to establish and authenticate connection against Kubernetes API. So client uses this private key to create a client certificate signing request, which client sends to administrator or to use a PKI enterprise PKI to get back an actual signed certificate. So that signed certificate needs to be signed using Kubernetes certificate authority private key or key, which is trusted by Kubernetes API servers, needs to be set as a parameter of the API server. So downside of this approach is that that private key now essentially has to be exposed to some outside vector. So, and if it's an enterprise PKI, it's acceptable. If it's a manual certificate signature, it's probably not what you want to do. Still, we'll walk through the process and as I know, so I tried to make this presentation as practically useful as possible. So in a few slides, you'll see a set of command line commands that you can use to try to experiment with that. So this slide shows essentially a sequence of signing certificate and using that certificate that we will try right now. So we'll start from scratch. So I as a client create my own private key and using that private key I'm signing a CSR certificate signature request. Note that in that certificate I use a command name and organizations. So Kubernetes will essentially map the command name into user and organizations into groups. So if this certificate request is signed, so that client certificate will be identified by Kubernetes as a user one who belongs to two groups, group one and group two. Now I as an administrator who has just received this certificate request can sign it using cluster CA key certificate and certificate certificate. Okay, so the certificate is signed, now I have my certificate and my user key and I can use them to send a request to my Kubernetes cluster. So I can just pass them as a command line option to kubectl. You see the result is that well Kubernetes identified us as a user one, but because as a user one we don't yet have any permissions within the cluster, we are not allowed to do anything. So just to make things interesting I'll go to that cluster and give me permission and so this is a user interface of Kubler. I'm using just to make it easier for me to manage roles and role bindings and I'll talk about roles and role bindings a little bit later. So for now what I'm going to do, I'm just going to create a new role binding which gives user one a cluster admin role and I'll talk about what cluster admin role is in a little bit. So for now it's just enough to know that cluster admin is someone who can do anything within the cluster. So we are saving the role binding, now user one is associated with cluster admin. Let's see if it helps to fix our issue. Yes, now the same command with this certificate returns the results, so we can see the nodes. Another way of using client certificate authentication is to get client certificate from the cluster itself. So how it works, again I as a client create a certificate signature request but this time admin administrator or external system doesn't sign it, instead they send it to Kubernetes cluster itself and Kubernetes cluster has a process that signs that certificate and issues that certificate so that administrator can extract it from Kubernetes API and send it back to the client. The client will use it, so the process like that. So for this we use a special object called certificate signing request in Kubernetes API. So let's try and run through the process here. Again we are generating another private key and we are generating another certificate signing request. This time it's for user two who belongs to group three. Now as an administrator instead of signing this certificate directly I am sending a request to API server asking to create certificate signing request. That's that and server obliges the certificate signing request is created, we can check which requests are there and in what state they are. Here it is our user two request is in pending state. So now as an administrator maybe another administrator I can approve it or I had an option to deny it actually. So now I approved it and let's check its status. So the condition briefly becomes approved and then based on that Kubernetes issues a certificate and condition becomes approved. So now if now I can extract and we can actually quickly check how this certificate looks. If we ask Kubernetes to show it in YAML format let's say so you see this is our object which has a request inside base 64 encoded request and in the status section it has actual issued certificate also base 64 encoded. So I can use this comment to extract it into a file and again this is part of the presentation and now I can use this file to try to authenticate in the Kubernetes cluster and you can see again I am authenticated as user two I'm just not allowed. Oleg are you still there? I think your audio might have cut out Oleg are you still there? Yes sure. Okay now we can hear you we lost you for a second there. Okay I apologize for those technical issues. No problem. Not not true if it's on my site or internet but okay just let me know if there is anything like that again. Oh I'll let you know if there's anything else but I can hear you great and I can see your slides you're doing great. Perfect thanks. Okay using tokens and the easiest way to get a new token is essentially to create a service account in the Kubernetes API and Kubernetes server will automatically issue a token associated with that service account and will identify then anyone who uses that token as using that service account to access cluster. So and a comment creating a service account is as you can imagine very simple keep cut will create service account and then the name so I have just created it. Let's check now you see when I'm checking which service account I have right now I see my service account I have just created and a default service account so default service account is always created in every namespace you create in Kubernetes and that's a default service account associated with pods usually that run in that namespace. Now as we look at those service accounts we can check what's inside and we will see that service account object has a link to a or reference to a secret object and that secret object is actually the one that holds the token so here it is here is that secret object and if I want to coincide with that secret object here is the token as a part of that object structure so I'll use a bit of bash scripting to extract that secret token secret name and the token itself into an environment variable and now I have that secret token in an SA token environment variable I can use it again either providing that token through kubectl command line and we see already familiar response that yes you are authenticated and identified as a service account in default name space with name SA1 but you're not allowed to get those nodes. I also need to mention that to work with kubectl you often or in most cases you will specify these secrets and credentials not via command line but via kubectl configuration file and you can modify that configuration file using kubectl commands itself so I can put that token into my kubectl config file as another set as an additional set of credentials I can create a context that uses these credentials and with the cluster I am currently working with and then I can switch context to use these credentials and now I can just run kubectl without specifying any secret information in command line but relying on the current context set within that config file so you see now I'm again I'm using the same token authentication but this time the token is taken from the from the config file so you can always again operate with your config file using various command line option so for example I can see the list of contexts available in my config file let me switch back to my admin account who has all permissions in the cluster so all these commands are mentioned here in the slides so you can experiment with them your own time and on your own schedule the last way of authenticating through tokens is actually using an external identity provider or ADC or also identity provider and in our case I will use an key clock identity provider which comes as a part of kubler so key clock is great in that you can create so-called realms there and realm is essentially a completely separate domain for your users groups syndications federations etc etc so key clock is a very flexible identity broker which can provide which is compatible with SAML or ADC allows you to manage users internally as a part of the clock itself so it is both an identity provider as well as it can work as an identity broker so for this demo what I did I created a demo app realm apologies let me login again simple so I created a demo app realm I created a so-called client so client is essentially a configuration object that allows certain identity provider clients to connect using a certain protocol so it can be a SAML client or in our case I created a client called Kubernetes which is configured to provide OIDC endpoint authentication endpoint and it's very flexible again you can even configure which attributes will be placed into tokens issued by this client etc etc I also created a test user called gay admin and put it into a group I also created and named gay admins so how authentication with OIDC works so first of all your Kubernetes API server needs to be configured to talk with an OIDC endpoint with an OIDC provider so and in our case so I essentially just changed through configuration of my cluster created in Kubler I provided those parameters as a part of the cluster specification so they were added to API server start up options now when client wants to connect to a Kubernetes API client actually talks to identity provider and uses one of the flows defined in OIDC to get an authentication access and refresh token so identity provider gives tokens back and now client can use that token to authenticate with Kubernetes API and Kubernetes API server can talk through back channel with OIDC identity provider to verify that token or that token may be JWT token which can be verified by by the API offline in any case so that token provides all required information to Kubernetes API server saying who this client is and client also can refresh that token using refresh token so let's see how it looks in a command line world so we'll just use curl to talk to identity provider on the client side although yeah IDP specific tools may be used in real life and the first thing you can do is to actually log in sending this command to API server we get a request which as you can see includes access token refresh token and ID token ID token and refresh token can be used to refresh the access token so access token is usually very short lift in this case five minutes refresh token is has longer lifetime in this case it's half an hour and I'll use a little bit again of scripting to send to run the same command so that it so that it saves those tokens in three environment variables and prints them so here they are and now as I have those tokens I can let's see how how refresh works essentially so I can refresh the token through command line sending ID and sending ID token and refresh token to the identity provider so I will get refreshed set of tokens which doesn't mean that my previous ones stopped working that may depend on the identity provider configuration I can also use an identity provider endpoint to introspect token so this is essentially an API that Kubernetes API server can use to check who sends a request to you so and here this is a typical introspection response so identity provider tells us that this token belongs to a user with the admin name who is part of the admin group so we also have these comments in the presentation and let's see how those tokens can be used to set up a kubectl to work with the Kubernetes cluster so you cannot this time use so refresh tokens for example using kubectl command line but you can put those tokens into a kubectl config file using this command again part of the presentation and this makes it possible so let's switch the context so now we are setting our kubectl config file to use this context with OIDC token so and now as we work with kubectl so kubectl will use these tokens and will automatically refresh access token when it expires as long as refresh token is active so in this case you see I'm trying to get notes and now again correctly identified as an ADA admin user registered in this OIDC provider so now user identity is a little bit longer remember it's not an object an API server it's just a string and the only problem is that this user doesn't yet have any permissions in our cluster okay so we talked about probably the last client useful way of authenticating in kubectl cluster and as I mentioned previously two more ways to provide access to kubectl cluster is using ours proxy so in this case so it's it's mainly used by vendors setting up different kubectl architectures so and this means that you would start an proxy server essentially that proxies all requests to kubernetes api and you would establish a trusted relation between kubernetes api and that proxy so proxy can authenticate users and clients anyway it likes it wishes and it will put user identification into the request headers for those requests that ascend to kubernetes api and this way so kubernetes api will know who they work with so in kubler for example this authentication method is used to proxy for example dashboard requests or web console requests and again I'm out of my session scope again so web console requests and in general provide a proxy kubernetes api endpoint so but in most cases it's used by vendors and another way of authenticating within a kubernetes cluster is impersonation so if you already have certain credentials and that are allowed to access kubernetes api and using authorization rules those credentials are permitted to impersonate users and we'll talk about those permissions in a little bit you can essentially send impersonation headers and kubernetes api will switch your authentication context to that impersonated user from command line perspective this looks as full so let's let's get back to our admin context aws demo so now i as an admin can send request for list of nodes but i as an admin can send also the same request with a set of impersonation headers so in this case for example i am specifying that i want to impersonate myself as a service account sa1 which in addition belongs to two groups and now i get an expected response that even though as an admin i am allowed to make that request then impersonating that service account i'm not allowed to to retrieve the list of nodes okay so that covers most important methods of authentication and then we are short on time so i'll briefly go through authorization mechanisms and we'll talk about on the two of them in a little bit more details it's airbug and webhook so there are ways to authenticate using node authorize using node authorization airbug authorization and then there are two more authorization plugins in kubernetes called always deny and always allow so node again it's mainly used internally for cubelets to a bug is based on a static file and is considered insecure and deprecated and always deny and always allow based on their name you can figure out that those are mainly good for testing now we'll talk about first about webhook mechanism so essentially through an additional option on kubernetes api server you can provide external authorization server service to which kubernetes api will call when it needs to decide whether a request needs to be allowed or not so and api for that authorization server service is well documented in kubernetes documentation moreover kubernetes api itself provides this api so and extension servers use that mechanism when they provide decisions on authorization on extension objects in kubernetes api so from practical standpoint the most useful way of authorization is airbug so airbug is based on declarative definitions of permissions based on cluster again api objects and the main ones of those objects are roles and cluster roles so role and cluster role both represent a set of permissions on certain objects in the api identified by api groups and resource names and verbs actions on those objects so you can have a number of rules like that within a role or cluster role object now the difference between role and cluster role is that role is namespaced object while cluster role is global spaced object so rule always exists in certain namespace and can only refer to namespaced resources and only defines permission to objects within that namespace cluster role on another hand on the other hand is a global object it is not namespaced it you're allowed to use it to provide access to global objects non-namespaced objects and also you can use it to provide access to to non-resource URLs so essentially to things like api versions healthy and points on on api service one thing that you need to know about cluster role is that you also can use a property called aggregation so while roles do not support that cluster roles can be aggregated in which case you just specify an aggregation rule for your cluster role which is essentially a set of label matching rules and kubernetes will find all cluster roles that match that label and aggregate them into this cluster role dynamically so you can add and remove matching cluster roles and aggregated cluster role will change the set of permissions accordingly so this is very useful for predefined cluster roles i'll talk about in a second you associate cluster roles permissions with actors service accounts users groups through role bindings and cluster role bindings and they have similar correspondence role binding is always created in a specific namespace and role binding can be associated with either role or cluster role if it's associated with a role clearly if it allows certain actor permissions specified in that role to objects within that namespace if it's associated with a cluster role it permits access to namespaced objects defined in that cluster role only within that namespace so it's very convenient when so we'll talk about that when we talk about predefined roles cluster role binding as you can derive from the name is a global object so it it associates cluster role with users groups service accounts it cannot be associated with a namespaced role so it is used to provide access to global objects non-namespaced objects or to namespaced objects in all namespaces so and every kubernetes cluster comes with three roles cluster roles defined by default out of the box one is cluster admin used to associate to provide access to everything in the cluster so an account that is associated with cluster admin can have access to everything which which which which sort of matches the name admin role is intended to define namespace admins so admin provides full access to namespaced objects within the specific namespace edit role is sort of same as admin except it does not grant you rights to change access control and permission objects so you can do anything within namespace except for granting access to other users and then view role is essentially read only access to namespaced objects within a specific namespace so just a quick overview of how you can start and make sense of the system the easiest way to do that is to just start with built-in role so those four roles is is quite enough to to to provide some basic use cases your operations team may have a cluster admin role associated if you want to give a certain team access to a namespace give them a namespace admin role and developers may have edit role etc etc so new roles you will define them as needed based on your specific use cases and based based on how you want to work around several gotchas so to say and this slide is dedicated to them so for example you need to be aware of a problem called privilege escalation via pod creation so if a person can start a pod in a namespace through that pod they have essentially access to secret objects within that namespace so for example it doesn't make sense to allow someone to start a pod but not allow them to read secrets in that namespace because people can just map that secret into that pod and extract information from there so it's something that needs that you need to be aware of about another thing to remember is that non namespace objects can be of potential interest to developers as well so for example things like crd priority classes pod security policies maybe may have to be created when we deploy certain frameworks so for example service mesh etc so if developers need some level of freedom to experiment with different frameworks they will most probably have to be allowed to manage at least some of the non namespace objects and in many cases it's just easier to provide them a separate set of clusters or a space where they can create clusters to experiment than to manage those non namespace objects one by one in a single critical cluster and one more thing that needs to be sought from the beginning is role and role binding name conventions in the sense that you need to decide how you will name those role bindings and roles if you create new ones how you create them for example whether you create one role binding per subject or you will manage role bindings with multiple subjects whether it's one per role or whether you are using mixed approach okay so we are close to wrapping it up i just want to mention a few things that we didn't talk about but that are worth looking at when you are thinking about security role based access control and permissions and those include pod security policies network policies limits and quotas admission controllers and dynamic admission control seems like OPA so all that allows you to further expand and extend what you can do with access control and security and Kubernetes cluster Kubernetes is an incredibly flexible and extensible framework i summarized objects that you need to be aware of when you are talking about access control and role based access control and Kubernetes so you can look them up in Kubernetes documentation and put several references that are a good starting point for experimenting and learning the subject so having said that i think we are open to questions and christy what do you think i i can start with the questions that we have yeah we're unfortunately we're actually out of time for the day um oh like is that okay if people connect with you on twitter i see your handle at the bottom of your slides absolutely yes and i apologize so it took a little bit longer than i thought no no problem at all it's super interesting well anyway thank you all for joining thank you oleg for a great presentation today as a reminder the slides and recording will be available on the cncf website later today and we hope to see you at an upcoming cncf webinar thanks again and have a great day everybody thank you bye bye