 Thank you, John. What's up, peeps? Back from break? Yeah, how is everyone? What is this, day two of KubeCon? It feels like day four for me. Yeah, yeah. All right, well, let's go do some learning. If you don't know me, my name's Lee. I contribute to Kubernetes sometimes, also pretty active with the Flux project. And then you have proud Filipino people in the room here. No? Hey, okay, John. What's up? Yeah, Filipino American, I live in Colorado. And I'm gonna kick this talk off with something that's a little bit different, a little weird. It's performance art. Yeah, so let me see what, let me just make sure I got the tempo here. All right, I'm gonna need help, like this. Should be about like right here, we just gotta keep it steady, yeah? Sit back, deploy apps, ship in it, relapse, trying to update a hump day, but I need a new play. The app's on fire, this is so not dope. I can't debug teams losing hope. I just clicked the button, it wasn't disabled. I was on cloud nine, then it dropped the table. I'm big surprise, just a simple reflex. Now production's down and I need some Kleenex. It was a wallet text, needed my glasses. We on Kubernetes, but the app still crashes. It worked, got the cert, took some classes, but I can't help now, because I don't have access. Just clicked the button, now it wasn't disabled. I was on cloud nine, then it dropped the table. Big surprise, just a simple reflex. Now production's down and I need some Kleenex. Walls of text, needed my glasses. We on Kubernetes, but the app still crashes. Did the work, got the certs, had the classes, I wanna help now. Can I get some access? Things down, man. All right, I don't know, man. So why are we talking about pain and working together with people on hard systems, right? We're working together, we're using Kubernetes. We gotta figure out how to use the tools to have the people in process part of getting work done. We don't use the infrastructure, we don't use fancy computers because it's fun. Well, sometimes we do, because we're nerds. But the bigger problems is to solve in society, humanity, and making money with capitalism are working together to do useful things. We wanna make apps, have people use them, store information, and so we gotta figure out how to share and scale things for our organizations. And so we're using a computer, maybe it's just one computer. You go log into that thing and it gives you the access to the thing and then you start using that computer to do all this imperative stuff. You're writing Google Docs or whatever. And it's just kind of you who uses this laptop. Maybe you share it with your cousin or something too. But this Kubernetes thing is the operating system that turns lots of these servers or whatever, cloud, data center, into one big computer. It's an operating system, it gives us a distributed computer. And you can't program it the normal way that you use a laptop or maybe you run some Python on a single machine. A lot of people say we configure Kubernetes but you configure things that are already doing stuff. Kubernetes is a computer and it doesn't actually do anything useful unless we program it. So we give these configs, even though they're declarative, they're source code. And this source code is in the form of a declarative promise. So Kubernetes makes promises happen for us. But the kinds of promises that it does are pretty useful. It deploys apps, it can promise us networks, can manage lots of servers at different pools with labels and have namespaces, schedule over here or over there. You can enforce policies and stuff. And we talk a lot about these objects. We're super focused on the tools. There's deployments, services, nodes, network policies. It kind of maps up to those use cases. They fit into different API groups. We use the same technique to go and declare these promises into this one computer. So we have the same programming model. Now, these things are all so different and useful that we tend to have a lot of people involved in using this one big, expensive, but super fast formula one car of a promise machine. And so you've got developers, you've got operators. It's like a whole community. It's like a city to run the Kubernetes race card around the track. And the API to help us share the computer is called role-based access control or RBAC. So there's the first bit of jargon in this really API introspective talk. So when we first log in to a computer, a computer wants to know who we are and it wants to also then start granting us access. And there's these two kind of other jargony words that people use called authentication and authorization. So when we authenticate, the computer knows who we are. It attests that, oh, I really believe that you are who you say you are. I know your identity. And then when we grant access, we're talking about authorization. Are you authorized to go and do something? Now that I know who you are, let's talk about what you can do. So there's who and what and then linking that together. Now role-based access control is the authorization part of this. So now that we know who people are, it's completely unconcerned with how we figured that out. We're gonna try and figure out how to authorize these people. And again, this problem, the reason that this API is important is because it is going to let us share and block off parts of the computer to this whole city that it takes to run this thing. Right? So if this question is, what are the people or apps able to access? We need to figure out a more generic term for this people or apps thing. So the RBAC API, role-based access control, it calls it subjects. And this is not a royalty or a king or queen thing. Like I don't have subjects, unless maybe the top is the CEO or something. But there's all of these people. So what's the term of subject? What's the type of subject that these people are? They are users and groups. So Kubernetes has some objects. Now, these objects are sort of special. There's not a route that you can go into Kubernetes and you can make users and groups. And we'll talk a little bit more about that later. It happens sort of indirectly based off of what you integrate Kubernetes with. So these words, they kind of look like the same as how we talk about the rest of the Kubernetes objects, but these things aren't real objects. You sort of have to imagine them and use the API to describe and target them. So hopefully that's making a little bit of sense. Just nod if you're kind of following. The other type of subject is an app. And for the most part, apps don't need to talk to Kubernetes. But we actually have a lot of apps that we deploy to Kubernetes to extend it. And sometimes when you're trying to solve distributed computing problems, it can be helpful to use the distributed computer parts of Kubernetes. And so you need to have your apps be able to access the API and learn things about itself, what's around it in its own namespace or elsewhere. I mentioned namespace. If this idea is weird to you, this is just the way that we section up the computer virtually in our heads so that teams can not step on each other and apps can not step on each other. It's a really important concept, right? So in this case, I'm just drawing out the mechanism of what the service account actually does. So there's other objects like pods, replica sets, deployments and stuff that actually run code inside of the computer that Kubernetes lends you. And basically the cooblet is the thing that sees the service account and sees the pod and it gives a token to every pod so that the pod can authenticate with the Kubernetes API. So that's the auth piece. Kubernetes does have an identity system to authenticate workloads inside the cluster. Yeah, just little details. Now we got these three types of subjects. For the people, we have users and groups and for the apps, we have service accounts. The subjects of this API are who. It's the who part of who has access. And then we can ask about who has access to what. So now we gotta talk about a different object. And this is a real object. We don't have to be confused with imagining things anymore. Service accounts are real as well. It's just users and groups that are sort of imaginary. So roles are an object that describes what should be accessed, right? You can think of a role like maybe viewer or editor. And the what of how something should be accessed also includes things you can do to that object. So roles have a list of rules and that list has API groups, has the kinds of things in those API groups, those resources. And then it has the verbs that you're allowed to do to that thing. There's a couple other stuff too that I'll talk about. So roles list the allowed APIs. They list the verbs. And specifically there's an object called role and this is a namespace thing, right? So if you got a team that's in their part of the computer inside of their own namespace, they can make roles that have rules about access and they can list stuff that's inside of that namespace. Resource types that can be used inside of namespaces. There's a different object called a cluster role. A cluster role is a special type of role that doesn't live inside of a namespace because sometimes we have to talk about things that affect the entire cluster. Whether it's just talking about say all of the configs or secrets across every single namespace or if we're talking about things that don't have a namespace necessarily. Things like cluster or custom resource definitions or like network policies and there's a bunch of these kind of cluster level objects that can affect everything. Also extensions can add their own cluster level objects and it's important to be able to control access to all of this stuff. Maybe that doesn't make so much sense if you're just thinking about it in your head. So I drew you a fancy picture with something that is really good at making round rectangles. This is a Kubernetes cluster simplified. We got the blue cluster. You can have resources at the cluster scope that affect everything and then we split up little pieces of it and then you can put all different kinds of things that are namespaceable objects inside of the namespaces, right? There's the dev team one and dev team two. Roles can only control access to stuff that points in, or that's within namespaces and in fact the roles themselves live in namespaces but cluster roles can affect everything. So now that we've answered kind of the question of who and what, how do we describe who has access and how do we describe what they have access to and what they can do? We have the missing link which is we have to map all of the connections, right? So we need this object called a role binding. The role binding binds a subject to a role. It means that, hey, this dev team has viewer access to this namespace and they can update this policy with this name or whatever. Similarly, you could say that this app has access to all of the configs in this namespace so it can go and update interesting fields and it will use a role to describe what it can do updating the interesting fields in the contact map and the subject would be that service account name from the service account object. This is basically authorization in action, right? Now we have the ability to ask the full question and make full statements, hey, these people can go and access this thing. So there's a little bit of just like kind of one-on-one, how is this API shaped, how do we use it, right? We're not even looking at like YAML configs and stuff. With role bindings, there's a couple of gotchas here and some interesting variations, right? So role bindings live in namespaces and they are fully concerned. When a role binding is in a namespace, it only authorizes to the stuff in that namespace and that, therefore, you can only really reference access to namespaceable objects, right? You're not gonna be role binding to something that has cluster scope, but cluster role bindings allow you to say, hey, this subject across the entire cluster scope for all of the namespaces, for any namespaceable objects in all namespaces and for stuff at the cluster scope, if it's listed inside of the cluster role, we can grant access. Now, there is some really common misconceptions and it's actually because of this particular shape of API, right? We have cluster roles, cluster role bindings, role bindings, roles. It seems like they would just kind of only pair up together, right? This makes sense. Any subject, any person, any group or application, we can role bind it and within a namespace give it access to stuff. Similarly, anything, any app in the cluster, any person or group that we can attest and authenticate, we can cluster role bind it for like across the entire cluster and this seems like it's enough to start doing some really good work. But if you say, wanna give somebody access to like five namespaces or 50 namespaces, but you actually have 100 and you don't actually wanna give them access to 100, it seems kind of like tempting to just go for that cluster role binding. But it's important to be remembering the principle of least privilege. We only really wanna give people access to truly what they need and we definitely don't wanna make any missteps and give people access to stuff that they don't. Did you know that you could role bind to a cluster role? This means that we don't have to define roles 50 times, right? And you're saying, oh, Lee, cluster roles also talk about cluster scoped objects. In this case, the cluster scoped objects are just ignored and this is actually still very precise and this is actually a very recommended thing that you should be doing, right? Maybe when you looked at these names, maybe you were using it and you learned it at some point, if you're a beginner, you actually have an advantage probably over most practitioners right now because you're like learning the basics instead of just pretending to learn something while you use it. I've had this misconception before, it's very common, you wouldn't believe just how many people do not know that you can do this. So the behavior here is that the cluster role, whatever it describes as namespaceable objects, you will only get access to those namespaceable objects in the namespace that you roll bound from. So it's just a single namespace role binding. And this at least, if you have to roll bind in 50 different namespaces with the principle of least privilege, at least you only have to make 50 role bindings. You don't have to also redefine the same exact role, the thing that talks about access policy 50 times. You can just find that centrally in the cluster role. And Kubernetes actually has a bunch of default ones that I really recommend you should use. This is a beautiful picture. I'm really proud of this one. The best thing about this picture is that it is complicated, but there's only one subject. And so there's a bunch of these purple and pink things. Those are the roles and the cluster roles sitting in and out of the namespace at the very end of the arrows. You have somebody who can administer policy, or a rule set about viewing things across the cluster, regardless of what you're talking about, or a rule set about updating quotas that might be just in this namespace. And we can use cluster role bindings. We can use role bindings. We can use role bindings. I didn't draw any arrows from the role bindings to the cluster roles, but you could do that. You can do that all on the same subject. And this is a common misconception that there has to be a one-to-one-to-one mapping of this, but it would actually be encouraged that you're being very precise about what you're doing when you're giving people access, right? So you can do that, lower the volume. Another thing that's sometimes not obvious is we're really used to giving access to apps within their namespace. But how do you give access to an application to stuff outside of its own namespace? Well, the truth is that apps are just users. Service accounts, they're just user names. There's nothing special really about them except for that Kublitz can give them tokens. And so you can actually have an app with a service account or even a service account group in another namespace. And you can role bind it from the place that you want access. So say I own a namespace. I have some interesting juicy configs or whatever inside my namespace. And some other team has an app that we want to give them access to. Well, service accounts in a namespace are actually all part of the same groups. And so you can actually role bind to the group of that namespace, and you don't even have to know what the name of their service account is. Another common underly used feature and sort of a misconception, you can role bind to apps from other namespaces. Service accounts are just users. Their user names look like this beautiful templated string over here. System service account, this is a reserved prefix. Most authentication systems will not spit this out, which means that it only mostly works inside the cluster and using cluster mechanics. And then it's got the namespace and the name. And then service accounts also get groups. So there's a group for every service account in the cluster. If you say wanted to give literally everything access to something, you could bind to system service accounts. And you can bind to system service accounts from just a couple of different namespaces if you want. Underused features. Now that we've kind of gotten past the basics of the API and are getting into a little bit esoteric stuff, just to make sure I know what the time is here. There's resource names. So we can get very specific about our rules inside of our roles or cluster roles. You can restrict your resources that you get verbs to, to particular resource names. This example is from the Kubernetes documentation. There's a config map updater. This is something that you maybe want to do with an application to spit out something into a consistent at CD storage. So then that could be consumed by something else. Here we are having the update and the get verb in the core API group, which is just what that empty string is for. But sometimes that string is really long and it talks about other API groups. Resources is just the config maps. You'll notice that this is spelled a little differently than normal. It's not capitalized and stuff. This is because it's just like the section of the rest endpoint when you talk to the HTTP service for the API server. And then resource names, we're actually restricting that we can only update the my config map instead of all config maps in this namespace default. Also, this one is kind of funny. You can't filter what people create with resource names, except that you actually really can. But it only works with server-side apply. So the reason for that is because when you're in a server-side apply, it knows about the whole object that you're sending it so it can do diffing. But if you are not using the server-side apply, like if you're actually doing the create verb just normally, then this is not going to work because the API server doesn't know the name of the thing you made yet. So it's sort of a weird limitation. I feel like that's a pretty useful feature. And for it to only work with SSA machinery is kind of odd. Cluster role aggregation, I can't really explain this on a slide, so I'm just going to show you in the cluster. And now my friend Rich is going to be happy because he came for a live demo. So in a Kubernetes cluster, if you don't actually even modify anything at all, and you get all of the cluster roles, so these are the centrally defined roles that can talk about cluster-specific things as well as all of the namespace things. And you can roll bind to this from just one namespace or you can give access across the entire cluster with the cluster role binding. There's all this stuff, and there's actually a bunch of extra stuff in here as well that got added from other add-ons that I installed to the cluster, external DNS, whatever, I am. But if we kind of grep for the system prefix, you can see there's all of this cool stuff that Kubernetes uses internally to have all of its components talk to each other. So you can see that the identity of things is very important. All of these things have various role bindings and cluster role bindings so that they can do like compute problems and make the Kubernetes cluster keep promises for us. Now extending this lets you really use the power of the distributed computer yourself, and it's also very necessary then for you to also be able to, what am I saying, also be able to give people access to the information from the computer. So here's all the cluster roles. We can look at the cluster role bindings, and you can see then that every sort of bit of access that we've needed to open up for somebody it's all in here. Now I'm not interested in that. I'm mostly interested in how I can open access up for other people because the cluster works great and if I need to program an app to talk to Kubernetes that's fine, but I really wanna enable my team. So let's get rid of all the system stuff and just look at what's in here. There's a bunch of still stuff for add-ons and that kind of thing, but if you look at the YAML, I'm gonna pipe this to neat and also just syntax highlight that for ya. Let's like look at the viewer role. Guess I mistyped it here, view. Double gets, thank you. Pair programming. And I need to also type cluster role. This is why you script demos. Cool, so you can see there's, if you say like wanna have the view cluster role, there's like getting and listing and watching for pretty much a bunch of API groups and resources across the cluster, but there's also these labels and there's also this aggregation role. That's weird, what is that? This aggregation role has a cluster role selector that matches the label of other cluster roles and what it does is it aggregates them together. So if there's any other cluster roles that have the label aggregate to view, then we can actually, from this default cluster role, if people bind to this and we add to the cluster and we extend it, we can also let them view other things, stuff that didn't ship by default with Kubernetes. Similarly, this viewer cluster role, as soon as you give access for something to be viewed in this rules list, it gets aggregated. See how this is labeled aggregate to edit. This is an official Kubernetes API and you can use this and it works fast. It's in the API server. And so these aggregated cluster roles, they're very precise. There is another, let me get this label selector here. I did this beautiful command earlier. Why am I, you can, we don't need to do that, yeah. Let's just do this and that, that's a T. So these are all of the bootstrap cluster roles. It's very simple. Everyone's heard of cluster admin because it's notorious, but there's admin, there's edit, there's view. And you can see how not having to figure out how to tell somebody that you can view every API resource or inside of a namespace would be very useful, right? So now, if say I wanna give a development team access to view everything that they own in their namespace, a very good default thing to do would just be to roll bind against the view cluster role. And again, if you do a survey, a lot of people do not know that you can do this. So let's use RBAC, this is really good. Let's talk a little bit. Oh, there's other underused features here. Sub-resources, there's a bunch of these. They're very powerful, mainly the status, the proxy and the port forward ones, also attaches and binding. And then there's also these weird virtual verbs. This is how you can do some interesting stuff with RBAC. You can, for instance, implement pseudo access using these verbs. Also impersonate is very useful for debugging RBAC. This is a very underused feature and I will talk about it later. But let's talk about then, okay, we've talked about using the API together and about how to grant access. Kubernetes doesn't actually authenticate people out of the box. Maybe you installed it in a cloud and it's integrated with the cloud's identity provider just out of the box. But you actually have to do work with default Kubernetes to make it know who your people are so that people can actually log in to the thing. There's some great projects out there, kube login. I'm really a believer of the pinniped project because you don't have to modify the API server. Just go check out pinniped.dev. But we need to be able to authenticate people. And we want to probably do that with the identities that we already use because we probably have multiple Kubernetes clusters at some point. I don't really recommend using Auth and service accounts outside the cluster because those things are for apps and integrations and cluster internal things and they will never mean anything else in another cluster. You should use the bootstrap cluster rules like I mentioned. We should be binding to groups. Binding to single people is good for ad hoc things but we want to enable teams. We want transparency. And ultimately the tools that we use start to shape our habits. So if we use a declarative computer, we should probably be using declarative configurations. So I would recommend to really get off to your access control. We can store all of these objects in some place where some or maybe even everybody could reach them. Everyone could see them. Maybe not everyone can write to these things. You probably want to really protect them but you should be thinking about how you can onboard people. How you can give people access when there's an emergency and they accidentally dropped the table and didn't have access, you know? So yeah, you want proper process because we need to be able to work with each other. This is how we share the computer. This is how we get stuff done. We need to empower people with information. Can you actually raise your hand if you've tried to do your job before and you didn't have the information to make good decisions? Yeah, right? So let's give people the info they need to be successful. If you're a manager, maybe you should be able to find out for yourself if some software got released. If you're a dev team, maybe you should be able to delete pods so that you don't have to page SRE if something's a simple fix and you know that there's a memory like there. You have more context on your app. Similarly, I mean, if you are... This is some of the most important intellectual property in your organization. It's like who has access to what? What do people actually do? If you don't have a good handle of how people get access and when people had access and what people have access to currently, you're never gonna be able to recreate your clusters, move your cluster somewhere else. You're gonna get very tightly coupled in an imperative state that's difficult to recover. Especially because all of this stuff really needs to be commented and annotated with things that are readable by humans. You can't just get this from like a Valero backup of your Kubernetes cluster. You need to preserve this intellectual property. I would say that you should consider using some integration tests. You should run these periodically. So check that people and apps have access to the stuff that they need. This is how we can keep stronger promises to each other. Actually, I'll just show the slide of what a command that you need to run. This is kubectl off can I. It's a way to probe. If you have the impersonate verb and you can also restrict the impersonate verb with resource names, then you can go and say, does this service account for this application have access to updated config? Do these people have access to these important business-critical objects like deleting pods? And if you put this in your CI pipeline and you have a transparent way to update access and to request platform admins, give you a little bit more permission to be able to solve harder and more intricate problems as you learn more about Kubernetes, then running these sorts of things in a CI pipeline gives people a very clear indication that they have what they need and that they don't have access to the things that they definitely should not have access to. So there's a bunch of limitations. List and watch, you gotta be careful with this. If you have list and watch on secrets, you can actually read all of the secrets at either the namespace or if you cluster roll bind it, that's bad. You cannot filter lists with resource names. If you need something like this, like filtering namespaces for secrecy, you should maybe be thinking about using V clusters or API server filtering proxy, like the Classics project with this open source. Resource names with the crate verb, I mentioned it only works with server side apply, although maybe you could also just put this somewhere else so that you don't have to filter what the name of the thing is. And yeah, you know, I've suffered a lot with RBAC. I've helped run Kubernetes and make it more usable for lots of dev teams as well as really seeing some interesting stuff from our users in the Flux and Kubernetes projects. You can definitely reach out to me on Twitter, my DMs are open and yeah, I think I'm probably out of time, but feel free to find me after this. Cheers friends.