 Hi. I'm Tablethysable. Unfortunately, Tim was not able to make it, but his contribution will be here. He'll be here with us by video in a moment. So I want to start by showing you one of my favorite tweets, which is this tweet. And the thing that I love so much about this tweet is, on one hand, it's like a big pile of line noise. On the other hand, if you have Kubernetes API access, then you can paste this tweet in and then you become root. So that is super fun. And let's actually do that. We're going to do this blind. So here we are. And now I think we're root. We can get in here. We can Etsy, Kubernetes. Well, anyway, there's a bunch of stuff in here. Trust me, we're really root. And that's creepy. That's not super good. So what can we do about that is the question that I have. And we have an answer to that. And the answer to that is that I can put this label over here onto this thing. And I am changing things. This is also bad. All right. We are just going to get away from that completely. The point is that without admission control, bad things are going to happen. And so therefore we want to have admission control. And so I'll clean up my windows here for you. So yeah, we want to have admission control. If we don't have admission control, bad things are going to happen. And so now we actually have admission control. I can label a namespace and it will block me from being able to create that pod. So yeah, thank you so much for joining me here virtually or in person for pod security policy replacement past, present, and future. Like I said, Tim wasn't able to join us physically, but he'll be here with us virtually in a couple of minutes. He and I have prepared the session to show how we came together as a community to make the demo that you couldn't see because it was hidden from me happen. So we'll start with a problem that we're trying to solve. Like what? What does this even mean? Why do we need an admission controller? And the easiest way to describe that is because without an admission controller, Kubernetes is a system administration tool. Everybody who can use it as a system administrator, which means everybody who can create a pod is root on every node in the cluster. And if that's what you want, you're perfect. Out of the box, everything is golden. But that's not really the way that we think about Kubernetes. If we think about Kubernetes as being for scaling workloads and things, we tend to think in terms of the pods and we want to imagine that the nodes are just a fungible resource and having everybody be able to do whatever they want to them really breaks down that understanding. So if you don't want root on every node to be the same thing as creating a pod, then you need to create an admission control. And the reason for that is that fundamentally, the API server is like a really opinionated object database. It has a bunch of object types. It has pods, deployments, replicas, that's all the things that you're familiar with. And that's the level that RBAC works at. It just treats it like a database. But the reality of it is that the objects in these databases become things in the real world because there are controllers that actually see these and act upon them. And RBAC has no opinions about the contents of those objects. But you might have opinions on the contents of those objects. So what are the kinds of use cases that we would have for deploying admission control into a cluster? We want to be able to encourage deployment best practices. And to say encourage in this context is a little heavy-handed because what we really mean is block non-conformant things from being deployed. But if you're doing this in a way that is caring to the users, you make some accessible way for them to ask for the exceptions that they need. And so in that sense, it is encouraging best practices because if you don't file an exception, you have to follow the rules. And if you really do need to break the rules, then you write down why. And that's reasonable. One of my favorite reasons to be going with admission control is to reduce the value of your credentials being stolen. So if we trust everybody, nobody is ever going to do anything bad, then we just make everybody cluster admin and it will be fine because nobody does anything mean. We just presume that everybody is going to be trustworthy. And like maybe actually I trust all of my teammates that much, but I don't trust the person who just phished one of my teammates that much. And so like as a system administrator, like I've been a system administrator since the earth cooled, it's always a good feeling to actually get some keys off of the key ring and make it so that I'm not as worried that I can do things that I shouldn't be able to or someone who has adopted my account can do things that they shouldn't be able to. And that's another thing that you can get out of admission control. And then another really common reason is for multi-tenancy. And I very carefully avoided putting multi-tenancy on the slide here because there are so many different things that multi-tenancy can mean. And like if we are talking about two users who absolutely hate each other and want to hurt each other constantly, setting up careful RBAC and careful admission control and giving them both Kubernetes API access, with sufficient effort and sufficient testing, you can probably get most of the way there, but it's not really the level of strength of isolation that you probably want. But we have a lot more cases where we have multiple groups of users who are more or less friendly to each other and need to be discouraged from accidentally harming each other. And in that case, that's the sort of thing where, again, if you make it so that users don't have trivial access to root on every node, then that can be useful for that. So these are the sorts of things. And there's a lot of good options for admission controllers now. Like you can write your own if you really want to. There is, gatekeeper is well known, caverno is well known, there are some other open source ones that are a little bit less well known. And pod security policy was sort of the earliest one that really had a lot of weight behind it. You know, it's built in, but PSP is deprecated now, and it's not going to be around forever. It had plenty of challenges, but like now it will eventually just go away, and that's a real challenge. So if you didn't adopt PSP, but you're here because you're thinking about it, because it was quite a newsworthy event, let the fact that everybody is thinking about PSP be an opportunity to think about, well, maybe I can deploy admission control. And if you did adopt PSP, you need to be planning your move off. In either case, it's really important for Kubernetes to give a simple to use built in choice, so that that way folks aren't required to adopt some external controller, just to be able to add these capabilities. There's there's a great value in having those options. But making that complexity mandatory really wasn't the way that we needed to be. And so therefore, we have a replacement. And I am going to bring up Tim here so that he can tell you about it. Thanks, Tabitha. And with that, I'd like to introduce security admission. Pod security admission is a new built in admission controller that went to alpha in 122 designed to replace pod security policy. And I'd say the primary goal of pod security admission was to offer or is to offer out of the box security on Kubernetes, so that create pod does not equal route in the cluster. And the way we do that is through a minimal configuration where the user chooses one of three profiles per namespace. Those profiles are privileged, meaning that anything goes, anything is allowed. It's the equivalent of not enabling the admission controller on that namespace baseline, which allows the default pod settings, but doesn't allow you to escalate privileges beyond that. And restricted, which enforces various security best practices, such as running as non route. And another goal of pod security admission was an emphasis on roll out and upgrade safety. This was something that was really problematic with pod security policy. It was really difficult to roll out and difficult to make changes to policies. So there's three different features of it of pod security admission that we've added to make this easier. And I'm going to demo those in a few minutes. And some of those are audit and worn modes, which let you trial new policies before enforcing them. Version pinning lets you decouple cluster upgrades from profile upgrades on namespaces. And dry run lets you preview your changes against existing objects in the namespace. So with that, let's switch over to the demo. That font is too small. There we go. It looks more readable. All right, so I already have a cluster running. But if you'd like to follow along with this demo later, these are the settings that I use to start up a mini cube cluster for this demo. So you see it's a 122 cluster. And because pod security is still alpha in 122, we need to explicitly enable the feature gate. I'm also going to set the current namespace to the demo namespace. And with that, let's get started. Let's take a look at again that pod escalation that was demoed at the very beginning. So here's a privileged pod that has enabled hostpid and privileged. And we see that if we create that, that enables a super trivial privilege escalation where we can grab the cubelets credentials and impersonate the cubelet. So we don't want to allow our users to do that. So let's go ahead and apply this label to the namespace, which tells pod security that we want to enforce the baseline profile on the demo namespace. Now if we try and create that privilege pod again, we see that it's denied and calls out those two fields that I mentioned earlier, hostpid and privileged. Now let's take a look at a baseline pod. This is basically the minimum pod that you can specify. We just have a name and a single container without any settings on it. So we are allowed to create this minimal pod because it's using just the default security settings. And if we try our privilege escalation from before, we see that of course it doesn't work. So let's take a look at the restricted profile now. Going to update the label to enforce the restricted profile now. And if we try and create our baseline pod again, we see that it fails with a bunch of error messages. That's because restricted places a bunch of requirements in it that go beyond just the pod defaults. So this restricted pod is basically the minimum viable restricted pod. So this is sort of the equivalent of the baseline pod that I just showed, but adding the minimal additional security constraints in the security context to for it to be allowed. So if I go and create that, of course it's allowed now. And we can see that it is in fact running as non-root. Now I actually cheated a little bit and before this demo started, I had already labeled added an additional label to the namespace, which is this enforce version. So we've actually been looking at enforcing the 119 version of all of these profiles. Let's see what happens if I remove that label and try to run that restricted pod again. We see that now it's failed because actually in 122 we added an additional constraint to the pod security standards that you must drop all capabilities for restricted pods. So this is a problem because if we hadn't pinned the version to 119 when we upgraded our cluster to 122, suddenly pods that were working before would start failing. So let's take a look at how we could do this upgrade more safely. Let's back up and say we have add that version pinned back. And now let's take a look at warn mode. So to enable warn mode we're going to say that we want to warn on the restricted profile. So same as enforce, but we're going to set the warn version to latest. This is actually the sort of implied version if there's no version pinned label, but I'm just making that explicit here. Now if we create our restricted pod, the pod is created, but we still get that warning saying that this would violate the latest version of the restricted profile would since it's not actually being enforced right now. Audit mode works in much the same way, except instead of returning that warning to the user, it adds the warning to the audit logs. Now this time I've left the pod running, and let's take a look at what happens if I update that enforce version to latest now. Except this time I'm going to set the dry run command to the server side dry run. And if I go ahead and do that, we see we get a warning that the restricted pod, this would be all pods running in the namespace, actually violates the new version of enforce. So this is saying that this new version that I'm enforcing, it is enforced now, or it would be if it wasn't dry run, but we have existing pods that violate it. So this would help detect if you go to upgrade and see that, oh actually there's pods in this namespace that are going to start failing if I upgrade this. That can let you roll back, or if you have done dry run, or in the dry run case you can see that I need to go and update these pods before actually applying that. And that's the demo. So with that I'll hand it back over to Tabitha. Thanks. Thank you very much, Tim. Yeah, so this is a little bit about what pod security admission controller looks like, a little bit about how it's used, and the story of pod security admission controller is really wrapped up tightly with the story of PSP. So let's talk a little bit about the history and the story of PSP. So it kind of evolved out of the security constraints that were in open shift from very early days. And pod security policy was added as an admission controller in beta in kubernetes 1.3. This was before RBAC was a default, the default service account was cluster admin, like this was wild west days. And it evolved really rapidly over time. So by 1.6 the resources to create pod security policies were available in the API server by default. You didn't have to turn on a feature gate for them anymore. By 1.7 you could bind them into namespaces, and there were tons of fixes in 1.8. And then development of pod security policy slowed down a little bit, but it was still moderately active. Up through 1.15 you see occasional feature additions, more pod fields that can be controlled with PSP, and things like that. But the fundamental constraints of pod security policy continued to be in there. Now those feature additions couldn't really make up for the fundamental struggles that adopting pod security policy had. I'll briefly wave my hands at them, like the fact that you must create those policies before turning on the admission controller because it hard fails closed. And so if you simply turn on the admission controller your cluster is immediately broken has been difficult to to deal with. And you know there's a lot of other sharp edges. For more details about this there is a there's a SIGAuth talk from several Kubecons ago that's linked from the PSP deprecation blog post. And in a little while I don't know if it's the next session or a couple of sessions down SIGAuth has a maintainer track session where they will get into a lot more detail about like what those PSP challenges were. But the challenges were there. And so discussions about how to replace PSP started around 116. And those discussions were not able to progress rapidly. So we got to the point eventually where other Kubernetes administrative changes made it extremely likely that PSP was going to be deprecated. So like what even is Kubernetes deprecation? It's the first step towards having a feature go away. And it's just a notice like it's a promise we are going to continue to support the feature for a certain amount of time but you need to be aware that it will not be around forever. And that is what is currently happening with PSP. It was deprecated in 1.21. It will be removed entirely in 125. But its story doesn't end even in 125. PSP's story ends when the last user turns it off. And so some of you might be wondering even whether you are using PSP today. One way if you can inspect the command line arguments that are being passed to your Kube API server, you can look for pod security policy in the enable admission plugins command line option. If you only have Kubernetes cluster access, K get PS, Kube CTL get PSP will probably tell you if you're using PSPs because certainly there must be some PSPs defined in your cluster or it would be broken. It is possible that you might not have access to them. So Kube CTL get PSP is not 100% trustworthy but if you see it, it does prove that you're using PSP. If you don't see it, it mostly proves you're not using it. So if you are using PSP, it's going away. You're going to need to replace it with something. And because PSP was such a flexible feature that could really be used in a lot of different ways, some of which were quite straightforward, some of which were quite clever, some of which were too clever, you can't simply replace it. Nothing could just replace PSP except PSP. But you can think about what are we trying to get out of PSP? Like it has mutation features. It can default certain pod settings to newer, more hardened settings than what the Kubernetes defaults are. Is that what you're using PSP for? Are you using it to exert really fine-grained control over pods in different namespaces? Like for example, if you are running a CNI plug-in that requires capnet admin and hostnet on its Damon set, have you made a namespace for that, made a PSP that allows only those features and enabled that PSP to be used in that namespace? Perhaps you are. Or perhaps you let everything in that namespace use all features. You need to think about what are you getting out of it? What do you want to get out of it? Because with whatever you replace it with, you need to know the answers to these questions. So once you know what you want to get out of PSP, then you can start to examine the alternatives that you could use to replace it. If pod security admission covers your needs, you can walk through the documentation and have some guidance on how to migrate to it. If you are using advanced use cases that are beyond what pod security admission can do, then the various CNCF open source alternatives like Gatekeeper or Caverno, I haven't actually read all of those docs, but I have glanced through them and I'm pretty confident that all of them have some documentation about how to switch from PSP to them. But whatever, whatever you choose, the key is to make sure that you are choosing something with intention that will serve the needs that you have. So that's what's going on with PSP, it's going away, what you can do to make sure that you can get safely off of it. And with all of that said, I would like to switch and instead talk about the story of pod security admission. And the reason for this is that PSP deprecation is one of the biggest breaking changes ever made in Kubernetes. And I absolutely loved Locky's keynote from the other day, talking about getting dual stack to GA support and how it was a very difficult process that required bringing the community together, iterating, understanding what was really needed and changing the work that was being done. And the story of pod security admission is very similar to that. So like how did we do that? Fundamentally, a core group of folks came together and showed reasonable possibilities in a way that helped to focus the discussion in a practical way so that it didn't continue to swirl. Then like we came together as a community and iterated until we got a rough consensus. And once we had a rough consensus of what it needed to look like, then it was ready, it was baked, it was ready to go into the cap process. And it proceeded through the cap process and now it's implemented. So the details, the details about that. They look like, first the SIGAuth leaders started asking about the future of PSP. They tried a few times to get a design process going, but it just wasn't happening. And then in late 2020, there was a new policy to get rid of PERMA beta. Features can only remain in beta for a certain amount of time and then they have to graduate or be deprecated. And Tim rallied SIGAuth again to find a solution. He wrote an outline doc with background and options, put a discussion topic on the SIGAuth meeting and people showed up. We had like a couple dozen people in that meeting. We had a great discussion and the discussion didn't really get us any closer to an answer. It was very similar to the previous discussions. And so my SIG security co-chair and I, Ian, we said, well, we will write a proposal that will stop this churn. That's going to be easy, right? No, no. Of course that wasn't easy, but we said that we would do it. And then we were, then we were, then we have to keep our promises. And so the idea was that if the proposal was specific enough that we could start asking, what if we adopted this? Or like, why can't we adopt this? Kinds of questions. And when we bootstrapped that, it was based on our experiences as end users. We've both run Kubernetes in production in several places and really have felt the pain. And so we put our end user feelings into this. And we had advantages. The situation had gotten a lot better since the earlier meetings. Now there's a bunch of third-party admission controller choices that are really good, so we don't have to solve everybody's possible use case in the way that PSP had to be everything to everybody. And now we have the pod security standards document, which gives some good guidance on places that you can think of stopping along your pod setting hardening journey. So there's some, there's some guidance there from that. And everybody agreed that Kubernetes needed to have some kind of built-in option. We, you know, we had many folks who were on board with that and we eventually came to everybody agreeing with that because built-in offers an unmatched simplicity of deployment. And we have to have something built in if we want to move Kubernetes closer to being secure by default. And for users who can't or don't want to adopt some external complex admission controller. So as Ian and I were working on this proposal for a PSP-like feature that was fairly complex, had a lot of flexibility in it, but had rounded off the really sharp edges that would cut you on PSP, Tim and others were also writing a very specific proposal for an extremely simplified feature. It was almost binary. A particular namespace could allow everything or allow something that was quite restrictive. And we coordinated the release of these two competing basic proposals so that neither one of them would get an unfair advantage due to timing. We shared our proposals together with all of the rest of the Kubernetes community in Google Docs to the SIG-Auth and SIG Security meetings, Slack channels and so on so that they could get improvements. And both of these proposals generated a lot of enthusiasm and discussion. We took over a couple of weeks worth of the SIG Security meetings and a couple of weeks worth of the SIG-Auth meetings. And at that point Tim said, hey, let's have a regularly standing breakout meeting so that we can talk about our feelings about this and our respective SIGs can continue to do the rest of their business. And this actually worked out really, really well. I brought back to the SIG Security meeting how it was progressing. I think Tim brought back to the SIG-Auth meeting how it was progressing and over the course of a few meetings, this ad hoc group really sharpened both proposals. We just focused in more and more on what could we agree with. We built group consensus. And as our goals and values became more clear from the kinds of discussions we were having about both proposals, Jordan came and proposed and wrote a prototype for a feature that combined almost all of the strengths of both proposals that had been being discussed and dropped a lot of the shortcomings of both of them. And it's very, very similar to how pod security admission is today. It had this very holistic design where user needs came first. That means how to maintain it is considered and the fact that there are other options was considered. We were able to simplify because we knew that if somebody's needs were beyond what we could do, that they had other choices. And so these are really the lessons that we learned from that. That big decisions have a life cycle and you need to respect that life cycle. That when you see a problem, it may seem obvious to you what the problem is, but there are plenty of other smart and caring people in the Kubernetes community who also think the problem seems obvious to them and you don't all agree on what that is. And you need to come to a shared understanding of what that of what that problem is and how to solve it. And a KEP review is a really hard place to talk about your feelings. So like I like to joke when I'm trying to solve an engineering deadlock problem, we're going to get all of the people who know what they want but don't know what they want into a room and we'll talk about our feelings and we'll figure it out. But like the reason that I love that joke is like it kind of disarms people, they laugh a little bit. But the real truth is we actually are talking about our feelings. If the technical decisions were obvious, we wouldn't have to have a meeting about them. We wouldn't have to have a discussion about them. And so big technical decisions do involve our feelings. And so therefore exploring big questions together in a more form more informal setting like meetings, slacks, Google docs, building consensus that way around an understanding of the problem and understanding of a good potential solution, then you can feed that into a cap that is set up to be able to actually get refined and implemented. So try something, ask for help and work together with folks. So you can get involved with pod security admission. It might be good for your needs. If you think it might be, try it. It's in 1.22. We're targeting beta in 1.23 which would mean a beyond by default. You can come to the SigAuth kubernetes slack channel or come to the SigAuth meeting to tell us how that's going. There is a webhook implementation that's in progress. You can look in the issues. I'm sorry I don't have a link for it. And there's a lot of room to improve the documentation especially around migration from PSP and advanced configuration. And we'd really love your help with that. So thank you so much for hearing us and please play with our tools. Like they're your tools. They're our tools. And let's make sure that we make them as good as we can. Thank you. You were lovely. We don't have time to take live Q&A in here though if you asked a question through the meeting play app, Tabitha's co-speaker was in there answering those virtually. Thank you Tim. I'm so glad to have done this with you. I thought that was actually really convenient and useful. But Tabitha's welcome to hang out in the hallway and answer questions if y'all have them. Yeah, I'll be around here for a few minutes and happy to talk to folks. And otherwise we're Kubernetes. We're in the internet people. You know, please come to Slack. Thanks a lot.