 Okay, let's get started. Welcome everyone. Thank you very much for taking time under your day for joining today's CNCF webinar, Secure Self-Service Kubernetes. I'm Jerry Fallon and I will be moderating today's webinar. Just a few housekeeping, we would like to welcome our presenter today, Mr. Jim Baguadilla, Founder and CEO of Nermada. Just a few housekeeping items before we get started. During the webinar, you are not able to talk as an attendee. There is a Q&A box at the bottom of your screen. So please feel free to drop your questions in there and we'll get to as many as we can at the end. This is an official webinar of the CNCF and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that are in violation of the Code of Conduct. Please be respectful of your fellow participants and presenters. Please also note that the recording and slides will be posted later today to the CNCF webinar page at cncf.io slash webinars. And with that, I will hand it over for that's where our presenter for today's webinar. All right. Thank you, Jerry, and thanks everyone for joining. So today we are going to talk about secure self-service Kubernetes. So just to kind of unpack that and set the context a little bit. If you think about where we are today in the cloud-native journey as the space of walls, there's several ways that we all know and that we all use to bring up Kubernetes clusters and kind of use them for our personal use. So it's fairly easy to spin up Kubernetes, whether it's on our laptops, on different virtual machines, cloud instances, or even using a cloud provider. It's possible to get a Kubernetes cluster running fairly easily. But the challenges that when we work with enterprise customers, larger organizations, the challenges that you typically end up facing are what we think of as day two challenges. So this is once you're past that initial deployment, it's dealing with Kubernetes configuration management, especially around the security and best practices of configuration, having visibility and governance into the cluster configurations, what's running on clusters, and making sure that the required services are running. And then dealing with cost compliance, as well as managing, of course, the amount of clusters and the amount of resources that get used. And it's interesting, with every new technology, it seems like when virtualization came out, we had VM sprawl. So now, certainly, we're starting to see the signs as Kubernetes continues to explode in popularity. We're starting to see Kubernetes sprawl also happening. So what's the way to deal with this? We all know that in our space, especially, and given where we are, going back to a traditional IT approach where you're creating IT tickets and you're waiting for resources to be provisioned or some central team to manage things, those days are gone. So there's no way we want to go back to that style of doing things. But at the same time, completely going in the other direction doesn't work either. So making all of this, Kubernetes management, cluster management, developer responsibility is just not going to work also. And as we continue to, as Kubernetes evolves, if you go to any offering. So I was looking just a few days ago at the Google console, the GKE console. And it's amazing how many different features, how many options, how many different items are now available there. So how do developers know what to provision, what's required for their enterprise, and how do you manage this at scale? So certainly, this is not the best practice either. And there has to be a proper balance. So we want to get, as developers, as a software developer myself, I don't want to, if I need some resources, I want to be able to self-service, create them as needed, get them at the right time, and dispose of them if I'm not using them. But I want to make sure that I'm compliant with my enterprise policies. I want to make sure that these clusters are secure. And I want to make sure I'm, of course, managing costs and expenses correctly without having to take on the additional burden of managing things at scale, but do this in an easy manner. So that's the topic we're going to cover today. And just to actually get to the punchline, and we'll go through these in each detail. But what I'm going to show today is at least three things you're going to need to achieve what we are just saying. So to get to secure self-service Kubernetes, you require policy management, and especially workload policy management. You require some form of virtual clusters. And I'll talk about some of the work going on in various Kubernetes working groups, as well as commercial tools and things that are available for Kubernetes virtualization. And then you're going to need add-on service management in a central and scalable manner. So these three are the fundamental things that we believe are required to do secure self-service Kubernetes. And of course, as the space evolves, as more things come in, this list may grow. But at least as of today, these are the fundamentals that you will need. So just a quick introduction. I'm Jim Beguaria, founder and CEO at Nermota. Within the Kubernetes community, I work in the policy working group, as well as in the multi-tenancy. And some of the things that I will present today are directly related to ongoing efforts, things that are happening within these working groups. Certainly, if you are on the Kubernetes slack, feel free to reach out. If you're interested in any of these topics, feel free to reach out and join in in one of our sessions. All right, so before we get into the three things I mentioned, I want to take a small detour and talk about cluster sizing. Because this is often a debated point in terms of should you have large clusters, small clusters, or how do you manage clusters within a larger organization. So of course, it's possible to say, OK, well, let's just do the spinning up clusters as easy. So let's do one cluster per app. Or maybe we have a hybrid kind of model where we're doing one cluster per team. If you happen to be working with on-prem clusters or private data center type deployments, you are more likely to be on the other side of things. You'll have larger shared clusters because these are fixed cost resources. They are assets which have already been deployed. And you're going to end up trying to securely share these clusters across your teams, across your organization. So some of these you can think of as single use clusters, either one team or one application using those. And the idea is that you make it easy to provision or you just allow your teams to go to your cloud provider of choice. And with some governance, they're able to bring up their own clusters. And that seems easier because there's less to do from a central point of view. But of course, it's going to lead to inefficient resource usage because everybody is going to over provision based on their capacity requirements. And the net resources that you're going to end up using are going to be far greater. And you're going to have many more clusters to secure and manage. So perhaps if you're dealing with three to five clusters, this is not a problem. But once you get your double digit clusters or in some cases even dozens or hundreds of clusters, this becomes a daunting challenge in terms of how to manage and secure them across all of these. On the other side of things, on the other side of the spectrum, if you're using extremely large clusters, I mean, yes, you can get better utilization. You kind of now will require some team to manage these clusters. So perhaps that's a drawback or that's a benefit, depending on how you perceive that. But there will have to be an operations team managing these clusters. And then these larger clusters can be more complex to manage as well. So because of the multi-tendency, and we'll talk specifically about some of those details, what's required to achieve multi-tenancy based on namespaces and other Kubernetes constructs. So both options have some trade-offs. And really what it comes down to is that, of course, you want to share clusters whenever possible, but you have to invest in the automation to be able to do this. And we'll look at some ways of achieving that. But even if you have shared clusters, you are likely to end up with multiple clusters for DevTest, for staging, for production. Perhaps you have several regions in which your applications are deployed. So multiple clusters is going to happen. The question is just how many and who's managing them. And independently or regardless of how you end up organizing your clusters and organizing access to clusters, every cluster needs to be secured and properly configured. Because if you're running your applications on it, most likely they're accessing secrets. Even if it's a test cluster, if somebody gets access to that, bad things could happen. So every cluster needs proper security. And as you'll see, there are tools and there are things available to help with these. All right, so let's dive into the first topic that I mentioned, which is workload policies. So the goal of these policies are, if you're familiar with Kubernetes, of course, there's the Workload API and there's several constructs available as we're defining and manage applications. So like with everything in Kubernetes, there's plenty of choices and there's plenty of ways to perhaps make mistakes or even insecurely configure various things. And this is where workload policies become extremely important to make sure to audit your configurations, to make sure that they're secure. Your workloads are well configured and they're following best practices. Now, Kubernetes itself has certain policies like pod security policies or network policies, things like that. But very often, these are pod security policies in particular is right now in alpha state and there's no road map or there's no plan to take that beyond because of some of the limitations and challenges with adopting PSPs, which the way they were designed was they tie back into role bindings and to configure PSPs. Really, there's no way of rolling these out the way it's defined right now without impacting existing workloads. So the idea is, and moving forward, there will be policy management tools and I'll profile Kivarno as a policy management tool. There's, of course, open policy agent, which is also an alternative and I'll discuss briefly the comparison between the two. But there needs to be other ways of securing configurations, auditing, reporting, as well as checking for configurations and best practices. Now, the other thing to keep in mind is that these policies are not just something you want to apply at admission control or even just within your cluster, right? They need to be things which you can check offline in your CI CD pipeline. So just like you would with image scanning, the best practice would be to not even allow as bills occur, as yamls are generated or checked in and to get. You want to make sure that you're checking for the right security and best practice configurations. And you want to then, of course, have another layer of enforcement at admission control. And even with admission controls, because there are ways to get around admission controls, as you might have seen in other KubeCon sessions or if an admission controller happens to be down for a certain amount of time, to be able to also have a layer of background scanning. So all three of these are required. And this is what we'll talk about as we're looking at some of these configuration security. So just to quickly introduce Giverno, the policy management tool that I'll talk, and we'll look at this live. We'll look at some demos in action. So the idea here is to use Kubernetes itself and to use the extensibility of Kubernetes to be able to define and manage policies, right? So what this does is to be able to, with Giverno, you can validate configurations using overlay styles. So if you're familiar with customize, overlays are a way of defining some YAML fragment, which you can use to then check whether the expected YAML exists in the configuration that you're inspecting. You can also mutate with Giverno so you can, Giverno can plug in as an admission controller. So you can mutate and be supported. Giverno supports both JSON patches, as well as a strategic merge patch, which, again, is what Kubernetes uses, KubeCuttle uses, as you are dealing with configurations. Another feature here is to be able to generate configurations. And this is especially useful for namespace-based multi-tenancy. So to generate and synchronize configurations across namespaces. And Giverno, I mentioned it acts as an admission controller. But it also does background scanning. And it also has offline scanning to be able to use this within your CI CD pipeline, right? So there was a question on chat, which was also on scanning. So if you're using Giverno, it covers all three. If you're using OPA, there's Conf test, which you can use in DevTest outside of the cluster. And I'm not sure if Gatekeeper or OPA do scanning within the cluster. But they are able to do this within the CI CD pipeline and then as admission controls. All right, so just a quick overview of what a policy in Giverno looks like. And again, we'll see some of this live. So policy is composed of one or more rules. And each rule can match certain resources. And there's a lot of flexibility. The whole point of designing Giverno is to make it as Kubernetes-native as possible. So there's a lot of flexibility in how you would match resources, exclude resources, based on namespaces, label selectors, kinds, even user roles, groups, things like that. And then once you match the right set of resources, you want to apply that policy to, you are able to mutate, validate, generate configurations through Giverno itself, right? So a sample policy looks something like this, and we'll actually, again, look at this live, look at the policy in action. But just to, if you see the structure, it's very simple to understand, right? So it's got a message that will be shown. It's saying that it's checking. It's using two patterns, right? So that's why we have the any pattern declaration. So if any of these patterns fails, then the policy itself will fail. And here we're checking the security context at the pod level. So this policy is applying to a pod. And we're checking a security context at the container level. So it's doing checks at both. And if at least if one of these is defined, then the rule passes and this configuration is, you know, at least passes this policy. In the Giverno project itself, we have about, you know, 20 or 30 best practice policies. It covers all of the PSP, you know, checks that you would want to do. And by the way, pod security policies are evolving. There's a proposal to have, you know, standard profile levels for pod security. And Giverno will support that and be able to indicate what profile level your workloads, your pods are compliant with. So just a quick comparison of, you know, how Giverno works and with OPA, which Open Policy Agent, right? So on the left, I have, you know, the same policy as on the right. On the right, we have the Giverno example. On the left, we have the OPA example. So the big difference here is if you're using Open Policy Agent, your policies are gonna be written in Rego. And this is something that either you would, you know, get a policy bundle, or you would need to manage and, you know, understand Rego to be able to, you know, configure and administer these. And then, you know, you would need something like Gatekeeper with its constraints language and things on top to manage that Rego itself. So the complexity and the common, you know, kind of feedback we've heard, the reason why we developed Giverno as an alternative is that, you know, Rego of course has some learning curve. It is a programming language, you know, and it requires understanding it and being familiar with it. And also like using, you know, other tools on top, like Gatekeeper and Conf Test, et cetera, they start adding some complexity as well, right? So in contrast, Giverno is pretty straightforward if you understand Kubernetes resources. And if you're familiar with how, you know, different deployment spots work, stateful sets, et cetera work, Giverno is very, very much, you know, in tune with that and uses the other advantages you can use all the same tools like KubeCuttle, you can use, you know, GitHub, style workflows, customize policies themselves become Kubernetes resources, right? So it's very easy to write a customization and change your policy, which you can't really do if you're looking at Rego, right? So that's, you know, kind of the main trade-off here, but it's, you know, something that you should explore and kind of see which one works best. And what we're doing is we're, of course, as we, as these tools mature and as these get built out within the policy working group, we're looking at ways to standardize certain aspects of policy management for Kubernetes. And that's a very, you know, interesting area of development as well. So some information on Giverno, but I wanna kind of go to a demo here and we'll come back to this. And, you know, also, I'll show you what the repo and things look like, so where you can find info. But let's, you know, switch to a quick demo itself, right? So the first thing, what I'm gonna do is I'll bring up, I'll show you, you know, and we'll just follow the instructions as we would as a new user. So let me hide my meeting controls because that always gets in the way. So if you go to Kiverno.io or to the GitHub, you'll see, you know, our documentation is all in Git. And all I'm gonna do on my cluster is I'm gonna just use the simple way of installing. It's a one-line, you know, configuration. And let's take a look, let's make sure that I have a cluster available. So here I'm just using on my Windows laptop, I'm using Docker for Windows and I have a Kubernetes cluster which doesn't have anything in it right now, right? So I'm gonna paste that one line, you know, kind of configuration and this is gonna, you know, install Kiverno. So if we go back and do get namespace, we should see there's a Kiverno namespace. If I do, let's see what's running in there. We should see, you know, the Kiverno pod running, which is good and it looks like it's ready and everything's initialized, right? The way this works is when the pod comes up, it has an init container which also registers itself as a mutating and validating web book. So at this point, every API request that, you know, is sent, Kiverno has visibility into, but we haven't configured any policies yet, right? So like I mentioned, on the Kiverno site, there's several best practice policies. If you go scroll down to our documentation and look at sample policies, pretty much everything you would find in pod security policies, plus other policies are all available here. And we're constantly adding more. We also have, you know, feel free to submit a PR or even a feature request if you feel that there's a policy missing and there's more being added, you know, kind of pretty much on a daily or weekly basis. What I'm gonna do for this demo is I'm gonna go back and I have here one of these policies. So in my, in my, you know, kind of scratch folder, I have a lot of different policies. I'm gonna use the same policies I showed in the slides, right? So it's a little bit familiar. So all this is doing is it's checking to make sure that, you know, a root user, a pod is not, cannot be run as a root user. And this policy is written to authenticate or to match pods, right? So let's do, let's go ahead and apply that policy. So what I'm gonna do now is I'll say, create minus F and this is in my temp folder. So I'll pick up, you know, the policies from that. So this is the policy I wanna apply. And now if I do, you know, in my cluster, I do get C Paul cluster policy for short. I see that I have one policy, right? So let's try and run a workload at this point and see what happens, right? So I'm just gonna, I have another, you know, an engine X deployment I usually use for testing and policies and different things. So I'm just gonna say create minus F and C temp engine X YAML, right? So what you see is I have one policy and what that policy did is it scanned this resource as it was being applied and it said it was blocked and it, because, you know, running as root user is not allowed. So let's take a look at the YAML and see what we have over here and why it got blocked. And a few things I wanna kind of point out which are somewhat interesting. So this is a YAML, very, very simple, straightforward, right? So I have some resource quotas, but the thing here is I have a deployment and, you know, that deployment is, it doesn't have, you know, the required configuration from the policy, right? So run as non-root is, you know, not set to true and which is why this particular policy failed itself. The other thing I wanna mention is this, what I ran was a deployment, right? I did not run a pod, but Kivirno was smart enough to detect that, you know, a workload controller was running when there was, or was trying to run and there was a policy defined at a pod level. So what Kivirno does is it automatically generates policies for workload controllers. You can manage, you can control which, you know, workload controllers at targets, but by default it will do this for most of the common types like deployment, stateful sets, job, crown jobs, et cetera. And because of that, I got this message back when I tried to create the deployment. If Kivirno did not have this feature, what would have happened is my deployment would have been accepted, but then I would see a failure when I tried to, you know, when the pod was created, which is just a little bit kluji and hard to debug, hard to kind of figure out, right? So it's several, and that's just one little example of a feature in Kivirno. There's several such features which make it super easy to, you know, manage these policies. Because it's a Kubernetes native tool, it can take advantage of the knowledge of Kubernetes workloads and how they are structured to do some of these kind of, you know, tasks, right? So, you know, feel free to browse the repos, you know, laid out in terms of there's a number of different things documented here. You can try some of the simple examples, but that's, you know, where that's the first thing that we believe is required when you are, you know, trying to configure secure self-service. So with something like Kivirno and your clusters, now you can, you know, you have not only, so by the way, here I was running Kivirno in enforce mode. I could run this in audit mode and just have it report back policy violations, which I could then, you know, later choose to, you know, manage or, you know, based on those policy violations, I can go inform the workload owner that something needs to be fixed in their particular policies, right? So lots of flexibility in how things are done and very, very native to Kubernetes and easy to use care, right? So again, that's one of the things you need, but not the only thing you're going to need to get to secure self-service Kubernetes, but something like Kivirno, you can now, you know, sort of have the peace of mind that configurations are following best practices as workloads are running in production, you want to enforce and block the things which are non-secure and which, you know, don't have things like resource quarters or probes, all of that you can do. And in DevTest, you might want to run in audit mode and just inform users of this. Oh, one thing I should show, and this goes back to the scanning. So before we kind of move to the next topic, it is the same, and here I did, I tried to apply this through KubeCuttle, right? When I created this, but I can also do something like I can say use the Kivirno KubeCuttle plugin, so I'll just run this. So this is a CLI, which you can find from using Crew and you can install as a KubeCuttle plugin. And what I want to do is I'm going to apply a policy now in my CI CD pipeline. So I'll apply that same policy to that same YAML and we'll see what happens, right? So in this case, I want to do, let's go find that policy. So we'll say this allow root users and I would say minus R for resources and you can even apply an entire folder of policies if you have these or you can apply one at a time depending on whatever is easier. And the workload here was just this Nginx workload that we wanted to do, right? So Kivirno will also, this runs in an offline mode and what happens is it applied. So there's two resources, the deployment and the service and it said one pass because of service pass because there was no policy for that, but one failed and it's telling you exactly what failed, right? So this is how you would run Kivirno in an offline mode and be able to apply and kind of manage policies as part of your CI CD pipeline itself. Okay, so let's go back to the next section and we'll take a look at virtual clusters and cluster virtualization as the next topic that you might wanna consider as you're looking at self-service. Yeah, so a couple of other questions. Let me answer on that this section before we move in. So the one question was how do you spell the, what is the language that OPA uses? It's rego, R-E-G-O, rego or rego. And the Kivirno GitHub link, if you look at the Nirmata GitHub repo, it says Kivirno. So you can go, let me just display that. It's github.com. Kivirno, I can also add that in the chat or we'll make the links available or if you just go to kivirno.io, you'll also be able to get to the GitHub from there. All right, so let's go back and talk about virtual clusters next. So Kubernetes, as many of you probably already know, the origins of Kubernetes were from basically trying to schedule and bin pack containerized workloads on infrastructure, right? So the idea is you have a pool of resources, a pool of infrastructure, and Kubernetes is gonna figure out the best placement, the best bin packing algorithm for making optimal use of those resources. So that's where, I think in that it's sort of unfair to think of Kubernetes only in that sense, but that's one of the core functions. The scheduler portion is one of the core functions of what it does, right? But of course, as we were just discussing policies and workloads, we know Kubernetes also has a fair amount of complexity and rightly so because it solves some very complex problems. So there is a learning curve and there's learning to do things to master within Kubernetes itself. And what this has done is because of that security, the challenges in deploying security, more and more, like what we've seen is teams have said, okay, I'll just deploy my own cluster. We don't sort of, and you kind of bypass that central management, right? So this is where, if you kind of look back again in the progression of most technologies, virtualization and almost every, like when you think about networking or storage or even compute, things started in the physical world and eventually we had virtual kind of abstractions for those resources, right? So similarly, we believe that Kubernetes is going through the same curve and eventually, I believe it's gonna be more popular to use virtual clusters versus actual physical clusters. Because virtualization just allows a lot of flexibility and typically when you think about virtualization, you're what we're discussing is more defining a software defined form of a resource itself. So rather than having to manage an entire cluster, if as a developer, I want my own sandbox, I can get my own virtual cluster or if I even as a team or as an application, I'm using a virtual cluster. To me, it's a Kubernetes endpoint as long as it's compliant with the Kubernetes APIs and there's no unnecessary abstractions, that makes it super easy to manage Kubernetes, right? So here what we're specifically gonna discuss is namespaces-based virtualization and how to achieve that, right? So this is more suitable and you might all have heard that, like hard multi-tenancy with Kubernetes is hard, right? It's difficult to achieve. Some might claim it could be even to completely secure Kubernetes and have a virtual cluster, you have to start replicating things and segmenting things and there are ways to do that, but that's not something you would typically do within an enterprise, right? So the type of sharing and the type of reuse we're looking at over here is for teams within an enterprise, within an organization that are looking to share a cluster and that model we see as being very popular in terms of a request and to be able to mature that and use that easily is what the multi-tenancy working group is focused on. There's various projects, there's a project called HNC which is hierarchical network controller or namespace controller. There's also the Kubernetes multi-tenancy benchmarks which I'll talk about. That's one of the tracks that I work with and that I'm leading within the working group. And then there's solutions being developed for control plane type virtualization. There's a proposal from Alibaba on actually having separate API servers and different to allow control plane based virtualization. So there's also other solutions for data plane virtualizations but again, those are more advanced topics and you require, it's not just standard Kubernetes at that point, you're extending Kubernetes in some way to be able to leverage those. So let's focus on namespaces, right? So what do you need if I have, you know, I wanna use namespaces across teams. So obviously you can deploy a workload in a namespace and if you look at the Kubernetes doc in fact says Kubernetes supports multiple virtual clusters backed by the same physical cluster and these virtual clusters are called namespaces. So problem solved, right? Well, not that easy. Namespaces are just one part of what you need to fully segment and isolate workloads. What you're also gonna need of course is some way to automate and manage access controls. So role bindings as, you know, namespaces are created, requested, things like that. You're gonna need network policies. So you don't want workloads to be necessarily talking to each other or even having access to each other. So you need network policies for that. Then you're gonna need limits and quotas, right? So you need to make sure that one sort of noisy or one, you know, namespace that misbehaves doesn't take down everything else. So you need proper limits and quotas configured and you're gonna need, you know, the pod security aspects of this runtime security to make sure that if somebody pods are not accessing host resources, pods cannot, you know, escape out from their, you know, their PID or their namespace itself. So all of those things need to be checked and done, right? So there's quite a lot of configurations and things which have to be managed. But the good news is Kubernetes does have constructs for each one of these and there are ways to automate this and there are ways to measure this. And I'm gonna talk about, you know, this multi-tenancy benchmarks is what we, you know, are developing within the working group as a way to measure this. So the goal there is independently of how namespaces are configured or which tool you use or how you use to create these namespaces. How do you measure that a namespace is properly configured and, you know, doesn't have any multi-tenancy type violations, right? So this is, you know, the link that I'm sharing is within the Kubernetes 6, there's a repo and actually let me, I'm gonna do another demo where we're next gonna look at this, right? And here, I'm gonna hide the meeting controls again. So if I go to the multi-tenancy repo under Kubernetes 6, within that folder, within the benchmarks folder, so within multi-tenancy you can get your benchmarks and within benchmarks you can get to, you know, this tool called kube-cuddle-mtb. So it's another kube-cuddle plugin and the goal of this plugin is to measure multi-tenancy, right? So what I'm gonna do is I'm actually gonna, let's run this in a different namespace and I'm gonna show you how this tool works. So here, let me check my namespaces again. I'm gonna create, you know, a namespace call, let's call it multi-tenant test, right? Empty test, oops, I need to tell it what to create. Okay, so now the next thing I'm gonna need in this multi-tenant test namespace is I need some roles, some user roles to use to create or to build this, right? So I'm gonna go ahead and by the way, the instructions for what I'm doing right now are all on this website. So here, notice it says create a namespace user role, create a namespace. So I'm just kind of following along and doing exactly this, right? So I'm gonna use let's do minus n in multi-tenant test and I'm gonna say create minus f here under my inner root. I think I have this role, so role binding. So let's go ahead and create that. So now if I look in my namespace and if I say get role binding, or let's actually just describe it so you see what it looks like. What I'll have is one role binding for this user Allie who is a namespace admin in here, right? So the first thing now, and I, by the way, already just to save some time, I have built this too, but if I wanna just run this, if I do kubectl, mtb and if I do get, it's gonna show me all the available benchmarks and check that this tool can do, right? So there's about 16 or so benchmarks that it will do. And what I can do is now if I say, if I just look at the help, the other thing we can do is we can say run. I'm gonna say namespace multi-tenant test and just to start with, I'm gonna use a role which is not defined and it should produce some errors, right? So if I say minus as, or I think it should be minus, minus as, and we'll say let's try Jim and there's no role for Jim, right? So it should show me some errors that it can't run these checks and none of these can be applied for this particular namespace, right? So that's what you're seeing. It's going through that bunch of errors. A few things passed because I did configure a role, right? So, and I did configure a namespace. So as that user, I'm not able to, well, if the user doesn't exist, so I'm not able to go ahead and do anything meaningful with that. But now let's run this as this user Allie who we configured before and see what happens, right? So at this point, instead of errors, what I'm expecting is, my check should actually fail because the user exists, but I'm not able to pass these tests based on the user. So if you just create a user by default using, or namespace by default, this is what you're gonna get. Now, how do we fix this, right? So to fix this, you can again use a policy management tool like Giverno or like OPA Gatekeeper. And there's examples for both here. Like if you want to try Gatekeeper, you want to try Giverno, you can do that. But I'm gonna apply all of the Giverno policies that are required to start passing these particular checks, right? So just following the instructions again on the multi-tenancy benchmarks website, I applied all of the policies required and there's about, if I do, kubectl get sepal again, I'll see there's a whole bunch of policies now configured, right? And let's run that test again to see what happens this time around. So we're running again as user alley in the same namespace to see is this namespace properly configured for multi-tenancy? And what I'm expecting to see is for everything where we have a standard policy, there will be, so you see a lot more passes. We still see a few failures because in this namespace, we just created it. I don't have any quotas. So let me go back and actually, we'll add some quotas, right? So here I have like namespace quota. If I add that and let's rerun this test one more time, we should see more passes again, right? So I think you get the idea of how using some of these tools now gets you to a point where you're able to apply these policies and be able to configure namespaces in a quite a secure manner where the only failure we have right now is there is no policy there's an issue logged on this. So right now there's no policy for node ports. So once we have that policy defined in here, this check will also pass. And the reason why this check is failing is because we do with kubectl with MTB, you need to define a label for how to identify multi-tenant resources. So that's expected to show an error in this case, right? So again, a simple way, a very quick way of creating namespaces and managing them. And that's how you can get to namespace based virtualization using Kiverno or other policy engines as well as you can use the benchmarking tool to now audit for this. The other thing I wanna show is and now I'm gonna show actually how this works in Nermaka. So just as an example for what can be built using this, you know, I have a Nermata account. So for those of you who are not familiar with Nermata, Nermata is a multi-cluster management tool. This is my cloud-based account. It's a SaaS-based product as well as can run on-prem as a private edition. I have, you know, a few different clusters like three clusters in this account. And what I'm gonna do is I'm actually, I've configured a user as a DevOps user which is a restricted role, right? And that user, so although I have three clusters in this account, if I do NCTL, which is the Nermata command line, or wherever I say clusters get, I only have access to one cluster. But what I can also do is I can, you know, create environments which are virtual clusters in Nermata. So I can do environments, create, and that will tell me how I can create an environment. So in this case, I could say something like minus minus. So let's say, you know, we'll call it demo ENV one. And then I'm gonna say minus minus cluster. And I'm gonna use prod demo. Whoops. And I need to give, in Nermata, we have to give it an environment type. And these types are configurable by the admin. And in my account, I just happened to know I have a type called medium. So I'm gonna create an environment of that type, right? So really, all it took is now in a few seconds, as a developer, I can request a virtual cluster. I got in a virtual cluster. And this is a properly configured namespace with all of the policies, everything required, for security, for resource quotas, for network policies, for the roles. And only I have access to this namespace, right? So now I can invite others from my team as required to be able to provision this. So everything I did, of course, you can also do from the Nermata UI. But I just kind of wanted to show what that would look like if I did this from the command line itself. All right, so that's the second thing I wanted to cover. And I know we only have 10 minutes left, so I'll speed up a little bit. And I do wanna cover very quickly the last portion, which is add-on services and why that's necessary, right? So what I add on services, I think all of you know, like when you bring up Kubernetes or if you use the Kubernetes dashboard, there are some standard add-ons, like metric server, et cetera. And even the dashboard itself would be an add-on if you're using Minicube. But so typically within an enterprise, there's cluster-level add-ons. And these are shared services that are used by every team, by every tenant. And they must be installed. They have to be present for security, for governance, for policy management, like with Kiverno. And then there are other add-ons, which you might need which are maybe optional, but useful for on an environment basis or on a team basis, right? So good examples of this are like using Prometheus for monitoring your app, using perhaps Loki for your log management, ingress controllers for your namespace, so on, right? So the challenge with add-ons is who manages these? How do you make sure that the right version is running, that the workloads that are being deployed, again, are secure, they're maintained properly? And how do you update these across all of the clusters, right? So what I'll show you as kind of the last demo that I wanna cover here is, again, I'll kind of show first how this works in Nirmata, and then we'll talk about some of the procedures behind it. So this time what I wanna do is I wanna say NCTL cluster types, so instead of environment types, I'm looking at the available cluster types. And as a developer right now, what I have available to me is this one cluster type, which it says it's on GKE, it's a Google Cloud and Kubernetes Engine cluster. And what I wanna do is I wanna request a new cluster, right? So I'm gonna say cluster create. And if I look at that command, it requires a few flags. So I'm gonna say, let's say I'll call this GKE demo one, and then I'm gonna call, give the required flags as required has. So here I have to say cluster type, and we'll say GKE demo, because that's the only cluster type available to me as this particular user, right? So much like before we saw with virtual clusters, what happens is now Nirmata has received that request. If I go into back into the Nirmata account, I'm seeing that GKE demo cluster one has started provisioning. In fact, if I go to my Google console, it should have picked up the request and I'll see that here too. But what's interesting here is not just that, okay, we're able to create a cluster, that's fine. But what happens, how is this cluster configuration managed, right? How is it that I'm only able to specify, I really didn't specify anything other than the cluster type. And the way this is done is through separation of controls, right? So if I go look in Nirmata and I look at the cluster type that we use, GKE demo, the administrator has provisioned all of this and this is possible to do through the UI, or you can do it through Paraform or you can do it through the command line. But this is where all of the details of what's required in that cluster, including the add-on services that are required, are configured. Now, once this is configured, what this enables is when somebody wants a cluster, they don't have to repeat all of this information. They're just saying, hey, give me a cluster of this type and here's how many nodes I want in that cluster, right? So again, the idea is to have a central team manage this. And if I quickly look at how again, let's say one of these add-ons is managed, if I go into my catalog, and let's look at this application. So I wanna see how the Walt injector is managed. So this actually is pointing to a GitHub stream which controls, which has the YAMLs for this. So I can use GitOps, I can change the YAMLs and the platform team would be able to manage this application. And that would automatically then get rolled out to one or more clusters which are listening to this. And I can, as a platform team, I can create a release whenever I want. So if I have some new changes, I can also publish this release in any of these channels. And that's where, so I think this already exists. So let's call it two. And if we publish this, this will create, it'll pick up the YAMLs, the latest YAMLs, create a new release and start a rollout across all of the downstream clusters that require that particular application. Now all of this, of course, you can do also through GitOps, you can do through customize and your own CI CD tools, but the idea here is to centralize the add-on management and make it super easy for one team to be able to, a central team to manage the add-ons, the required add-ons like wallet, et cetera. So if you go back and GKE typically takes about five minutes or so to bring up the cluster, this cluster should be coming up fairly soon. And the nice thing is when that cluster comes up, not only is it provisioned with all of the required add-ons, but if I go in a separate window, I have walled open, if I actually go and look at my policies here, I will see that once that cluster is created, the authentication path is automatically created, the cluster does its handshake with vault, which I have configured. And then at that point, it's able to securely get secrets from vault, right? So again, this is not something then each developer has to manage or each team, but it can be centrally managed through a platform team. All right, so just a quick summary of what we covered and really enabling self-service security. I guess my hypothesis and what I started with was it requires at least three things. So workload policies, virtual clusters, and add-on services management. And the good thing is Kubernetes has some pretty well-defined and great constructs for all of these, as well as in the community through tools like Nirmata. There's a lot of things that you can leverage and make this easy across different cloud providers, across different infrastructure vendors. And this is where I love this kind of quote from Grady Butch. For those of you familiar with programming concepts, Grady Butch is one of the, I guess you can call him a father of object-oriented and other theories, where he says, you cannot reduce complexity, right? Complexity is natural, but the best you can do is manage it. So similarly with Kubernetes, I think we need to shift our thinking from trying to reduce that to be able to come up with the right ways of managing it. So definitely, there's a lot of topics we covered. So feel free to, again, reach out to me on the Kubernetes Slack or on Twitter or LinkedIn. You can contact me on pretty much everywhere as Jim McGuadier. And if you are interested in Kiverno or in any of the other projects I mentioned, feel free to check it out from the Git repo. Also, if you wanna try Nirmata, there's a 15-day free trial at try.nirmata.io. So feel free to sign up. There's no credit card or anything required. You can just sign up and register your clusters and start using it. All right, so I think that's everything I wanted to cover. And I know we're up on time. So let me check and see if there's any questions. There's one other question which we haven't answered. The question is what exactly was the environment that we created? It's a good question. And I kind of rushed through that. Environment is really a virtual cluster using namespaces. So it contains a namespace. It contains network policies, resource quarters, role bindings, pretty much everything that we audited for using the MTB, the multi-tenancy benchmarks too. So all of that is preconfigured in an environment. And in addition to that, Nirmata gives administrators a way to manage environment types. So this makes it easy to have either different T-shirt sizes or different quota levels. And you can manage which, you know, where these environments are registered. Okay, thank you very much for your time, Jim. That was a wonderful presentation. That's about all the time we have for today. As I said before, today's recording and slides will be posted later today to the CNCF webinar page at cncf.io slash webinars. Thank you all for attending. Thank you again, Jim, for that wonderful presentation. Everybody stay safe, take care, and we'll see you all next time. All right, thanks Sherry. Bye everyone.