 Hello and welcome to the Kubernetes working group for multi-tenancy. Today we're going to be doing a project overview in which we will cover how you can get involved with our working group, which is part of the upstream Kubernetes community. And we will also be going over some of the three main projects that the working group has been working on, information about how you can use them, how you can get involved in contributing to them. Just to quickly kick off for the working group, the chairs are me and Tasha Drew. I am the director of product incubation at the Mware. I also work on project, I used to work on project Pacific. I was responsible for launching the Tanzu Kubernetes Grid Service for vSphere. My co-chair is Sanjeev Rampal. He's a principal engineer at Cisco. And then we have the project leads on this, on this presentation today. We have Adrienne Ludwin from Google, who works on the hierarchical namespace controller. We have Faye Guo from the virtual clusters and tenant controller project. He's at Alibaba. And then we have Jin Baguadia, who works on the multi-tenancy benchmarks. And he is the founder and CEO at Nermada. And each of those project leads will be giving a more detailed introduction of both themselves and the projects later in this presentation. So you may be wondering how you can get involved in the multi-tenancy working group. So for new contributors or people who are interested in getting started with open source, what I would, my number one recommendation is join our Google group. So that's the second link on this slide. You can go to groups.google.com and just search for us for working group, multi-tenancy, Kubernetes. When you join the Google group, that will give you access to our documents, our meeting agendas. You'll see links to go and watch our meetings that we've posted to YouTube. We'll post a link to this presentation in that same YouTube channel as well. But you will also get a calendar invitation to all of our meetings. And so a lot of times people get really confused about how to join our meetings. We have bi-weekly meetings where we go over project status. You can come and present anything you're working on by adding it to the agenda or suggesting on the mailing list that you'd like to talk about something. But that calendar invitation to that meeting and the Zoom link will all come to you if you just join our Google group. So if you're not going to do anything else, but you want to join that, definitely join the Google group. It may take a few days for that calendar invitation to show up. There's some latency in the system. Adrienne's working on it. Google will solve it any day now. But yeah, just be patient. I promise if you join the Google group, you will get that invitation. We're also in the Kubernetes Slack in our multi-tenancy channel. You can join that by going to slack.k8s.io. And yeah, we also have a GitHub repo where all of this code is available. You see that link at the bottom of this slide. So just kubernetes-sigs-multi-tenancy. So yep, that's how you can find us. That's how you can talk with us more. We are part of the open-source community. Super happy to have more people involved, more interest, and new ideas and new projects as well. So yeah, join us. And now over to Adrienne to do a deep dive into HNC. Thanks, Tasha. So I'm Adrienne. I work at Google Waterloo up here in sunny and warm Canada. Come visit next year. And I'll be talking a little bit about the hierarchical namespace controller, which is our project to add a concept of hierarchy to the Kubernetes namespaces that you know and love. So why would you want to do this? Well, the reason is that namespaces are the foundations for all policies in Kubernetes. So RBAC, network policies, quota, limit ranges, and also most of the CRDs that you create will have namespace scope as well, unless they're a cluster-wide operator. So everything really centers around namespaces. And yeah, sometimes you can subdivide things at a subnamespace like within a namespace, I should say. But by and large, most of these features either work best or only work at the namespace level. And you can see my talk from QCon EU from earlier this summer that goes into this in more detail. So namespaces are great, but sometimes you want policies across namespaces, such as to represent a tenant who has access to multiple namespaces in your cluster. So adding hierarchy to namespaces is a great way to enforce this idea of ownership and express tenancy, and also allows you to create self-service, what we call subnamespaces, which give you permission to create a namespace, even if you don't have cluster-level privileges, all while adding as little as possible to core Kubernetes. So agency, it's a great solution if lots of teams want to share a single cluster. And if you combine with a get-off solution to spread policies across multiple clusters, that's when it works well in a multi-cluster environment as well. So how does this work? Well, what it does is it creates a little agency, gives you this little customer resource that you can use to take existing namespaces and arrange them into parent-child relationships. And we call these full namespaces because you can modify the hierarchy. Or as I mentioned, you can create these self-service, what we call subnamespaces, where you just have permission to create a subnamespace underneath this particular existing namespace, and then that one's locked in place. You can't move it around. And so that's really good for those sort of self-serve contexts. And then once you have that, you can create policies in the ancestor namespaces, and they will just get copied into the descendants. You can pick which ones get copied in a cluster-wide config. Things like RBAC, limit ranges, network policies. And agency also puts some guaranteed labels onto the namespaces that mirrors the hierarchy. And you can use those as label selectors and things like network policies to get that kind of hierarchical enforcement. As I mentioned, I gave a talk that went into agency in-depth at QConiU. It's on YouTube. It's fantastic if I may say so myself. So please go look at it. But we are not sitting still, and we've been working pretty hard ever since then. So the big changes since that talk have been, well, the agency has been increasing in stability as we get more users, more people evaluating it, trying to break it and poke it in weird ways, which is great. We're finding some corner cases and getting rid of some of the patches that made it harder to use. We are adding exceptions and selectors. What this means is that, remember how I just said you put an object in an ancestor namespace that gets copied to all of the descendants? Well, sometimes you don't want it to go to all the descendants because strict hierarchies are too restrictive. So with exceptions, you can basically put a label selector on objects and that will stop it from being propagated everywhere in the hierarchy or even to know where at all if you don't want it for some reason. The last thing that we're adding is an improved API. So we did an API review after the API grown in a sort of ad hoc way for about a year. It survived pretty well, but there were a couple of nice changes that we made, things to make it more Kubernetes compliant. And so we're introducing this in the next version, which is 0.6, not gonna be, we don't think there will be many changes before we go to a beta. And if you already have each NC installed, we're gonna upgrade all of your objects for you automatically. So if you wanna get this, you do not need to get a new version of Kubernetes. It's just an add-on, so it's pretty easy. You can add it to any Kubernetes 15 cluster. I think it's on point and we'll be switching that to 116. For, if you want it on open source, you can go to that multi-tenancy repo that Tasha mentioned and just click through to HNC or go to the releases page and grab it from our releases. If you happen to be using GKE or Anthos, then you can get it as a product called hierarchy controller in config sync or Anthos config management. And it comes with a really nice plugin that you can install using crew, which is cool and it's available for Linux and Mac OS. If you wanna learn more about this, as I mentioned, I have the video from QConIU. Just after that, I wrote a blog post on kubernetes.io. So please go check that out and read through and learn more about the thinking behind HNC and the concepts around it. And if you would like to contribute, we are looking for contributors who wanna work on features that we like the idea of but we haven't quite been able to get to yet, things like setting us per subtree configuration. If you only want secrets to be propagated in one subtree but not another, creating subnets spaces with default policies that are not inherited, hierarchical resource code. Lots of people talk about that one and just generally improving productionization support. I was actually working on Prometheus today, but more eyes are always welcome, plus more to help with testing, more help with documentation, any patches that you wanna provide, we are always looking for people who want to help out with that kind of stuff. And so please stick around and I hope to hear your questions afterwards. And with that, I will pass it over to Faye to talk about virtual clusters. Just quick check, can you guys see the screen? Okay, so, hello everyone, my name is Faye. I come from Alibaba. In the past one and a half year, I was working closely with the multi-tenancy working group or project called Virtual Cluster. As Ingrid mentioned in previous presentation, the Kubernetes namespace is the primary API abstraction to support multi-tenancy. But by sharing for many tenants, users to share one API server themselves bring some problem that cannot easily handle the by namespace. But then they may have a performance interference between among each other. And when we really use namespace to organize all the resources and only give tenancy users to access to namespace, it is hard for them to create a cluster of resource, such as CRD or Web Hooker Central, which gives some trouble for their management stack. I see, we do see some cases that the namespace-based isolation is good enough, but we do see some other cases, people may want very strict or complete isolation and they won't have a solution which is compatible with the existing Kubernetes API. So people may do this by just creating individual, you know, cluster of the tenant, but in that solution, there is a problem that if each people has individual cluster by managing their individual nodes, it is harder to do the global optimization for the node utilization. So ideally, we'd like to have kind of complete control plan isolation solution where we can share the node resource among all the tenants. As I said before, we wish the solution has almost zero integration efforts for the existing system, which means we'd like to have a solution that is fully compatible with the existing Kubernetes APIs. In this project, so we introduce a new kind of architect for the virtual cluster. In short, the idea is that we will have a tenant operator which we are assigned a dedicated control plan for each individual tenant and the tenant, which we call the tenant master, the goal of the tenant master is to do, to maintain all the objects that are created by the tenant users. And there is a single controller which synchronize all the tenant created objects from the tenant master to a super master which is the Kubernetes cluster which manages the actual node resource. So in order for the different tenant to call the Kubernetes API to access or monitor the pod or do the logging or excc API code to the tenant in the past, we introduce a component of Covian agent, which proxy all the tenant request to the Kubernetes APIs. From a high level perspective, you look at this architecture. Ideally, the tenant master is the object state maintainer and the super master is can be as abstract as a resource provider only. It only provided a part of resource to each tenant users. With this solution, it is clear that with this solution, each tenant user has a complete view isolation. It won't be available for all the objects that are created by other tenants since they don't have direct access to the super master at all. With this solution, we also limited the velocity radius. If any tenants hit any security vulnerability, only that tenants get affected and other tenants won't be get affected. Also with this approach, since each tenant user has a dedicated control plan, it will have access to all the resources that presented in the tenant master, including all the CRD scope resource, RBAC control, et cetera. Since super master only works as a part of resource provider, it doesn't involves in the control plan management step. So this is what we do in the virtual cluster. What the virtual cluster really does from the high level architecture perspective. Next, I'm gonna give a brief status about how this project goes. At this moment, among all the three major components, tenant operator, single operator, and a VM agent, the single controller and the VM agent are pretty much as of now. They are kind of production ready. We are actually working on switching the tenant operator to a new cluster API based design and implementation. The idea is that we are trying to provide a more formal way to do the tenant master provisioning management by leveraging the all existing cluster API offering that cluster API C groups has been done in the past about a half year. They have a very good tooling around it and we want to follow their specification about how to define a tenant master component. This project was starting as a multi-faceted working group incubator project. And now we already get a C cluster API support and we are working on moving the entire code to a new repo called cluster API provider nested. So all the future development and the enhancement we have been done in that repo and this is an open source project and this is a community effort. We are very welcome. Anybody who are interested in this project could join the project by finding the issue by giving VC virtual cluster a try. So I'm happy to help people to use it and making this project better. So next I'm going to hand over to Jim for the next presentation. Thank you, Faye. Yeah, so hi everyone. I'm Jim Baguardia and I lead the multi-tenancy benchmarking effort within this group. And so both Adrian and Faye showed us different ways of implementing multi-tenancy. And as everyone probably knows, there's several constructs and Kubernetes that you can use for security, for multi-tenancy, for segmentation isolation. But one of the common challenges that we heard from folks in the community from customers is how do I know that my cluster is properly configured for multi-tenancy? Are there ways to measure this, test this and report this? So the benchmarking effort, our focus is to create a set of guidelines for, first off, of course, configuring multi-tenancy and then to also create tools to easily be able to validate and test whether multi-tenancy is properly implemented. So in terms of what are some of the definitions we have come up with and these are, of course, things that are evolving in the community. But right now we have two profile levels defined where the first profile level, the idea is that given any namespace, independently of how this namespace was configured, how do you make sure that that namespace has the right constructs as all of the right, again, the isolation, the segmentation required for multi-tenancy different teams using that same cluster. And then with the profile level too, the idea is to extend the first profile level and also add self-service for that. So not only do we allow in that profile level teams to be able to isolate and segment their workloads but also to be able to then do self-service operations like maybe defining custom network policies for their application, custom roles, things like that that would be required across different multi-tenancy constructs. Also in terms of the categories of the benchmarks and tests, we identified seven different categories. So everything from isolating the control plane from workloads, isolating tenant workloads from each other, network isolation, storage isolation, making sure host resources are not accessed from pods. So these are following like pod security best practices, ensuring fairness through configuring resource quotas. And then finally, self-service, which is profile level too, as I was discussing earlier. And then in terms of the tests, what we saw is just checking configuration is not always enough because there's an increasingly number of different ways. Just take, for example, network segmentation. There's a number of different ways you could configure a network segmentation or implemented. So instead of just checking for configuration, what we also do is run some behavioral tests in terms of how multi-tenancy is configured within the cluster. So today, the tests are mostly focused on namespace-based isolation, but as the virtual cluster project evolves, we'll also be looking at adding behavioral tests to be able to check on that. So with that, and I also mentioned that part of the effort of this track was to also come up with the validation tool. So we have a kubectl plugin which you can run on any cluster. It just requires a namespace and a role, and I'll give a quick demo of this. And what it will do is it will allow us to check and it will run about 15 or so different tests to ensure that multi-tenancy is properly configured. And this list of tests, again, all of this is on our repo. So feel free to browse through, look at the details, provide feedback and comments there. But let me just show what this looks like in action and then we'll come back and talk a little bit about the roadmap, et cetera. So here I have already a cluster configured and what I'm gonna do just in sake of time is I'm gonna show what this looks like when running. So here I'm using a particular user role. This is a tenant admin alley and the namespace here is test. So there's, it's gonna go, the MTB plugin is gonna run through a number of tests. And as you can see, what happened initially was it's a few of the tests pass because I have a quota configured, I have some limits configured, but a lot of the tests are failing at the moment, right? So what I can do and all of this, by the way, is documented on our web page. So if you browse into the project and if you look at the MTB webpage, you will see instructions for going through this exact same demo. But I'm gonna apply some policies now to for segmentation isolation within my cluster. I'm using Kevrono policies, there's also OPA gatekeeper policies, so you can choose which policies you apply. And as you can see, what these are doing is they're providing host isolation. There's different policies for also disallowing, things like adding capabilities to your pod, et cetera. So with that, let's run these tests again and what we would expect to happen is we would get much better results this time around as we're running the tool. And since we have the policy engine in place, which is guaranteeing some isolation segmentation. So this time, and as expected, we went through all of these 15 tests, we're seeing a pass. So this gives a very easy way for us to now validate that the namespace is properly configured. So a few of the things that we're working on and this is where, again, we're looking for folks to join in and help and extend some of this or even just contribute by giving feedback is we wanna add more behavioral tests for storage and networking. There are some pending requests for also converting this tool to run inside of the cluster and produce periodic reports using some custom resource for reporting. And then also, there's interest from other working groups to be able to reuse the benchmarking tool for auditing and benchmarking across different other types of security standards like the pod security levels, things of that nature. So that's a quick overview of the multi-tendency benchmarking project. And I think up front, Tasha gave some links. I'm just showing the repo links and other things again but we're here and we're open to take your questions in chat. So feel free to jump in and ask questions there.