 Okay, good morning, good afternoon, and good night, everybody. Thank you for joining us at this discussion of the Multitenancy Working Group, where we'll be talking about the similarities and differences between multi-tenancy and multi-cluster setups. Before we go on to the actual panel, I just want to basically introduce the group that all of our panelists are a part of. We are all members of the Multitenancy Working Group, which is one of several working groups that's organized under SIGAATH and a couple of other specialist groups in the Kubernetes community. This slide shows how you can join us. You can come join us on Slack at WG Multitenancy. We have a Google group. If you join that group, you will automatically get the calendar invites for our bi-weekly meetings, which are every second Tuesday at 11 a.m. Pacific, where you can go peruse our video, like the recordings of our meetings on YouTube. We also have a repo where a couple of our projects live. Those projects, we might have a chance to talk about them later in this panel. But those projects are the virtual cluster project, which allows you to take one Kubernetes cluster and subdivide it into a bunch of virtual clusters. There is the Multitenancy Benchmarking Set that will test your cluster for its compliance with multi-tenancy policies. And there are hierarchical namespaces, which is one of the things that I work on, which allows you to cascade policies across namespaces and treat them as though there's an ownership structure of namespaces. So please come join us. We're all pretty friendly, or at least I think we are. And we'd love to see you. So without further ado, I would like to introduce my fellow panelists today, who will be talking about Multitenancy versus Multicluster. Why don't we go in alphabetical order? So, Jim, would you like to introduce yourself? Sure. Thank you, Adrian. Hi, everyone. I'm Jim Beguaria, co-founder and CEO at Nirmata, and I work on the Multitenancy Benchmarking Project. Also, I'm a contributor in Kivarno, an open source policy engine. So nice to be here. Thanks, Ryan. Why don't you go ahead? Hi, I'm Ryan Bezecek. I'm a software engineer at Medtronic. I kind of help out everywhere in the working group. And I'm the co-chair for the Multitenancy Working Group. Thanks. And I am Adrian Ludwin. I am an engineer at GKE at Google Cloud. And I am the tech lead of hierarchical namespaces. We're HNC. We're HNC. We're HNC. We're HNC. We're HNC. We're HNC. We're HNC. We're HNC. We're HNC. We're HNC. We're HNC. We're HNC. We're HNC. We're HNC. And we're HNC. And we're HNC. We're HNC and the tech lead of hierarchical namespaces. We're HNC, the hierarchical dopamine space controller. So why don't we get started on our discussion topic today? So the first question that we wanted to talk about was what is the difference between mullticluster and multi-tenancy? So why don't we start in the reverse order we just went? So, Tasha. I want to give your — your reviews on this. Yeah. a bunch of different, potentially single or multiple use clusters. And so I just have a lot of clusters, I may have clusters for all. And so if that's kind of the model that I'm veering towards as far as how I leverage Kubernetes as a service in my organization, then maybe every time a dev team wants access to Kubernetes, we just spin them up a cluster, they can customize it however they want and it's their cluster. Meanwhile, when I think about multi-tenancy, what I really think about is the shared cluster use case. So I might have different development teams, different organizations all sharing a single Kubernetes cluster. And in that case, you really need to start thinking about how you're securing the control plane and securing access to underlying resources in order to make sure that you have a secure and you're not stomping on each other when you're sharing that cluster. Thanks, Brian. Do you want to add anything to that? I don't know, she covered quite a bit. Yeah, so I think I'd point out, right? I guess it might be like multi-tenant and multi-cluster aren't necessarily mutually exclusive, right? So there are multi-cluster kind of scenarios that you can have that include some form of multi-tenancy, be it how you're accessing, who has the cluster admin privileges, etc. And yeah, I think Tasha pretty much summed it up well. Jim, you want to go ahead? Yeah, I think one thing just I would add to that and it's interesting because, you know, again, as we talk to customers in the space, it's interesting to try out different terminology and see what resonates with them and, you know, how customers think of this, right? So one thing that we found which, you know, seems to resonate quite well is just looking at what is the end user ultimately obtaining or using, right? And so they want some form of shared Kubernetes, whether it's an entire cluster, whether it's a slice of a cluster, or even, you know, there's other emerging models where there's some set of shared infrastructure like we can talk about with the virtual cluster model. But ultimately, it seems like it comes down to either you're consuming a namespace, you're getting a namespace, which is part of a cluster, perhaps a set of namespaces with something like agency, or you're getting an entire cluster and then your responsibility as the end user within an enterprise is to manage the cluster and its lifecycle itself, right? So one way of looking at it is also either namespaces as a service or clusters as a service. And those seem to be the two popular models of consumption that, you know, I think we can further elaborate on. Yeah, I agree with everything that everybody has said so far. I think that the two are not mutually exclusive. In many cases, multi-cluster can be a way to implement multi-tenancy. And that becomes even more clear when you start looking at virtual clusters where there's only, where there's one underlying Kubernetes cluster and you're essentially running multiple Kubernetes control planes on top of that. And it becomes very clear that like the boundary or like the lines between, you know, single cluster and multi-cluster can actually be blurred. I think the only thing I would add to what others have said here is just the definition of a tenant often. And this was, I think, the subject of the very first panel that I was a part of as a part of this group was what is a tenant? And I've seen basically two broad definitions. One of them is that a team, like a development team, is a tenant. So a team runs either one or more services. And then the others that in the, like a software as a service in a SaaS context, every consumer of the service could be considered a tenant. And so maybe that doesn't matter if all of your services are implemented with native multi-tenancy. But if you have a single tenant service, let's say a database where you would not want to share one instance of a MySQL database between Coke and Pepsi is our two canonical, you know, competing customers, you would want to give them each their own database, which is to say each their own deployment. And so does that mean that they each get their own cluster as well, or could they each get their own namespace in their cluster, in a shared cluster? And that will depend on wide variety of factors, which I think we can get into now. So yeah, why should I care about the difference between, so now we've talked about what the difference is, and why should it matter what the difference is between the two? So why don't we start with Ryan for this one? Sure. Well, I mean, so there's a lot of reasons, right? So, and it really kind of, to talk about Adrian's point is it comes down to who is kind of leveraging this cluster, right? You know, there are certain things in Kubernetes that are cluster-wide and like notably CRDs, right? So CRDs are cluster-wide. So if you have a bunch of teams that want to deploy an operator, but different versions and different CRDs, it gets really messy really quickly. And so some sort of conformance there. But the other issue is kind of typically comes down to like security and auditing. Typically, you're going to want all of your clusters to have certain level of services running for, you know, your security teams or whatever and implementing like policies and all that other stuff, which is a lot easier on a single cluster that you do multi-tenant and harder on multi-cluster, but doable. And it so really kind of depends on what you're looking for and what attack services you're preventing against. But it's also notable to remember, right? At the end of the day, these are containers running on a single VM or bare metal node share in a kernel, right? So they're still sharing a kernel. There's no like CPU or hardware level isolation that we get with hypervisors, at least not yet. And so, you know, understanding your security posture and what you're running, if it's untrusted, trusted, you know, somewhere in the middle, that's really one of why you should care. Jim, do you want to... Yeah, sure. And it's, I think just to kind of extend on that, it's really about finding that balance and what's right again for each, you know, like a development team or any end user within an organization. And of course, if you kind of look at the whole sort of, you know, guts of Kubernetes itself, I mean, ultimately Kubernetes is meant to share a collection of resources and to do some bin packing and scheduling across those, right? Well, that's one function of it, but that's part of the core function. So obviously there is, you know, value in sharing resources within enterprises, within organizations, for efficiency, for agility. And it really comes down to also then, okay, if you're sharing these resources, how do you make sure that the right workloads are segmented or the right users are segmented? And folks have not sort of, you know, tripping on each other or stepping on each other, so to speak. So from that point of view, it's, you know, there's no right or wrong answer. I think what we're starting to increasingly see is that even within the same organization, some teams may just want, you know, a namespace and that's perfectly fine. Other teams may want, or perhaps there's a need in production, you want to dedicate an entire cluster to a certain workload, like your management tools or your security tools, things like that. So it's really finding the right fit and there's, again, there's some interesting trade-offs as you look at these different models. Yeah, thanks. Why don't I go next? I'm going to riff off of something you just said, Jim, like, there is no one right answer. And I see this a lot with people who are starting to either switch to Kubernetes or expand on Kubernetes. People are interested in what is the right answer and not only will I say there's no one right answer for everybody, but the right answer, even for one team, will change over time. And so why do people care? Well, it might be that you shouldn't care. It is, if you're just getting started, this might, like this, the right answer might be just do whatever is easiest for you at the time, but try to avoid closing the door to improvements in the future. So for example, probably the bare minimum I would say is don't run everything in the default namespace, because that will make it much harder if you ever need to share a namespace with somebody else on another cluster. But yeah, watch for, watch for those kinds of points where you are running out of steam with your turn approach. So if you want to give every team their own cluster, because that's easy for you, great. If you then later find out that that's very expensive because all of the clusters are regional, so there's a minimum of three nodes and everybody's running in multiple regions. So now each, every single workload that you have is taking up a total of nine or 10 VMs, even though your total aggregate load is less than one CPU, that's probably a good time to start considering multi-tenancy if the cost is important to you. If you have put your workloads into different namespaces, you will have an easier time moving in the future than if you didn't. But even if you did, even if you did put everything in default, I mean, there's going to be a lot of changes that you need to make anyway. So basically, it's good to be aware of these things so that you know when you're starting to run into whatever limitation is of your current approach, and just be flexible and know that it's going to change without getting too hung up about getting it right from the beginning. Tasha, why don't I give you the last word, although I think there might be a follow-up, so Tasha. We're all getting excited about this topic. Yeah, so I think the other interesting thing to me is dependency management in the Kubernetes control plane, and how do you really manage it? Because a lot of the things that we start running in the Kubernetes control plane stomp all over each other, right? So one example is CERT Manager. CERT Manager is a dependency that a lot of applications and services want to have installed if they're going to run, and if you have two versions of CERT Manager running on the same Kubernetes control plane, then nobody knows which one to call and you enter a broken state, right? And so just starting to think about Kubernetes as an operating system, right? Like starting to think about like how can I really understand what are all the dependencies of these tools and services I'm running here, and then how do I basically coexist with other tools and services? It gets really complicated fast, and I think it's going to be a big challenge for people who are writing services to run on Kubernetes to be leveraged by more than one team to really understand how to peacefully exist with the kind of the dependency modeling that exists today. Great, I'm going to follow up. So yes, one thing I think is worth pointing out. A lot of the conversations we have and a lot of the people that use multi-tenancy are concerned about it are really kind of looking at like dev systems and how do we reduce costs for sharing. But I guess I'd point out like it's actually probably a good idea that you've been considered using a multi-tenant solution in production even if you don't care about multi-tenancy because you get that isolation. So us specifically, right, Metronik, we deal with a lot of PHI data and some of our containers or applications that we run in Kubernetes are actually class three medical devices considered. So we have a lot of compliance around that. So adopting kind of a multi-tenant solution in your production workload, even if you're not multi-tenant, but having some of that isolation and segregating things off and only opening the holes that you need are just good practice. And I think some of the tools that we're working on kind of benefit at a larger level than, oh, I just need multi-tenancy. It's more of, how am I going to better, I guess, isolate and secure my production Kubernetes cluster? Yeah, that's an excellent point, right? Because there's often this misconception that, okay, maybe if I don't need to be multi-tenant, I no longer need to care about security in my cluster, which is not right, right? So you do want to make sure you're using namespaces correctly, you're using, you know, isolation for different components in your application, you have pod security properly implemented. So all of that matters whether you're running a single app or multiple apps, right? And so it doesn't, not being non-multi-tenant doesn't eliminate the need for security or reduce it, I feel. Yeah, and I'll go even further than that. Like even if you are running a multi-cluster setup, it's not like the different clusters can't interact. It's not like, oh, I've gone multi-cluster, now all my security problems have gone away. No, you might be sharing a network, like a network space, and you can send network to the wrong customers. I suppose if you're particularly paranoid, you could worry about VM escapes, although hopefully somebody else is helping you out with that. So yeah, there are, like, this is never the only aspect of this question that you need to be concerned about. Yeah, another interesting note on that, and this is still evolving as we speak, but with, you know, pod security, pod security policies being deprecated in version dot 21. There are new proposals being, you know, drafted in the community. There's a cap which is, you know, just coming out. And namespaces are going to become even more important than before for isolation and for that segmentation boundary, right? Because really, you're only as secure as the namespace that the pod is running in, and anyone who has access to that namespace gets all the privileges that namespace allows. So the new, the evolution of pod security seems to be moving towards using namespace segmentation and isolation. So projects like HNC as well as making sure that your different parts of your application are correctly namespaced with things like the right roles, the right network policies is going to become extremely important for any security or production deployment. And to get a little more specific, if you don't mind, right? So like, one of the biggest problems with, you know, pod security policies in general, right, is they're at a cluster level. So you're only really as secure as you're least secure because pretty much anyone can just say, Oh, give me that one. But that's actually a problem that we have in namespaces that a lot of people forget about is your service accounts generally, unless you add extra policy, you any container or pod running in that namespace can use any of the service accounts in that namespace, right? And people kind of forget about that. So if you're running cert manager in the same namespace as whatever, you might have some cluster wide privileges to alter the CA or whatever, because, you know, they have ownership. So segregating by namespaces in general is super important. And a lot of people kind of forget that at our back level. Yeah, I am there's an engineer I work with who works on GK who often talks about that namespaces should be like the atom, the smallest possible thing that you deploy that is indivisible, because everything other than that might change the permissions you want to assign to the service, the secrets that it might have access to the config maps, all of these things are at the namespace level. Now, of course, the challenge with that is that there have been policies that you want to have that span multiple of those atoms. And that's where things like HNC can come in to help you out, even if you don't want to then go to the virtual cluster or the true separate cluster approach. I think that I wanted to maybe let's poke a little bit more at some of these specifics, because we've been getting a bit more into detail. So let's start with Jim. We've talked a little bit about when you would you want to go from multicluster sort of down to multi tenancy? When would you want to consolidate? Would you like to say anything about what would push you in the other direction? What would you say to a customer who started out multi with multi tenant clusters? At one point, let's say other than CRD, should they say, well, maybe we should be considering either virtual clusters or entirely separate clusters? Yeah, and I think it comes down to also the again, back to the usage model, back to what the person or the team sort of getting the cluster is trying to accomplish, right? So very and as of course, you know, cloud providers and different software vendors make and even projects like Cappy make spinning up clusters easier and easier, it is very, you know, why not get a cluster, use it and sort of throw it away or dispose of it when you don't need it, right? So much like we've seen that evolution with VMs where at one point, I guess none of us remember this now, but servers were hard to get and you would get a server and you would keep a server, right? But now we're used to spinning up machines and disposing machines when we don't need them, right? So similarly clusters, it's definitely if you just need a sandbox, you want to do some tests, you want to run some batch jobs that make sense to spin up a cluster, it's completely isolated, at least at that boundary, of course, you might be, you know, sharing other things like networks and other underlying resources, but you have that entire cluster and you're not, you know, worried about impacting anything else running at that point. So there are those use cases and it comes down again to the type of, you know, workload or what that team or person is trying to accomplish with the cluster. Why don't I add something very briefly to that? So yeah, I think when you would go from a multi-tenant cluster to a virtual cluster is exactly what you're saying, Jim. When would you want to go from a virtual cluster to a so like a completely separate cluster, which is a separate set of VMs? I'd say that on the on the data plane, there's not a lot of isolation between your virtual clusters, certainly much less than there is between any two VMs on the same host. And so if you started seeing sort of noisy neighbor behavior, where one workload from one virtual cluster was impacting another, that might be a case where you would want to switch to fully multi-cluster solution. Obviously, if you have regionality, you need to be running in different locations, that's another thing that can push you out. Those are probably the biggest things that I could think of that might push you out of the virtual cluster level to the true multi-cluster level as well. And I think I've already spoken about what might drive you in the other direction, which is mainly cost savings, sometimes ease of use depending on what you're going for. Tasha, anything that you wanted to add to that? As far as like multi-cluster versus virtual cluster, I think that there's just some inherent complexity over sort of which projects you want to adopt and also just how you're managing your infrastructure, right? So as Jim was saying, like if you're using a public cloud, it's so easy to just spin out machines and also to add nodes, right? And so that's something that we've heard from some of our users is that when they're on-prem, it's actually really hard for them to add a node to an existing cluster if they want to add workloads to it. So it ends up being easier for them to spin up an entirely new cluster, like which ends up being provisioned on a different rack, right? Versus just adding a couple nodes over here where there's no extra capacity. So it just kind of seems to be pretty impacted by how your development teams are interacting with their infrastructure and the constraints on that infrastructure. And then also, and that's just kind of for internal use cases. And then when we start thinking about external use cases, there's just a lot of things to kind of unpack around security and, you know, like how sort of how open your customers are to like sharing resources versus do your customers even know they're on Kubernetes? Like what APIs or where is their point of interaction, right? Like they may not have access to the control plane. So who, maybe it's totally fine. So yeah, use case can really drive, I think, which of these tool sets and kind of paradigms you end up finding yourself in. Thanks, Ryan. And I think you get the last word. I think it's almost the time. Yeah. So I think, you know, there's a couple of reasons and in kind of like I said before, it might not be too exclusive, right? So, you know, for our specificity, right on our dev machines, we have kind of some shared clusters that are multi-tenant. But then, you know, when we look into like performance testing or like verification testing, those get, you know, production grade level clusters by themselves so that we can have that parity. So that's one of the reasons why, you know, those tenants are, I guess, separate. But the other major concern or not really concern is at some level, when your Kubernetes cluster gets large enough, sometimes it might not be worth it to kind of be like, oh, well, let's just keep, you know, figuring out how to scale this thing. And even though it can scale, right? Like we see Kubernetes clusters with 5,000, you know, workers, you know, fine. But it's important to remember, like every CRD you add and mutating webbook or validating webbook, all that's contention, you put it on the FCD and the API server. And at some point, you know, you might like, we certainly hit it and we have to hit it, you know, with less than 100 nodes, right? So it's, at some point, you just kind of like, well, maybe it's easier just to have another cluster for some of this stuff because 90% of or like 50% of our tenants don't need, you know, certain manager or something. And yet we're adding extra, you know, contention to their workloads because of those shared cluster-wide services. So I think at that point, you kind of get to a level to understand, you know, what makes sense to separate out. Yeah, thanks. We are almost a time, I just wanted to give maybe 30 seconds so that we could quickly talk about the projects that we're working on in the multi-tenancy working group and how they apply to some of these challenges. So, Jim, why don't you go first? Sure. Yeah. So the project that, you know, I'm most involved in is the multi-tenancy benchmarks project. And the whole thinking here was to, you know, of course, there's a lot of things you can configure in Kubernetes for isolation segmentation at the namespaces level. So the benchmarks projects is trying to make sure that that can be measured within a cluster through both configuration as well as behavioral checks. So it's a tool you can download either as a CLI or you can, you know, there's a container available to you and you can run that periodically. It produces a report which says, is your namespace properly configured for multi-tenancy? So you can check it out on the Get repo and happy to get, you know, more feedback and thoughts on how to extend. Thanks. I'll take one step up the stack, which is HNC, hierarchical namespaces. As I said, if you think of the namespace as the atom, you can think of HNC as the ability to create molecules, which is groups of related namespaces with related policies. And so it makes it easier to create a, like a multi-namespace network policy, for example, or RBAC policy. And we are working our way towards 1.0. I think we're our next release is 0.8. Does anyone want to talk about virtual cluster space in here? And he's the lead for that. Tasha Ryan? Yeah. So I was just pulling up his blog post but to show kind of the architecture. But basically the idea with the virtual cluster project is how can we have all of our infrastructure managed by one, what we call, I think, super control plane. But then everybody who is a tenant of that super control plane gets their own virtual control plane that they can customize however they want to. So as an individual user, I never actually hit the shared control plane because all of my requests are being proxied. And so then I can have the customization and sort of the experience of having my own control plane. But then the Kubernetes team, the Kubernetes platform team that's managing all of the Kubernetes based infrastructure still has the super control plane that they're uniquely managing and that nobody else has access to. Yeah. Thanks. Yeah. And if you are interested in any of these, please come visit our repo on GitHub. We're at Kubernetes SIGS slash multi-tenancy. You can find links to all of these projects. And we have some time now for additional questions. So thanks for watching everybody. And please add your questions to whatever method we add questions to.