 All right, welcome everyone to the talk Beyond Namespaces. Virtual clusters are the future of multi-tenancy. I'm Lukas, and I'm the CEO of Loft Labs. And I'm based in San Francisco, but originally from Germany. So, you know, excuse the funny accent. Together with my team, we're working on some really exciting things in the Kubernetes space. Our commercial product Loft essentially enables large companies, you know, to provide self-service, isolated namespaces to a large number of engineers, really like an internal provisioning platform for Kubernetes. So, we really care a lot about multi-tenancy, about user experience, developer experience, and about self-service. And because we care so much about these topics, we're also heavily involved in open source. We're working on four different open source projects. Our oldest project is called Dev Space. It's a developer tool for Kubernetes, and it's essentially a replacement for Docker Compose that is designed to be, you know, for Kubernetes, that you can streamline your workflow with Kubernetes the same way as you can do with Docker Compose with your local Docker daemon. But now you can also do that with a remote Kubernetes cluster. We're also working on Kiosk, which is a multi-tenancy extension for Kubernetes. And on JS Policy, which is a policy engine that allows you to write policies in JavaScript or TypeScript, making it much easier to write these policies and maintain them over a long period of time, even where someone in the company leaves that was the re-go expert writing policies beforehand. And one of the projects that we're working on is VCluster. And that's what we're actually gonna talk about today. It's a really, really exciting project around virtual Kubernetes clusters. So let's dive in. Beyond namespaces, virtual clusters are the future of multi-tenancy. Gosh, that's a pretty heavyweight title, right? I don't even know why you're attending this session. The future of multi-tenancy. Ooh, wow. What was that, right? Virtual clusters, never heard of that. You're probably all here because of first part, right? Beyond namespaces, right? You know, namespaces, yeah, I got this, right? I think everybody attending KubeCon probably knows what a namespace is. But do you really know what a namespace is? If we actually ask Google, what is a Kubernetes namespace? I mean, Google came up with Kubernetes after all, right? So let's just take a look. And if we go to the Kubernetes documentation, it reads there Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces. So namespaces are virtual clusters. Dang, we don't even need this talk, right? It's built in in Kubernetes, you all know about it. I guess we're done here, right? If you ask me, this is pretty outdated. Namespaces are not clusters. So they are not virtual clusters. Namespaces, and that's also part of the documentation, provide a scope for names in Kubernetes, right? For your policy, et cetera. Names of resources need to be unique within a namespace, but not unique across namespaces, right? So I can have two deployments called database and they live in two different namespaces and I don't have any naming collisions, right? That's great. That's what namespaces do and that's about it, right? They don't do a lot more. But as we all know, a Kubernetes cluster is much more than just naming objects. So what is a cluster? Not all objects are in a namespace. There are a lot of cluster-wide resources. They include nodes, persistent volumes, and even namespaces themselves, right? Namespace is a cluster-wide resource. If you actually want to check out which resources are cluster-wide and which ones are namespaced, you can actually ask kubectl, right? You can run kubectl API resources and you can either use the flag namespace true or namespace false to get the respective resources and Kubernetes will tell you all the resources in your cluster that are available. Hands up if you're a Kubernetes contributor. There's a couple, yeah, a couple more. So this slide is not for you because we heard earlier Stephen Augustus in his keynote was talking about we need help with documentation in Kubernetes. That docs page that I was showing earlier in particular, that sentence about virtual clusters is over five years old. It hasn't been touched pretty much since the inception of Kubernetes, right? And really a lot of people need to get involved to actually maintain this vast set of knowledge that we're building in the Kubernetes documentation. So after this talk, everyone should know the differences between namespaces and virtual clusters and should be able to fix this. And whoever is a first-time contributor to the documentation fixes this will get some kind of a little surprise present from me. So pay close attention to this talk and hopefully there's an edit page button directly on the docs page, right? Just Google Kubernetes namespace to the docs, open that pull request on GitHub. Well, let's get back to the topic. Multi-tenancy, why are you even here? Why should you care? Well, in my opinion, spinning up thousands of single-tenant Kubernetes clusters is a really terrible idea because it's very, very costly and it's very, very hard to manage. Just imagine you're spinning up a Kubernetes cluster for a thousand engineers, right? And they're spread across like 200 teams. You're gonna end up with 200 clusters per team or even a thousand clusters if you provision one for each individual user. And none of these clusters really is gonna scale down to zero. And then you gotta maintain things like ingress control or CERC manager, Prometheus, metrics, all of these things, right? In all of these clusters, that's a lot of work and it's really, really hard to keep in sync. And what happens if someone breaks a cluster, throwing away a cluster, recreating that? Typically, the user, the actual engineer wants to just connect this Kubernetes cluster to the CI-CD pipeline or just do some pre-production development work there. Doesn't have the permission to destroy a cluster and recreate it. It's pretty tough. So what about namespaces? Can't we use multi-tenancy, create shared clusters and essentially separate things by namespaces, right? That's the alternative. And a lot of people are doing that today. If we're comparing those two approaches, we see that namespaces have a lot of advantages. They're much cheaper. It is much easier to share resources, like for example, shared ingress controller for all of our tenants in this shared cluster. But a lot of other things are really problematic, like isolation of tenants, right? As we saw earlier, a virtual cluster is essentially just a naming strategy, right? The scope for our names. That's about the isolation level that a namespace provides. So we need to add a lot of things like network policies, our bag, right? We need to really lock these namespaces down so our tenants can't get out of there and actually bring the cluster down or harm another tenant, et cetera. And that's a lot of work. And if we're doing it well, then we're restricting that very same tenant that now cannot do certain things, right? So let's say there's a team in the organization that needs five namespaces and needs to do things across five namespaces, but you set up network policies in each namespace to lock the traffic down to that particular namespace. Now you've got to make exceptions for that team and who's going to clean up these exceptions later on, right? It's a pretty big mess when you're using namespace-based multi-tenancy. So if we're comparing those two approaches, separate clusters for each tenant versus multi-tenant clusters and separate namespaces for each tenant inside these multi-tenancy clusters, then we see that the benefits and the downsides are kind of like opposite on both sides, right? A lot of times we see problems like that. Can't we just get the best of both worlds? That's what we're trying with virtual cluster and with our implementation vcluster. vcluster is an open source project. You can access it on vcluster.com. You can find it on GitHub. And it tries to be multi-tenancy solution for Kubernetes that is beyond namespaces. It is more extensive than what a namespace offers, but at the same time, it is as lightweight as a namespace. So let's take a look at what that means. What do I mean when I talk about virtual clusters? For me, a virtual cluster is a control plane that runs inside another Kubernetes cluster. What is a control plane? Well, it is an API server, a Kubernetes API server. There's a data store where our namespaces, pods, et cetera, are stored in. That's typically at CD. Then we have a controller manager, which controls our replica numbers. If I create a deployment with five replicas, someone's gonna create these five pods. That's what the controller manager does. And then there's a scheduler, which then takes these pods and assigns them to nodes to actually where the keyboard starts these containers for us. And then we have the workloads on top of these control planes. And typically, we wanna create a namespace, we wanna create some pods in there, we wanna create a couple of other namespaces. And then what we always do is we talk to this API server. That's our gateway to Kubernetes. That's our single source of proof in a way. So, for thinking about virtual clusters that run on top of another cluster, and we have the capability to spin up workloads inside namespaces, then why not create a control plane that runs as a pod inside a namespace of our cluster? That's what a virtual cluster is. That means we have an API server and potentially a couple of other components. And it really depends on your attitude towards virtual clusters and how you technically wanna implement this, but you're certainly gonna need some kind of entry point that you talk to. So now we're not talking to that API server of our real cluster anymore, we're talking to this API server up here. So when I run a kubectl command, I talk to a pod, right? That runs a separate API server. So let's take a look at how this actually works. I recorded this demo and we'll create a namespace in a local, let's say, mini-cube cluster. We call it host namespace because it's supposed to host our virtual cluster. And then we're using the cluster CLI, which is a very lightweight binary to create a virtual cluster. We call it VC1 and we tell it, create it in host namespace, right? That's our namespace. We see that the cluster under the hood executes a health command, but you don't even need that. You could just run kubectl apply or create a customization out of it, right? But for ease of use, just running helm under the hood pulls a chart, deploys it to your cluster. And then we run recluster connect VC1 and specify our namespace again. And what this does, and you see that here in the output, it actually starts kubectl port forward to now connect to this API server, right? And the second thing that it does, and that's why the CLI is pretty handy to get started with virtual clusters over just using kubectl, it also creates a kubectl config file in our current working directory. So we're split the terminal here and I'm exporting the kubectl config environment variable so that all my kubectl commands now point to that virtual API server and not to my real clusters API server. So any kubectl command I'm running now is inside the virtual cluster. So let's just list all namespaces and we'll see there's a couple of namespaces in there like kubesystem and default and they've been created 51 seconds ago when we started that virtual cluster. We can also list pods in the kubesystem namespace and we may not even have the permission to do that in the underlying cluster. We're not admin in the underlying cluster, we just have access to a namespace and now we're admin in this virtual cluster. That means we can also create namespaces in that virtual cluster, solving our problem with network policies. I can now do things across 10 namespaces although I just have access to one namespace in the underlying cluster and of course I can schedule workloads and that's actually the tricky part, right? Creating a namespace is just an entry at CD or a data store but what happens if I create a deployment nginx in this case, I set the replica number to two and I schedule that in our newly created virtual namespace NS1. If we list the pods for these deployments a second later, we see that they actually get scheduled. The first pod is still creating and the second one is already running but where is it running, right? That's a question and I'm gonna dive a little bit into this, we'll see both of them are running now. Our virtual cluster seems to function like a regular Kubernetes cluster and that's really the goal with virtual clusters. So let's recap of what happened here from an architecture perspective. We created this host namespace in our regular Kubernetes cluster and then we deployed a virtual cluster with the vcluster CLI for ease of use, right? A virtual cluster has mainly two components. It has a stateful set, we call that VC1 just like the virtual cluster we created and it has a service. The stateful set essentially creates a pod. It has replica number one in this case and inside this pod is really where our virtual cluster lives now. We have two containers in this virtual cluster. The first container is a control plane. It has an API server as we talked about earlier that's definitely what we need to do any kind of Qtcl interaction and it has a separate controller manager and it has a separate data store as well so it has a separate state from the underlying cluster to store our pods, namespaces and everything we do inside the virtual cluster and then the third component is the so-called sinker and that is what is replacing the scheduler. You may have noticed our control plane is incomplete. The scheduler is missing. Instead we're gonna have the sinker and we're gonna talk about this in a second. Next part of the demo was essentially running Vcluster Connect which establishes this port forwarding connection through this Vcluster service so that we can now talk to our Vcluster API server. And this API server is, Vcluster is a certified Kubernetes distribution. It just recently became a certified distro. It's a very new project. We just launched it in Q1 this year and very recently became certified. That means this API server that we're talking to is 100% compliant with all the conformance tests that we have in Kubernetes and it behaves one-to-one like a regular Kubernetes cluster. You won't be able to tell the difference except I guess after this talk you should be able to know some tricks on how to find out if it's actually a virtual cluster. So let's get rid of our underlying cluster and just focus on the virtual cluster for a second. Let's get rid of this underlying control plane. The first thing I ran in this demo was creating a namespace. All right, that's an entry in our data store. By default, Vcluster is using SQLite because it's a very, very lightweight data store and by default we're using K3S's API server which supports SQLite as a storage backend but you can also use Postgres or even at CD if you need full-blown virtual cluster and we're currently working with K3S but we're also working on making real Kubernetes API server work with Vcluster as well as one of our next steps. So we're creating a namespace so that creates an entry in our data store and then the next thing we saw in the demo was creating a deployment with replica number two, right? So we have another entry in our data store, that's it. That's all we're doing with the API server and then the controller manager, part of our virtual control plane sees that new entry, right? It gets notified, it's listening on these objects and sees, okay, replica number was two, I create two pods. What does that mean? Well, it just writes two more objects into our data store. So we have four objects in our separate virtual cluster state now. We have a namespace, we have a deployment and we have two pods that are owned by this deployment. And in the beginning, these pods are just, they're waiting to be scheduled, right? They're nothing more than just an entry in a data store and now that's part of the scheduler to actually schedule these pods to nodes. But in a virtual cluster, instead of the scheduler with the sinker and that's actually what makes it really, truly virtual, what the sinker does is it copies the pods from within that data store that we have inside the virtual cluster to the underlying host namespace so that the host clusters scheduler can now schedule these pods, right? That means all the restrictions we have in terms of network policies, what we have in terms of our bag, anything we have in terms of service accounts being mounted into these pods and all these things can be restricted by the underlying cluster. You can have regular admission control, right? Everything is still enforced, but only at the pod level. The deployment doesn't exist inside the underlying cluster, only the pods. And because we can obviously create multiple namespaces in our virtual cluster, what about naming conflicts, right? We were just talking about that earlier, namespace, being the scope for names, how do we solve that? Well, what the sinker does when it sinks from multiple namespaces down to a single namespace in the host cluster, it actually renames these pods and it adds a suffix with the virtual namespace in this case and has one for both of these pods and even with the name of the virtual cluster because we could have just like three stateful sets with a virtual cluster inside the same host namespace, right? So we need those two suffixes. And it's really important to understand that sinker only sinks the minority of things, right? Very, very few things, mainly pods. So sinker sinks only low-level resources. That includes pods plus everything that pods need to start, right? So we typically take a look at that pod in the sinking process and we see it has a config map that is mounted. So we'll sync that config map as well, but the other 5,000 config maps don't need to be mounted. Same for secrets, of course, and same for persistent volumes. We also sink services to make service-based communication possible. And there's two optional sync resources which are also really basic resources in Kubernetes and that's ingresses. We added that because there's a lot of need for like shared ingress control or across different virtual clusters, right? So this is actually by default on NV cluster but you can disable it with a flag pretty easily. And then the question is, yeah, what are the nodes inside virtual cluster, right? What if I run kubectl get nodes? That's actually up to you as configurable because sinker sinks in both directions. It doesn't just sink pods down, it also sinks the status up. That's why we saw earlier, one of these pods was container creating and the other pod was already running, right? And we saw that inside the virtual cluster, although that state actually lives in the underlying cluster because that's where these pods are being scheduled. And it's really important to understand that those are only very, very, very few resources in our Kubernetes cluster. All higher level objects are not being synchronized. Only very basic resources are being synchronized. And high level resources include everything that is replica controlled, right? Deployments, stateful sets, steam insets, all these secrets and config maps which have nothing to do with pods where we just store some kind of state service accounts, jobs, a lot of these things are not synced. And most importantly, CRDs and the custom resources that we create from them. And that actually creates a really interesting possibility. If you want to install HelmChart today, you know, any kind of machine learning framework, there's a very high probability that you will need to add CRDs to your cluster. But if you're not cluster admin, you can't do that because CRDs are cluster-wide resources. But in your V cluster, you can do that. And the great thing is we're shifting left, right? We're shifting this permission and also the responsibility for these CRDs to the application-specific teams that actually care to work with these custom resources. And that's really great for IT teams because the underlying clusters can be pretty dumb. You know, they don't need any CRDs, right? They may need to add an ingress controller there and some basic monitoring for these pods that we're scheduling and obviously some network policies, right? But we don't need anything specific to applications. We are not going to run into any conflicts of CRDs either, right? Two teams can have two versions of CRDs and run side-by-side in the same host cluster but within two virtual clusters, very, very interesting. So the vast majority of resources exists only in the scope of our virtual cluster. So what about pod networking and DNS? Well, pod-to-pod communication is a given, right? Pods are actually running in the underlying cluster so they can communicate VIP just as they can in a normal cluster. Pod-to-service communication, as we heard earlier, we, by default, sync services and we do wire them up to the actual write pods, right? So that works out of the box as well because the sinker takes care of this. And then what about cluster or V cluster internal DNS? Well, as you may have seen earlier, when we listed the CUBE system, the pods in the CUBE system namespace within our V cluster, there was one pod already running and that was core DNS. So V cluster, by default, has in its CUBE system namespace a core DNS deployment and that one creates a pod, that pod is being synced down to the host cluster, right? That means we have a pod and also a service for DNS down there in our host namespace which now is managing our DNS. That means when the sinker synchronizes down to the host cluster, the only thing we need to do is modify the pod spec and point it to that DNS service instead of the regular cluster-wide DNS service and then DNS-based communication and name resolution works as well. And then, as I mentioned earlier, ingress as sync is pretty interesting as well to even get more capabilities out of this. So now let's get to a very interesting question. What happens when I run CUBE CTR get nodes inside a virtual cluster? Well, it's like always an IT, right? It depends, right? There are multiple options here. The first option, that's the default option, is fake nodes and they're dynamic. That means what we do is we let the schedule of the underlying cluster schedule our pods and then we see which nodes do they end up on and those nodes we add in our data store because a node is just an entry in our data store, right? Inside the virtual cluster and we tell this virtual cluster this node exists and then we fake it, right? In a way that we rewrite the name, we rewrite certain things in this node specification to make it obscure for the user what this node is about but the node for the user virtually exists. There's also the option to not do that obfuscation and just have the plain node being synchronized into the cluster. We can also enable a flag that will synchronize all nodes from the underlying cluster if we wanna see our tenants, I have our tenants see all nodes in the host cluster or we can add a label selector for our nodes and essentially enforce our pods during the sync process, right? To automatically get that node selector so that a certain V cluster can only use certain nodes and only can see certain nodes inside the cluster and all of this is configurable. We are pretty straightforward flags that you can pass to the sync container. What else is possible with virtual clusters? It's very, very exciting because there's a lot of things that you can do with it. One thing you can do is also run kubectl inside these pods which is very exciting because when you're thinking about it, this is a pod that actually runs in the host cluster, right? So if I run kubectl there but I started this pod from within the virtual cluster so if I run kubectl and I talk to, with a service account being mounted, I talk to the host cluster. So that's why we actually changed this environment variable in Kubernetes so that when you use the in cluster config for kubectl, it talks to the virtual clusters API server instead of to the real underlying host API server. So we get the true cluster experience even when running kubectl inside a pod. We can expose virtual cluster API servers. It's just a service in Kubernetes, right? So we can add a load balancer to it, type load balancer or we can put an ingress control in front of it and make it available to outside of the cluster, not just via port forwarding. There's a couple of things upcoming which are really exciting as well, making network policies work. If you're creating a network policy inside of your virtual namespaces and that has a namespace selector, that's a little tricky, right? Because you may not have the same namespace in the underlying cluster but we can also do that with changing things during the sync process and we're actually working on that right now a couple of weeks away from making this work. Pot disruption budgets is another hot topic that we're working on and non-rude vclusters. Right now, K3S runs as rude and they have the option, I think they recently released the option to run K3S as non-rude and that will be possible with vcluster as well. It's one of the most important steps in making vcluster very, very resilient and isolated and unless the virtual cluster is there is something very interesting, right? You could now talk to this virtual API server and run vcluster create again, right? Create a stateful set there, create a service and connect to this virtual cluster, right? You can see it, you can go pretty crazy about this and Rich Bowers, which works with me at Loft Labs, he's also attending this talk if you wanna chat with him later. He made a pretty great video about, it's called vcluster inception, nesting virtual clusters. You should definitely check it out. There are a lot of use cases for virtual clusters. You can create them for ephemeral CICD environments, any kind of integration end-to-end acceptance tests. You can spin up preview environments per pull requests and get an entire Kubernetes cluster. You can even have different versions of Kubernetes being tested in your CICD now on the same Kubernetes cluster, right? Because that API server you can enable alpha and beta flex is independent of your underlying cluster. The only thing your underlying cluster needs to do is creating pods, creating services, right? Very basic things, right? I guess if the pod spec changes with a future version, right? There may be some incompatibilities, but that the pod spec is having a breaking change in the future that probably we, very more problematic on other edges than virtual clusters. You can spin up these remote development environments that your engineers directly work with Kubernetes. So instead of having them maintain their pet mini cube clusters or Docker desktop, have them just spin up virtual clusters, right? Their throw way, they can create one, they create another one for a different project, right? That's actually what we do with our commercial project. So with loft, engineers are exactly doing that, right? They're spinning up these virtual clusters and namespaces. Before we open sourced V cluster, virtual cluster was for about six months already part of our commercial offering. So we already have companies working with this technology and essentially providing these deaf environments to the engineers and propping the local host environment entirely. And what we do as well is we put those environments to sleep when engineers are not working with them, right? So automatically pausing these virtual clusters, purging all the workloads and automatically spinning them up again when new cube CTL requests come in so that when your engineers leave the office at 8 p.m., everything turns off 30 minutes later because they're not working again or they're shifting between their virtual clusters, but they're only really working with one at a time and we really wanna make virtual clusters cattle in a way and make them disposable, right? And you want the engineer not to have to treat their pet local host cluster as they do it today. We see a lot of experimentation where you really need a lot of compute workload, MLAI workloads, we see cluster simulations as well. It's a very interesting concept, adding fake nodes to the cluster and things like that, right? There are so many things that you can do, like have like 10 virtual nodes mapped to the same underlying physical node, right? All of these things are possible, even virtual cluster spanning across different Kubernetes clusters, right? There's so many creative ideas of what you can do with this and you can really make multi-tenancy more resilient. Wing-Wing, that's kind of the name of the conference and you can do things like sales demos and higher level things as well with these virtual clusters. So how to get started? I hope you all got excited about virtual clusters and we try to make it as easy as possible to get started with them. You can essentially just download the VCluster CLI and run VCluster create and VCluster connect in any, pretty much any Kubernetes cluster. You can head to the documentation for more detailed getting started guide. Source code is available on GitHub if you wanna dive a little deeper, if you wanna open issues. We also have a VCluster website with some information. You can follow Loft Labs on Twitter. We regularly post blog posts about virtual cluster and shoot them out on Twitter. And definitely if you have any questions, reach out, right? That's why we're all going to a conference. We want this interaction and I'm so glad that we have this personal hybrid format again, right? Where we can really connect to each other. So if you have any questions, I don't ask now or join us on Slack, reach out via Twitter, come by the booth. I'm gonna be at the booth right after this talk. We're, when you're entering the exhibit hall to the right, close to the project maintainer area. Booth number is SU-49. We have a couple of pretty nice swag and I especially wanna point out, we have KubeCon postcards. They look like this and that really meant to be sent out to your colleagues who cannot be here in person. So feel free to grab one and just send it out to whoever cannot be here in person. And since we're a startup and we're working on really exciting technology in a lot of different open source projects as you've seen on the first slide, we're so hiring. So if you wanna work with my really brilliant CTO, definitely check out our careers page or approach any member of our team, reach out to us and see if we can work together. And I think we have about two minutes left for questions. So if you have any questions right now, just raise your hand, happy to answer them. Thank you.