 Felly, we are going to talk to you today about how we scale up using VCluster in Argo. Our use case is pretty niche, to be honest. So what I've tried to do is give you a few Argo hints and tips, which just help you with your general scaling. And you can obviously follow our journey as well, as we talk through it. By way of introduction, my name is Tim. I'm the infrastructure guy at Pipekit. I basically live in the CNCF Slack channel, so if you've ever asked an Argo workflow question in the last three years or so, I've probably spoken to you. And I do the same as part of the Pipekit services team, helping our customers just do better stuff with workflows. Hi everyone, I'm the co-speaker. My name is Lukas. I'm the CEO at Loft Labs. I'm involved with a couple of open source projects that my team and I got started. VCluster is one of them. That's the main piece of this talk. But there's also two exciting other projects you may want to check out. One of them is called DeafPod. It's for essentially spitting up deaf environments, kind of like Terraform style client only, but deploying pretty much any infra. It's a very exciting project we launched about six months ago. And then DeafSpace, a developer tool for Kubernetes. We handed it over to the CNCF end of last year, I think. And that one's a sandbox project now. I'm originally from Germany. You can probably spot the funny accent. I do live in San Francisco, though, since about five years ago. And yeah, come to our booth at C15 if you want to chat about anything in detail that we're going to talk about today. So we've both got funny accents. I guess before I explain the problem, it's probably worth explaining just what Pipekit is and what we offer. So Pipekit is a SaaS solution. We offer a control plane for our workflows, specialising in things like multi cluster fun, easier RBAC, that sort of stuff. And quite early on when we were developing Pipekit, we realised that our customers don't want us to manage everything. They don't want us to host their workflows and pass on the charges and that sort of stuff. Our customers in general want to still host the stuff themselves, want to manage that, want to keep it all nice and secure. And so the way that we operate is we give you a small pod to install in your cluster or clusters. And that pod is the conduit between our solution and your workflows. So you get full control over your stuff, but you still get the nice control plane to manage everything at a much bigger scale than open source workflows offers. The challenge we find is that those same customers who want that security also don't happen to have Kubernetes clusters just lying around to try new software. And they also usually have some form of red tape that prevents people from running up a mini cube or something like that for the same reason. So we struggle to get people to try our software to see if it's suitable for them and then obviously go on to hopefully buy our software. So that's our main problem and our goal is obviously to try and get people using Pipekit. So we came up with some really high level goals. We would like new trial lists of Pipekit to be up and running with their first workflow within five minutes of signing up. Sounds fair. And we want to reduce the friction of getting started with Pipekit by we want to offer them a hosted cluster so they can just play around, go nuts. And then if they're happy with Pipekit, then they can take it away and set up their own infrastructure with their actual real workloads. So we came up with some solutions. As a AWS shop, we offered EKS and we decided, well, we could just give every potential customer their own EKS cluster. That would definitely be nice and secure and isolated between different trial lists. But obviously that's reasonably expensive and spinning up an EKS cluster is not quick. It certainly wouldn't take under five minutes. We're realistically looking at 20, 30, 40 minutes maybe, which is a long time to wait if you're trying a product. You don't want to sit there just watching that happen. So I guess another solution is we could offer one EKS cluster and just put all of our trial lists in there. That's definitely cheaper and it would be quicker because the cluster already exists. But then we have this problem of, well, what if one trial list goes nuts and uses all the CPU and then all the other trial lists are stuck in the corner, the CPU's just been eaten up and they're not getting a great experience with Pipekit. So hopefully there's another way and luckily it was in the title and luckily the Lof Labs guy is here. So he's going to explain it. So one alternative to having single tenant clusters and namespace-based multi-tenancy is having Kubernetes virtualisation. You know, if you're in a company today and you're asking for a server, chances are pretty low that someone's going to go into a data center and plug in a server for you. They're going to run a Terraform script or something like that to spin up a VM, right? They're going to hand you out an EC2 instance or something like that. So why are we doing something super heavyweight and legacy sounding for servers and kind of abandon that towards virtualisation? But with Kubernetes, we're essentially making that same mistake, right? We're spinning up these heavyweight clusters and hand them out to folks, whether for trial or whether for deaf environments, right? There's a lot of replication here. So two and a half years ago, my team and I looked at this problem and thought that's kind of crazy. Multi-tenancy is not really solved in Kubernetes these days. Namespaces are not an isolation boundary. They're merely just like a, you know, space for names, right? Literally just like prefixing things. So now two objects can be called database, but that's about as much isolation as you get from a namespace, right? Virtual cluster is a virtualisation technology for Kubernetes that essentially creates virtual clusters on top of an existing Kubernetes cluster. We got this started about two and a half years ago. There's about 3,500 people that start the GitHub repository. You can check it out here. vcluster.com is a kind of landing page to get some more details on this project. In the first 12 months after launching this project, we had about a million virtual clusters created. Now two and a half years in, we just celebrated 40 million virtual clusters created. So there's a huge uptick in usage regarding number of virtual clusters spun up. And that's not even including, you know, like we're essentially counting image polls here, right? So if you pull that image once and then you instantiate it 100 times, that's, we just counted once. So that number is likely underestimated as well. I think you do very extensive image caching. So probably in your case, we count one, but yeah, they spin up like hundreds, hopefully thousands in the future of virtual clusters. So yeah, if you're interested in virtual clusters, here's the GitHub link as well to check it out. vcluster, what does it essentially do? It establishes secure multi-tenancy and Kubernetes. You know, if you're looking at single-tenant clusters today, they kind of look like this, right? We have like three clusters, and then we have three times, SIRC manager or policy agent, Istio or vault, everything is replicated. Everything runs idle most of the time, right? Very lowly utilisation, a lot of fleet management, a lot of things that you have to maintain and upgrade individually. These clusters look kind of uniform and the same. It looks actually way better in this picture than it looks in reality, because if you're like 12 months in, these clusters look really wildly different usually. And one solution is multi-tenancy. But namespace-based multi-tenancy, as we said earlier, is really not secure enough, not isolated enough. So instead of hosting your application stack down there and then running namespaces on top of it, you're essentially putting virtual clusters inside these namespaces. They just run as containers. It's literally just starting a pod. And then instead of giving a tenant access to the namespace, you give them access to the virtual cluster instead. That's at its core what Vcluster is an open-source project does. And it has a lot of huge advantages. The first one, it's a hell of a lot faster than spinning up separate clusters. They typically, these virtual clusters, unlike EKS or so that takes 30, 40 minutes to start, a Vcluster is just a pod. It starts in less than 30 seconds and you cache the images really well. It's going to go a lot quicker as well. He's going to talk about that in a second. It's a lot cheaper. Typically, we see cost savings coming from single-tenancy of 40% to 70%, sometimes even more. But although we run multi-tenant clusters of Vcluster, it's still very, very secure because everybody has their own isolated control plane. So we're not running with the same API server here, we're running with the same data store. Everything is separate, essentially. And that makes Vcluster a lot more robust for production. And they have been a lot of people proving this resiliency and production greatness and really battle-tested the solution. Here are a couple of logos of our customers and users. CoreWeave, I actually gave a talk with them this morning at the AI and high-performance computing day. They're actually a cloud provider and they hand out, they're building something like an EKS, basically, and they hand out Kubernetes to their customers. So if that's safe enough for them as a cloud provider, it's probably safe enough for everybody else to run in their internal use cases. There's also a couple of other really great folks on here, like GoFundMe, for example, CodeFresh last year gave a really amazing CoupCon talk about how they're using us. And now PipeKit has a really great solution out there as well, which has a really impressive startup time which we're going to see in just a couple of minutes. I'm just briefly touching on the cost savings here. Again, this is all open source V cluster, so there's no license fee or anything attached to this. It's just a patchy two license product. If you have single tenant clusters today and you have 100 of them, what you're paying is 100 times the EKS, GKE, AKS fee, and that's typically about $900 per year. And then you don't have any notes, right? It's just the API server, it's just a fire up that EKS cluster. You're paying $900. So if you run 100 of them, that's going to add up to about $90,000. That's quite some money just to turn it on, right? Nothing's running in there yet. Now we've got to deploy our platform stick on policy agent Istio, all of these components that you deploy to typically every Kubernetes cluster. What we found in our customer base, the cost is actually way higher than $1,600 per year, but we're going with a very minimal kind of startup example in this case. And that cost really, really adds up quickly. If you're switching to multi-tenancy and using virtual clusters, that's the graphic on the right-hand side, you essentially just have one Kubernetes cluster. If you're a really, really big organization, you may have like 10, but not 1,000, right? And then you run that platform stick on top. Instead of 100 platform stacks in 100 clusters, you just need one. That one is highly utilized, which is actually great, right? Because that Istio is not running idle all the time, it's actually busy all the time, and that's a good thing. That means that a platform stack may be a little bit more resource-hungry than each one of these individual idle running ones, but it runs at a much, much greater efficiency. And then you've got around 100 virtual clusters. We did our tests internally, so virtual clusters cost about $90 to run per year each, right? Which is about a tenth of the cost of a regular EKS or GKE control plane. And if you add all of that cost up, that's just, you know, essentially saving 85% of the cost that you had with a single-tenancy scenario. And then obviously you've got to pay for your workloads on top. We don't make your application any faster. Sorry, Tim, I wish we could, but hopefully they built the application in a way that actually runs really, really smoothly on Kubernetes and takes the benefits of running it as a container-based application on Kubernetes, and that runs effectively pretty smoothly as well. So let's tempt fate and try a live demo, shall we? So this is Pikeit. I've signed up, referring me to the cluster that we're just talking about. So I'm going to click the yes button and cross my fingers. So if we look behind the scenes, hopefully, we can see that a workflow has started and that workflow has two nodes in it. Really simple. A create node that creates the v-cluster and then a configure node that starts configuring the v-cluster, installing a software on top of it. And I have to talk really fast because this happens really quick. So v-cluster is here. It's pending. And if I refresh the page quick enough, it'll be in here somewhere, which I now can't see. You can have to believe me because it's probably ready by now. There you go. Oh, no, not quite. I've got time. Yeah, it is. That's deploying the software into the cluster there. So if we look back, the workflow's done and hopefully it's ready. Of course, I could have faked all that. So let's run something in it. So I've got a simple workflow written in hereer just to be a little bit different. So for those who don't know, here is a Python SDK for workflows. And we're going to run this directly against PipeKit. So this hereer workflow is really simple. It's designed to show you the versions of the software installed inside the cluster. So the workflow controller, PipeKit agent, we also installed Minio and it also just outputs the cube CTL version just to prove the version of the cluster that we're running. So I'm just going to run that here. This gives me a nice URL to go to. Skip all that. So we can see that the workflow has started and it's got a few nodes all to output that stuff. It's finished. And if we quickly look at the logs, for example, there's the PipeKit agent version. There's some cube CTL versions, Minio and Argo workflows. And that's all come from within that virtual cluster that we were just spinning up a second ago. Our average startup time is 39 seconds. So that includes starting the B cluster, installing Argo workflows, installing our PipeKit agent and installing Minio and all of them being ready for use. So as I say, 39 seconds. Sometimes faster, sometimes a little bit slower, but we haven't seen it any slower than a minute. So here's where I try and sort of help you out a little bit. As I say, our use case is quite niche. You're probably not trying to create V clusters for the purposes of offering free trials to customers. But I obviously, as you saw, I used Argo to do all of that. So I'll try and guide you through some of the steps and hopefully you can gain some nuggets of information out of that. So first of all, all of those virtual clusters are in their own EKS cluster. They're completely separate from the rest of our infrastructure. So if somehow someone managed to escape from a V cluster into our cluster, they couldn't get into the rest of our network. And we did all that with Terraform, which isn't exactly surprising. And we use Argo CD to manage the cluster. And importantly, Argo CD manages itself. Now, most of you are probably going, yeah, that's pretty standard. Argo CD manages itself. Everyone does that. But actually, in the Slack channel, people ask quite often questions like, how do I back up Argo CD? And the answer is, well, basically if you're asking that question, you're doing it wrong. Argo CD is a GitOps tool. Argo CD should be in Git. And everything associated with it should be in Git. And sequence management is the slightly confusing element of that. And there are lots of solutions for that. Unfortunately, I don't have time to go through it today, but there's a link up there to guide you through sequence management. The last link at the top there is actually an example designed to show how you could use Argo workflows for CD, which is irrelevant here. But the whole example is Argo CD managed, including the Argo CD that comes with it. So you can take that and you can play with it and you can hopefully see what I'm trying to go on about, basically. And then in terms of triggering the workflow, we use the workflow event binding to do that. Very simply, a workflow event binding is a webhook endpoint on the Argo server, essentially. You fire it a call message or similar, and it will start up a workflow template. And that's exactly what we did here. So we have two. We've got one to create the V cluster, which is what you just saw. And we've got one to delete the V cluster at the end of the user's trial. And that's it. That's the only thing we have. They're great for relatively small workloads, I guess, event loads. If you want something big with more scale, then you probably need to start looking at Argo events, probably queue management and things like that. But the sort of scale that we're running at, this sort of thing is fine. And it just means we don't have to add the extra complexity of Argo events. Then, as you saw, we had two steps in our workflow. The first step created the V cluster and the second step sort of set up everything inside the V cluster. So in terms of creation, the way I did it is used the Kubernetes resource element of Argo workflows. That allows you to create any Kubernetes resource. In our case, that was an Argo CD application that then creates the V cluster and deploys it into the cluster. You shouldn't do the thing I've just said to do because that's not GitOps. What you should be doing is putting something in Git and getting Git to do that. So in getting Argo CD to pull it from Git to do that because then everything's backed up. You can still use the Kubernetes resource, though, to monitor the deployed Argo CD application even though workflows didn't deploy that application, if that makes sense. Finally, we used Argo CD application sets to actually install stuff into the V cluster. So what happens is when the V cluster is installed into your cluster, the QPCTL secret is installed into your cluster as well. You can take that and you can give that to Argo CD and Argo CD thinks it's a normal cluster like an EKS cluster or anything else. You can do loads of stuff with application sets, but all we did was add a little V cluster label. Argo CD knows everything with a V cluster label. I should install Pykot agent, Argo workflows, MinIO, and off it goes. It also means that we can upgrade those things across the board with a click of a button, a Git push basically. So does it scale? I'll be honest, I wrote the slide before I found this out, but yes, it does. I essentially wrote a really noddy test to try and prove this. So what I did was I wrote a workflow that uses the Kubernetes resource that I've just mentioned before, uses it many, many times, and uses the Kubernetes resource to deploy another workflow saying hello world, hence the slide. If you do that on a normal EKS cluster, I was managing to run hello world about 2,000 times in two minutes. I then did exactly the same thing in the EKS cluster. I added 30 V clusters to the EKS cluster, ran the same thing again, and managed over 5,000 hello worlds in the same time. So the test is stupid. If you're running hello world this much, then you're doing something wrong. But the point is you can expect something like a 2x improvement just by putting V cluster in place. And that's because the workflow controller is talking to the V cluster API and not the Kubernetes API of the host. The host is a lot more relaxed, can handle the request coming from V cluster and you get that extra scale. Really quickly, here's some other tools that we used just to help things along. The image caching, as Lucas mentioned, an open source image cache. If your image has been pulled once in your cluster, this image cache will allow you to use it on another node somewhere else without having to go back to the original source and pull the image again. It's a lot quicker. The ArgoCD lovely plugin is a plugin that I worked on with my colleague Alan. It just makes ArgoCD lovely. It does a lot of stuff, so I'm not going to bore you here, but go check it out. It will probably be useful. Then the last two go hand in hand. Renovate is a bit like dependabot, but good. It works for Kubernetes resources. For example, if you've got an Argo workflow and you're pulling an image as part of your workflow, and the tag changes, Renovate can change that tag for you and give you a PR to update your workflow for the future. It also works on Helm charts, so Renovate can tell you when there's a new version of a Helm chart out. What Renovate can't do is merge in the changes from the upstream values file into your values file. That's where the last thing comes in. Again, it works on it with my colleague Alan. This is very opinionated, and you probably won't want to use it directly, but use it as inspiration. Renovate your values file for you and allow you to just have a nice PR where you understand what the upstream change was before you push it into your cluster. Let's take a look at what actually happens under the hood. We've been talking about the cluster and virtual clusters and enabling multi-tenancy, but what does that actually mean? How does that work? One of the things that Tim was saying is that that EKS control plane is actually under a lot of fire if you're spitting up a lot of and the way we resolve this is essentially by creating virtual clusters on top. Let's take a regular EKS cluster. It has an API server, an LCD controller manager, a scheduler. We pay at a best to run this for us, so we don't have to. Typically, as an admin, we talk to this API server. It means we have a cube context, plane cluster, nothing in there. We can create a namespace in here, and as we said earlier, we could hand that namespace out and make that a tenant. We're talking to the same API server. There's going to be a lot of load on there, so let's not do that. Instead, what we're doing is we're launching a V cluster, and as I mentioned earlier, that's just a part. That's why it starts up in 30 seconds and that demo on the PipeKit website is ready when you want it to be ready in less than a minute. What's inside this part? Inside this V cluster part, we have another Kubernetes control plane. That means we have another API server, we have another data store, and we have a virtual cluster. Instead, we have a sinker. The sinker is what connects things to the underlying cluster. I'll get to that in a second. The first thing I wanted to mention is if I'm a tenant now, whether that's an application or an actual person running CubeCTL commands, we're now talking to this API server on the top. We're talking to the orange API server, not the one at the bottom. It doesn't really matter whether it's an ECRS or a cluster internal IP, but you talk to this API server. That means there can also be different credentials and authentication. You can also hook it up via your OpenID Connect to your SSO, et cetera, but it's separate from the ECRS control plane. That means we get a new context. In this Cube context, we're not seeing that namespace or that V cluster part. But we can use it. That's the really cool part. That's where it comes in. We can share things on the underlying cluster, but we're not locked into using the same stuff, same versions, CRD upgrades, different Kubernetes versions. All of that is possible now because the V cluster is isolated. We can only connect it to things in the underlying cluster as we want it to be. Let's create a namespace. We can run some CRDs in there and set them up. That controller manager in the V cluster is going to say, oh, I saw there's, like, you know, replicle numbers too, right? I'm going to create two pods, right? The big question now is how do these pods actually get launched? Because the V cluster doesn't have any nodes. It's just a pod in a Kubernetes cluster. That's where the syncr comes in. The syncr effectively copies the pod down. But instead of having all the objects replicated and a lot of conflicts and this API server being under a lot of fire, we have separate API servers. And the only attack vector we have is the actual pod. And while syncing the V cluster in isolated mode, does network isolation that can really isolate this workload that it syncs down to the underlying cluster to really isolate it from all the other workloads in the cluster. And that's pretty much how V cluster works under the hood. And this V cluster on the top is a certified Kubernetes distro that means switching from a real cluster to a V cluster just means switching your cube context and it just works. We pass all the CNCFs conformance tests and it's really, really easy to switch over. Sometimes we see customers swap out real clusters of V clusters and their tenants don't even realize it. We just head straight to this. There's a lot of free stuff. Follow the links. We've run out of time. Thank you. And if you have any questions, let us know. We're also going to be around after the talk but I think there's a mic over here. So you briefly brought up the usage of Istio. I am curious how you would share Istio running on the host cluster inside of a V cluster. Yeah. So that really comes down to syncing more than just a pod. So by default we synced a pod because that needs to be scheduled by the underlying cluster. But if you're looking at Istio and things like a gateway resource or virtual service resource, what you can do and you can do that in the V cluster configuration, V cluster essentially you can just deploy it via a home chart, right? And in the values you can specify what resources get synced and how. So you can enable syncing for virtual service or for gateway resources and then these resources will be copied to the underlying cluster. You can apply modifications to them. We typically do some standard modifications to some of the resources to prevent collisions and to prevent the smooth isolation and then the underlying controller, the underlying Istio would reconcile these resources that are now available inside the V cluster. But unlike in the underlying cluster, so in the underlying cluster you would see all of the resources just like you see all the parts from all the V cluster but inside the V cluster you only see your own. And you can have N namespaces in the V cluster map to a single namespace in the underlying cluster. It's very flexible on what to configure how that syncing mechanism works. That's kind of the magic sauce in V cluster. Thank you. Hello. Question about the benefits of virtual clusters. Earlier you were saying that one of the big benefits for users if you've got hundreds of clusters virtual cluster is that you save on resources. So if you've already got like a large multi-tenant environment what's the big benefit of virtual cluster? So I think if you're using namespaces already that's what we're probably referring to large clusters, multi-tenant clusters with namespaces you usually don't gain a lot in terms of cost. You don't gain anything in terms of cost actually. You've got to run the recluster pod additionally. But what you're gaining is developer autonomy. So if you're in a multi-tenant cluster today and you're using namespaces for isolation and someone in your team wants to install Argo they need to install a CRD and they may not be permitted to do that they may have to go to the admins and to the platform team to kind of request this but then the question is like who's now managing it? Versus in the recluster if you want to install Argo you're going to be admin inside that recluster. So if Pipekit deploys any CRDs into these virtual clusters they're independently versionable like some of your tenants may be on a legacy version of Argo others may be wanting to test the new better version. All of this is possible versus if you're in a multi-tenant environment you're kind of stuck with having to agree on the minimal consensus for everybody and that becomes really tricky the bigger that multi-tenant environment gets. So cost savings if you're already using multi-tenancies not the biggest advantage that's more when you're coming from single-tenant environment. When you're in multi-tenancy it's more about gaining that admin access and the velocity and essentially the developer autonomy. Got you. Thank you. Customers bring up V clusters. It runs basically it's like the master running on a pod sort of without the worker nodes, right? Do they get a list of worker nodes or they don't? Or do they share? I would recommend watching the talk I gave this morning with CoreWeave on how they build up their GPU cloud because there is two options you have and then multiple sub variations of those. The default option is you're just using all the nodes from the underlying cluster but you can also have the V cluster apply automatic node selectors for example to divvy up things. One thing you can also do is you can join dedicated nodes into a specific V cluster and you can say well all of this node pool is shared and this subset node selector is available to these three V clusters but this V cluster gets these five nodes purely dedicated joined in. So there's a lot of flexibility on how you can handle these kind of node scenarios. And then you're leveraging anti-affinity rules and everything as such to do those separations between the workload. So one interesting thing is if you're using anti-affinity and those kind of scenarios you can actually strip these things away during the sync process as well. So you can leave it to your tenant to use these. You can also validate them in the sync process just like you would do with admission control. You can actually do it with the underlying host cluster as OPAR or other admission control as well but you can also strip it away during the sync process or modify it during the sync process. So what we do in terms of metrics is so in the sync process each of the pods gets labeled with which recluster comes from. And what the original namespace was inside of the V cluster. So when you then hook up things like Datadog and you want to filter, you can do that by V cluster and that makes it a lot more easy to manage in terms of metrics lock streams etc. As I said it's very flexible in terms of how you want to configure it. I think if you look at our standard helm values there's probably 1000 lines of different options on how you can set that up to that specific need that you have. I think we have one last question over here. My question might be a quick one. So I'd like to know about the network for the virtual cluster because previously the cluster have different physical IP and they are using flano or something to provide virtual IP to solve this issue. And how about the network in the virtual cluster? Yeah, so since the parts are synced to the underlying cluster, the underlying cluster's network, CNI etc. is still being used. So there's no difference. The only thing we have to actually look at is cluster internal DNS because DNS obviously you want to have addressable names and service names inside the V cluster that are stable that you can talk to. So the IP level layer is going to be handled by the host cluster but for DNS we actually deploy core DNS inside V cluster package with V cluster. That means each V cluster has its own core DNS and in our V cluster pro distro which Pipecat is using we even have like a built in core DNS plug-in that we ship as part of the V cluster control plan which allows you to do cross V cluster communication with stable service addresses, right? But yeah, by default you're just using the underlying cluster as CNI networking, same for storage as well. Okay, thank you. Thanks everyone.