 Thanks for coming to my talk about orchestrating multi-tenancy Kubernetes environments with Flux. My name is Prankaravi. I am a developer experience engineer at Weaverx. If you're, yeah, I saw someone, so yeah, I like to pose my dogs and make them model Flux shirts. If you're not familiar with Weaverx, we're the company that created Flux and also coined the term GitOps. And we have a bunch of other tools as well, such as a tool called Flagger, and then we have a lot of other things around the Flux ecosystem as well, such as our Weave GitOps UI, which is open source as well, and then the Terraform controller, which lets you do Terraform deployments and the VS Code extension as well. There's a bunch of others, but yeah. And then we also have paid support and other things around Flux. So I'm going to start with what is GitOps and what is Flux, and then we'll go into multi-tenancy. And before I get into that, I guess I'll share, yeah, I'll go back in a second. So along with being a developer experience engineer now at Weaverx, I was actually a software engineer at State Farm before this. And I was on the team that stood up GitOps there. And one of the things we did was set up Flux multi-tenancy on our on-prem Kubernetes solution there. So that's my experience with this topic. So what is GitOps? GitOps is an operating model for cloud-native applications such as Kubernetes, but it's not limited to Kubernetes. If you're doing multi-cloud infrastructure like we were doing at State Farm, you can still make it work for other platforms. But since this talk is about Kubernetes, we'll be focusing on that. And it also utilizes a version-controlled system such as Git, like most commonly Git, as the single source of truth. But there's other things you can use like OCI repositories and S3 buckets and stuff. So yeah. And then it also enables continuous delivery through automated deployment monitoring and management by a version-controlled system. So you basically manage your infrastructure and applications declaratively, which brings us into the GitOps principles. So these principles were actually set by talking to end-users and people in the industry. Yeah, y'all can come in, it's all good. We just started. And so they're created by the GitOps working group. And you can find out more about that at OpenGitOps.dev. We welcome people to join us at our meetings and stuff like that. Please, please come chat with us about GitOps. So the first one is that a system managed by GitOps must have its desired state expressed declaratively. So everything is written in code. It's reusable. There's an audit trail, things like that. Desired state is stored in a way that enforces immutability, versioning, and retains a complete version history. So in this way, there's no sneaking and changes. Everything's, you know, there's a trail. Software agents automatically pull the desired state declarations from the source, and they also continuously observe actual system state in an attempt to apply the desired state. So this is where something like Flux comes in and actually pulls your changes in. And if you're not, you know, fulfilling all these requirements, don't feel like you can't get started with GitOps because we didn't have all these set yet, and, you know, it's like a process. But this is kind of like the end goal. Like, this is what you want to have with GitOps. All right. So why? Why should you care? There's a lot of reasons. Like, there's a lot of benefits that come with GitOps. So individuals, like people that use it, experience many things such as stronger security guarantees. So because of the way that GitOps treats everything as code, it also creates a direct impact on security. So for example, if all configuration and security policy are treated as code, then everything can be held in version control, and everything, every change that's made is reviewed, input, and it's all automated. There's no manual processes, which means you're probably less likely to be at work on a weekend if anyone's done Friday deployments, they get it. And also there's like, so with that comes increased developer and operational productivity, makes the developer experience better, you can focus on things that you really care about, and there's more stability, higher reliability, consistency, and standardization as well. So now let's talk about Flux. Flux is a Git-centric package manager for your applications, but like I mentioned earlier, Git isn't the only source that you can use, and it provides a set of continuous and progressive delivery solutions for Kubernetes. So it's a natural, it was created with Kubernetes like entirely in mind, so it's a very natural extension of Kubernetes, and the core, it basically just continuously monitors your version control system, or your source, S3 bucket, whatever, and it applies the desired state that's been declaratively expressed there. And the nice part of this is that it actually runs on a schedule at reconciles, the terms of reconciles on a schedule, and you don't have to worry about configuration drift, because if something gets out of sync or something, then it'll actually set it back, which is nice. And it also really reduces developer burden, because like I mentioned earlier, it removes the need for manual deployment processes as well. Also the CLI, the Flux CLI is a really easy way to interact with Flux, so you can bootstrap the cluster, which is to get it set up really fast, and you can also access the custom resources that make up the API. So Flux is getups for apps and infrastructure, like I mentioned, the idea is you just push to get, and Flux does everything else. It is declarative, automated, and auditable. Flux and flagger, so I mentioned flagger earlier, using flagger you can actually deploy apps with canaries, feature flags, and AV rollouts as well. Flux can also manage any Kubernetes resource. Structure and workload dependency management is built in. Flux can even push back to get for you with automated container image updates to get, so like image scanning and patching. You can describe the entire desired state of your system in get, and this includes apps, configurations, dashboards, monitoring, and everything else. So you use YAML to enforce conformance to the declared system. You don't need to run kube control because all changes are synced automatically. And everything is controlled through pull requests, so your get history provides a sequence of transactions allowing you to recover state from any snapshot as well. It's also designed with security in mind, and so the whole pull versus push argument, so there's the least amount of privileges in this case, it adheres to Kubernetes security policies, and there's also a tight integration with security tools and best practices. You can read more about that in our docs under security considerations. Also we say that it's multi-cluster, multi-tenancy, and we like to say multi-everything, which we're going to talk about. Flux can use one Kubernetes cluster to manage apps in either the same or in other clusters, spin up additional clusters themselves, and manage clusters including life cycles and fleets. And it works with any Kubernetes and all common tooling that you're probably already using. Flux works with your Git providers such as GitHub, GitLab, BitBucket. Like I mentioned earlier, you can even use S3 buckets as a source. All major container registries, OCI registries, and CI workflow providers. There's also support for customized Helm, Harbor, custom webhooks. You can set up notifications on Slack and other chat systems as well, RBAC, and different policy driven validation like OPA, Kyver, no, admission controllers, whatever. And it is created with Kubernetes controllers, so it's very modular and you can tailor it to your needs. And also dashboards love Flux, we say that because it's really easy to set up whatever you're using your Flux situation on. It's really easy to set up dashboards and visualize your situation. There's also a lot of offerings out there for such a thing as well. And so the benefits of Flux is that it reduces developer burden. It also removes the cube control problem, like I mentioned a second ago, where you don't have to worry about cube control versions in order to interact with your cluster. And it's also really extensible, so it's versatile. I mentioned that it's Kubernetes controller, so it's like a microservice architecture. You can tailor your experience with Flux, you can pick and choose what you want to use in order to make your experience the best that it can be. And like I said, it was designed for Kubernetes and it comes with out-of-the-box support for Customize and Helm. If you're not familiar with the Kubernetes controller, they handle the lifecycle of objects in Kubernetes. So what should be done when an object is created, updated, deleted, et cetera. And so this is kind of like the breakdown of our little controllers. So the source controller fetches resources from whatever your source is. So get S3 bucket, whatever, and it stores them as artifacts. So it just grabs them and puts them, grabs them. And the Customize controller goes in and applies those manifests. And the reason it's called the Customize controller is because it's using Customize. So if you have a Customization.yaml already there that says, like, these are the files that I want to apply, overlays, whatever, it'll apply those. If there's not a Customization.yaml specified in the path that you tell it to apply from, it'll actually recursively search the folders from that path and it'll pull all the yamls that it finds. And it kind of creates its own Customization.yaml on its side and then applies it. And then the Helm controller manages deployments of Helm charts. And it's actually using the true Helm lifecycle. So if you use the Helm, like if you do a Helm release, you can actually interact with the Helm charts using the Helm API like you normally would if you did like a Helm install. So you can still like list things. You can use the API to interact. And then the Notification controller does notification dispatch. It handles inbound and outbound events. And so it can do like Slack notifications or like whatever you want to notify to. So let's say you make a change and you push and you want to be notified like, hey, a new change was just pushed and like Flux actually picked it up and deployed it. Or like if something gets out of sync, you can be notified. But another cool thing that the Notification controller can do that maybe some people aren't taking advantage of is you can set it up to listen for web hooks. So let's say I push to get and I don't want to wait, I don't know if my sync interval is like 10 minutes, I don't want to wait possibly 10 minutes for my change to be picked up. I can actually set up the web hook to notify the Notification controller, which then tells the source controller to go pull the change in. So yeah. And then the image controllers work together to update a get repository when a new container image is available. So the image reflector controller scans image repositories and reflects the image metadata into Kubernetes resources. And then the image automation controller updates YAML files based on the latest images scanned and commits the changes to a given and get repository. Okay. So now let's talk about multi-tenancy. Obviously, I'm sure, I'm sure mostly you know that there's different forms of multi-tenancy. There's hard multi-tenancy, which is, you know, each tenant has their own cluster. And there's soft multi-tenancy, which is where a cluster is shared across many different tenants. And my own experience, like working at State Farm and like what we did is actually soft multi-tenancy. So that's what I'm focusing on. And with soft multi-tenancy, the key here is that tenants need to be isolated. You don't want people being able to mess with other people's namespaces or whatever is in there, like their deployments, their applications. So, all right. So Flux differs to Kubernetes native RBAC to specify which operations are authorized when processing its custom resources. So by default, operations are constrained by the service account that the controllers run under. And it has the cluster admin role bound to it. So this works for a model in which all users are trusted. It's not always the case, obviously, that's usually not. Since tenants control Flux via its API objects, so RBAC roles need to be attached to Flux API objects. To give users control, in a multi-tenant deployment, each tenant needs to be restricted in the operations that can be done on their behalf. So to give users control of the authorization, the Flux controllers can actually impersonate or assume the identity of a service account mentioned in the apply specification. So there's a field that you can set in the customization object or in a Helm release object if you're doing Helm for both accessing resources and applying configuration. So this lets a user constrain the operations performed by the Flux controllers via RBAC. So in the tenancy model, there are two types of users. There's platform admins and tenants. So besides installing Flux, all the other applications, such as deploying applications, configuring ingress, policies, et cetera, do not require users to have direct access to the Kubernetes API. So Flux acts as a proxy between users in the Kubernetes API. So it uses Git and OCI as the source of truth for the cluster-desired state. So the platform admins would have unrestricted access to Kubernetes API. They're the ones that are responsible for installing Flux and granting Flux access to the sources, such as like Git, Helm, OCI repositories. And that makes up the cluster's control plane-desired state. The repositories owned by the platform admins are reconciled on the clusters via Git Flux under the cluster admin Kubernetes cluster role. So some examples of operations performed by platform admins would be bootstrapping. Like I mentioned, Flux onto the clusters, extending the Kubernetes API with custom resource definitions and validation webhooks, configuring various controllers for ingress, storage, logging, monitoring, whatever. They set up namespaces for the tenants and defined their level of access with the Kubernetes RBAC. They're basically the ones that are onboarding tenants and registering their Git repositories with Flux. And then the tenants have restricted access to the clusters according to the Kubernetes RBAC that's set by the platform admins. The repositories owned by tenants are reconciled on the clusters by Flux under the Kubernetes accounts assigned by the platform admins. So they would be doing things like registering their sources with Flux, deploying their workloads onto their namespaces, and setting up webhooks for alerting, for their release pipelines, configuring the release pipelines, and stuff like that. And so my experience, like before, was more on the platform admin side. It was kind of interesting. We had a separate GitOps team. So we were kind of in like a middle state between the platform admins and the tenants. And I guess, yeah, this is probably a good place to mention it, but the thing was, before we started applying multi-tenancy with Flux, nothing was declarative. If you wanted to go look at how the clusters were set up, it was all done manually or through API calls. There was no way of knowing what was on those clusters, any custom resources, like no CRDs. You couldn't see it unless you went and actually did API calls. So once we started adding Flux multi-tenancy, it became a lot clearer to be able to see what's on this actual cluster. You can see what namespaces are on there. You don't have to do calls. You can just go look. Anyone can go look at the repository, right? So yeah, so like I mentioned earlier, it's very important to make sure that there's tenant isolation in this situation. You want to make sure that nobody can mess with anyone else's deployments or anything like that. So a platform admin can lock down Flux on multi-tenant clusters during Bootstrap with the following practices. So I mentioned Bootstrap earlier. It's the easiest way to get set up with Flux initially. It's a simple command to run, and then you can basically, if you point it to a repository, and if the repository doesn't already exist, it uses your token to create the repository and then push the files that Flux needs up to it, and then it sets itself up to listen to itself. And so that command's actually idempotent because you can upgrade the Flux CLI version later and then rerun the Bootstrap command, and it'll update all the files to the latest version. If the repository already exists, and even if the Flux system folder already exists, it'll just Bootstrap into what's already there. And update the file if needed based on the version. So with the above configuration, Flux will deny cross namespace access to Flux custom resources, so it ensures that a tenant can't use another tenant's sources or subscribe to their events. And that's with the dash dash no cross namespaces refs equals true flag that's on this patch. It also denies access to customize remote bases, which ensures that all resources refer to local files, meaning only the Flux sources can affect the cluster state. And then also all customizations and helm releases, which don't have spec.service account namespecified, will use the default account from the tenant's namespace. So tenants have to specify a service account in their Flux resources to be able to deploy workloads in their namespaces as the default account has no permissions. The Flux system customization is set to reconcile under a service account with cluster admin role, which allows platform admins to configure cluster-wide resources and provision the tenant's namespace, resource accounts, and RBAC. So by default, Flux RBAC grants Kubernetes built in view, edit, and admin role access to Flux custom resources. This allows tenants to manage their own, to manage Flux resources in their own namespaces using a service account with a role binding to admin. So if you wish to disable the RBAC aggregation, you can remove the Flux view and Flux edit cluster roles with this patch as well. All right, so I'm going to switch back to mirror and get rid of this and then go back here. Okay, all right, so these are some good resources, but I wanted to show this because also this repository here is like an incredible resource. It's how like when we were first trying to learn multi-tenancy with Flux, we definitely relied on this and we just used it like we followed all the steps in here. So this repository is still maintained, so it's like up to date. It's a really great example if you want to just like get started with using like, if you want to like set up a kind cluster or something and just try it out, whatever your cluster is. And so if you go in here, oh no, it's doing the thing it was earlier. It's not scrolling. It's lagging. Sorry, y'all. Okay. Dang. I guess I can scroll. Sorry. So it says a lot of the same stuff I've already talked about, but I just want to scroll down here to show y'all like how to, you know, get started with it. So the first step is to bootstrap. And this, if you're bootstrapping with this, like with a repo that's like this, it already has like the Flux system folder. This really doesn't want to move, right? So you're bootstrapping with the like existing thing and there's a Flux system folder. And so like I said, it uses your git token to have access to edit the files. So this is the bootstrap command right here. And as you can see, there's, you know, like a owner, which is like your, wherever you're trying to clone to, and then your repository and then the branch personal and then this path is like what's important because that's where you're telling it to go, like set those files. And then, sorry, it's so slow, it's just so laggy. I should have restarted my computer. Okay. So if you go through this on your own, I'm really sorry y'all. There's a bunch of commands in here where like it helps you kind of like learn also if you're new to Flux, the interactions with Flux. So like this command right here is how you would like check your like what customizations you already have set up. And so remember, those are the ones that are actually applying your files. This is like a command to see like what sources you have that are pointing to git specifically in this case. And then so on and so forth. So like Helm command rate, Helm releases, I want to get to the on board new tenants part. It just won't scroll. Okay, give it a, okay there. So the way, no, now you're scrolling too much. Okay. So the way that you, sorry y'all, the way that you create a new tenant is with this command right here. So this Flux create tenant dev team with namespace apps. So that's telling it to create this tenant that's called dev team. Oh my gosh, now it's just moving. And this team that's called dev team will have access to this namespace called apps. And if you actually give it more dash dash with namespaces, it'll create multiple namespaces that this team then has access to. So and then this export command, you can run this command without the export. But what that'll do is it'll just actually create the tenant and then you won't be truly doing get ops, right? Because it's just there and it's not been pushed to get. So if you do this export command, you can put it in this location, then push it to get, which is what this is saying to do, create a source, which is how you tell it, go listen to this at this URL, go, you know, pull whatever is there. And then this is telling the customization is saying, hey, within that repo that the source is listening to only apply what's in this case, everything, because it's not slash, but you could specify, like, you know, whatever path you want within that repo. And then you push it. And so let me just show you real quick what that command creates. Everything. Okay. Well, so it creates this namespace. So this is, like I said, in this case, there's only one namespace that it's creating. And I ran this command saying team one, I think, and I said team one namespace. So this label is to say that everything is tied to this team one tenant. And then it's also creating the role binding over the, well, sorry, I didn't scroll it. No, scroll. Yeah. Sorry. So this is the service account that it creates, the team one service account. And then down there was the role binding that gives it the accesses it needs to the service account. Okay. And then, yeah, what's the lagging? That's all I can, I think I can show on that repo, but I do recommend going and testing that repo out for yourself. It's really simple to use and like it's quick and easy to get started using multi-tenancy really quickly. If you want to reach me and ask me any questions further or chat about flux, chat about multi-tenancy, these are the places you can reach me. I'm also in the CNCF Slack as well as we have a channel in the CNCF Slack, just hashtag flux where you can reach us if you have any questions about flux as well. And then that QR code, if you scan it, will take you to all of our upcoming events. So we have a few talks actually at coupon coming up as well. And then also, please come visit us at the project pavilion booth. We will be there all day, every day probably other than our talks. And so we'd love to chat with you guys and hear your experiences or give you help or anything like that that you need. Or just chat. We're happy to chat. So that is everything I got. Thank you guys. Are there any questions? I know I'm kind of, I'm sorry, I went to like almost the end of my time. I think we have two minutes, sorry y'all.