 Hello, I'm Stefan Prodan, I'm a Flux Core maintainer for a long time now, very happy to be here with you and I'm super excited about Alec and what he will share with us, he will tell us how Cisco uses Flux XK, and after he is doing his presentation showing us how he is using his Flux, with Helm, with Customize, with Advanced Kitripo Setups, I will come up with some crazy ideas on how to optimize the current setup and hopefully some of you will try these crazy ideas and tell me afterwards if they work, okay? Thank you. Go Alec. Thanks, Stefan. Yeah, good afternoon. My name is Alec Carzon and I'm principal engineer in the Cisco IoT group and two years ago I had to come up with a plan to deploy a lot of workload on Kubernetes and in our case we have a lot of data centers to cover, probably around a few hundred clusters and we have not different wide variations of type of workloads to deploy, so pretty tough problem, but two years ago there was not much documentation on how to do this with Flux and I'm going to do two days, I'll kind of show you how we did it and how it works and in our case we have a big team, we have over 200 people working on this platform, going from developers to testers, people who release code, people who devops on SRE teams and it was really a challenge to see how to build something around Flux that allows all these different teams to work together without, no, with as little friction as possible and in terms of what we're deploying, we deploy a mix of open source home charts and a lot of home charts developed internally, around a few hundred and on every cluster that we have to deploy to they can be different types, different size, some data centers have different loads and in our case we had to deal with no disparity of targets and all of these data centers currently are running across the world, we have some in Europe, Asia, North America, Middle East and we also are trying to have a common model to deploy on-prem and on public cloud as well, so the other concern we had is when you define production, you have to make sure that not everybody can touch these deployments, so the way to organize the Git repositories when you use Flux is very critical in ensuring that you can control exactly what goes into production, so we're going to cover all of these different aspects. So if you're new to Git, the first thing that you'll have to do is kind of decide what kind of Git repositories and what layout you're going to use, so if you start there's a single repo that's very easy to use to start with, so a lot of people use single refos and then they start adding more home charts, more clusters and they start to run into issues, like for example, as your team grows you'll have single refos, we'll have a lot of commits on it and you have a mix of commits coming from developers, from your leads teams, and the problem is there, it's very hard to enforce that only certain people can commit to productions versus a task response and if you look at the Git log it becomes a mess because maybe 90% of your history will be developers testing deployments and then maybe 5% is just the production clusters, so not pretty much the only way for large scale is to have multiple Git refos and multiple Git refos kind of rest most of these issues, the challenge is how do you keep these Git refos in sync, so and the other challenge we had is how do you link Flux, deployments to the way that you build software, you can have a lot of users developing changing code and home charts, how do you tie all of these together, the way that we do in terms of Git repositories we use a two level model, two levels of Git repositories and what you see here is on the left side you see all the source code of your applications, that's where 99% of your code resides, in our case we have one Git repo per application or per home chart, and there you see a mix of open source time charts and internally developed time charts and we have a CI system that builds these home charts so every time somebody commits something there the CI kicks in and builds the home charts, the container images and go into container registry and the home repository, so that's the lower side of it, on the right side you see a lot of clusters and we have two types of two groups of clusters, let's say one that's on the production clusters and another group that are all the test clusters that we use for testing and for that we're going to see a little bit how these Git repos work together and let's start with the level two, this is cluster, we call it Blueprints, Git repository and this repo contains deployment instructions on typical workloads that you want to deploy on every type of cluster, for example you have two examples of Blueprints, one that deploys our vault server and another one that deploys an application that is comprised of several home charts and you see in this example we deploy the vault Blueprint in one cluster in production and then the application Blueprint we deploy it in three different clusters, so the role of this Blueprint repository is really to contain instructions of how to deploy a certain type of workload, like which home charts to deploy, what are their dependencies and these repositories is not very big in size, maybe a few thousand lines of code, it contains mostly flux, custom resources like Helm release, Helm repositories and dependencies between them and what the owners of these Git repositories are really people who develop applications, so developers own it and every time they add a new home chart they usually create a new Helm release on that Git repo and then they attach it to a Helm repository and from there it's ready to deploy to any cluster and this content there is also agnostic of the target cluster because we want to reuse the same Blueprint to run on multiple types of targets, so this repo also we don't want to release teams or SRA teams to touch it, all they know is that there's a Blueprint to deploy and we use intensively Git branches and tags to version our Blueprints. Next level is the level one Git repo, so this is really a place to link together a cluster and then a Blueprint, so we have one Git repo per group of clusters, so we have one Git repo for all the production clusters and one for all the task clusters and clearly the production Git repo only the release team, SRA team can make commits there, so it's much easier to enforce roles based on this. And then the cluster of the test one is developers and tests they can access it and they can make changes there, so in addition to for each cluster you have one folder in that Git repository and that folder contains basically a link to which Blueprint to use, which version of that Blueprint to use and it also contains the configuration parameters specific to that particular cluster, so that's how you kind of distinguish what goes into each Git repository. And so we're going to see on the next slides how we manage to control all these different configuration parameters. And then the second part is also how we attach a release, how do we release versions of Helm charts to departments. And as you can see, there's also two different models of departments we support. For testers, what they want is that whenever they commit something in the Helm chart, they want their test cluster to update automatically to the version that they just built. So this is kind of a live updates, and that's something that we will use mostly on the test cluster side. On production clusters, that's something you don't want to use, because there you really want to make sure that whatever you deploy is really pinned to a certain versions of Helm chart, and you don't want your developers to be able to commit something that would impact your production clusters. So in terms of how do you differentiate your departments per cluster? For that, we're using a feature in Flux, which is called substitution variables. So it's a very nice feature that allows us to control exactly what we run and how we run on each blueprint in each cluster. So to see here, on the left side, we use the example of one Helm chart called FODINFO. And this Helm chart has a list of values, the default values of the Helm chart. And typically what you do is you want to deploy this somewhere, you have to create the Helm release file. And that file we point to the Helm chart to use. And it also contains a section called suspect.values. That allows you to override the different values in your Helm chart. So here are examples of two different fields and values that we want to be able to customize per target. And there, instead of putting the values we use for each cluster, we use a variable, a Flux variable. And then, optionally, we can also put a default value. So you see that there, the default blueprint, you can end up with a number of different variables. And then, in general, when you have a blueprint, a blueprint is comprised of possibly dozens of hundreds of Helm releases. And that gives you a list of Flux variables that you can use to customize each target deployment. So that's on the blueprint side. On the clusters, the triple side, that's where you define the value to use for each cluster on each of these variables. And so these come as form of a config map. And you see here an example of a production cluster. We want to override the number of replicas from default to three. And then, in the test cluster, we want to do something else. We want to change the color of one of the attributes of the Helm chart. And what happens there is that when you bootstrap Flux on these clusters, Flux is going to pick up the Helm chart, the Helm release and config maps. And it's going to merge everything together and then form the final set of values for that particular Helm chart. It's a pretty simple way to control all of the values that you can use for all of your Helm charts per blueprint. So in terms of controlling these config variables, a lot of people use a certain way to organize their clusters. In this case, for example, you have clusters organized by data centers. And in each of those centers, you can have multiple clusters. And sure, what's nice with Flux variables is that you can have overlays of config maps. So, for example, you can have one config map containing global variables that you want to apply to all the clusters in all data centers. And then for certain data centers, you also can specify variables for each data center. For example, if you use a common secret store, you put URL of that secret store, and then all of the clusters you deploy there will be using that value. At the bottom, you'll have the cluster level config map where you can really say, for this specific cluster, I want to use these specific values. So this way to overlay multiple config maps is really very convenient. And that allows you to also avoid repeating the same values for different clusters. So the other side of all this deployment is how do you link a version of your applications or the home charts to a certain blueprint. So when you point to a blueprint, you can specify either a branch or a tag. And in our case, when you point to a branch, that means every time we change something in the branch that gets reconciled into your cluster. Well, if you point to a tag, then it is really fixed with that tag. So let's see an example of typical workflow when you build home charts. So we can have a lot of home charts. When your developer commits something, a new stable version, what happens is that that home chart is built. And it goes into a home repo that contains a lot of home charts. And for each home chart, it also contains a lot of versions of it. And the main branch on your blueprint, you have a home repository that points to that live home repo. So that means that when you want to deploy these home charts on the main branch, whatever is on that home repo is going to get deployed on your cluster. In the home release, you also specify which version of home chart you want to deploy. That level, you can use a range, like greater than a certain version. And that allows you to kind of automatically deploy when you have new versions of home charts coming in your live home repo. They will get automatically reconciled and deployed on your clusters. So that's the mode that we use for all the test clusters. And for production clusters, we use a different way. And there are actually two ways to pin the version of home charts to your production deployments. The first one is to go to each home release. And then each of them, you change the range version, you put the exact version to use. So that's very inconvenient because as you get more home charts, you can have hundreds of them. It's really hard to go and make sure that all Android of them have the right version. So instead, what we do is that we version the home repositories. Which means that when we cut a release, we take a snapshot of the live home repo. And taking a snapshot means that for each home chart in that home repo, we take the latest version. And then we create a new repo for it. And at that point, the only thing you have to do when you create a release is you create a new branch in the blueprint directory, the repository. And in that branch, you just change one file, which is the home repository. So that file instead of pointing to your live home repo, you just point to the new repository that you just created. And you don't have to touch all these release files because they're still a range, but using that repository, there's only a choice of one. So that's why you can be sure that that's going to be pinned to a certain version and that's not going to change. And the next time you cut a new release, it's the same thing. You create another snapshot and you just change your home repository. So it's a pretty simple way to kind of make sure that whatever happens in development, that impacts only a different home repository. And when you do updates in production, you go from release one to two. Between these two, you might have maybe one home chart that changes or 50. It doesn't matter because you can just reconcile and then update those that have changed. So the challenges with this approach is that when you have two levels of Git repos, sometimes it gets difficult to get them in sync. Initially, when we deployed this, we had a lot of update failures because, for example, you can have developers add a new home chart and then add a new variable. And in the production Git repo, we forget to add that value. So that happens or sometimes you add the wrong type. And in our case, we use also variables to specify a path in the secret store. Those kind of mistakes can cause update failures. So what we have to do there is really try to fill that gap and what we did there is really develop a small tool that will be in charge of kind of validating your config maps. This tool, I just pointed to a cluster's repository and we passed to it the cluster we want to test. And another option is to pass which blueprint you use and which are tagged to update to. So you do this before you update. And that tool is going to scrub all the config maps or the variables, and it will also connect to the blueprint and then look at the tag that's used. It will extract all the variables that are used there and then it will make sure that all the variables that are missing are pointed out. And we also added some validations in terms of making sure that the right type is applied to each value. For example, here you see the important four replicas. There's a default value of one. So it's really smart enough to guess that if you put a value to that variable it has to be an integer in the string, right? And those that don't have complex types like a color code, right? You can specify a syntax of that variable. So we use a Q language for that. So Stefan's going to talk about a Q in the next slides. And with that tool, that cut down quite a bit of the updates we have, mistakes we have in the updates. To summarize, so with that kind of organization we deployed to currently a few hundred of clusters and we have maybe a dozen blueprints, right? About 200 time charts. And we have over 200 people working on it without too much issues. And that's really, I don't think that was possible using any other method, right? Because we use intensively git branches to control all these different eolutions of ham charts that we have. Limitations of this approach is that you still have to write a lot of boilerplate code like for example the ham release. Probably 100, 200 lines of code. So there we could use some tools to make these less verbose, right? And the other limitation is the flux variables are really very simple in terms of values, right? They only support the integral, Boolean and string, right? And when you want to do things like lists or maps, then you're kind of on your own, right? So that's a big issue for us for complex deployments. And as we said, you need some additional tooling to make sure that your git repos are in sync, right? And some of the complex deployments we have are also required some big variations of ham charts like for example, for certain data centers, you only want certain combinations of ham charts to deploy. And from that point of view, the only option is really either to patch your ham release or to... There's not very good options there, right? Or to add more Blueprints. And lastly, the clusters, they still access to Git repos, right? So you have a lot of hits on your Git repos, Git servers, so that sometimes can be a problem if you have a lot of clusters, especially when you use enterprise Git server that are used for other purposes, right? And sometimes you can have... They can be down for maintenance and then you cannot do certain things Yep, so that's all for me and... Yeah, Stefan, you want to... Yeah, thanks, Alec. So we saw that it's quite complicated, right? Deploying to hundreds of clusters, mixing all these apps, bundling ham charts, it works, but it's... It's hard to comprehend how to separate. You need to iterate and create your own structure of Git repos. That's one. Second, we have this problem of Helm, which, and YAML, of course, nothing is type safe, so when you have separate constructs, but on the cluster they have to be bundled together to be applied, you get this un-sync feature, right? So in my opinion, we can improve this kind of complex setup, and I don't think it applies only here for Cisco, but for many other flux setups, which imply having a base app configuration, then deploying that to hundreds of clusters or hundreds of namespaces in many clusters, all that type of hard multi-tenancy when you have one cluster per tenant or soft multi-tenancy when you have one tenant per namespace, and you always have some kind of app that needs to be slightly modified depending on where it lands. So one evolution improvement would be desired state consolidation. I'm going to talk to me, it's about it. So we see that with GitOps and what Alex said, usually you use Git in your dev pipelines, right? Your developers push code there, then you have some CI process, and that CI process produces an artifact, and that's the thing that you are using in production. In most cases, that's a container image in Kubernetes space, right? So Kubelet does not go to your Git repo. It does not fetch the code from there to run your app. It goes to the container registry. In the GitOps approach, you run something in your cluster, that's Flux, and Flux goes back to that Git repo. What that means is that the Git server now becomes a hard dependency of your production system while for the development process is not a dependency of your production system. So there are a couple of challenges here. You have to make sure you can scale out Git. The more clusters you have, the more connections will go there. You need to ensure HA. You need to do upgrades. If the upgrades fail, everything fails, right? So how can we keep using Git for configuration as we use it for code? We have the same advantages. We can collaborate on it, but we don't have to have the Git server as a dependency in production, we could use the container registry. Most container registries out there right now are compatible with the OCI standard, and that means we can store there more than just container images. In our case, we can store there the Helm charts, Helm version 3 has great support for container registries. You can do a Helm push, and instead of using HTTP repos, you can use the container registry and push there your charts. But for the Flux configuration, there is also this option. We have an extension in the Flux CLI which allows you to bundle your... One or more director is a whole repository or more repository. It's up to you how you mix all these things together and you can produce an OCI artifact for Flux. Then Flux only goes to the container registry and usually those container registries, RHA, you already have sorted how you are upgrading them because they are already part of your production system. What it means to migrate from Git to OCI artifacts, for Helm is simple, you do a Helm push, but the target is the registry and for your Helm releases and Helm repositories, you will do a Flux push. It's not that complicated. There are also some API changes that you need to do in the Flux configuration. The major one is changing, switching from Git repo story API to OCI repo story API. But luckily we created the OCI repo story to be a mirror of the Git repo story. It's very easy to switch from one API to another and back and forth. The difference is in a Git repo you have branches that Flux follows. How you can do that with OCI repo stories, you have mutable tags. Let's say the main branch becomes the main tag. Every time you do a commit to main, you override the latest main tag and Flux resolves the digest of that. Like with Git, when you follow a branch, you resolve the commit and in the same way Flux for OCI repo stories is the latest thing, which is what's immutable is the OCI digest. OCI digest inside the Flux system is the same as a Git commit shell. It's a one-on-one mapping of those. We figured out the first improvement. The second one is about simplifying the whole way of how you define variables, you merge them, and you need to do some changes which are only for data center than only for some cluster in the data center and so on. We saw how we can do that with variable substitution and lots of lots of YAML. My proposal is instead of using Q to validate some part of the YAML, what Alec was showing, we can use the Q language, which is great for validation. You can also use the Q language for generating the final YAMLs. Q is a superset of JSON. It has a steep learning curve. You'll need to invest some time into learning Q. But once you do that, you have this powerful language where you can have complex types, you can express logic, and you can have all these variants of a particular application very well-defined typesafe in a compact form. So why would you use Q for Kubernetes? One of the things that I really want to highlight here is the fact that the Q language can import a Kubernetes API spec, being a deployment or a custom resource definition. It can import that then when you want to write a template for, let's say, a Kubernetes service or a Kubernetes deployment or any Flux custom resources, it can validate every single field in there as you would have the Kubernetes API as your disposal. But it's client-side. You don't need to do any kind of apply to itself when you build it. Then you'll get the same validation as you'd get when you apply it on the cluster, which is very powerful. You can basically validate everything way, way before it gets deployed. So we want to catch all these errors early on in our pipeline. And Q can do that because it really integrates with OpenAPI and can generate these schemas for you. So we have the Q language. That looks appealing to Kubernetes, but now how are we going to generate all these things? So for this, I'm working on a tool called Timoni. He's been in development for over a year now. My goal with Timoni was I was so frustrated with writing Helm charts all day, so I said, OK, it needs to be a better way. I discovered Q and Timoni's in nature feels like Helm, but it's completely different. But it has the template directory in there. We don't call it Helm charts. We call it modules and so on. The idea with Timoni is that it can bring the Q-type safety to Kubernetes templates and can make this more sane, pleasant way of defining all these Kubernetes objects. It comes with a couple of concepts. Modules, which are like Helm charts instances, which are like Helm releases in the cluster. A bundle, which is like a Helm file if you want, or an umbrella Helm chart. And a runtime, which is something you tell Timoni, hey, I don't want to give you secrets or give you all the configuration up ahead. You should look up in your cluster and fetch from there the configuration at the playtime. So instead of having to pull a secret down and store it in your Git repo with Timoni, you say, this variable comes from this secret in the cluster at this field. And using Q, you express that query and Timoni runs it, and that's how it deals with secrets. I made a bunch of modules to be able to generate flux objects. So it's released. You don't have to learn much of Timoni or Q. And you can use, for example, the flux Helm release module to generate Helm releases. You can generate tenants. You can generate OCI things. All the, almost all the flux APIs there will be at some point available at this URL and you'll have abstractions built on top of it. And you can generate all these complex flux objects easily through Timoni modules. Okay, so we have, we've learned Q. It's a big step. Then we've learned Timoni. It's even a bigger step. How we migrate now. So it will be a hard proposal here, right? We actually delete all the YAMLs from Git, which is... Okay? We don't store YAMLs into Git. There will be only Q definitions. Your developers will have to understand a little bit of Q and what's happening there. Then, but you don't, in this step, you don't get rid of Helm charts. So they can still do Helm charts, right? For now. So in the blueprint repo, where you have the Helm release definitions, there you'll be having only Q code that generates Helm releases, Helm repositories, customizations, OCI repo, all the things that you need. And then how you put them all together and how you change their tiny bits for each cluster is with this bundle concept where you can refer to all the modules and say, oh, if it's this target, I'm going to change this little thing here. What Alec was showing with Flux variables. Now, how you distribute all this stuff? Of course, to the OCI repo. And Timoni has all these build commands. So you can do a Timoni bundle build of all my variables. And you pipe that to the Flux push artifact command, push to the container registry. Flux on all these clusters, see the new configuration and starts and reconciling. Next step for Timoni, there are a couple of things that are not there yet. The API is not quite stable. There are alpha APIs, so I may break them. There is a Flux IO distribution for Flux where Timoni can deploy Flux in a single pod. It works on the edge, it doesn't need a CNI and I'm trying to promote this distribution as an alternative to the official Flux distribution. It will not replace, it will be just another option. And finally, what most people are asking for is a Timoni controller, so you don't have to build the YAMLs and push them to the OCI repo, push the whole Q definition there. So I don't know when a Timoni controller happens, it all depends on adoption. What I'm asking you is, like, give it a try, try to migrate the easiest Helm chart that you have to a Timoni module, see how that goes. Let me know. There are lots of people who love it, but they love it because they actually are into Q, right? I still think there is a major gap switching from Helm templates to Timoni and Q. I think it's worth it. Maybe spend a day or two. And yeah, let me know. I would really love to know. Thank you very much.