 Hello and thank you for joining this flux deep dive session. My name is Philip Lane and I work as a DevOps engineer at Senate we are Swedish based company who builds developer centered experiences around cloud Kubernetes and get ops. I'm also part of the flux maintainer team. And that's what I'm here to talk to you about today. So what is flux. Well, flux is a tool for keeping Kubernetes clusters in sync with sources of configuration. What this means is that in basic terms that that you have configuration stored in some sort of external repository for the most of the time this will be get. And then you have flux the tool running inside of your Kubernetes cluster, it continuously synchronizes with that external repository and applies it into the cluster. And this has the benefits, partially due to auditability when it comes to get because you have a history of changes that you can see who made changes to what configuration, but also an observability perspective where it is very easy to look inside the get repository and see what is actually being applied inside of the cluster. This is a quick overview of how how might look like to work with flux. So, in one end you have your deployment manifest that you are going to commit to a get repository. On the other end of this you have flux flux is pulling the get repository constantly looking for changes. If a change is detected here it will apply it to the cluster. And as you can see here you have your point manifest on one end, and then your deployment on the other. This also works in the sense that if somebody were to go in and delete this deployment in the cluster, flux will eventually detect the drift between what is wanted or expected and what is actually there in cluster, and you have the possibility of really doing this as simple or as complex as possible and this example here it's very simple, but you can make you can scale this out to run across multiple teams and have different types of functionality and automation, but there isn't really any requirement to do so so that's kind of the beauty of flux is that it works for smaller teams but also works for large enterprises at the same time. So quickly before we get started just want to go through a quick timeline of flux a flagger. So currently flux is incubation project. It has 14 containers of five companies. But it started it's been around for a while now so in 2016 and started developing a we works. Since then, we've had the flagger project showing us a sub project for those that don't know flagger is a progressive delivery operator that works really well with flux. If you're looking for project to get involved in I really recommend getting involved in the flux project it's it's really fun and we're always looking for for contributors and we're happy when we see new contributors join the project. So quickly then about my journey into becoming a maintainer about in flux so in 2018 I was working as a software developer building microservices that were deployed on a cluster or consider, which was not kubernetes. So I was kind of looking around looking at what other opera like cluster are considered were out there. And I stumbled upon a blog post about Sony building a recipe pie base to witness cluster. I found it kind of interesting I wanted just to find a hobby project so I started building one of my own. After that I realized that I wanted to have continuous delivery to my new cluster, but I realized that the only way currently it was possible was to expose the qapi server to the public Internet. And I wasn't really ready to do that mostly because it required me to open up before my home router. So I was looking around and I'm kind of ashamed of admitting this, but the only reason that I found flux was because it was a continuous delivery tool that didn't require you, or didn't require an external resource to have access to your cluster. So, as flux works by pulling an external and get repository work really well with having a small cluster that could not get public IPs. Moving forward to around. Well now, my cluster does have the ability to get public IPs, but it still runs flux because what is a great project. I think somewhere along along the line there I realized what a novel idea flux was. So I started getting involved in the project I was using it as at work and eventually around when flux V2 started picking up steam is when when I got more and more involved and became a container. So once again if you are interested in getting involved in the source project. I think what is it is a great project to get started in. So, at Senate we develop and operate a multi tenant Kubernetes solution that obviously uses flux in our solution each tenant usually represents a team or a product. Sometimes the product is usually because the teams are working on around a central product together and tenants can raise can rise range anywhere in size between a couple of applications to 50 plus. We are heavy Azure users so as you can see, we have a case cluster and a couple of resource groups so each tenant is confined by default to their own main space within the Kubernetes cluster. And it's linked together to an Azure resource group. So this brings the challenge of how we expose these namespaces or tenants to our end developers in a safe and secure way which has a reasonable maintainability in the long run when it comes to maintaining the clusters. Before I joined Senate, we were not using flux there. And I think this is a technical deployment method that the most companies have been using for a while now with Kubernetes and well before Kubernetes. The development teams were maintaining their own CD pipelines, they would have multiple applications that would have each have a continuous delivery pipeline that was pushing in, or that was first of all building the image, and then pushing that image to a central registry, and then it was running some sort of qctl apply or help install upgrade command towards the cluster. And this resulted in applications having their manifests and source code tightly coupled as each application basically have their own unique manifest. It would also require each application or at least their CD pipeline to know about the cluster. The changes to the cluster or if you wanted to replace the cluster would require you to go into into each and every application and update them to point them towards a different cluster. And any small manifest change would well first of all require you to rebuild the whole image as adding custom logic to detect manifest changes and skipping the build was complex and dependent on if the teams were willing to build a type of logic. But also require you to trigger all of the applications. So, for example, we all know that API versions sometimes get deprecated in a situation where we were using an old API version, and we need to upgrade the Kubernetes cluster. And we need to make sure that each development team was upgrading their manifest that would require somebody to go into 60 applications and manually update or retrigger that application to deploy a new home chart to the cluster. So you end up building new automation tools around automation just to manage manifests. And this is around the time that I came in and tried to bring some of my experiences that I've had in the past around flux. So, my idea was that flux would solve a lot of the bigger issues that we're seeing with development teams, mostly around trying to keep what developers already knew and worked but replace things that were still new for for certain developers but could simplify their lives and our lives also. So, after we flux one of the main benefits that we're getting is we're getting a unified enter point towards Kubernetes. As it is a pool based deployment method, we don't have to export any service account tokens outside of the cluster. We don't have to configure individual deployment pipelines to tell it which cluster to look at. So, this opens up opportunities of earning, for example, blue green deployments of clusters. It simplifies observability of what is being deployed. So this means that we can run automated automated scanning tools to look at our developers manifests to help them be better at their jobs and changes to manifest are easy and fast. So, if we, for example, come up with a new recommendation or we want to run new API versions, being able to just go into a single repository or a couple of repositories is way much easier or and way much better than just having to do that for for 50 or so. So one of the cool things with flux is that you can run it in a multi Tennessee solution. What this means is that you can configure a single git repository to only have access to a namespace. In this case, a tenant would have one git repository and one namespace and these would be linked together with flux. So anything that you place into this git repository would then be synchronized into that namespace. This allows for a good separation between teams where they can cooperate through opening up their namespaces with network policies, but only a specific team would have access to their own namespace. And because this configuration is all based on YAML, it becomes very easy to automate. So, you can configure flux in multiple ways. So one of them is by using the flux Eli. Another one is using the Terraform provider which helps you bootstrap the cluster with Terraform. And the last one is writing manifests on your own. So basically demo onboarding a new tenant by jumping into this Terraform here. So this is this Terraform sets up a single ATS cluster that has three environments and it's a dev, a QA and a production environment. If we call it out this we're going to set up a new tenant called tenant B. This is linked out the git repository that it's supposed to synchronize from. And we first run a plan. While this plan is running, we can look at the fleet-tempfer repo. So for flux, the fleet-tempfer repository is the main repository that is initially synced from by flux. We have a tenant directory and right now we have our tenant A configured. So the tenant A deploys or creates a service account, a role binding, a git repository and customization for this tenant. And as you can see here, it's pointing towards a separate git repository. This git repository is located here. So this is tenant B's git repository and they've been super nice that they've actually set up all the manifests and configuration before or ahead of time. So after a plan it should be ready to run. So, tenant B is just going to deploy a simple guest book application and it's going to set up a certificate and the ingress and a couple of pods that's going to be deployed. But all these changes will be automatically reconciled. So we don't really have to interact anyway with the cluster or the teams repository after we set things up because all flux is going to do is going to reconcile or it's going to synchronize from its fleet-tempfer repository and it's going to apply the new tenant configuration that it's committed to the repository. And then it's going to go off to the next git repository and for the tenant B and synchronize the manifest and configurations from there. So now the terraform has completed, we can actually jump in and look what's being changed here. So jump into production database or production in a space for the tenant and let's get the ingresses. And you can see, oh great, an ingress has been applied to the cluster. We can get pods. Now we can see that we have our guest book application running where the red is. If we jump and get the DNS name, we're now seeing the guest application running here. So what has actually happened? Well, if we go to our fleet-tempfer repository and refresh the page, we can see there is a new devB YAML file that's been applied by terraform. This creates the new tenant configuration with new role bindings to a different namespace and a new git repository. And from there on fleet-tempfer repo kind of loses responsibility over what else is created here. So then flux is going to reconcile this git repository and look at the B tenants repository. And here we see, and for example, production that we have created an ingress resource for guest book, a certificate, and we have our base here and the base is deploying our guest book application. So all this is basically automatically reconciled. So from the tenants perspective, they have their own enter point into the cluster that is configured by the cluster administrator. To successfully introduce flux, it requires buying from all participants. Initially, flux can seem a bit foreign in the way that it works to some people. And it's mostly because it's trees away initially from the traditional CD promotion pipeline where you have a first deployment to a dev environment, and then a QA and then a production and there's like a logical pipeline of promotion. One of the earlier issues with flux was that there was no feedback loop between a commit and an actual successful deployment. Today, flux has a component called a notification controller. What it can do is that it can link together the reconciliation events from a specific commit to feedback, the actual status of that deployment. In this image to the left here we have a git history of commits. And for each commit you will have flexible reconcile that commit, and then it will post the status of that commit back to your git provider if it's GitHub, GitLab, Azure DevOps, or Bitbucket. This uses the same APIs that the CI does so it shows up as a green check mark or a red cross. If it's unhealthy so as you can see here this commit would be reconciled and it might be that your application has a bug in it or that it can't pull the image and then the reconciliation would eventually fail because the health checks would fall and it would post back a unhealthy commit status back to it. So, with this feature implemented, it reaches a middle point between the new GitOps deployment methodologies and maybe the older ways where there are very big benefits of getting a feedback between a commit and a success or what is used to be missing in flux but isn't anymore. What's also cool with this feature is that we're now able to build promotion flows around this without actually needing access to the cluster. Everything is pulled in by flux so flux just needs access to the Git repositories, but also the feedback is posted by flux so there's nothing going back to the cluster to check the actual deployment. So, in theory, what you can do is you can have a pipeline configuration that after a change to a dev manifest, it could take that change create a PR to the QA manifest and wait for the dev chain commits to have a successful status posted back. So we can actually demo this now. If we go here and what I've hit here is the pod info application that I've for and if we jump into code quickly, and then we can change the color of the background. Let's say, there we go at this change change back in color push the code. So this will trigger a CPI plan is going to build a Docker image and then it will trigger a promotion flow. If we get the ingress and we jump to our browser. So, this is currently running the older pot info version. We can see this in QA and production. So if we could jump to pipelines. I think we're all familiar with with this type of builds very we're just building a dark image. But what's different here is that we have our tenant a get ops repository. So the tenant a is the one deploying the pod info application. If we jump into apps and dev. There is a override of the image tag here. This annotation is a custom annotation used by the tooling that runs in our CI to update. Also, in theory, do these types of motion flows with image automation controller as it comes more stable. I think currently it's in a beta or alpha release, but probably soon it will, it will reach a more stable version that that will come by default in flux. So what our goal is here is that after the pipeline has been built, we want to update the tag override in this dev customization. And after that is that change has been synchronized into the cluster, we obviously want to go forward and move this tag override to the QA environment and then to the production environment. And that's what we built here is that we built a CI chain that falls outside of Kubernetes and outside of flux. So this is something that you have to build yourself. But what it enables you and it shows you is that there are a lot of possibilities here and I don't I don't think there is a one good way or best way of doing this. I think you have to build these types of things to fit your type of use case. So we're just going to wait for the bill to complete now. So the bill is now completed and we can jump over to our dev environment and we can see that it is now updated. Hopefully we will also see a change being done in QA. So if we jump into our commits in the 10 day get ops repository. We see a couple of things here. So one of the things is that we have our status objects on our commit and this looks slightly different form get up get lab and other get providers. But as you can see here, these three commits or the statuses are sent by flux. So we see your app step QA and production that are posted back when a change is synced and successfully applied. We can now maybe see that. Yeah, let's wait for QA. So what is happening is that we're running a pipeline that is creating pull requests for us. So if you look at completed pull requests, we can see here is a pull request that changes the image tag. So this pull request can be automatically completed. If a status check passes, we can merge it and the change will then be applied to the cluster. So this allows us to build a promotion flow outside of having access to the cluster itself and here we can see also that QA has updated and maybe production will eventually update to and this is all done by by our commit status feature. So the commit status as I said before, as changes are reconciled it will it actually knows the hash of the commits that it's reconciling. So that way we can post the status of the commit back to the cluster back to the get repository. And then it's kind of up to you you have all the opportunity and all the options to build whatever automation tools or non automation tools you want around this feature. But the main thing here that is it closes the loop in the same way that a traditional continuous delivery pipeline would be expected. A major benefit of running flux is that you have the options of doing proper blue green deployments of clusters without having to cooperate with all your developers to re trigger pipelines. If you weren't using flux, and you'd have multiple continuous delivery pipelines that were pushing applications into your cluster. If you want to set up a new cluster you'd have to tell all your developers to basically then target the new cluster. Instead with flux, all you do is set up a new cluster pointed to the exact same get repositories allow it to sync, and you're set to go. So, these things couldn't do if you run during after hours behind the backs of the end users or the clusters, and the next morning they wouldn't even realize that they had a new cluster in the first place. So this let's actually run a new cluster and see what happens. So, we're going to first plan and apply our new cluster. And then we're also going to update the DNS records because external DNS is creating our DNS records so we need to allow the new external DNS instance in the new cluster to take over these DNS records and point to do the new load balancer. Let's just first wait for our cluster to be planned and applied. Okay, so the cluster has now been created and everything's been provisioned so if we just check. We've now configured in new contexts we have two clusters a case two and a case one, the case two we have our hot info deployments if we jump back to a case one. As you can see, we also have them running here. If we then scale our deployments down in a case one, because DNS has triggered. We should now reach the other cluster, as you can see certificates haven't been provisioned yet. But that might take a couple more minutes. This is basically a blue green deployment without any interaction we can actually look for example in our old cluster and see that we have for a new cluster sorry and see that we have all our name spaces here now deploy again. So that's all for today folks. Thank you all for joining. If you want to get involved or started discussion within the flux project, I recommend going to the discussions and get up. We're also on slack so the flux and flagger slack channel is where you can find us. Yeah, thank you and I hope I will see you there.