 Hello and welcome to today's cloud-native webinar on Flux and OCI. My name is Max. I'm a core Flux maintainer. I've been on that role for about a year now, and I'm a senior software engineer at Weaveworks, and there I'm working mostly on multi-cluster Kubernetes with Weave GitOps and also on Flux itself. Yeah, as it says on the slide, I love working on distributed system. I love programming and Go, which is a language I quickly learn to like very much because of its simplicity and ease of deployment, I should say, and I'm very passionate about free and open source software. Our agenda for this webinar will be as follows. I will introduce you to terminology and semantics of GitOps and Flux itself, what Flux is and how it got to be. And I will introduce you also to OCI, what these three letters stand for and where is it coming from. And then we're going to get our hands dirty looking into what Flux can offer with OCI and conjunction. I'm going to show you that it's still GitOps and we're going to run through two demos, one for Helm and one for Customize. So what is GitOps to begin with? GitOps evolved into these four principles that are outlined on the slide. The first one is a GitOps system is declarative. That means the system managed by GitOps must have its desired state expressed declaratively. For example, Kubernetes where each state, for example, a deployment is declared in a manifest file. GitOps means state is versioned and immutable. So the desired state is in a way that enforces immutability as with Git where an object reference cannot change. It provides versioning and retains a complete version history. The third principle pulled automatically means that software agents running on the system that uses GitOps pull the desired state from the source. Which is from a Git repository or as we'll see today from an OCI distribution. GitOps also means and that's the fourth principle that the state is continuously reconciled. So it's not only that you declare your manifests, you store them in a versioned way and then an agent pulls them. But this agent continuously observes not only the state of the source of these manifests, for example, a Git repository, but it always tries to match the actual state of the system with the desired state of the system. So any system that follows these four principles can be called a GitOps system. And why would people want to use GitOps in the first place? And I've put some bullets on this slide who lay out the major benefits. You get stronger security guarantees. You have much more developer and operational productivity. You have a better developer experience. Developers can just write their manifest files, push them to Git, have a CI pick them up and then the system will reconcile them. You have an improved stability, especially because of the immutability principle. You have a higher reliability because the agents continuously try to reconcile the desired state with the actual state. And it also provides a way to be consistent and standardized. And now what is Flux? Flux is at first an implementation of the four GitOps principles. And it also is a CNCF open source project. Flux in one sentence is a Git-centric package manager for your applications. And it is a set of continuous and progressive delivery solutions for Kubernetes. So it's very much fit for any Kubernetes deployment. The benefits are very much aligned with the benefits of GitOps in general. You reduce developer burden when you use Flux. It is very extensible. It even provides a good API where you can build your own components on top of the core Flux components. It comes with out-of-the-box support for Customize and Helm. And as I said before, it is designed for Kubernetes. So Kubernetes is like a first class citizen in the Flux world. So it provides GitOps for not only your apps but also your infrastructure. So you can have your cluster completely declared and get repository and Flux will make sure to install all the applications and infrastructure components. It integrates well with existing tooling. So it is as opinionated as necessary but as dynamic as possible. It works with Kubernetes. Multitenancy is built in. So the Flux CLI, for example, provides commands for managing multiple tenants on a cluster. Flux has ways to alert and notify users upon certain actions. This is done using CRDs. So you are very flexible with which actions you want to trigger and alert from. Users trust Flux. So we have a huge user base among the cloud native community. And Flux itself has a very lovely community. And I myself experienced that when I first got in touch with Flux around one and a half years ago. It's just a very lovely experience being in touch with the Flux community in general. Flux is a set of controllers. So it does all what is laid out here. In one of these controllers. So there's source controller, for example, that actually clones a Git repository or fetches a harm repository index. Or fetches an OCI image from an OCI registry as we'll see later in the demo. Then we have customized controller that implements Flux's opinionated way of applying customize. It has a home controller so you can very easily use Flux to deploy home charts and create home releases. Notification controller is a separate component that is configured using CRDs. Then there's image reflector controller that you can leverage for image automations, but also for other things. So image reflector controller basically tracks an image registry, which is also an OCI registry. And, for example, reflects the latest tag of a certain image in the cluster and makes it very easy to act upon new images being pushed to a registry, for example. Which is something that image automation controller does. So image reflector controller and image automation controller taken together let you automatically update your applications based on the images that are available. And Flux can even commit to a Git repository automatically if you configure it so. So the next thing that I quickly want to dive into is what is OCI? So OCI stands for the Open Containers Initiative that was founded in 2015 by Docker and others. So basically Docker, the company took what they had and brought it to the OCI to have a bigger community around Docker containers. And it is part of the Linux Foundation. It provides several specifications for image formats, distribution and execution. And these three brought together provide a very powerful tool set. And as you can see here, when I issue Docker Run and provide it with a certain image and registry name, these three specifications work together. So there's execution, which is symbolized here by Docker Run. So there's a runtime running on the computer that takes care of executing the image. And then there's the address of the registry Docker.io, which implements the distribution specification. And eventually the image itself that is pulled from the registry is specified by the image format specification. And ever since the OCI was founded, usage of its specifications as soon as they came out has transformed from only Docker containers but one a certain application in a certain way in a certain runtime to a very generic application and configuration container format. So this becoming an application delivery format. There's several prototype and also product ready solutions out there that show what is possible with OCI. So this car, which is a personal project from Chris Nova, where you can save your configuration of the application together with the actual application in the form of a container image in one container image. And then when you want to run it, you have like only a single image that you pull or that you distribute it to your users. And they have everything in that single image, the configuration as well as the executable application. And then there's customizer, a project by one of fluxes core maintainer Stefan. And that's very, very similar to what flux does with customize and what we're going to see in the demos. There's also quite new project and a company that has been founded around it called Acorn. And Acorn uses OCI for bundling applications as OCI images. So you have a very nice front end, a very simple way of declaring an application. And then Acorn pulls all the definition coming from that front end configuration, creates an OCI image from it, pushes it to a container registry. And then the Acorn agent warning and Kubernetes makes sure to execute the application as a disconfigure. And Google is also heavily invested into OCI, for example, with config sync, which is part of Google Anthos and can also consume various kinds of OCI images. And the great thing about OCI besides its open specifications is that OCI registries are already a commodity in the cloud space. So every cloud provider has an OCI registry warning that hosts the application images, so the Docker images, if you will. But there's also lots of other implementations out there. And the Aorus project actually provides a list of these implementations. The simplest one is shown here below, which is distributed in the form of the registry container image, that you can just run using docker run registry column two, and then you have a full blown OCI registry warning on your machine. And now, when OCI and Flux come together, oftentimes people ask, how is this still GitOps? Like Flux, when it only pulls images from a container registry, it's not pulling anything from Git, so can we still call it GitOps? But if you look closely at the principles that I described above, OCI satisfies all four principles and eventually Git is still the source of truth. It's just that an OCI registry now serves as an intermediary where all the manifests are pushed to and that Flux pulls from. So before OCI, you push to Git and then Flux directly cloned the Git repository and reconciled the cluster state with the desired state in Git. And now you push to Git, you have a CI running that creates artifacts, pushes them to an OCI registry, and Flux pulls from there. So it's not very different, but eventually it provides better security and much better scalability. So pulling an OCI image is much less resource intensive compared to full or even a shallow Git clone. High available registries, as I said before, are available everywhere. So if you're running on AWS, if you're running on GCP or Azure, you will have an OCI registry available for your purposes. And in terms of security, Flux leverages Kubernetes workload identity and IIM when pulling OCI artifacts. So you don't have to manage any keys anymore on your cluster or even in your Git repository. There's no more SSH keys to wrangle and to generate to make sure they are revoked from time to time. And there's no more proprietary API usage, for example, having to hit up the GitHub API to create a token with a certain scope. It's actually using the same mechanism that's used for pulling containers. Because pulling containers and pulling anything else from an OCI registry is just the same thing in terms of security authentication. And very soon we'll be merging a PR that lets you verify the images that Flux pulled down using Cosign. We're currently working on that and trying hard to make it available as soon as possible in Flux. I now want to show demo how you use and leverage OCI in Flux for Helm and for Customize. We are at the terminal now and I would like to start this demonstration by showing how you can leverage OCI registries to deploy Helm charts. I have a cluster running locally here, a simple kind cluster. And we are going to start by creating the Helm demo namespace. Now I'm going to use the Flux CLI a lot. However, for 90% of the actions that I'm showing here, the Flux CLI is not really necessary because all it does in this demo is it creates Kubernetes objects. And in a production workflow, you would just store these Kubernetes objects in your Git repository. Alright, there's one exception and I'm going to call that out later when we get to that. Alright, let's create a source of type Helm, call it Pod Info because that's what the application that we're going to deploy is called. And now the difference compared to HTTPS-based plain old Helm repositories is that the URL starts with the OCI scheme now. And it's pointing to GHCR, which is GitHub's own container registry. And as you can see, it just generates a home repository that we can just inspect. And if you're used to using Helm with Flux, this doesn't look very different with the exception that the type is now set to OCI here. And the URL, as I said before, starts with the OCI scheme and not HTTPS. And now based on this source, we can create a Helmese. We're going to call it Pod Info as well. We're pointing it to the Helm repository that we just created and to the chart that we want to deploy. So what you can also do if you don't want to maintain a home repository per chart or even per OCI namespace, is you can create a home repository that just points to the domain or to the base of the OCI registry. So basically just point it to ghcr.io. And then if you do that, you have to provide the complete path to the chart. So in that case, we would have to do something like this in the Helmese. That's something that has come up in the community and being asked for and one of the latest additions to Flux so that you don't have to maintain a lot of Helm repository objects in your cluster. For the purposes of this demo, we're going to keep it simple. It's generating the homilies. The homilies Pod Info has become ready and we can have a look at it. And again, if you're used to using Helm with Flux, this is 100% looking the same as before as it would if this was based on a chart served through an HTTP Helm repository. So if you're already using that, the only thing that you would have to change is the source, the Helm repository and not your Helm of Isis, which makes migration to OCI a lot simpler. Let's verify that the pods are there and that the services there, everything has become ready. And it's as simple as that. So consuming Helm charts from OCI registries doesn't change very much in terms of your workflow, especially if you're consuming third-party charts and you don't have to upload them to an OCI registry. Now two things that are different though are supporting workflows. And one of these is updating chart versions automatically. So you might know that in the homilies, you can specify not only the exact version of the chart that you want to deploy, but also a semantic versioning range. But in that case, the concrete, the actual version being deployed wouldn't be reflected in your source, in your Git repository for example. If you need this to be the case, you can now use automated updates to Git with OCI repositories because Flux has an image automation component built into it. And it should be explicitly enabled though, but it's just a command line flag when you're using the Flux CLI. So it's one more controller that's running in the system then that watches container registries and that of course can also watch any OCI registry and it can apply image update automations based on a chart version for example. So now if you're consuming health charts from OCI, you can use this image update automation in that workflow too and you will get commits for every version bump and you don't have to do that yourself. You can also let it, you can specify the branch that the change is being pushed to here in the spec of the image automation object and have it push to a certain branch that another workflow will take off from and create a GitHub PR for example if you don't want Flux to directly push to your main or production branch. And one more thing that's possible with OCI based homo repositories is that you can use the provider field. So the dot spec dot provider field of Helm repositories, let's you specify one of these four providers and we support the three major cloud providers AWS Azure and Google GCP and the current version of Flux and then you can make use of your worker node authentication so you don't have to store any secrets, any credentials in your cluster and provide them to Flux in order to access private registries for example. Right and we have examples for all these three cloud providers in the documentation. So I want to encourage you to have a look at this documentation if you're on one of these three cloud providers and don't want to maintain any credentials in Git or on your cluster anymore. So that's it for Helm. Let's clean up and let's proceed to how using customizations now works with OCI because that's even more exciting in my book. All right, so we are in the pardon for directory. This is the source code I forked pardon for from from Stefan's repository. And it has a customized directory here that has customization file. That doesn't look very exciting if you're used to customize. It pulls in three Kubernetes manifest files creating a deployment horizontal part auto scaler and the Kubernetes service. So how do we deploy this customization using OCI info? Okay, so let's say we want to deploy version 002 of our deployment customization manifests. Now with OCI support for customizations, the Flux CLI also got some new artifact some new commands. One of these is flux push artifact artifact rather that is provided with the URL of the artifact that I want to push. The path that I want the artifact to contain or to be built from. And I'm going to tell it this is the source of this artifact. We're going to see later how this comes into play and the revision. This both of these annotations that are put onto the resulting OCI image are there to facilitate tracking back an OCI artifact to its source. So that you can know, okay, this is coming from this Git repository. So if I need to change anything, I'm going to clone this repository and push the changes to it. And here is the output of the successful run of flux push artifact. It says I'm pushing the artifact to this registry. It has been unsuccessfully and it also spits out the actual image digest. So this looks very similar to the workflow that you use when you push a Docker image. And in fact, it's not very different. Let's have a look at the registry real quick. GitHub provides a nice user interface for its registry. And you can see version OO2 has been pushed. Here are all the tags that I've created before. So OO1 has been created and like one day ago and now we are at version OO2. You can also make use of a tool called Quaint to check all the tags. But the flux CLI also has a command for that. I'm sorry. You need to provide the OCI scheme. And then flux is going to pull down all the tags and show the source and the revision. These are the two data points that come from dash dash source and dash dash revision when we push the artifact. So this is very valuable information when you need to work with these artifacts, right? You don't have to provide them by the way, but it is highly, highly encouraged to do that because otherwise it'd be hard to track these back to their sources. All right. So now we have pushed our artifact. Let's see how it looks like. What does a flux OCI artifact actually look like? And I'm going to use the Quaint tool again to pull down the manifest file. And this looks very, very similar, if not equal to a manifest file for a Docker image. It has a config. Flux is currently not making use of these configs, but it's still there. The important thing is the layers. Flux by default creates one layer, but you can actually push OCI artifacts with several layers and tell flux which one to pick based on the media type. So you can have a layer for configuration, a layer for your artifact, and maybe a layer for your actual Docker images. And then you tell flux, please pull the artifact from this media type. And let's have a look at the actual image content. So every content in OCI is addressable via a digest. And of course that's a binary blob. So flux basically just takes the path that we provided it with, with this flag. It creates a tar archive and gzips it, and that's the content of the layer. So what you see here is just the same as we have in the customized directory. And it's as simple as that. So a flux artifact is basically an OCI image with the tar archive as a layer. So let's consume it. We're going to create a source of type OCI, and we're going to tell it to consume this tag. And you'll see here that it's telling us that it's creating an OCI repository. OCI repository is a new type that we created to support these kinds of workflows. See here, this is kind OCI repository. Also as a spec field as to all flux resources. And it looks very similar to other source CRDs where you can provide an interval. You can provide a provider here as well. So this is the same as with OCI based helmet repositories. So if you want to access a private OCI repository and you're running on one of the three major cloud providers, you can just tell it provider AWS, for example, and then Flux will use the cloud provider's authentication mechanism. For example, IAM and the AWS case to authenticate against the OCI repository. Since this is a public repository, the provider is just set to generic. And then you can also provide a secret ref that contains the actual credentials. And here you'll see that Flux also pulled down the annotations from the image and stores them in the status of the object, which is also very helpful. You don't have to pull down the image yourself to know where the source comes from, where the OCI artifact comes from. All right, so I have another tunnel open on the right side. We're going to watch for pods being created in the default namespace because now we are finally going to create the pattern for customization. This now comes from an OCI repository that's called Pod Info. The path is customized as we saw when we pulled down the actual layer contents. And we want the target namespace to be default so that everything coming from this customization is created. This namespace dash dash prune tells customize or Flux to remove every object that the customization created as soon as the customization is deleted from the cluster. Now the pod is being created. It's running and it's going to get ready in a couple of seconds. The deployment is scaled to two pods. So we have two pods running of Pod Info now. We have servers running in the cluster and everything is fine. Let's have a look at the customization. And this also looks very similar to all the other customizations that you use that are based on, for example, a gateway repository source. The path here tells Flux to only create the customization objects from this path. And this actually helps you when you want to deploy the same OCI artifact on different clusters. For example, a dev cluster and a prod cluster. So you can store the configuration for both in the same artifact just a different path. And then on the dev cluster, you point it to the path dev. And on the prod cluster, you would point it to the path prod just as you would with the get repository. Now we see that a service of type cluster IP has been created. So if you don't have an ingress controller warning or don't have any kind of ingress on that cluster, we don't have a way to access the Pod Info UI without any port forwarding, for example. So let's change that. We're going to change the service type from cluster IP to loadPalancer. And we are going to change the version that we are going to push. So I changed the service manifest. And now I'm going to run Flux push artifact again and create another version, 003. And now since we pointed Flux to the specific version 002, I need to change the reference here to 003. And as soon as I exit here, take a look at the right side. And now the Pod Info service has changed from cluster IP to loadPalancer. So you can take the IP address, 9898 is the default port. And we get our greetings from Pod Info. So everything that I had to do was change the manifest, push another artifact and change the OCI repository. Now if we want to skip the last step, changing the OCI repository, we can just tell Flux to use a semantic versioning range. In this case, the asterisk tells Flux to always use the latest version. Now let's change our service back to cluster IP because maybe we now have an ingress controller running in the cluster. So we're going to set the version variable to 004 and create and push another artifact, no version 004. And now since the interval of the OCI repository is set to one minute, it can take up to one minute for this change to be applied. And here we go. And we can also inspect the OCI repository, get too much. And we can see that the revision is this digest starts with A9A. And this is actually the same revision that we just pushed when we pushed version 004, this one here, A9A. Now, oftentimes the question comes up how to track back a certain resource to the source that is being created from in a get-ups workflow. So let's say we have this deployment here that is called PodInfo. How do you know which actual source repository or get repository it is coming from when, for example, I want to change it from two replicas to three. And Flux has this handy trace command which I can ask this exact same question. So I run Flux trace and search in the default namespace for the deployment PodInfo and tell me where it's coming from. And here you can see Flux prints out the path back to its actual source. So this is the deployment that we are asking information on. It is being created by the customization called PodInfo in the namespace Flux system. And that customization actually consumes manifests from the OCI repository called PodInfo in the Flux system namespace. And since we annotated the artifact when we pushed it, we also know the exact revision in the get repository. So this contains the branch and the tag, I'm sorry, the char of, so the commit char and the path like the URL of its repository. This doesn't need to be a URL, by the way, it can be anything that makes sense in the environment that you're working in. But usually it's a URL pointing to the actual get repository that you can clone. So in this case, we would know I need to clone this repository and check out this commit to know what exactly is being created. And now if I want to change that, I can just change the files in that repository and create another artifact. So Flux trace and Flux push artifact are any artifact command in the Flux CLI rather like Flux list artifacts, Flux push artifact, and there's also Flux pull artifact, which is very much the same as Flux as the crane blob command that we saw before. Are the commands that I called out before when I said like everything can be done without the Flux CLI. In this case, in order to, for example, trace back sources without having to use the Flux CLI, you'd have to build obviously something yourself, but that is pretty simple because the Flux CLI also just consumes the objects in the cluster. So there's no magic behind that. It's just a nice way of representing, for example, the trace path here. This isn't specific to a CLI. I just want to point that out real quick. This has always been the case with Flux. It applies labels so that you can track back each resource back to its source. So here the label says this resource is being created by the customization of this name coming from this namespace, and there you can trace back your way using cube color, for example. And for Flux push artifact, it's very similar. You can create your own artifacts that just have to have the same structure. So like the layer that you're pointing Flux to has to be a tar archive, tar gzipped archive containing the manifest files or the customization. So this is how you use OCI and Flux today. And I want to point out one more benefit of using OCI if all the other benefits that are called out now haven't convinced you yet to at least try out OCI with Flux. And one of them is that more and more cloud providers and OCI implementers are starting to support immutable image tags. For example, here's the announcement from AWS from two years ago where they announced that they are going to support immutable image tags. And this is a major benefit over Git, for example, because when you consume a Git tag, there is a chance that you're not consuming the exact same content as maybe in a week or one week ago, because Git allows you to just force push a tag and override the tag with the new content. Using immutable image tags and OCI, this is not possible anymore. So this is a major step forward when you need to have a better accountability in your deployment pipeline because when this tag has been deployed and immutable image tag policies are in place, you exactly know the content that is being deployed, which is very important, especially to security or rare people. Now, if you found all this helpful and if you want to try out Flux yourself, I encourage you to have a look at the Getting Started Guide. This is part of our own documentation that we maintain at fluxcd.io and it helps you just getting started installing Flux bootstrapping with your own cluster and everything. And then if you want to take it a step further, you can browse our docs a little more. We have documentation on all the available components, source controller, customize, and so on. So this is also a very, very helpful reference when you're using Flux on a day-to-day basis. We also have covered use cases that we usually stumble upon and that we hear from people using Flux. We have guides on, for example, using CL secrets or Mozilla SOPS if you need to store your secrets in a secure way in your Git repository and all that. So I really encourage you browsing the docs. And if you want to get in contact with the broader Flux community or the Flux maintainers, just join the CNCF Slack at cloud-native.slack.com and join the Flux channel there. I myself have experienced the community to be very, very polite, very, very welcoming and helpful. So if you have any questions, please don't hesitate to reach out in that channel. You also have a mailing list where we put out monthly updates and announcements and all of Flux's source code is spread across GitHub in the Fluxity organization. But it's best to start with a Flux to repository where we also have many discussions going on. For example, enhancements to Flux. So if someone wants to propose an enhancement to Flux, they usually start a discussion on GitHub in the Flux to repository. And with this, I want to thank you for taking part in this webinar and have fun with Flux.