 Hello, everyone. My name is Adrian Vakaru. I'm a software engineer with Fidelity Investments, and I am working in the public cloud platform team. We are working towards enabling our internal business units to safely and securely adopt cloud at scale. I'll give you an overview of why and how the Crown project started. We'll deep dive into how it works. And we'll also have a look at some of the features it currently has. First of all, to better understand the Crown and the need for it, you need to know that Fidelity is embracing a multi-cloud strategy. And the approach we took to achieve that was to build a multi-level platform on top of Kubernetes and other CSF technologies. We started to work on building conformity across all the platforms that a business unit may consume. The goal of doing that was to achieve a consistent developer experience, no matter what platform the developers are using. In other words, if they are using Kubernetes and they deploy on top of Kubernetes, it's Kubernetes. So we worked to eliminate some of the complexities or differences between running Kubernetes on-premises or running Kubernetes managed by a specific cloud provider. In this way, when the application developers want to deploy to the cloud, they shouldn't have to modify their applications to suit depending on the cloud provider they are trying to deploy to. While the focus is on the user experience, which is the application teams, the platform developer experience is also improved when working in layers. It allows for usability of components and collaboration between different business units and teams to improve the platform experience. For example, here, the security layer can be developed and tested by a dedicated team of cybersecurity specialists. Once a new version of this layer has been released, the platform developers can integrate that in their systems as a dependency. They will have the confidence that that layer was tested and certified to be working on a specific Kubernetes version and on all the cloud providers. Before we move on to the next slide, let's take a moment to look again at the big picture here. So with the help of the color coded boxes, you will notice that as a user of the platform, your experience would be very similar no matter what the underlying cloud provider is. Your application can leverage the same tools, the same services on each platform, and you can easily migrate your applications from one to the other with minimal or zero development effort. Now that we've seen what the objective was, we needed to find a solution to implement that. I think you might be wondering why do we need another deployment tool to achieve that? So in the beginning of our cloud journey at Fidelity, we found ourselves constantly adding new features or as we call them add-ons to the platforms we were developing. So managing the deployment of those add-ons became more complex and harder to maintain as the system expanded. Using Helm and Customize at the beginning works very well. You get a quick MVP, you have fast development lifecycle, and ultimately you get working software. Problems start when you need to handle all the dependencies between add-ons, all the preconditions before installing or deleting an add-on, and implementing remediation retries if something goes wrong and doesn't work as expected. For example, you might have applications or services that are depending on cert manager being installed before they can deploy correctly, or you might want your OPA constraints to be in place before you deploy any other applications to your clusters, and so the list can go on. Crowd comes into play when you need your suite of application to be deployed following some very specific business logic. It can simplify your deployment logic, but it can also speed up your deployment process by grouping those add-ons in layers. Before we move on from this slide, there is another thing it's worth mentioning here. So the difference between customized Helm and Krann is that Krann is a Kubernetes controller. So it's deployed inside the cluster and all the deployment logic it's happening in a reconciliation loop. Let's have a look at this example. So your deployment script could turn from what you see on the left side here, a lot of ifs, while loops, to one single line with Krann. QCT will apply dash f layers.yaml. Krann will then take care of all the prerequisites you define in the layers manifest and only deploy your applications when they are ready. More than that, it will also reconcile your deployment application for further upgrades. All you need to do is define the layers, define the dependencies and let Krann and Flux take care of the rest. Let's see what's inside the layers.yaml file. So two of the most important configuration options here are the source and the prerequisite sections. The source will reference a Git repository which contains the layers contents. These are Helm release files inside a Git repository. All those Helm release files will be installed as part of this layer. The prerequisite section is Krann's brain. It is what is driving Krann in applying those Helm releases. They will only be applied, the Helm releases will only be applied after all the dependencies here are fulfilled. One other thing to notice here is the version of the layer, but also notice that the version is using the depends on property as well. This is what we call the version dependency system. The common layer here depends on the security layer at version 120. So unless this layer here, it's already deployed to the cluster, the processing of this layer won't start by Krann. At this stage, all you need to know is that managing the prerequisite section wisely combined with defining meaningful layers in the Git repository can help you achieve even some of the most complex deployment scenarios. We'll dive into how everything works in the following slides. So let's see some technical details about Krann in order to understand how the magic takes place. Krann is built on top of the GitHub toolkit. It is using two of the key components of the flux family of projects. Those are the source controller and the Helm controller. This means that on top of the features Krann brings to the table, you can take advantage of the previous experience of using the flux projects. When deploying Krann to a Kubernetes cluster you'll get the source and Helm controllers out of the box as well. They all work together to bring your cluster to the desired state during each reconcile loop. To better understand how all these components interact let's have a look at the following diagram. So the dotted lines here just separate the controllers from the resources they manage. We start from the bottom up. The Krann controller reconciles add-ons layers. The add-ons layers construct is the only manifest file that a user would directly apply to the cluster. Everything else happens automatically. So as a user you apply the add-ons layers manifest file to the cluster and that add-ons layer contains a source reference which is in fact a Git repository that is managed by the source controller. Inside the Git repository in a source control management system like GitHub, for example, there is a path that contains a bunch of Helm release files. Those are the manifest files that Krann will need and it will apply to the cluster. Once Krann obtains a copy of those files from the source controller it will apply those files to the cluster. And from here on the Helm controller will react to this change and it will work together with the source controller to get the Helm charts from the Helm repositories and apply them to the cluster. Let's have a look at the Git repository and GitOps reconciliation. So because Krann is built on top of the flux projects the source controller it's the one that's handling all the GitOps operations. When defining the Git repository source for an add-ons layer we have a choice of how we want to source the source controller to track that repository. For example, if we decide that our add-ons layer will track the develop branch of a Git repository every time there is a new commit to that branch when the source controller reconciles the repository the Krann controller will pick up the changes and decide if there's something to do to bring that layer to the desired state. In this case here you've seen there is a new add-on added there's a new Helm release added to the layer and it will be deployed to the cluster. Everything happens automatically starting from the Git commit to the develop branch the source controller pulls the latest changes from the Git repository the Krann controller gets a fresh copy of the files and notices the difference. It then proceeds to apply move or delete an add-on from the layer in order to bring it to the desired state. If you want to use if you want to track stable released versions of a layer then a tag or a commit hash should be used. You have probably noticed how the version of the layer is tightly coupled with the reference the Git repository is tracking. Besides playing an important role in the version dependency system it also gives the visibility to the cluster administrator into what versions of each layer are deployed. The next slide will describe how an add-on can be configured. Familiarity with the Helm controller would be an advantage if we simply put it this way an add-on is just a Helm release. Here are some of the features the layer developers cannot take advantage of. So the remediation retries when installation or upgrades fail. They can automatically run Helm tests after each installer upgrade. They can provide dynamic values to the chart using values from a config map inside a cluster. And they can also define dependencies on other Helm releases. Just to clarify once again what I'm talking about Helm releases or when I'm talking about an add-on this is what is actually behind the hood. In other words, all the information all the metadata about how an application should be installed. The application itself has to be packed as a Helm chart that is developed and maintained separately. Let's now talk a bit more about the business logic of CRUN and we start with how CRUN handles the dependency relationships. So the main construct that CRUN defines is an add-on layer. And inside that add-on layer there can be a number of different add-ons. But the most important piece of the business logic is that you can define dependency relationships to from a layer to another layer or from a layer to a specific Kubernetes version. Let's take an example. So we have layer X and layer Y and we want to only install layer X if layer Y is available. But also we don't want to install layer X if the Kubernetes cluster version is lower than 120 and we don't want to install layer Y if the Kubernetes cluster version is lower than version 121. This helps with reducing deployment complexity deployment script complexity and also gives developers confidence that the add-ons layers have been tested and certified to be working on a specific Kubernetes version. One other thing to notice here is the interlayer dependencies that can be added with the help of Helm Controller. You can see them on the screen between add-on two entry and between add-on four and five. This adds to the flexibility you have when defining your layers and helps with the confidence that all the add-ons will deploy in the expected order with minimal remediations. Further down into the dependency system I would like to talk about a small but very powerful feature of CRAN and that's the version dependency system. We'll make a comparison between the version dependency system and a non-version dependency system. We have as an example two layers where layer X depends on layer Y. The dependency is needed because all the add-ons in layer X depend on cert manager from layer Y. Between the two layers there's just a simple dependency relationship that allows layer X to start deployment only if the layer Y is already deployed to the cluster. Please also notice the versions of each layer, both of them being on version one at the moment. Also another thing to notice is that cert manager in layer Y is at version one at the moment. Given that both layers are applied at the same time as soon as layer Y is deployed successfully then layer X can start deployment as well and it will complete the deployment successfully because add-on one, two and three were able to create all the certificates they needed because cert manager was already installed. Now, in the scenario of an upgrade we develop add-ons one, two and three to use a new version of cert manager. They're gonna be dependent on version V2. If we put those new add-ons in the version two of layer X, layer Y, we're gonna leave it as it is that it will have deployed cert manager V1 and that's it. It's already deployed in a deployed state. So because the dependency relationship doesn't indicate a specific version the layer deployment will still go ahead. So layer X will start deployment because layer Y is already deployed. And of course, the layer X deployment will fail. At deployment time, add-on one, two and three we're trying to create a new kind of certificate that only becomes available in cert manager V2. So how can we avoid this kind of errors? If we go back to the initial state we can avoid the previous error by defining a version dependency relationship as follows. The first step is to say that layer X depends on layer Y at version V2. With this condition in place now ground one start processing layer X V2 until layer Y V2 has been successfully deployed to the cluster. In the current state, layer Y V2 doesn't exist in the cluster. So layer X won't go ahead with deployment. Won't go ahead with the upgrade. It will have to wait until we deploy layer Y version two to the cluster. Once that is done, you can see here cert manager V2 once that layer is applied it will complete successfully the deployment successfully. Only at that stage, layer X will get the green light to deploy and complete successfully. All because now cert manager version two is deployed by layer Y and add-on one, two and three were able to create the certificates they needed. Another interesting feature of CRON is that it is possible to change your deployment strategy and you can move an add-on from one layer to another without having to remove and reinstall the Helm chart. Doing this, you can avoid any unwanted downtime of your applications or even worse potential data loss. An example for data loss would be if you want to change your deployment strategy. So with one of your add-ons you are deploying has CRDs deployed as part of the Helm chart removing the add-on could cause the CRD to be deleted and all the related custom resources will be lost as well. So you don't want that. How can CRON help in this scenario? So let's say you've defined the layer Y to contain only two add-ons instead of three. So layer six, it's now outside that layer but you want to move it to layer X. So instead of just deleting that add-on when it's removed from layer Y and instead of adding the Helm release file, applying the Helm release file again because it's defined in layer X, CRON is waiting for a period of time before taking any action on this add-on. It's waiting for a period of time and it will observe that, okay, layer X is now referencing this add-on. I'm not gonna apply the file. I'm gonna just change the referenced owner of this add-on. And it will simply do that and we call this the orphan and adopt feature. As I said earlier, this helps with unwanted application downtime or potential data loss while allowing you the flexibility to change your deployment strategy. Wait, what about deleting an add-on? So it works in a similar fashion only that if at the end of the grace period no other layer is adopting the add-on, it will be marked for deletion. And whenever that grace period finishes, now this will be the desired state is gonna be reflected in the cluster. The add-on will be deleted. Yeah, so now last but not least, I'd like to mention that CRON is an open source project. So we'd be glad if you check it out on github at github.com slash fidelity slash CRON. That's where you can find all the resources or the source codes of the CRON controller and also a Helm chart that can help you deploy CRON, the source controller and the Helm controller so you can start building layered platforms as well. I'd like to thank all of you very much for joining me for this talk and I hope it has helped you in some way. I'll see you in the live Q&A section and thank you again. Bye-bye.