 Today I'm going to speak about how we manage our Kubernetes clusters. We'll talk about cluster architecture, what components we're using, and what future enhancements we're planning to implement in our Kubernetes infrastructures. We started to look into Kubernetes in 2019. We went a long way from manual cluster deployments in single data center to automated management solution based on Kube Spray. It's a set of unstable roles to manage Kubernetes and GitHub's toolkit to manage cluster components and cluster applications. Let's talk about our cluster layout a little bit and basic components we're using. So it's really nothing special and it follows usual deployment models. We have three VMware VMs for control plane nodes, then we have three nodes for each CD and then countless worker nodes to run the workloads. We don't schedule our workloads on control plane nodes and each node so that if something happens to the workload, it doesn't kill our cluster or our each CD deployment. You can notice that our worker nodes are not that powerful. This limitation comes from VMware. The automatic virtual machine rebalancing, I think it's called Vemotion, doesn't work very well on bigger VMs. So that's why our worker nodes are just 16 CPU and 24 gigabytes of memory. Currently we are looking into having bare metal Kubernetes deployments to overcome this compute power limitation. And basically just today my team messaged me that they were able to deploy fully blown production ready Kubernetes cluster on bare metal, so this is really great news for us. As for the cluster components, which are listed on the right side of the slide, it's pretty standard. I'm not going to dive deep into too much details there. Just mentioned that the reason why we're using two ingresses, HAProxy ingress controller and engine ingress controller is for convenience only. Some of our developers requested us to enable engines ingress controllers in the clusters because they do some fancy rewriting for their application and they needed that functionality. Our default is HAProxy though. If you don't specify ingress class in your ingress definition, HAProxy ingress will be handling incoming HAP and HPS requests to your workload. For persistent volume support, we use deprecated entry driver from vSphere. The reason for that is because our VMware installation is a little bit old and it doesn't support the fancy new up-to-date CSI drivers. We are waiting for that to be fixed for us by another team. Hopefully eventually we'll be able to switch to cloud storage interface. Another more interesting components we are using is GitOps tool kit that I mentioned already, sealed secrets controller from Bitnami and Prometheus operator. Sealed secrets controller allow us to store secret resources in Git. Basically the way it works is that encrypts a secret in a special format. You can store it in Git and run kubectl, sorry, apply against it. The sealed secret resource will be created in the cluster and sealed secrets controller will decrypt sealed secret resource and create the secret resource for you. The Prometheus operator is used for cluster monitoring, nothing special there. GitOps tool kit is our bread and butter and we'll be talking about it in a little bit more details. This picture is small. Nevertheless, GitOps tool kit also known as Flux CD or simply Flux is a set of controllers that provide you an API to manage your cluster from a Git repository or from a number of Git repositories. There is no hard requirement to use just a single repository to manage your manifests. Installing Flux in the cluster means, excuse me, installing Flux in the cluster adds support for several custom resources which define where your manifests are stored, how to install your help charts, how to handle updates, and so on. Flux also allows you to automatically update your applications whenever there is a new image built by your CI and push to your Docker registry. Flux is able to watch a Docker registry and automatically update an image that is running in the cluster. I don't think we'll have time to talk about this one today. We'll see. The very basic concept is that you create a Git repository which stores your manifests and description of helm releases, install Flux pointing to that repository into your cluster and then your stuff is working up and running after some time without any manual interventions. And what's the most important is that your cluster is always in the state which you define in Git. You can always be sure that it's up to date that no manual changes were made. If someone will make a manual changes to some resources in the cluster, Flux will notice it and will roll it back to the state that's in Git. Here are the main Flux controllers and custom resources they provide. There are more into this, but these are really like the working horses of Git obstacle kit. Source controller provides you a Git repository API. This API describes the Git repository. It's URL, credentials, which branch to watch and how often the repository should be queried for the new changes. Customize controller provides you customization API. This is different from the Kubernetes customized set of APIs. They just used the same name for some weird reason, not sure. Nevertheless, customization API tells Flux where to find your manifests and how to apply them, how to find your manifests within the Git repository. And helm controller provides support for helm release API. This API describes your helm release. If you're familiar with helm, it describes where to find a helm chart, which values to use, and so on and so forth. We also use Flux notification controller and image update automation controllers from Flux, but I won't be talking about those today because we only have limited time. Before we talk in a little bit more details about how all those APIs work and how they look like, I want to show you our repository layout. We are using multi-tenant concept in the repository. There are several sub-directories created which correspond to the name of the namespaces. Each application in our infrastructure gets three namespaces, which equals to three environments that the applications are running in QA, Dev, and Pro. The directories, like the naming convention is not enforced. It's just our agreement that will name the directories as namespaces name. The actual namespace definition is stored inside the YAML file that's stored in the directory. This is basically our usual repository. The system components, as we call them, like interest controllers, primitive separators, everything that's not applications, basically. Each gets only one namespace, which is application name, their system, and they only get just one namespace. We have a separate staging cluster to test upgrades and changes to the system components in the production clusters. We do have certain namespaces where engineers prefer to manage their manifest themselves from their own repositories, and the way it looks like is basically the namespace definition is still present in our repository, but we delegate the rest of the object's management to another team repository using customizations and Git repository resources. This is how usual Git repository definition looks like. Actually, any Git repository definition will look like this. It's very shortened up to the point. This definition basically tells Flux source controller to fetch branch main from the very specific URL every minute. Flux doesn't care about what exactly is in the repository at this point. It will happily clone kernel sources, for example, for you. So this is all that resource is doing. It just clones the repository or fetches updates from the remote repository. Then we have customization that I mentioned already a couple of times. This is connected with the Git repository through the source ref definition in the spec dictionary. This guy already cares about what is the actual content. It will ignore non-YAML files. It will do nothing with them, so you can store like readme or some pictures describing the application architecture in your namespace directories. But it will fail if it encounters poorly written YAML or poorly written Kubernetes manifest. It performs certain validations before it actually tries to apply the changes. And if the validation is not passing, then it will fail. You can have any number of customizations watching different Git repositories for manifests for you. In our setup, it's just our organizational agreement that we always have a Git repository named FluxSystem, which is located in the FluxSystem namespace. And anyone basically can create a merge request to our FluxSystem Git repository. Our team will preview it. And if it's accepted, the objects will appear in the corresponding cluster. Before I go to helm release discussion or description, I want to mention that we currently have six Kubernetes clusters and we have six FluxSystem repositories. So we don't manage all our cluster from single Git repository. We use one repository per cluster just for management convenience or maintenance convenience rather. Until now, we saw how FluxSystem handles plain manifests. Let's talk about the helm charts. There is a helm controller in Flux that I mentioned before, which provides you the helm release API. You can see it on the screen. It's really nice and special. I'm not sure if it's visible to everyone, but if you're familiar with Helm command line tool, you should pretty much figure out the format of the manifest. There is one new resource that I didn't talk about before. You can see it in the source ref definition. This kind helm repository. It's another source controller set of APIs that basically describe a helm repository. It can be anything. It can be a Git repo container or helm charts or it can be some upstream helm.sh helm library. It's pretty flexible which sources you can have to store your helm charts. The helm release object or rather helm controller using the helm release definition will check the state of the installed helm release in cluster every hour and run helm upgrade command against it. This comes with it like pluses and minuses because helm upgrade is not always working well, especially if you change some sensitive values of the values. But usually it's usually pretty seamless in this management. The values can be stored in the same file as the helm release definition as you can see on the screen or you can refer some remote values file that is stored somewhere in Git or is available at some web URL stuff like that. It's again very flexible. We have limited time left so I skip right to the known issues that we hit with this setup. Normally you would expect that when you deploy, when you do disaster recovery for a cluster, you spin up a bare bones cluster, install flux and it pointed to your flux repository and stuff should work. Well, yes and no. We recently found out that we, and it's just our infrastructure, you might have different situation. We have circular dependencies in our repositories. For example, Prometheus operator is using sealed secrets resources and it provides service monitor resource and sealed secrets controller is using service monitor resource and provides the sealed secrets resources support. So this creates this circular dependency for us where both of the controllers cannot be installed because the helm release resource or object fails because it cannot create stuff for you. We don't have a good solution for it yet. We just literally removed the service monitor definition from the repository for sealed secrets controller, wait for stuff to be installed and return the service monitor definition back. We are using storage driver which is deprecated for a long time now. This I mentioned already and we have some of the security issues with this setup because flux allows you to use third-party, more than one Git repository to manage a single cluster and we actually use this feature. We provide, like, we manage certain resources. We owe our developers to manage their workloads. There is a chance that these resources, these repositories can contain some malicious manifests like someone wants to bump up their namespace limits or what not and we have zero control over those repositories. We don't participate in code reviews there and so on and so forth. A solution for that is to enable, to disable, actually to disable cross namespace references in flux. It's pretty well documented in the Flux documentation. It's just we have certain workloads that actually need this functionality and right now we're trying to work this around in the nicest way possible. I'm going to wrap up. According to my timer, we have one minute and a half left for the questions. The slide deck will be available from the organizers if you need it. Thank you very much for your time and if you have any questions, raise your hand right now or find me during the party. I will be happy to talk about our cluster setup. Thank you. How do you manage the infrastructure below Kubernetes? Well, as I said, it's VMware virtual machines and VMware infrastructure is managed by separate team. Answering your question, we don't. The configuration management, like we do manage the OS layer on the virtual machines. We use Puppet as our configuration management system, but it really doesn't touch any Kubernetes related things. Maybe a couple of CCTLs, but not more than that. How do you upgrade your clusters? How do we operate our clusters? Can you elaborate it? Upgrade. Oh, yeah. This is a very good question. We are using CubeSpray. We basically follow CubeSpray's official documentation and the upgrade itself consists of a couple of manual steps that we upgrade the CubeSpray version that we're using and we upgrade to the version of the Kubernetes cluster that comes with the version of the CubeSpray. We don't override it in the group variables files. So we... Thank you and hold on. I want to finish. We don't do anything fancy that goes beyond the official guides because we want newcomers to onboard quickly so that it's enough for them to read the official documentation and be done with it. Oh, yeah. Another question in regards of upgrades. So which Cube version do you use? And so... Yeah, which Cube version do you use? And as far as I'm concerned, that entry drivers would be disabled in a couple of Cube releases and you will have a couple of problems. What's with that, I guess? So the question was which Kubernetes version we're using right now and what we're going to do when the storage driver will disappear from the Kubernetes version that we're about to upgrade to. I don't remember the exact version that we're using right now. It's what was latest in Cube's prior repository two months ago. Two months ago, we finished our quarterly upgrade cycles and it's 1.29 something, I think. I'm not sure, really. What we're going to do when the driver will be removed from the tree? Well, as I mentioned, the VMware... We hope that our VMware will be upgraded and we'll be able to switch to CSI drivers, cloud storage interfaces. If we won't be able to do it, well, we'll just stay on the latest version where the driver is enabled because we cannot upgrade without having the persistence. We have certain stuff like hardware, for example, that we use for Docker images hosting. It requires persistence and we cannot just upgrade and drop the persistence support from the clusters.