 All right, welcome to our talk on cost and resource management on CI, CD infrastructure by using OpenShift. So a few words about ourself. My name is Salus Prasmanthas. I work as a SRE at SDX. My name is Marcel Harry. I work as a cloud architect in the services team at Red Hat. I mainly focus on customers within the Alps region and they're on infrastructure and OpenShift for a topic. Hello, my name is Rady Dono. I work as a cloud consultant in the services team as Marcel in the Alps region. I'm focusing mostly on OpenShift and cloud native ecosystem in general. OK, thank you guys. So we all work at SDX at the moment. This is a six group company providing financial market infrastructure services in Switzerland. There are over 200 employees and we build DLT based solutions. We're part of Blockchain Network. Primarily we use R3 Quarta. But as you can see, we also partner with Hyperledger and Enterprise Ethereum Alliance. So we build a number of products where we provide trading infrastructure for digitalized assets, also centralized security depository, market surveillance, market data, quite a bit of products which requires agile and scalable infrastructure. So companies started in 2018. We provided a group of concepts. And later on, we adjusted to our vision where we're striving to build next generation digital assets infrastructure, which bridges traditional financial world and new digital, including crypto asset world. So this requires scalable CICD infrastructure. And the first question, why does it have to be container driven? First, we want to build it once and run it everywhere. It means developer builds on the laptop, compiles it, pushes the code. CICD's system releases, securely signs it. So it has to be scalable, first of all, to accommodate weak or weak-end loads. There's a end of sprint when there's a lot of things happening. It also has to be rapid and with uniform tools, meaning we want to use same tools for building the code or deploying the infrastructure. And for FinTech, company security is number one priority. So we want to enhance it with infrastructure and we have to have flexible configuration to segregate development from release or deployment workflows. So improving security is extremely important for us. Also debugging, same as developer debugs on the laptop, they need same capabilities to have it in CICD pipeline. We chose GitLab. Here's the architecture of GitLab, which is on the left you can see the GitLab server, where developer pushes the code and there's an API where GitLab runners connect. So first of all, it registers with the GitLab token and then it constantly pulls if there are any pipeline jobs available. And based on the type of workloads, it starts a number of runners, jobs. It scales through the nodes in the pods. GitLab runner types. We can segregate GitLab runners into multiple types based on security requirements or based on type of sizes. So for example, developer feature branches, they could be run on shared autoscale build runners and release. They need more secure environments, which leads to OpenShift features. OK, secure builds. This is one of the, as I said, important features to run these builds separately from each other. So when we run a new job, it has to be separated. So there should be no previous artifacts or workloads. OpenShift provides a nonprivileged user by default. So we segregate by user ID. Also, separate projects and namespaces can run separate jobs. It integrates quite nice with a hash across vault where we store all the secrets. And it also provides dynamic capability. So when we log into GitLab or S3, every time a new secret is generated, for releases and for extremely secure loads, we run it in special hosts with a trusted execution environment. So why OpenShift? This is comprehensive offering, which provides the enhanced security and enhanced scalability. It has some extra robustness and performance. It also compliant and very development-friendly. And we also get great support from our friends at Red Hat, where Radu and Marcel could brief us in, how did we achieve scalability and reducing the cost of CI-CD workloads, Marcel? Yeah, thank you, Solis. So let's talk about one of the features that are coming with OpenShift, and especially when we combine OpenShift with a public cloud like Azure. Maybe next slide, please. And when we are developing in such an agile environment, we actually kind of need to address also very elastic demands. So people are kind of like busy, especially when it goes towards release. They're busy spinning up new features or actually fixing things. And so we get a lot of things being deployed. And when we looked at what SDAX deploys within a single deployment or for a member network, then it's actually multiple deployments, databases, CDU jobs, everything together. So it's quite some kind of resources. And if you are able to just spin them up with a finger snip, then it's kind of like you end up with tons of things being spawned since it's so easy. But still, development happens mostly only during business hours. So maybe during eight to nine to 10, maybe a little bit more hours, but at least hopefully not at the weekend, which in the end means more or less your cloud is going to be idle during around 2 1⁄3 of the time. And when you look at how you consume cloud, you consume clouds based on the usage. So you pay for what you use. So if you don't want to use it, you just have resources idling around and this can make a cloud environment quite expensive, and especially if it's idling too far off the time. So let's make that environment also more elastic towards cloud perspective. Next slide, please. So when we look at how we need this scheduler works, we can use this behavior to actually then drive how things are getting scaled out. And when a new part is getting scheduled, the scheduler looks at the part that is being scheduled. It checks for its value. It checks whether the part is valid or not. And then it looks at all the available nodes and checks which one might fit. And then if there are some found, it prioritizes. And the Kubelet will start the part on the node. Now, next slide, please. When we look at an integration with a cloud provider, we actually get OpenShift tightly integrated with our provider API, which means that we are able to spawn machines just like pods. So as we spawn a machine, we get it. It joins automatically the cluster and it will serve workload. Now, with a pod, you actually also have replication controller which just watches whether your pods are still alive and whether the desired amount of replicas is still running. And kind of like parts of that, similar is the machine sets for machines. And they are kind of like the template that define in which the availability zone, in which network based on which image your machine should run. Next slide, please. Now, if we look at the feature of the autoscaler, we have something that watches how resources are getting scheduled. And when there are no available nodes being found, a pod stays in a pending state with not enough resources available. And so the cluster autoscaler then sees that and actually provisions a new node to the cluster. That node will join the cluster and the pod gets scheduled. And we can see that on that animation on the right-hand side where the pod is pending, machine got scheduled, got deployed, and the pod is part of the cloud. Now, there is the cluster autoscaler, which is the global configuration. And then we can actually schedule, we can actually configure the cluster autoscaler to configure per machine sets. And so you can have a different autoscaling functionality for GitLab CICD runners, and you can have another one for your regular work nodes. Next slide, please. Now, when we see the cluster autoscaler in action, and this is the view on one week, we actually can see that it automatically adds quite a bunch of nodes, then removes them again, and so on. And on the right-hand side, we have the flat line over the weekend where no development happens. So this is kind of like we can see that we kind of like scale out our used resources and kind of like one third of the cluster, it goes up and down. Next slide, please. Now, this ballooning and up and down and releasing resources means we're able to cut quite some costs. In that case, it's actually some 10K zero per month that we're able to cut things. And as I mentioned, the cluster autoscaler also compacts cluster and compacting clusters in OpenShift always means that workload is getting evicted. And evicting workload in OpenShift means that parts are getting killed and new replicas are spawned on another node. Now, on a busy cluster when things are being developed, this actually can lead to failures in terms of, let's assume you have an integration pipeline running, testing particular parts, and then they are suddenly evicted. So this might interrupt things. And part of that is you can actually annotate at the beginning of a CI run, you can annotate your parts that they are not safe to evict. And so the cluster autoscaler will not try to evict them and the nodes will stay. And then you can actually just remove that flag at the end and then the cluster autoscaler is able to release the resources. And now, since people usually don't clean up when they just spawn, we also quite heavily invested into downscaling the way the deployed replicas and so on so that the cluster automatically starts compacting. Now, ballooning up and down, usually throwing things away is quite fast. Provisioning new VMs still takes some time, still also in the cloud environment. And one of the things where we've seen issues there are CI runs and Radoo is now going to tell you about how this looks like now at DESTIX and how we solved it there that we get faster provisioning times. Thank you, Marcel. Next slide, please. So, as Marcel mentioned, provisioning of VMs takes in the cloud from five after 20 minutes. When running the CI CD pipeline, the developer must wait already some time until the image will be built and then to deploy to the cluster. Imagine that the cluster now is full and there's no place to accommodate resources anymore. And the developer will need to wait extra another five to 20 minutes for a new node provision in order for his workload to be deployed to the cluster. In many use cases, this is not acceptable. Next slide, please. So in order to mitigate this issue, we came up with the idea of having empty nodes or pseudo empty nodes that are always started. And basically, they are ready to take over workload instantly. How did we do that? So we basically leveraged on the Kubernetes concept of both priority. For those who don't know, let me make a short parenthesis. So each pod, when it's getting deployed to Kubernetes, it gets assigned a scheduling priority. If this is not specifically defined, the priority basically is inherited from the cluster default priority. Now, the important part here is that if a cluster is full, so it's out of available resources and there are some pods with priority higher that needs to be deployed to the cluster. But in the same time, there are no available resources anymore in the cluster, but there are pods with lower priority, then the scheduler will evict this pods with lower priority in order to make room instantly for the pods with higher priority. Let me close the parenthesis and let's explain how we designed this dummy pod. So basically this dummy pod, it's a pod that runs any image which must be non-terminating and has a lower priority than the cluster default priority. And it also consumes all the resources in one node. So in order to have one-to-one mapping with a node. Another important bit here is that we don't specifically define custom priorities for the rest of the workload in the cluster. So all of them will inherit the cluster default priority. Next slide, please. So let's see this in action. Let's imagine we have an OpenShift cluster with free worker nodes. Two of them are real worker nodes. So they are basically have pods with real application workload. We say we have application one, application two, application three, and application four. And let's also assume that these four application consumes all the resources on node A and node B. We have an additional node, node C, which is the so-called pseudo node and pseudo worker node, and which has scheduled this dummy pod which we were talking before. Now, we have a GitLab application, a CI CD pipeline, that wants to deploy application five. Next slide, please. However, as the cluster is now full and doesn't have available resources anymore, the scheduler will check if there are pods with lower priority in the cluster that can be evicted. And of course, the scheduler will realize that actually the dummy pod has a lower priority than our application five, and he starts evicting this pod. Next slide, please. Yeah, so this pod is getting evicted, it's terminating. Next slide, please. And after this pod is getting evicted, two actions will happen. First, the application five will be instantly scheduled to node C, and then the cluster autoscaler will detect that the dummy pod now doesn't have enough resources, the cluster is not able to offer the dummy pod enough resources in order to get scheduled. And the cluster autoscaler will start spawning up a new node in order to schedule this dummy pod. The important part here is that the waiting time for provisioning the node is now transferred from the real workloads or from application five to the dummy pod, which is acceptable. Next slide, please. Oh, how did we implement that? So first we must define a new priority class. This priority class we will assign to the dummy pod. As we said before, this priority class needs to have a value lower than the cluster default priority in order for this dummy pod to be evicted. Additionally, these priority class and the value of this priority class should be higher than the cluster autoscaler threshold because we want the cluster autoscaler to react when the dummy pod cannot be scheduled anymore in the cluster. Considering the default value for the pod priorities is zero and the default ratio for the cluster autoscaler is minus 10, which is a priority of minus five for our new priority class, like this. And next we need to create a deployment for our dummy pod. So important bits here are the following. So first we need to define the replicas. The replicas is basically the number of these pods, but from the other hand side, because one pod mass one to one spot node, it is basically the number of the spot we need for that specific node category. Then the priority class name needs to be the priority we defined before. The node selector needs to be the type of the node for which one to create the spot nodes and the resources must consist of the node allocatable resources. So in order to fit the pod to fit exactly in the capacity of one node. The image, as I said, can be anything but must be non-terminating image. Next slide please. So let's draw some conclusions. First, when we have a new workload, that needs to be scheduled to the cluster. If the scheduler detects that there are no available resources, it can still make room for these resources by evicting these dummy pods which have no priority. After the eviction, the cluster outer scaler will detect the unschedulable pods and will start a new node in order to schedule these dummy pods. This will also help when we encounter a situation like this in the future, we already have some spot nodes ready to be able to set workload. Now, when we consider that the cluster outer scaler is doing downscaling, so it's basically computing which nodes can be scaled down, we must also think that the dummy pod will fill out the capacity, the full capacity of a node and the pod that are getting rebalanced by the cluster outer scale will not have room to be scheduled on this spot node. And will be scheduled on another worker node. So the dummy pod will remain scheduled on the spot node until it will be eventually evicted by the scheduler when we'll encounter a scenario like we presented before. And that's it about the spot nodes. I will give the word to Solius to do the run up. Thank you, Radu. So just to finish a few takeaways. So outer scaling, as you can see, it works, but it requires proper configuration and some tuning to be done. GitLab turns it out to be a very flexible framework which allow us to configure building software for developers. And we found that container-based hybrid infrastructure suits very well for these kind of workers. So thanks a lot. Thanks for tuning in and see you in the next ones.