 Hello everybody, my name is Iliad Jan, working at CSCS, the Swiss National Supercomputing Centre in Lugano, Switzerland, and my talk today is about orchestrating Kubernetes clusters on HVC infrastructure. So first of all, some words about me, I'm 29 years old, I'm Swiss, and I'm assistant engineer about, since about 10 years, and working at CSCS since 2020. My hobbies are everything, what's motorcycle related and to travel, and if you want to contact me, here is my email. So just to introduce what we are doing at CSCS, we have this new HPC system, which is called Alps, and it's an HPE machine, Cray, EX, using Shasta architecture, and the LinkShot network to interconnect its nodes. It uses the infrastructure as code methodology to configure all the system, and it's running different workload types in what we call V clusters, so it's like having an abstraction of the cluster, including classic HPC jobs, so login nodes, submit jobs on the nodes, and also serving grid-like workloads for WCG, for example, or CTA, and so on. Regarding Kubernetes at CSCS, we are leveraging Rancher in order to provide the Kubernetes service at the centre, and we have three main scenarios, the bare metal one, the virtual one, and the Alps one. So the bare metal one is the scenario where we want to use commodity hardware, but running Kubernetes natively on it, so without an hypervisor or virtual machines, and we are using this, for example, for the storage element of the WCG, so it's running Kubernetes with the cache on it, or for our Elasticsearch cluster, for example, where we want to use local disk and have the maximum performance out of them. On the Alps side, we can deploy Kubernetes clusters on the HPC machine using Rancher as well, using K3S or RKE, and a benefit of the HPC infrastructure on that cluster. We are currently having some performance issues, for example, I will talk in the next slide. And the virtual scenario where we integrate Rancher with Harvester, also by SUSE, and VMware in order to spin up Kubernetes clusters on the fly. Everything, all the cluster is now moving to Selium. We started with Calico, but now we are moving to Selium CNI, and all the application and configuration is managed with Argo CD. So about the main challenges, we have diskless nodes, for example, on the HPC machine. So we had to use SAF as external storage to preserve the state of the clusters. This is something that we had to deal with. Regarding the performance and scaling, when we start having many nodes, we saw also a new G impact on the storage, and having SAF to cope with high load is not so easy. Regarding the management, Rancher really helped us, because having this overview and manage the deployments and the upgrades from a central point was very useful. All of this and bringing Kubernetes on top of an already pretty complex system like the HP EX machine using Shasta is pretty complex to manage. So these are the challenges. We are, as I said, using Argo CD with the app of apps to manage all the applications. And in fact, we have a single top level application called controller, which then configures Argo CD itself and then deploys the underlying application on the downstream clusters. And then forcing the state as well. In conclusion, we are building Alps going with the aim of using cloud-native technologies. And we are also waiting, and we just upgraded the LinkShot network to the version 2 in order to leverage the VLANs as well on the Alps machine. And Rancher, Argo CD, VMware, and Harvester are the tools that really helped us build this Kubernetes infrastructure in an easy and manageable way. And it's very fun to work with that. As you can see, it's a work in progress. Nothing is written in stone. We are still learning. We are still exploring. We are still finding new technologies, and we are continually evolving. So this was my presentation. I hope it was useful. And I'm around for any question. Thank you. Any question? I'll jump in and ask one. You mentioned this class nodes. Is that something that works out of the box for Kubernetes? Or did you have to do something special? Actually, we started mounting in-memory to store, for example, the HCD database or so on. But then at node reboots, you lose anything. So we wanted to have also the persistence of the cluster when we reboot the nodes. And so we had to mount, for example, self-RBD volumes coming from an external storage in order to have this persistence. The nodes themselves have no disk at all? Exactly. And the bootstrap of the nodes to run the kubelet and have some sort of configuration? This is done via Rancher using the custom cluster. So you actually get an URL that you can download the bootstrap script. And it's bootstrapping the Kubernetes cluster using Rancher with this script. Thank you. Could you give us some numbers about the scale you're running at? What challenges, if any, you're facing with scheduling or queuing? We are seeing this impact when running the WCG workloads, where actually we are also not really aware of what we are going to get as jobs. And the underlying storage is SF. We have around now 100 nodes of Alps dedicated to WCG. And when there is a high load on the storage backend, here is where we saw the problems. So we are actually exploring new storage backends for this. This is the real point. Any other question? We have time for one more, maybe? Thank you. Thank you.