 Hello, everyone. Welcome to Qubeday. I am Ashik, and I lead the internal compute team at Cloud Era. Hello, everyone. Are you able to do it? OK. Hello, everyone. I'm Sibi. I'm a staff engineer at Cloud Era, working on a container compute cloud. What do you ask? OK. So we can all agree that we live in a container era, right? Everything is moving into microservices or almost already has moved. And containers are at the heart of it. And even Cloud Era transformed, right? OK, so why containers work for Cloud Era? As you can see, our product moved from a monolithic architecture to modular, adopting microservices and containers were the de facto choice. The compute platform, which provides infra for all the product development, testing, and release certifications, also followed course. And it simplified a lot of things for us, right, in terms of operations and all those. We saw better hardware utilization because of the less overhead. We saw optimized software packaging due to image layer reuse. Dev self-service improved because they had more control over the runtime environment. We also moved out of the dependency management cycle. So now let's take a look at the container cloud that I was talking about. At Cloud Era, we use Apache Yarn for resource management and scheduling. And Docker was a container runtime. We provide VM-like containers to engineers. We have a routable IP and a DNS name with access to access and external connectivity. Today, this platform is a core part of our product development and testing, and is almost adopted by most of the products across the company. Let's take a look at a few numbers for reference. Capacity-wise, we have around 700 bare metals, which give us around 220 TB of memory and 27,000 cores. And in terms of scale, at any given day, we will see around 2,000 clusters running with around over 8,000-plus containers with an average memory usage of 80%. And during peak release cycles, we also reach around 95% of average memory usage with around 3,000-plus clusters running. So just like Cloud Era, the container brought a massive shift from VMs to containers. So does this mean that virtual machines are going to die off and become a relic? So these are search volumes for VMs and containers from Google Trends. And we can see that from 2013-14, we have a steady growth in container adoption and its effect on the VM popularity. So the interesting fact is that even after eight years of container adoption, we see that there is still a considerable amount of workloads that run on VMs. And they have not. And why is it? Why did VMs not flatline in a container era? One is isolation. So any workload that needs strong environment and network isolation tend to choose VMs. The flexibility with the OS kernel and especially in places where they have to tweak the non-names-paced kernel parameters, et cetera, VMs are the go-to choices. Simplicity in terms of networking and storage is another strong feature of VMs. And we can see that why VMs are still a crucial part and cannot be discarded as such even in the container era. In our container cloud, we didn't support VMs. And the VM use cases were rising. For example, the last one that we had was around FIPS-enabled kernel for FedRAMP. Let's look at a few other things, the gaps that we had in our container cloud. We didn't have precision volume support for stateless application. Similarly, for the same ephemeral storage was hindering our host maintenance. And it was making it very difficult and time consuming, both for users and us. Apashayan, by design, deploys an application manager for each cluster. Although it consumes only a couple of GBs, at our scale, it used to come in order of TVs. And last, most of our control plane components were going EOL. And in case of any Docker upgrades or kernel version of grades, there were a lot of incompatibilities and things which we had to hack around. So with all this, we started our pursuit to find that silo bullet, which would solve all these issues for us. And that is where Kubernetes came into picture. Kubernetes showed promise. And it kind of covered all the basic aspects for us. It gave us good performance in terms of resource management. It could run at scale. It was highly available. It had support for containers by default. Qbert gave support for VMs, persistent workload support for stateful machine applications, loads and loads of plugins and controllers from choose from, and an exponentially growing community. But it did come with challenges for us in terms of our use cases. The major one was network. Our use cases need a stable IP address and external connectivity and SSH access and DNS. So we also need a stable name resolution for pods because users have to SSH and need connectivity to the VMs and containers. CubeSecondary DNS comes by default with Qbert, but only handles for VMs. So what to do with pods? That was an open question. Qbert uses something called data volumes for provisioning the root volume from image disks. We found that it is slow, and our pre-warmed images are around 15, 20 GB, and it was considerably slow for us. Live migration support, the Qbert VMs again supported live migration eviction policy, but there was nothing clear in terms of how to make it work with zero downtime. And multi-tenancy, this is one of the critical things. Apashayan helped us manage complex hierarchical quota management, and we needed something that would do the same for us in Kubernetes. And this is where our journey gets more interesting. And Sibi will now walk you through all the innovations and the workarounds that we did to get this working and migrate our production environment. Thank you. Thank you, Sibi. Green one. So we will now discuss how we address our platform and revamped and innovated by addressing the challenges. So this is a high-level architecture, and we use a vanilla Kubernetes flavor, and that comes with the default control planes. And on top of that default vanilla Kubernetes cluster, we introduce a Qbert HCO and then Apache Unicorn. And this Qbert HCO allows us to run the VM workloads and keep to stay the VM as a first-class citizen on the cluster. And Apache Unicorn is the powerful resource scheduler for managing the resources of VM and pods in the cluster. On top of that, we have a Cloudera infra control plane that includes Provisioning API. That is a layer where user interacts with our cluster to provision the VM and pods. And we have a quota manager, and that is our Cloudera CDP part of product. And that takes care of providing the hierarchical base of quota structure. And the DNS manager, it's a controller to manage the DNS for pods and VMs. And there is a workflow manager. And we built that over using the Sackstrom as a default workflow manager. And this is the high-level picture of how the VM and pods works together in that default Kubernetes architecture. So network architecture, external connectivity is the main deal breaker for us. Because by default, when you create a VM and pod, that will default attach to the Kubernetes internal network. And to address that problem, we introduce multiple CNIs part of this network stack. The first thing is the multis CNI. That is a CNI which takes care of attaching the secondary interface on the pods and VMs. And there is a bridge interface that is configured in all the nodes to connect all the multis CNI interface to that bridge. And there is a variable CNI that takes care of IP address management for pods and VMs. It's always ensured to provide the unique IP address for that requester resource. So DNS architecture, Cuba recommends to use the cube secondary DNS for managing DNS or name for VMs. But it doesn't have support for creating the DNS record for pods. So that is how the challenge is addressed. So we patch the upstream cube secondary DNS by forking that open source cube secondary DNS repo. And we added the pod support. And by default, it comes with the VMA controller. That is the one that coordinates whenever there is a VM request comes and creates the DNS record. And at the same time, the pod controller watch for the new pod creation. And it creates that records in the zone file. That is a persistent volume in the cluster. And between that, there is a zone manager which coordinates between the core DNS. And the core DNS will auto reload when there is a changes to the records. So provisioning time. So by default, when you use the Cuba recommended the provisioning way, it uses the data volume to import that image into the PVC. So for creating the few MBs, for example, one to 10 mehabits of image, it would take more than five minutes. To address that, we use the backing image support provided within the longhorn. And that allows us to create the storage class based on the OS image we request for. And then there will be a PV attached to that storage class. And ultimately, the provisioning takes only a few seconds to present the entire VM. So live migration. So ultimately, having the perfect live migration use case, it's better for the stateful applications. So we configured the eviction strategy in the live migration manifest. And that interacts with the persistent root disk. And also, that is configured with the ORWX mode in the VM side so that the live migration can happen between across the nodes. And the future side, we are still exploring how we can provide the dedicated network for segmenting the data traffic and optimizing the transmission of doing the live migration. So quota management. So having the cluster with managing both VM and parts, it should have the proper resource management. So for that, we are using the Cloudera's CDP product that takes care of providing the efficient quota manager and the Apache Unicorn for resource scheduling. So quota manager, it takes care of providing the API for a hierarchical quota. And Apache Unicorn is the powerful resource scheduler that provides the quota enforcement, and preempts, and priority, and access controllers, and more. Yeah, this quota manager and Unicorn both interacts together to perform the efficient quota. And our migration journey started towards migrating into this keyword-based platform. And probably we'll come up with the next talk about how we fit with the size and scale and performance numbers. We would like to thank our open source communities. These are all our key pillars to form this cluster. And without them, we will not be able to build this cluster for combination of VM and parts. Yeah, thank you, everyone. Thank you. Thank you, guys.