 Hello everyone, I'm Kevin Wang, leader of the Huawei Cloud Native open-source team. So glad to be here to share with you. I'll start with introducing our speakers. For me, I focus on developing open-source projects and maintaining the community. I'm a CNCF ambassador and the co-founder of several CNCF projects, including Kamada. Next, Shen Yifan will join me to talk about the multi-cluster management using Kamada. Okay, thanks Kevin. I'm Shen Yifan from the Software Development Center of the Industrial and Commercial Bank of China, or ICBC. I work on the design and R&D of the ICBC Container Cloud Platform, and I'm also the co-founder of the Kamada project. First, let me start from the construction of ICBC's cloud platform. ICBC cloud services support the running of many applications, including online activities on holidays, core service applications, MySQL, Redis, and other technical support applications, and the ones in blockchain, AI, and some new technical fields. ICBC has been developing a customized cloud platform based on mainstream open-source projects, ensuring overall independence and controllability. It is also the largest container cloud in the industry with more than 280,000 containers deployed. ICBC services run with specific requirements and on a large-scale cloud-native infrastructure. There are four typical requirements, namely high availability or HA deployment, cross-cluster autoscaling, cross-cluster scheduling, and a specific dependency on Kubernetes versions. Here's how ICBC's cloud-native infrastructure is running now. First, high requirements on the reliability of a single cluster. The number of nodes in a single cluster is less than 2,000 to reduce the fault domain of a cluster. Second, rapid growing of resource pools as services are migrated to the cloud. New and core applications have been developed or running in the cloud and existing applications are being migrated. Third, production-level heterogeneous clusters. ICBC services depend on specific Kubernetes versions and we are using a large number of heterogeneous CNIs, CSIs, and some heterogeneous underlying hardware. Fourth, our banking services are distributed in multiple regions, centers, and clouds. ICBC cloud services run in the headquarter branch and ecosystem clouds. In terms of fault domains, geo-redundancy data centers are constructed and there is a finer-grained division of multiple fault domains in each data center. For now, the number of Kubernetes clusters in ICBC has reached more than 100. They are managed by the container cloud platform in a unified manner. However, we also face the following four problems. First, limited availability. A Kubernetes cluster is also a fault domain, so automatic recovery across fault domains is required. Second, limited resources. Only application scheduling and autoscaling are available only to single clusters. Third, non-transparent clusters. With heterogeneous resources, fault domains, and other attributes configured for clusters, the service team has to distinguish all the underlying clusters to find out the target cluster where they want to run services. Upper-layer applications are not aware of cluster differences. Fourth, duplicate configuration. Although our services are configured on the cloud management platform in a unified manner, the specific configuration needs to be delivered to each cluster, which must be synchronized. To address these challenges, we formulate the following objectives for multi-cluster management. For multi-cluster management, we want to manage all clusters in one place and their entire lifecycle and use unified standard APIs. For resource management, we need to support multi-version and comprehensive Kubernetes resources and multidimensional resource override. For cross-cluster autoscaling, it should be realized based on the fault domain and resource margin, partnering with cross-cluster autoscaling. For disaster recovery, cross-cluster resources should be able to auto-recover and the control plane and service clusters should be decoupled. For compatibility, smooth management is required for a large number of existing heterogeneous clusters and the open-source software we use should be highly scalable and adopted by a large user base. With these objectives, let's think about how to achieve them. First, commercial products bring in vendor lock-in, which do not meet our requirements on independence and controllability, so pass. Other open-source software can be a consideration, such as KubaFed. According to the survey on KubaFed, it uses non-native Kubernetes APIs, which makes it difficult to migrate existing clusters. In addition, its community is becoming less active. It doesn't become the de facto implementation standard in the industry. So we decided to develop Canada to satisfy our needs. As we are building the ICBC financial cloud based on open-source software, we hope to closely work with members from the community and make more contributions to boost the open-source development. Some of you may wonder, why don't we choose to go further with KubaFed, but initiate a new project? Actually, we did develop the Kubernetes Federation version 3 at the early stage and quickly completed the prototype development. However, after communicating with many community users, we found that the Federation itself cannot fully satisfy what's expected. In addition to multi-cluster workload management, we wanted that Federation has capabilities such as resource scheduling, fault migration, autoscaling, service discovery, data automation, and multi-platform cluster lifecycle management to provide users with ready-to-use open-source software for multi-cluster, multi-cloud applications. Therefore, a neutral open-source project hosted by CNCF will be more suitable for long-term technology evolution and community development. In terms of the core architecture development of Kamada, we have learned from the experience of multiple core sponsors in multi-cloud and multi-cluster management. We focus on the native API support of Kubernetes and the scalability of the core architecture. The architecture of Kamada is similar to that of the Kubernetes single cluster in many ways. Both of them have a control plane, an API server, a scheduler, and a group of controllers. The underlying layer manages single Kubernetes clusters. In Kubernetes, the underlying layer manages single nodes. The API server of Kamada provides Kubernetes native APIs and policy APIs extended by Kamada. Kamada scheduler focuses on fault domains, cluster resources, Kubernetes versions, and add-ons enabled in the cluster to implement multi-dimensional, multi-weight, and multi-cluster scheduling policies. In addition, the scheduler framework has a pluggable design so that users can customize and extend scheduling policies. In terms of member cluster synchronization, Kamada implements agent-based pull mode. In this way, the working pressure on the control plane can be effectively reduced and users can easily manage large-scale multi-cluster resource pools. As for Kubernetes clusters in various network environments, such as public clouds, private clouds, private networks, and edge networks, Kamada also supports centralized API calling, such as Execution Controller or Kubernetes Integration, to implement network penetration and direct management of Kubernetes clusters in these different networks. To allow multiple clusters to run securely, Kamada has an execution space to isolate the access permissions and resource views across multiple clusters. Here are the core concepts in Kamada. All workload-related resource objects specified by users are defined in a resource template. They are exactly the same as native Kubernetes APIs, including deployment, service, and config map, and CID objects that are enabled by operators. In this way, users can use the YAML or APIs in the original single cluster to create multi-cluster applications without any modification. The service platforms developed based on Kubernetes APIs do not need to be modified either. They can be directly transformed from the single cluster architecture to the multi-cloud and multi-cluster architecture by interconnecting with Kamada. For cross-cluster deployment models and scheduling, Kamada provides independent propagation policies defined using a policy API that can be reused across applications. For example, ICBC has its independent platform team and service team. This model can well adapt to their organizational structures. The platform teams can set common policies for typical application deployment models, such as high availability model for applications with multiple fault domains. The service team can still use the Kubernetes native single cluster APIs to manage their daily service rollout and version upgrade. Let me use an example to illustrate how you can use Kamada to manage your services. This is a propagation policy. In the definition of this policy, the platform team sets a resource selector to restrict all deployments. If an application has a special label and its HA mode is multi-zone replication, the applications are strictly propagated to the three zones. The YAML manifest on the right is the API definition of a standard deployment. From the two definitions, we can see that the platform team focuses on the settings of the common application deployment model and the service team focuses on the definitions of images, versions, and container pods in the applications. Kamada combines the requirements of the two teams to achieve cross-region HA of applications. If any underlying cluster is faulty, the missing clusters or pods in the availability zone can be automatically supplemented with the dynamic scheduling capability to implement cross-culture fault migration. During Kamada's positive cycle consisting of design, R&D, practices, redesign, and R&D, ICBC has summarized some of its advantages and lessons learned in the following four aspects, resource scheduling, disaster recovery, cluster management, and resource management. I think the following three points deserve special attention and are especially prominent in the actual implementation process. The first is the binding and scheduling of multiple types of resources which can ensure that Kubernetes resources required by service nodes can be scheduled at the same time. This greatly improves the real-time resource provisioning. The second is the support for Kubernetes native objects which can ensure that a large number of Kubernetes external clients do not need to be reconstructed. The third is the support of the Kamada for distribution in pull and push modes which adapt to multiple scenarios, especially in the scenario with a large number of clusters, the pull node can greatly reduce the performance pressure on the Kamada control plane. After talking about the current status of our project, let's take a look at the follow-up plans. In terms of large-scale production, we hope that the container cloud platform will serve as a user-oriented platform and the underlying layer will manage and schedule resources of multiple clusters in a unified manner based on Kamada. In this way, more than 100 existing Kubernetes clusters, including heterogeneous clusters, can be managed. We also hope to make continuous contributions to the community. We mainly focus on the feature which includes a smooth migration of existing applications and can be automatically incorporated into Kamada for Federation. Second, we hope to continuously optimize and implement cross-cluster scaling, application migration, and data linkage. Of course, the Kamada project also has many other exciting functions. Welcome to the GitHub of the Kamada project to view our release notes and community documents to learn more about the functions and details. If you have any suggestions or feedback when using Kamada, please join our WeChat group and Slack channel to reach out to us. That's all our sharing for today. Thank you for your time.