 Bonjour, everyone. Welcome to our session, and in this session, we are going to talk about the CNCF Commada project. I'm Hongkai from Huawei, and this is Xiao from Dogcloud. We are Commada maintainers. Actually, we have another speaker, but she didn't make it this So, in this session, since this is our first time present in Europe, so we are going to talk about the background of this project and the general concept of Commada and the core features and also the Commada community. So, let's get started, Xiao. Hello, everyone. It's a great honor to hear, share Commada in parents, the roles and the art of city. This is our agenda. So, let's begin. Why we need multi-cluster? Okay, from the 2023 lecture report, more than 87 of users are adopting their multi-cloud strategies, including hybrid public cloud and the private cloud. As cloud-related materials and the error are programmable, multi-cloud management services are coming. Just as Roma was not built in the day, the development of cloud-related multi-cloud is not achieved over one night. We are going through different stages and facing different challenges. In the beginning, no service interactive, isolated data and resource and traffic just like an isolated island. As cluster increase, management costs rise, driving demand unified for delivery, access and occur structure and scheduling. We are called the stage when it's water city. Here we are. With AI becomes popular in the corporate cloud in Europe, as we all know. As we learn, corporate infrastructure is increasing. There are more and more large-scale cluster. The final stage weighs maybe automatic scheduling, flexible scaling and free migration. We call this stage, garden set, age of sale. The future is here, so we need to catch this. Of course, there are many challenges in managing multi-cloud container clusters, such as numerous clusters, schedule service and restriction clusters. Of course, the vendor looking is also we should care about. The corporate community has never explored multi-cloud solutions. In 2015, the Federation SEG was established, there is no white paper. Federation V1 started in 2016, becoming a corporate sub-project in 2017. Due to limitation, like Federation information in an annotation of corporate results, which has a poor usability and has low independent version controller. Federation V2 in 2018, and the 2020 multi-cluster release of top service APIs in 2021, Commander was open sourced. It's developed in continuation of Kubernetes Federation V1 and V2. Some basic concepts are inherited from these two versions. This is the reason why people say Commander is Federation V3. Next, we will be sharing Commander design principles, architecture and the core features in detail. So, what is Commander? The corporate is Amanda. Commander is designed to train scalable container results pool, enabling and the user to multi-cloud as if using a single corporate cluster. Users can use the application from a single cluster to multi-cluster without any refactoring. And the two chains previously used in corporate strategies without any rules. These are most important design principles. Then, Commander will get out of the books ability and the capabilities and many building multi-cluster scenarios such as the active-active and cross-region and the disaster remote and so on. Also, Commander will support free operation between multi-cloud vendors and already the corporate network API thereby achieving no vendor locking and the central rest management. As a last, Commander has a built-in scheduling policies based on industry scenarios. So, okay, let's get a start overview of Commander. Commander also has a control plan deployed on corporate latest cluster which have Commander API server and the scheduling layer and the control manager just like the corporate latest with this components cooperation. Users can create resources and complete the construction of multi-cluster application through the native corporate latest API just with said first. At the same time, some clusters has limited network access. Commander supports push mode and deployed Commander agents in corresponding clusters. Based on this architecture, Commander also supports many features just like the cross-cluster application failover, multi-cluster service discovery and so on which we will share in detail later. Before deeply learn more about Commander, let's first understand the core concept of Commander. We need focus on this resource. Resource tablet just corporate latest native resource deployment demo set and the any other native resource and the propagation policy. The widely applicable policy for multi-cluster application scheduling configure which cluster their application should be propagated to and the override policy different cluster has the different configurations just like the image URL and the other configurations we also supported. At the same time, resource bonding and the work are internal resource of Commander and the users do not pay attention to them. Now let's look at the work flow of Commander. Let's start with the magic jewelry from deployment of NGX. Let's go. We have corporate latest native resource just NGX deployment. At the same time, we can declare or propagate policy to associate this deployment through label selector, their order doesn't matter. The Commander schedule will select the most appreciate cluster for the application based on propagated policy, create all resource bonding and complete the assessment, the resource and the cluster. At the same time, in actual scenarios, deployment have different configuration in different clusters just like the image URL. Therefore, Commander defends override policy to handle this situation. Finally, Commander controller will create a complete native application in the real cluster bound by deployment completing the entire work flow. The creation order of propagation policy and the Kubernetes native resource is not related and depends on the users through practice. One is that the cluster administrator manages and plans the scheduling strategies of the application in other ones, such as which cluster to schedule and what strategies. At the same time, it will create a propagated policy in other ones. In this example, the administrator wants NGX deployment to automatically deploy to three clusters besides distributed in two drones to achieve the application high ability. Deployers can just focus on deployment's native application as they normally would. Using the NGX access tools just like Kubernetes ETL and they can't go, they don't need to worry about whether it's a multi cluster or a single cluster. Let's learn more about the propagation policy. The propagation policy associated with Kubernetes native resource or the user customer resource through the label selector and at the same time, propagation policy provides rich multi cluster scheduling policies, just like the cluster affinity and the cluster tolerations and the dynamic weights and so on. Commander also supports the different overhead settings, just like the plan test, imagine overhead and the commander overhead, just as they say, we support different configurations in different clusters. And the last resource is the commander cluster, describes the cluster management by commander. The cluster support pull and push network mode depends on the user practice. The secret reference is used to include the certifications to commander connected clusters as well as the other configurations. So, by law, we should have a basic idea of what commander is all about. Next, welcome to Hongcai to share with the core concepts of commander. This is the most important fascinating part. Thank you, Xiao. Commander aims to provide the tinkering automation for multi cluster application management in multi-cloud and hybrid cloud scenarios. The key features are including cluster management and cross cluster scheduling and multi cluster service and so on. I will give a general introduction for these features. First feature is since commander are going to manage application across clusters, so the first feature is how commander manage a member cluster. So it is pretty painful for people to manage so many people and so many clusters and do a lot of repeat operations on that. So commander can control the cluster of various cloud service such as GKE and AKS, even private clusters. Additionally, commander can offer different registry modes that push and pull to meet the network situations. The push mode involves direct communication from commander to the member cluster and the pull mode delegates the management to a component named commander agent. The commander agent is usually deployed in the member cluster. Since commander accepts the component API, it is pretty easy for people to migrate the ecosystem to multi cluster. With the property policy, people can describe their scheduling rules on that in it and commander together will distribute your application across cluster based on the rule user described in the property policy. Additionally, as the results manifest may vary different in member clusters. So commander also provides the override policy to implement the different configurations including the registry and the different labels, different acts. So you can use the property policy to replicate application to model clusters or you can divide an application to multiple cluster. Commander manages multiple member cluster in the event that one cluster goes down. Commander can gracefully migrate your application from one cluster to another, ensuring that your business remains available and uninterrupted. Additionally, if you want to proactively migrate the workload from a cluster, you can such as time a cluster and trigger the migration. Beside of a cluster, an application can potentially fail. In such case, commander can automatically migrate the faulty application to another cluster. User can control the migration logic. User can control the migration process, including define application failures and specifying the migration behavior. Commander uses the proxy request to access the member cluster and provides a unified aggregated APS server to enable unified authentication and cross-cluster ONM. With this feature, you don't need to switch your cool config fare to access the member cluster. All you need is the commander's config fare. Another feature is that commander can cache all member cluster results by supporting third-party store and even local storage and provide a unified query. You can query all member cluster results in a unified manner and efficiently reducing the access pressure on member cluster. Commander can take open search or electric search at the third-party storage. With the power of electric search, you can build a one-fold resource dashboard across the cluster. There are many reasons why commander's user want to split their deployments across the cluster, but still remain material dependencies between workloads running in those clusters. The commander cluster is a hard boundary. A deployment cannot access a service from another cluster. The commander multicloud service which extends the service concepts across multi-clusters. Service can be consumed from, can be consumed everywhere as long as the cluster is managed by commander. For example, a client in cluster 2 can consume a service in cluster 1. Even there is no service back in the cluster 2. Okay, that's the commander call features. Next one I'm going to talk about the commander community. Commander was open source at 2021 and accept by CINSEF as a sandbox project also in 2021. We just moved levels to incubation at the last year. Commander joined iFORCE more than 500 contributors and built a fast-growing community. For now, there are 20, about 30 public adopters using commander in their production environment. There are six vendor successfully leverages the commander to build their service on public cloud or private cloud. One of the capabilities of a commander is that it accepts Kubernetes API, which allows commander to easily integrate with tools from the Kubernetes ecosystem. The community has tested ACOCD, Flux, Kavanaugh, and so on. They all work well with commander. Here is the public adopters. Now we have about 30 adopters. Commander community builds the connection with them and listen to their voice and provide support when they need. They also contribute back to the community. That's the way a community works. Commander community is always looking forward to hear feedback from users. If you want to use commander, please feel free to talk to us. Every year, the community will refresh the roadmap, but what's to note that this item might be subject to change over time and depend on how many feedback we get. Basically, we will focus on provide support for AI training area and building a multi-cloud workflow. Welcome to talk with us about any feature you want to have. We are always glad to help. Welcome to ask questions. You can find us from the GitHub issue, Slack, and so on. That's all of our presentation. Thank you very much. We still have some time. We are going to have one or two questions. Okay. Is it only for stateless applications or stateful applications as well? If it is, how do you handle volume migrations? Your question is how to manage the stateful application, right? For now, commander can schedule all kinds of applications across the cluster, but for now, for the migration across the cluster, there's a problem with the stateful application because you have to migrate the application and the problem is you have to migrate the data. That's the problem. We are working on it. Okay. Thank you. Hi. Thank you for the presentation and how granular is control over objects. Let's say we have a core DNS and one of the, I don't know, 50 clusters has some issues. The configurations are stored in config map, for example. Could we, for example, pose a propagation to one specific cluster for one specific object in kubectl, just disabling the label and then our administrators are, I don't know, changing the configs to see what is happening and why we have an issue for this one specific cluster or do we have to disable it, I don't know, by label or for the whole easy or can we just pose for this specific cluster and specific object? Thank you. Hello. You mean public application by level selector to 15 clusters and what's the problem? Yeah. So we have an issue with one cluster and it's a core DNS that gets config from config map and we want to pose a propagation for that specific config map. So for example, for two weeks that cluster, this one object is not synchronized with our general deployments and changes that we are pushing to the project. Is it possible in Karmada? Yeah. Karmada, the propagation policy can handle the situation. If cluster is down, you mean? Cluster is not down, but we just want to experiment with configurations for this specific cluster. You mean the cluster configuration already changed, Karmada can refresh the cluster in sub-cluster? Yeah. It's automatic. No, it does not change that specific cluster. That specific object in that specific cluster. Yeah, that's a question. If it's a specific override policy or can I just add a label to one object and Karmada will not synchronize all the changes to it? We can talk further after this session. Thank you. Another question? How does Karmada support propagating HPA and KIDA auto-scaling configurations? Actually, Karmada can manage the Kubernetes HPA resources, but we still have advanced HPA. That's the name of the federated HPA. The HPA will work on the Karmada control plan, and it can scale the workload across cluster. That is the future, I think. And do you support KIDA? We don't have an integration with them, but we will do. Yeah, thank you. Okay, for the time we can talk about it after the session. Okay? Okay. Thank you for the presentation. My question will be related with cross cluster failover and the migration. So the service will remain live until the migration is happening. For example, one pod goes down and then this pod appears in another cluster. So the service will remain seamless, like there will be kind of like a live migration or... I'm sorry, the service will... Will be live at that time. So for example, if your application... Remain live, right? Yeah, yeah. So is it similar to live migration or how does this work? Yes, service might... For example, application might have several replicas, and some of them will be done. So in this case, Karmada might be migrating the application, but during the migration process, the rest replicas will be alive. And after the application totally launched in another cluster, the previous replicas will be destroyed. Okay, so in case if the service is available in only one cluster and the service goes down, so if the migration happens, so it will start from the same place? No, no. I think if the scattered rules describe that the service only can live in one cluster, so the cluster failure will not happen. Okay, yeah. Thank you. Okay, thank you for your presentation. I have one question related to the authentication flow which you have shown in the presentation. You said that currently the problem is that you have to authenticate to the each cluster, etc., and with the Karmada control plane, you can use that to authenticate to the multiple cluster. So a question for me from my side is that is it possible to manage the permissions for the different clusters to the different users? So for an example, if we are building a platform and we want to provide that platform and use the Karmada to use that for doing the authentication and authorization to the individual cluster to specific members, so is it possible with that that we put this Karmada on top of the platform and then this is working in a way that, okay, you belong to tenant A and you have access to cluster A and B and you belong to tenant B and you have access to cluster C and D, etc. So is this possible there or not? Actually, it's a good question. Many users have different permissions in different clusters. The Karmada open source project has not supported this situation. Okay. Maybe you can check the Karmada windows. The enterprise product maybe covers the permission and the authorization or authentication to handle the permission difference with different cluster, different users. Yeah. Okay. Thank you. Okay, thank you. Okay. Very good presentation. Is there any limitation with the maximum latency that Karmada supports to the clusters or the maximum number of clusters that Karmada can manage? You mean the Karmada can manage how many clusters? Yeah, the maximum number of clusters that Karmada can manage. Yeah. The community have run a test at 2022. Yeah. We have run against 100 huge cluster, 100 cluster, but I think that we didn't run more clusters because we have a limit of the test results, but we have feedback from our users. They are using Karmada to manage more than 100 cluster. Yeah. That's totally a big cluster, you know, but more than 2,000 nodes in one cluster. Okay. I think you can find the test report on the Karmada website. Okay. Thank you. Okay. Thank you. Thank you.