 Hi, folks. Welcome to my presentation. The topic we are going to talk about today is best practice on Windows Worknode management in Kubernetes. We will share some experience related to Windows clusters in product environments. There are two speakers, including Benjamin Wang, Izumi, and Wenli Wei. We are both coming from VMware. I mainly focus on Kubernetes and CSI related work. And Wenli is focused on Kubernetes Windows related technologies. Wenli is busy with customer issues recently, so I will deliver this presentation myself alone. There are three items on our agenda. Firstly, is the background. And it's mainly about the issues from our customers' product environment. We summarize some experience and lessons based on the issues. That's exactly what we are going to share today. It includes some operation tips for cluster lifecycle management and how to integrate with the group management service account. First, let's talk about the background. We received some issues from our customers in the product environment. For example, the C drive may run out of space when there are a lot of workloads. Our PVE couldn't be attached to a new workload during rolling updates. Sometimes we see some updates after moving a cluster. It's also challenging to manage credential on Windows platform. And there are some other issues as well. We will share the experience on how it resolved all these issues or concerns. The first tip is about the eformat disk. Each container on Windows platform has its own scratch space. And actually a virtual eformat disk is created for each container. The C drive is 20 GB by default. Usually the root directory for container runtime locates on the C drive by default. So if there are a lot of parts running on a Windows node, then the C drive may run out of space. The seasonal files are in the C drive as well. So the C drive is important for the Windows OS to operate correctly. So we should avoid running out of space on the C drive. The best practice is to config the container runtime's root directory on a larger drive, such as eDrive. In this example, the C slash y slash v cap slash data is linked to another larger drive, such as eDrive. The target drive may change, but the soft link will always be the same path. Sometimes users may need to update the cluster. It's important to make sure that there is zero downtown on the work node. So we should perform a rolling update, which means to update the work node one by one. But if there are a lot of work nodes, it may take a long time to finish the rolling update. For example, if there are 50 work nodes in the clusters, it needs about three minutes to update each work node. Then you need about 150 minutes in total to finish updating the cluster, which is two and a half hours, is a long time. The solution is to let the user to set the maximum number of work VMs, which can be created or updated concurrently. We call it as maximum flight. In this example, we set the maximum flight as five, so it needs about three minutes to finish updating the cluster. All of this saves a lot of time. In the meanwhile, you can also meet the zero downtown requirements. The next operative is also for the rolling updates. When updating cluster, we need to trim the windows work node one by one. We assume the maximum flight is one for simplicity here. The rough workflow is described in the diagram on the right side. The first step is to come down the work node, so as to mark the nodes as unschedulable. Any new part will not be scheduled on this work node anymore. Secondly, to trim the work node, so as to evict all the existing parts, which are running on this windows work node, please note that we need to ignore the demon set, because the demon set cannot be dreamed. Usually, after the step two, all the parts on the nodes should have already been evicted, but somehow some parts may still be running for whatever reasons. In this situation, we need to firstly care for such parts, and this is exactly the third step two. Next step is to watch disk until all block volumes are detached from this work node. It's important, you know, if a block volume is not detached from this work node, then it cannot be attached to another work node. So the part assuming the PV is not healthy. Please note that we need to ignore the informatics created for the container, because we should only care about the block volume for the PV. The last step is to remove the node from the cluster. Okay, the next tip is about deleting clusters. Sometimes users may want to delete the cluster. We need to remove the PVs beforehand, otherwise it may result in often disks. If the reclaim policy is retained, we need to remove the PVs as well. Previously, we received some cases related to this one. You know, some users complained that they still pay for the disks, which are not used anymore. So actually, they are often disks. The next topic is about the GMSA. GMSA means group management service accounts. It could automatically manage a password and simplify service principle. It's widely used in Windows application. From Kubernetes 1.16, Docker Rom time supports GMSA for Windows working out, and it's stable in Kubernetes 1.18. In Kubernetes, GMSA credential spec are configured as customer resource. These are background information for the GMSA, and you are recommended to read the official guide on Microsoft to get more and more information on the GMSA. There are some preparations in order to use the GMSA. Firstly, you'll need to set up a Windows Active Directory server and create the KDAID root key using the domain admin account. You also need to create a script group and get the GMSA credential associated with the security groups. This page is the highlight of the diagram, highlight of workflows. The left diagram shows how the GMSA works for the Windows cluster. Firstly, each Windows working out needs to be added to the Active Directory, and then it also needs to be added to the security group, and credential spec needs to be created on each Windows working out. The left diagram shows how to integrate the GMSA into Kubernetes. Firstly, install the credential spec CRD, and secondly, install the webhook, including the mutating webhook and the ventilating webhook. The mutating webhook is used to expand reference to the GMSA into the full credential spec in JSON format within the pod spec. The ventilating webhook is used to ensure all reference to the GMSA are authorized to be used by the pod service account. Next, we just need to follow the standard RBAC pattern, create the cluster row and config the permission on the GMSA credential spec, and then create a service account and assign the row to the service account using a cluster row button. Lastly, we need to set the service account in the pod and set the credential spec name in the security context to the GMSA credential spec reference in pod spec. Afterwards, all pod running on this Windows working out could join the AD and the security group automatically. This page is just the detailed description on the diagrams on previous page, so I'm not going to repeat it. That's all for today's sharing. Please let us know if you have any questions. Thank you.