 Hello everybody and welcome to kubecon and cloud native con europe 2022. This is the kubernetes sig windows maintainer talk and today we're going to be giving some project updates and to talk about windows container users and also some updates on kube proxy next gen. Let's introduce ourselves. I will start my name is mark Rossetti. I am a software engineer at microsoft. There's some contact information for me too and I am the co-chair of sig windows. Next up is brandon. Hi, I'm brandon. I am a product manager at microsoft working on pretty much all things containers and kubernetes. Also have some aliases there and always feel free to feel free to take time and reach out to me. Hey, yeah, it's jay. I work with mark and claudio on the sig windows side as a lead on the sig windows project and I also do some things in sig network as well. I work even more. Hello, my name is claudio velo. I am from cloud solutions and I am working as a cloud engineer on kubernetes and container d and I am working together with mark Rossetti and jay vias as a tech lead for sig windows. Thank you. So here's an agenda of what we're going to be covering today. First up is some project updates. Start with improvements to gmsa or group managed service accounts. Since the last maintainer series update, here are some updates for gmsa. First one is what we're just calling v2, but that really is just a way to use gmsa identities in containers without requiring that the host be domain joined through the use of a container credential guard plugin. This makes it much easier to manage your windows nodes because you don't have to worry about joining them to the active directory domain before joining them to your kubernetes cluster. We have support for this in the admission web book that's part of the windows gmsa project linked here and support was added in version 0.3.0. We also have a reference plugin implementation available on github. This reference plugin is using an azure key vault to store the credentials for doing the domain off on behalf or in place of the having the node be part of the active directory domain. And then there's more information about how all of this works under the hood at the link provided below. The other big update is we now have a helm chart used that's used to deploy the all of the gmsa work that you can find in the same windows gmsa repository. Please check that out if you're interested. The next update is this pod os field. I think we've mentioned it at one of the last cube con talks. Now this enhancement has been promoted to beta, so it's on by default. A couple of notes here that this is not currently used for scheduling. So you still will want to apply node selectors, taints, tolerations, runtime classes, or however your cluster is steering windows containers to windows nodes to make sure that they do land on their correct node. Some of the big benefits of specifying the pod os field in your spec, there's some usability, some big usability improvements most notably if your windows containers accidentally land on linux nodes. Previously, you used to see some pretty, I'd say, convoluted error messages just saying that your image failed to unpack. And usually this is one of the first issues that people who are first experimenting with windows containers in a Kubernetes cluster run into. It's really not obvious what's going on there. But now you get this nice error that just says, you know, the pod failed because you meant to schedule a windows container, but it landed on a node of a different operating system. There are a lot of other improvements or functionality that happens at API validation too. If you're interested in that, check out the Kubernetes website for more information and full docs on that. Last update I wanted to give was on this operational readiness specification for windows nodes or what we're kind of, what we're calling windows conformance. Today, there's no good way to validate that your windows node is set up correctly when it's joined into a Kubernetes cluster. And this solves that. So what we've done here is we've wrote a bunch of tests that help validate the configuration of the windows nodes and bundled them up in a Sonobu plug-in to make it easy to run. This tests just things like making sure your network, like your network rules and everything are set up correctly, making sure your defender has the correct settings, all of that kind of node level configuration work that isn't tested by any of the Kubernetes IDE tests. There's some links in a video with more information here too. Next, I'm going to hand it over to Brendan and he's going to give a deep dive on how container users actually work in the windows operating system. Brendan? Thanks, Mark. Yeah. So one of the most common questions we get from the community is how user accounts work in Windows Server containers. So I'm going to spend some time today to provide an overview of how all this works. So to start, we need to first get a grasp on how Windows Server containers are built in the first place. Remember, Windows Server containers sandbox the entire user mode portion of the OS. Because they share the host kernel, they don't need to run through the kernel init process. On boot, after configuring the OB namespace, file system and registry, an entirely new instance of the session management subsystem or SMSS is created. SMSS manages the startup processes of all of the user sessions of Windows performing the same actions within the container as it would on the host. Next slide. So why do I mention this? Well, when talking about user accounts in containers, it's important to understand the isolation properties of containers they're running in. Unlike virtual machines that virtualize at the hardware level, containers virtualize access to various namespaces, and a namespace provides access to information, objects or resources via a name. The security of a Windows container is derived from the degree of sharing that occurs with its host. The security boundary of a container is constituted by the isolation mechanisms that separate the container from said host. And most importantly, these isolation mechanisms define which processes in the container can interact with the host. So looking at this chart here, we can see where on the isolation spectrum the different container types lie. While the design of Windows Server containers differ vastly from Linux containers, they offer similar degrees of security and isolation guarantees. So you can see on the left there, we have just regular processes on your system. They have direct access to the host and directly work with the scheduler and the OS and have access to all that information that are only limited by user accounts. Whereas on the other side, we have hardware-based virtualization, which totally separates a kernel, user space and everything else from the original host. Next slide. So Windows Server containers by default offer two user accounts, container user and container administrator, each with their own specific purpose. These user accounts are only visible to the container. The host does not have any awareness of the container user accounts as they're managed using a virtualized SAM database within the container. Container accounts can be created while running or during build time and be stored within the SAM database before deployment. So because each account exists purely within the container, the host is unaware of user accounts that exist within each container. The alternative is true as well. Containers have no visibility into the user accounts available on the host. This high degree of isolation is one of the benefits of being able to run an entire copy of the user mode OS in a dedicated space. The caveat here is that if the host ever shares resources with one or more containers, it's possible to hit conflicts between user and group SIGs. Mounting volumes with specific container user access rates is not possible under this model. All container accounts will be able to see what their well-known group membership allows them to see. So here we kind of have like what I would call a sphere of influence, where the container administrator doesn't have access to pretty much any of the resources on the host unless we explicitly allow it. And both, but the interesting thing here which kind of refers to the architecture and design of Windows versus Linux is both container administrator and administrator are members of the same administrative group because they're built in a way that they assume isolation from one another. So the administrator is part of the built-in administrators group that exists on the host. It's the same SIG as the built-in administrators of the container administrator, but it's all relevant to the scope in which they have access to. Go to the next slide. So let's go dive a little bit into each specific account. So the container administrator account enables you to use the container for administrative purposes, installing services and software such as enabling IAS within a container, making configuration changes and creating new accounts. These tasks are important for several IT scenarios that may be running in custom on-premise deployment environments. By default, container administrator does not have the ability to interact with the host. Any direct interaction must be first granted explicitly by an account of higher privilege on the host. Despite its name, container administrator has very few privileges enabled by default as we can see here. As the account does have several powerful privilege granted to it though. So this emphasizes the need to one, limit usage of the account to only trusted workloads, and two, ensure trust in the source of the container image. The container administrator account can install services and perform administrative functions within the container that persists across the build and deployment cycle. Even if you enforce a policy and production to run workloads under low privilege accounts, untrusted container images can use the container administrator account to install administrative services that can still run with high privileges on boot. Granted, the impact of these services will be limited to within the container workload unless access to the host was explicitly given. Next one. The container user account exists for all of the scenarios where administrative privileges in Windows are not needed. For example, in Kubernetes, if the user is container administrator and the host volumes are permitted to be mounted into the pod, then the user could mount the .ssh folder and add a public key. As container user, this example would not be possible. It is a restricted user account designed purely for applications which do not need to interact with the host. I strongly recommend that when you deploy a Windows Server container to any multi-tenant environment, that your application runs via this account. Container administrator and the container user operate on opposite sizes of scale when it comes to granted privileges. If an application needs something with more specific access, then a custom user account can be created with a set of specific privileges granted to it. Customer user account creation works just the same in a container as it does in the host. Only the account information is stored in the virtual registry and file system. By default, container administrator and container user are passwordless accounts. If you create a custom user with password credentials, these will need to be specified when starting the container. When you are using container administrator and you try to create a custom user while you are running the container or you do it from the Docker file or such, the changes you make, the accounts you create and everything will be stored at build time. You can store these custom user accounts within the container. When you want to run in the cloud, you can run with these specific accounts which have highly specific configurations and privileges that are tuned to the needs of your application. This gives you a lot of fine green control over what the container has access to and what the capabilities of the container are. It helps you restrict your security attack surface to the principle of least privilege. If you have any more questions, we are always available to talk about these things and dive deeper into your specific concerns. Now we will hand it off to Jay. Hey, everybody. This is a project originally created by Mikhail Klusso. We over at SIG Network and SIG Windows have found that it is solving a lot of potential long-term problems, both for Windows and for the broader SIG network community. The idea, my beautiful Myro board diagram over here, is that you have the kube proxy. It breaks the kube proxy into two separate pieces. One part that is like a brain that talks to the Kubernetes API. That is the kube to store thing on the left and the top. Then it can also sort of, and as it takes in information from the Kubernetes API, it can feed that over GRPC to a back end sync, which in the Windows scenario is a go-lang program that writes HNS load balancing rules. That is how the kernel space windows proxy works now, or a user space proxy that does NetSH rules and things like that, or whatever you want to do. The KPNG Windows work is now underway, and we have about 70% ported over to run in KPNG in this new architecture, so the Windows kernel space work. I have been hacking on that with a meme, NABN, and Doug Landgraf for a while. We have got it sort of compiling, and we have got the old code working inside of the KPNG framework as shown over here, and it is not fully working with the endpoints and everything yet. That is the process. Once we get kernel space windows proxy working in KPNG, we believe that we will have a new kube proxy that is totally modular that we can use for a lot of different use cases, and even entirely replace the kube proxy entry, which is the long-term goal of this project. Next slide, please. Yeah. If you wanted to test it, I have a blog post. You can go to my blog at the bottom, and you can just grab that. I will update it. A couple of things going a little lower level and looking at it from the Windows side. Right now, the way we test it and we run it is I am testing it without a CNI. Normally, your CNI, for example, if you are running Calico or entry or something like that, your CNI will go off and do some prep work to set up your Windows networking environment so that you can write kube proxy rules however need to be done. When it comes to looking on the right here, this is an example from Calico. Calico ultimately calls this HNS-PSM module, and that creates an HNS network, sort of manually as you can see in a firewall rule. After that is created, then the kube proxy can get to work and start using those network constructs to write its rules. One of the things we are talking about doing possibly is making the kube proxy for Windows maybe a little smarter so that you don't need to do any initial bootstrapping logic, and maybe it can just write those rules for you initially before startup so that it can just kind of run standalone without having to depend on any CNI or other external scripts to bootstrap it. That is an idea we are playing with. If folks have ideas, please reach out to us on that. Yeah, next slide please. Oh, yeah, that's I think you, Mark. Or Claudio? Claudio. Yeah, next we are going to talk about what's next for Windows and what we have installed for you in the next cycles. Right now, we are working into increasing the performance for HNS. There are certain scenarios to which it might take a bit of time to sync all the rules, especially have a lot of services deployed, and we want to improve the time required to deploy all the necessary rules and configurations required for HNS. Next, we are going to look into host process containers and how to improve its usability. First of all, we are going to add more options for run as users at the moment. You only have anti-authority local service, anti-authority system and anti-authority network service as available users that you can use for host process containers. Add more fine grained scopes for new potential users that you can use. Next, we will look into predictable volume mount paths. At the moment, you basically have to use an environment variable called container sandbox mount point, which is a bit less straightforward to use, which can hinder its usage. The idea is to potentially have a union of host OS and container file systems and to have the payload of the container in a special folder, but only visible to the container itself. CTL logs work. At the moment, if you actually want to inspect the logs from all your deployed nodes, you would actually have to SSH into those nodes and inspect the logs manually. This can be quite problematic, especially if you have a lot of deployed nodes, but instead the proposal is to use CTL logs to get all the logs from the nodes. Those can also be selected through labels or filters or services and so on. This is going to improve how users can debug potential issues whenever a pod or container affairs start on Windows nodes. In that case, you can simply get the necessary logs and inspect them and address the issues if there are any. Next, we are going to look into performance tests for Windows nodes. The purpose of this is to uncover any potential performance or reliability issues that could stem from the cluster being under pressure, for an extended period of time. The idea is to use Prometheus metrics under Windows node exporter to export a couple of metrics which will be useful to detect those issues. Some metrics like memory usage, no storage, open file handers, open processes and so on. Finally, one of the things that we are going to look into is file ACL improvements. At the moment, you could specify file permissions for volumes for Linux containers and Linux pods, which they are not currently applied for Windows containers. While the Windows ACLs are a bit more fine grained, we could potentially map the regular user group another to match the user seed, the group seed and the world seed, and also match the permissions themselves to generic read, generic write, generic execute. And that's what we have planned for now for the next cycle. Next slide, please. Next, we are going to give our special thanks to our community members and to new contributors. For David Schott, for all the help related to Windows network internals, Arvind Puttyaram Bill, sorry if I butcher the names, for all the help with the docs, bug triage, code contributions and more, Jamie Phillips for contributing for the GMSA webhook and including the web, the Helm charts. Ravi Gumi Gudimetla for code contributions and running our weekly CI signal triage meetings. Young Koon Gui for performance improvements to load balancers in Windows Q proxy. Danny Cantor for Windows Container D work and the host process container support, Scott Rosenberg and Tara Sky for work on multi OS cluster and Windows and cluster API providers. Doug Schilling, Michael Clousseau, Amin Naban for helping with Windows kernel proxy KPNG and to the new contributors, Shin Lee for improving for improving GMS state tests and operation readiness specification, Mario Soprin for his work on performance testing for Windows and Christian Glombach for work on the node log viewers enhancements. Next slide. If you want to learn how to contribute to Windows and see Windows in Kubernetes, please visit our Windows committee page, follow the contributor guide, join our weekly meetings, which is every Tuesday at 12 30 Eastern Eastern time and join our see Windows sessions, which are after the main meeting. You can also help us with additional documentation and specific user stories that you may have. You can file bugs and review open PRs on our project board. And we are always welcome to join new people and contributors. You can also drop by and say hello. Next slide. Next we have here the community resources. We can contact the see Windows chair or the technical leads, which is myself, Jay and James. You can also join us on Slack on see Windows. You can group our Google group and you can read the documentation for deploying Windows and having production environments for Windows. You can contribute to Kubernetes itself under the see Windows label and you can also check out the Kubernetes community see Windows repository. You can also check the YouTube playlist which we have all our see Windows meetings and you can join our weekly Zoom meeting at the following URL. Next slide. Now we will await questions. We have the Q&A session.