 Hello everybody and welcome to the Sig Windows maintainers talk and welcome to Amsterdam for kubecon and co-nativecon Europe 2023. I'm going to start with some introductions. My name is Mark Rossetti. I am the co-chair of Sig Windows. I'm a software engineer who works at Microsoft and I've been contributing to Kubernetes since about 2018. Hi myself Pramita Gautam and I'm from India, and I'm a staff engineer there. I've been a contributor to Sig Windows as well. Thank you. Hello my name is Claudio Bello. I am one of the senior cloud engineers at Cloudby Solutions. I've been working on Kubernetes for a couple of years and I am one of the tech leads for Sig Windows. And unfortunately JVAS, another tech lead from Sig Windows, wasn't able to make it today. Here's a little bit of an overview of what we're going to talk about. This is I think mainly for people who are just looking at the slides. So first we're going to give a little bit of an update on the enhancements that are going on in the Kubernetes project. For anybody unfamiliar with the enhancements process in Kubernetes, usually larger or especially user-facing changes need to go through this Kubernetes enhancement proposal project where they get reviewed by the appropriate folks and then kind of implemented through different stages, alpha, beta and then stable. So this is kind of what Windows has been working on for a little bit are the in-flight ones that we have. The first one is an enhancement to be able to specify the size of the root FS volume for your Windows containers. So today I think a lot of users know that that size of that root FS volume is kind of locked at 20 gigabytes and a lot of people have been asking for a way to specify that especially to get more space for this. So we're working on that and should have an alpha implementation of that in the next release. One of the next enhancements is this node log query enhancement. We've demoed it I think at the KubeCon Detroit. That one has kind of been in-flight for a long time and it's finally merged and ready for testing in alpha in the 127 release, which released last week. And all of these are short links. If you're interested, this is the full, kept right up the Kubernetes enhancement proposal to go and get more information to see how to interact with it, how to turn it on if it's not stable, etc. So next is this C-advisorless CRI container and pod stats. Yep. So there's this effort in SIG node to really defer a lot of the work that it's doing, especially around the container runtime to the different container runtimes and just have that the CRI API calls. Windows will have support with this in v128 when the feature goes beta. And last is, I think a lot of people have been waiting for this for both Linux and Windows containers, but in place pod vertical autoscaling. So this is, you can dynamically reassign limits, memory and CPU limits for your containers without needing to use like a horizontal or vertical pod autoscaler. The next release will have support for doing this with your Windows containers as well. Next, I'm going to hand it over to Permita. Yeah. So this is one of the project, SIG Windows DevTool. I'll start with what exactly is this, what announcements we have done. So let's say if you want to contribute to SIG Windows or you want to bring up your cluster, but you don't have a cloud account or anything like that. So you can really leverage this tool locally. You just have to download and it's actually builds a cluster for you from the source itself. So if you make any changes on the GitHub, it actually pulls it and it brings up the cluster for you. And it actually internally uses a vagrant, which is also open source. You can download that and you are good to go. I'm now talking about what's exactly new features that we have supported in it. So one of the support is QMU support. So basically earlier when we were testing it on Linux, Windows or Mac, we were using virtual box in it. But with a new Mac coming in and the ARM based system, we did not have that. We wanted to leverage the same boxes that we were using for the bring up. So on that M1 boxes, we are not really in your laptop. If it is M1, we are not really using virtualization. We are doing emulation by using QEMU. And as you all know that vagrant, if you have to download the vagrant, you need to have that plugin also. So what, how and you can do that is get clone our repo, check out the main branch, main QMU branch, install the vagrant and the vagrant came in and make all and just you are good to go. If you need, if you want, you can contribute to that and help us, you know, test it. We want that to be tested on Linux, Windows and every other type of OS or architecture. I would like to also thank our maintainer and the owner of vagrant QEMU, PPGGFF, her link that I have shared. She actually changed the plugin for support us. I would like to thank her. Second update is that our friend Malaskot, he's also working on the Windows side of it. If you want to bring up the cluster, he's, he's working on some doing R&D on that front via WSL. So he's been working on that. Okay. Yeah. Next slide. Second update is host process container. So as you can see, our data plane are like, if you want to, so right now we support two types of C&I's, Entria and Calico. So if you want to use Calico, we have like done host process container, we have really containerized the, the one that uses VXLan. So, but still, if you want to use BGP, you still have to, you know, install everything on the OS and then install the Calico. But for, yes, for the VXLan, we have done that automation for you. Entria, right now, yes, again, if you want to use Entria, you still have to use OVS, but we are kind of working on it and we kind of, in future, you can see that we'll make it host process containers for you as well. Okay. So this is also, so that was about, you know, bringing up whole the cluster and everything on based on, on what exactly you have in your laptop. Once that is done, now comes the operational readiness. This is also one of the projects that we've been working. So its whole, whole purpose is to validate whatever you are having in your cluster. Let's say you did some testing development and you want to test some changes that you did on the cluster. For that, it is very useful. So it certifies your windows, right? It is similar to what we have for Linux, right? K8 conformance test, which is there for Linux. It was nothing there for Windows. We're trying to build something similar to what we have for Linux. So it actually certifies you regarding storage, networking, if you have any AD or any network policy that you really did in your cluster, you wanted to test that. So you can download it and you could certify it. So what exactly, you know, how it does that. So you have this YAML file, which actually you can state that, okay, this is, this is the whole things that I want to test. You just have to feed that to the, to the code, code level and, and it actually internally takes the E2E test, which is there on the GitHub of, of Kubernetes, it builds that and generate XML for you. Okay. So how it does that, right? So you have to check out this code on based on what Kubernetes version that you want to use. Just do a make build. So it will generate a binary for you. And then you could use that binary and give the Q config file of your cluster. Now, again, the cluster can be anywhere local or on cloud. Just have to give the path to that, right? And your setup again can be anything Mac, OS, Linux, whatever. You can give the category also. But if you don't give any category, we are going to run all the tests for you. If you give any repository, like any, any directory, it would, it would copy all the things to the XML file to that directory. And that directory at the whole XML file can be passed to some, some dashboard, right? Like maybe I've given some example on the right-hand side. If you have Jenkins, you can pass that and see what exactly it did. Yep. Thank you. All right. Next, we're going to cover some Windows improvements that went in with Continuity 1.7, which just released, I think, probably last month. It's been a long time coming. So some, I think a lot of these features have been kind of been asked for for quite a while. So hopefully you guys are excited to consume them. The first one is a device assignment support in for your containers. So, and there's links to either the pull requests or the issues if you want to check out more information. But now you can through the CRI API. So if you have a device plugin for your Kubernetes cluster for those resources, it'll kind of get plumbed all the way through and you can attach devices into your containers. I think one of the big use cases for this is a GPU acceleration in your Windows workloads. So, and then you can also, we've also plumbed through all of the, kind of, through all of the different tools like CTR.exe and Nerd CTL to be able to test this. So you can just do a dash dash device and specify the device ID and it should show up in your Windows containers. Next is a graceful pod or container support for graceful pod termination for all Windows containers. So initially, as kind of evidence in the GitHub issue, this feature worked for Windows containers that were based on Nano server, but not Windows server core, which most workloads are based on. So actually took a bit of engineering work, but now graceful pod termination will work on all Windows containers. Next is we've improved the volume amount experience for host process containers. So anybody who's used them before is kind of, should be familiar with, if you want to access the volume amounts in your host process containers, you needed to route the paths off of this special environment variable that got set up during container activation. Now that's no longer necessary. Your volume amounts in host process containers are accessed, the exact same way as our in regular containers. And one big set of functionality that this enables is like in cluster config and any workloads that are looking for like your service account token itself. We'll just pick that up at the path that's expected. So you can assign a service account to your workload and authenticate as the API server without any extra work for your workloads. Next is support for that enhancement that I mentioned before to be able to consume all of your, or get all of your container metrics and stats directory from the container runtime. And with this, we did take the opportunity to kind of split it a little bit. So now there's more Windows specific stats that you can get and instead of just the stats that were defined for both Windows and Linux containers in Kubernetes. And last but not least, we have support for hyper view isolated containers, which is something that a lot of people have been asking for. So in order to use this, and yeah, again, if you're interested, please look at the issue. There's a lot more information about how to use it and everything. But if you specify a runtime class in your pod spec, like we have some examples here, it'll instruct the container runtime to use a different runtime class handler to start your containers. And there is a default runtime class handler for Windows hyper view isolated containers that's listed here. So if you just specify that, if you're running container D one seven with the default config should start as a hyper view isolated container. Now, since it is running in a utility VM, we also have the ability to specify the size of that utility VM, the resources for it through annotations, which are listed here, and also in the documentation. So that way if you want to, you know, you can scale the size of the utility VM to match the containers in your workload. So yep, those are some of the big improvements. I'm going to hand it over to Claudio. Yeah, thanks. I'm going to talk about one of the goals that we've had for Windows, and that's having unit tests running for Windows as well. When we started, we had a lot of failing units, more than 300 out of which maybe some of them were not even compiling at all. And at the moment, we only have somewhere between five or 10 units, which are still failing, which have been introduced lately with the addition of a couple new features in communities during the last release. We've also improved the test coverage a lot for a lot of Windows modules for Windows. And we've also ported a lot of unit tests from Linux to Windows to have additional coverage as well. The goal is to have in the end at the very least some release informing job, so we can actually know and detect any future breaking changes that have been introduced to Kubernetes. You can check out the test grid link, which will include all the birthday jobs, which have been run for Windows unit tests. And you can see all the requests which have been sent for this goal on that issue marked over there. If you are planning to contribute to Kubernetes, I do recommend and I think we all do recommend you to run the Windows unit test jobs. You can check it by leaving a comment with slash test poll ci when it is Windows unit test, which will trigger the job and see if everything is fine. If it's not, you might be introducing some aggressions for Windows, in which case, ideally you would ping us or try to solve it yourself. But if you want to run the unit tests on your local Windows machine, you can use the scripts we have listed here to prepare an environment for it. But basically you just need Golang and Git. Then running the unit test is just a matter of preparing the environment, cloning the Git, the Kubernetes repository, rendering and just running the unit test themselves. If you want to run the unit test manually, you can simply do it with Go test. Or if you want to run multiple tests recursively, you can do it with the second command. And additionally, something that's extremely useful for any developer is Debugger, which we recommend Delve. It works great. And it's a very useful tool to detect what's going on in particular tests. While solving those issues, those unit tests, we have noticed quite a few problems with Kubernetes code in general when it comes to how things work on Windows. For example, in a lot of cases, path functions were being used instead of path.file path. The difference is that path module is mostly used for Linux paths and even URL paths, while file path should be used for pretty much anything that's regarding any type of files. And they differ a lot in behavior. For example, in some use cases, the behavior of those functions will not work at all for Windows if they're not, they do not contain proper separators. For example, in some cases, if you are having backslashes in your paths, if you're using path functions, you might actually get something completely different. So that's just a good to know item for anyone who's considering Windows as well. Additionally, file path is absolute. It does not consider forward slash or backslash prefix paths as absolute, even though technically Windows does. A couple of issues were present in QBDM as well. For example, it's trying to do to copy a couple of files using CP, but now it's going to use X copy, which is going to work. A couple of issues for QBDM, for example, if you're trying to configure the NodeFS.inodes3 for the eviction, eviction hard configuration, it might not work properly in that case, which was actually the default value being set. In a couple of other scenarios, we did not have both DNS policies being applied properly if you wanted to use the host DNS configuration. Now you can actually use them if you just specified the dash-resolve-conf equals host as a configuration for QBDM, or alternatively, you can simply use resolve.conf file to have custom DNS configurations for your paths and have them applied through poddns policies. Additionally, the CPU usage no stats were not being collected properly for multiple processor groups. In the case of the QBDM plugin watcher, it was not working as intended simply because the way to detect if a file is a Unix socket is a lot different on Windows. We do not have a direct implementation for that, so we have to check manually if we can actually connect to that file. So there needs to be a different check than on Linux, so now it works with the new check and that now works as intended. Additionally, going through those a bit faster, Simulins were not evaluated properly if there were hidden folders in the in the subpaths. Log compression now works properly on Windows. Basically, the issue was that the file was being the file was being renamed before it was being closed, so that goes on an issue on Windows, and the voluminous mount points were not grouped properly because they were being sorted by ASCII values, which is a bit more complicated because the backslash is a totally different value from forward slash, and because of that they were not sorted properly, so that's no longer the issue, but that also affected Linux as well to some point. In regards to initial resources, if you want to join C-Windows and contribute, you can join, you can visit the C-Windows community page, you can join our community meetings which are on every Tuesday at 12.30 EST. You can join in, say hi, and after the meetings you also have the after hours in which we help new contributors get on board and set up their environments and see what's going on and have different discussions about different topics which are Windows related for the most part. You can also help us by writing additional documentation and user stories or comment on review bugs and pull requests on different project boards that we have listed there. You can also contact us, we have Arvind, we have Mark, Jay, James, and myself. You can contact us on Slack using our handles. You can join us on Slack on C-Windows. You can also send mails on the mailing list and you have all the meetings in the YouTube playlist. Finally, I would like to thank a couple of contributors. I think we all do, Amin Naben for his work for Windows Service Proxy, Fabian Fulga for his work on supporting in-place port vertical scaling and the Calico Windows host process containers, Pramita Gautam, which is here with us for her work on Windows DevTools and Operation Readiness. Jun Rhodes for his excellent presentation on Redpoint Kubernetes Manager, which he had had it working on WSL2. Mateusz Loskot for his involvement with Windows DevTools contributions. Before we open it up for Q&A, since this is a Windows talk, I also did want to say that tomorrow if there's a talk specifically about performance improvements coming to Windows containers and also how to kind of identify and tune your Windows workloads based on some of the Windows performance experts here. So that's tomorrow afternoon. But yeah, I think now we're ready to open it up for questions and answers. All right, and there's a QR code. The slides are also already on schedule, so if you want to click any of the links, I know people aren't paying attention to those during the talks, but feel free to get those or reach out to any of us if you have questions. Thank you.