 Hello everybody and welcome to the Sig Windows maintainers talk. Let's get started. First we're going to introduce ourselves. I'm Mark Rossetti. I am the co-chair of Sig Windows. I'm software engineer at Microsoft. And I'm James Servant. I'm the tech lead for Sig Windows and contributor to Capsi and cluster API. And you can find me on Twitter at Aspen Wilder. I can make fire in six different ways with just six in stone. So you can ask me about that later. And I'm happy to be here. I am the tech lead for Tanzu and contributor for Sig Windows upstream. Mine seems to be working. Hello. Okay. So here's some updates. We're going to just do some project updates. Demo a little bit of the work that's going to be coming. Share some additional resources and then open up for Q&A. So first I'll just give a little bit of an overview of the enhancements that Sig Windows is working on. The first one is the OS field in the pod spec. That one recently graduated in the last release to stable. Next is host process containers. We actually had a whole talk about using host process containers earlier in the conference. And I encourage everybody to check that out. New one is host network support for Windows containers. We're targeting alpha in Kubernetes 126 for this release. Then there's node service log viewer. We've got a demo of that coming. That will be hopefully going alpha in 126 as well. We've been working with Sig Node a lot to have parity with our CRI based stats support. So it works on Windows and Linux. And next is operational readiness. So this is a little bit of information about the pod OS field. This field was, I think, added in 122, but it's stable in 125 right now. So for people who aren't familiar with this, you can specify the OS that the pod is intended to run on directly in the pod spec. And this gives a number of different benefits. Some core components of Kubernetes like the kubelet have been enlightened to honor this field. The scheduler still isn't. But what this allows you to do is it allows you to, we added a lot more checks into different components to make sure that the container image that you're pulling is targeting the right OS and return a much richer error around that. The other really important thing that I'll highlight is this is tightly integrated with a new pod security admission. So a lot of the fields that are restricted, like that you can't set in the restricted policy are, there's a lot of rules about what fields you must drop or set. And those don't always apply to Windows. So this was a nice solution. So if you're, say your pod is running on Windows, we will allow you to run restricted pods without dropping all Linux capabilities, which doesn't make sense. Next is, we're going to do an update on host process containers. We are going stable in this release 126. There is a little bit of new functionality that will require a container to be 17, but everything is fully backwards compatible with the old implementations. One of the big highlights and one of the big reasons why we were waiting so long to go to stable is now the volume on behavior is much more kind of as expected with containers. And a big plus is we can do, you can use all the encluster clients and configs. Another thing I'll call out is we now have a very thin base, or a slim base image that you can use to build these host process containers on. It's about 25 kilobytes, which for people who are used to running Windows containers is extremely small. And we've got demos of all of this in the other video. There's a link to the KEP and more information and a link to the other talk right there as well. I'll hand it over to James. Yeah, so with the host process container support, we've seen it adopted very heavily across the community. And the Windows node exporter is one of the ones that we've seen it used very heavily. There's an open issue in Windows exporter for about three years to add support here. The Windows exporter typically had to run on the node, which meant you had to log in and install it. It was hard to update, hard to manage, hard to figure out if anything went wrong. And so we've added support to enable that as a daemon set. And you can also use it with any of the Prometheus components out there, in particular the Prometheus operator has support for it. And you get some really nice graphs if you use the Prometheus operator out there, which has integrated support. The really interesting thing is we released this just recently, and we have over 100,000 downloads already. And so, yeah, this is a very top, top ask for our customers. Another one is the Windows container networking stack inspector. This is a tool that's built with host process containers as well by the networking team. And it lets you track the packets that are flowing on the node. And you can narrow that down to an individual pod, so you can debug any of these problems that are happening. There's a whole set of things that you can trace with it, from the packets all the way down to the hns state. And it's using packetmon under the hood on the node. And then you can convert those packetmon into wire shock format so that you can kind of analyze what's going on with your traffic. So really cool tool. We did a demo of this at our previous talk, so you should go check that out as well. In this update period, we also have seen a lot higher adoption across the ecosystem with Windows containers. And with that higher adoption, we've found a few bugs. And one of those was NQ proxy. And you can see the time stats here and how much it has improved, particularly for Windows Server 2022. But what was happening was it was very expensive to make these calls into the OS to update the proxy rules that redirected the traffic. And with some changes to Q proxy, which we cache a bunch of that information and a few changes to the OS, you can get these sync times really low. And so what was happening was Q proxy would, a new node would join the cluster. And if you had a huge amount of services across that system, it would take a while to sync those. And now with Windows Server 2022, we're down to less than a minute to sync large amounts of services. Another thing that we worked on inside this semester was Perf dashboard. Using the Windows exporter that we talked about before, we enable us to run performance tests against the Windows cluster. And we can monitor this for any regressions in performance across the changes that we make. So we're tracking things like memory usage, CPU, network changes. And next up, we're planning on doing some work with larger scale tests, as well as doing some soak testing to catch any bugs that happen on the VM's long term. We did do this work with the Perf dash. So this is a public URL. I put the link down at the bottom. You can go view this. There's a bunch of other information out there for the Linux side of things. But this is a great step forward for Windows in our Perf story. So next up, we're going to hand it over to Averinth, who wasn't able to make it with us today, but he's been working really in depth with the node team to deliver a feature that we believe is going to enable Windows users to debug problems with their Windows clusters in a much easier way. So we're going to play a little video here, and he'll explain it all. Hey, folks. My name is Averinth, and I'm a staff engineer at Red Hat in the OpenShift organization. I'm also the lead for Windows containers at Red Hat. I'm here to give you a talk about a new feature called NodeLog VR that we are hoping to introduce as alpha in Kubernetes 126. Before we delve into the feature, let's talk about the problem the feature is addressing, and it has to do with debugging services that run on either Linux or Windows nodes. So what usually happens is if a service on a node is malfunctioning or acting up, the cluster admin has to SSH or RDB into the node to debug it, and then they have to sort of rinse, repeat, and log into all their nodes to figure out what's going on and solve the problem now. This is not the greatest user experience that we can provide for cluster admins in this area, and that is what we are trying to tackle here. So for the solution, what we decided was to add the capability to kubectl to view node service logs, similarly to the way it can view pod logs. This puts the cluster admin in a very comfortable place because they use kubectl to interact with the cluster in other ways. Once we implement the feature, there would be no need for the cluster admin to use SSH or RDB to debug these classes of problems. Given the sensitive nature of some of these logs, we decided to make this feature available only to cluster admins. So let's now delve into the design details. The kubectl already has a barlog endpoint viewer. It's just that there's no client enabled to view the data that this endpoint is returning. So what we decided to do was to supplement this with a shim that shells out to journal Ctl on Linux nodes and get an event which is a power shell command on windows that can be used to get application logs for services. We would then add a node log sub-command to kubectl with a query flag. This query flag implements a heuristic mechanism to return logs either from the journal Ctl or from a file under warlog. What this means is say you give a input foobar to this query flag. The heuristic mechanism will figure out did the foobar service log to journal Ctl in the case of Linux or windows events in the case of windows or did it just log to a plain file under warlog and it will return the appropriate information based on this heuristic mechanism. I do want to give a huge thank you and a call out to Tim Hawken for suggesting this mechanism. So it's thanks to him that we have this feature. In the alpha phase the feature is going to be enabled by a feature gate called node log view and it will be restricted to Linux nodes with systemd slash generally support and windows services that log to Ctl and warlog and the application logs. But this would work across all windows variables. I also want to call out that this is based on an implementation by Clayton Coleman which he did for OpenShift 4.0. So this is a feature that is present in the OpenShift product and has been for the last few years. So what's the current status of this feature? We have a cap which has been sponsored by windows and has been accepted. The implementation is under review and I have dropped a link to the PR here. The initial release will be behind the alpha sub command of Ctl and as we go towards beta and GA we will move it outside of this alpha sub command to its own sub command. So without further ado let me show you a demo of this feature. So what I have here is a cluster that I brought up using Sig Windows DevTools. It's a simple cluster. There is a Linux control plane and a windows worker and let me show you some of the features of this new sub command that we have added. So this is the help. It gives you a brief description about the feature and the various options available. I will hit some of the basic options here. One of the things you can do is you can query particular nodes that you want. So for example here I'm going to query the windows worker for a service called Microsoft Windows Security SPP. This actually logs to the windows application logs so that event will be used behind the scenes. And as you can see it's going to be a pretty large log and you can also do things like just see the last 10 lines of log and so that you don't have to see the whole log. You can also have different queries. So for example here what I want to do is I want to query the control plane. I want to query the continuity service and I want to only look at logs that have a particular pattern which in this case is error. So this means that I'm going to if for example if the control plane spans multiple nodes you're going to get messages from all these nodes of the continuity service. You can also do things like look at you know if you know that there's a particular file under warlock that you want to look at. The query mechanism as part of this heuristics will gives you that capability also. So here I'm going to look at okay these are all the directories that are available and then I can see what's under the cubelet directory and say I want to look at a cubelet error log. I can do that too. So in a nutshell this is the capability that this feature is introducing. So let's move back to the presentation now. So yeah we have been moving forward in the sub-projects of CIG windows as well and the first having these slides are the windows operational readiness where we are developing a operational conformance kind of test for windows. So it's easier now for enterprise to validate their windows Kubernetes distribution and the way it works is like you have a specification that defines a bunch of categories and tests and everything that your cluster needs to pass enable to be ready and operational. So check out the project we had another presentation about these two hours ago and where we deep dive and talk more about the project. So this is evolving this is starting and we are looking for contributors. Capping now supports windows. That's a pretty cool one. Jay, Dimitri, Mark and Douglas have moved the kernel space to the Capping. Now it's a backend for Capping. Now we support user space and kernel space on Capping. Capping for those who know is the next generation of CUBE proxy and this will allow us to provide a more scalable solution across the service prox that Kubernetes provides. One note here is that the windows and Linux user space are deprecated and are going to be removed on 126. So if you want to still check one of them, check on the Capping. We have migrated them for that. So in the next project we have the single windows development tools. This has been evolving as well. So now Luther implemented the double SL2 support. Now we can run the project inside windows machines. You can use like latest binaries to run your entire cluster locally or you can compile with a hash as well. That's one of the functionalities that was added. And we are using, we are always migrating for the latest CNI plugins like Entria and Calico. That's what support right now is only a YAML. You can change and can bootstrap a cluster locally from zero with the latest software and leave dangerous. I'll talk a little bit about what's next now. So in addition to all of the work that we've just highlighted in continuing all of those Caps, we are making progress on hyper-reisolated containers. This has been a big ask from a lot of people for a long time here. So previously with DockerShim there was kind of very experimental support for hyper-reisolated containers but there was a lot of rough edges. The biggest one being you could only run a single container per pod which isn't really practical. So we've started to plumb through all the necessary support for running hyper-reisolated containers kind of the right way in container D. Most of that work has merged in Continuity 1.7 which should be releasing this year. And there's a big issue with kind of outlining everything there including what's next and the state of that there too. So this is something that we're hoping people will start taking advantage of for those of you who aren't aware of this. Hyper-reisolated containers do provide a much better security boundary versus traditional Windows process isolated containers. And we're hoping this is going to help people enable more onboarding and more secure workloads into the cloud. Yeah. Yeah, I got it I think. Yeah, so we just wanted to say a special thanks to our contributors and some of the new contributors. So Fabian has been helping out with adding test coverage and some documentation to the SIG Windows DevTools that you can find some really interesting things there around how to use host process containers. And so, yeah, thank you. David Scott, he has been helping us on the networking side. He was the one that found that Qproxy bug and was able to help us debug it. Andrew Collins, he's came to SIG Windows a few times and helped us with some of the perf work that they're doing over at Red Hat. And so really appreciate the help there. Averint for all the work that he's doing as a really cool feature. I think one of the really good use cases for the feature that he's working on is GMSA, being able to query the event logs and identify any problems that you're doing when you're setting that up. And Dmitri, he's helped out with the WinColonel Proxier and the KPNG projects. And there's so many other people who have contributed. And so we forgot your name. We're really sorry. But we really appreciate your work. And come out and help us out. We'd love to help. And on that, so we'll talk about how to contribute. We have a SIG Windows community page. It talks about all the sub projects that we have, where you can get involved. It has links to the Slack page, all the bug triage and meetings. We have a contributing guide that helps you get started specifically for Windows. We have community meetings every Tuesday at 9.30 Pacific. And we also, after that, if you come into SIG Windows and you have a problem and you want to just kind of hack on something, we do do some pairing sessions afterwards. So somebody can drop in and say, hey, I've got this problem. And we'll just kind of hop in and start working on it. And so that's a really cool way to get started and get your questions answered. Some other ways you can get started is the documentation and other projects that we've kind of mentioned throughout here. We're looking for contributors across the board there. And every other Thursday at 9.30, we also do bug and issue triaging. And so there's a small group of us that just go through our backlog and triage it and start to figure out what's going on and which ones we're going to solve. So here's just a little bit more information about who some of the leadership is in SIG Windows and all the resources, like I'll link to all the YouTube recordings, meeting notes, the Zoom meeting, just where you can find us. I'll leave this up, but we can open it up for questions and answers. Hi, thank you. About the host network containers, is that integrated with host process containers that sit on top of host process containers? Or is it either or? No, so that's something different. Host process containers are currently restricted to running in the host network namespace because they run on the host. Currently, you cannot run your like normal Windows pods and join them to the host network's namespace. That's just that new functionality that we're enabling for non-host process containers as well. Got it. So it's sort of like host network without the volume out. Exactly. Yeah, it's host network without having everything else run in all the different host namespaces. Thank you. Anybody else? Question? Hello. Thank you for the talk. I would like to ask what are the options I have when I want to run a hybrid cluster and using network policies regarding the CNI plugin? That is still being considered, right? That's still TV. This was the primary reason not using a hybrid cluster for my community and the organization. So we've had distinct Windows and distinct Linux cluster. Do you know? Brandon, do you want to come up? Brandon, could you repeat the question? My question was what options do I have when I want to run a network policies in a hybrid cluster regarding to the CNI plugin? At the moment, I don't think we have a direct option for that, but that, like Jeff Mark said, it's in the works. So let's definitely talk and we can see what you're looking for. With the Hyper-V isolated containers, the first step was being able to launch them in container D. And those will be launched through like the run time classes and all that functionality for other, you know, Codd containers and things. After that is stable. That's when we're going to start looking at the rest of the Kubernetes experiences on top of that. Yeah, the first piece is just getting it to work with HCS. It is. That may come in. Yeah, I know Calico has network support. Just to clarify, per here said network policies are part of the CNI. I know Calico has support for network policy. Azure CNI does as well. Do you know if Antrade does? Yeah, Antrade does as well. Experiment to see how that all ties together. Anybody else? Thank you everybody for coming.