 Hello everyone, thanks for joining for CIG Windows Maintainer Talk. My name is Claudio Bello, I am a Senior Cloud Engineer at Cloudbase Solutions and I am one of the tech leads for CIG Windows. And we are going to do a very short retrospective of what happened from the last cycle. And some additional guides on how to build Windows images, different tips and tricks and some very niche tricks that you could use. This came up because there are quite a few questions regarding how to build the best Windows images you can possibly do. So that's why I'm here. We've been doing this for a couple of years, so we have a bit of knowledge and experience regarding this. And in the end, I'm going to try to recruit you to join CIG Windows. We are a fun-loving community, so you're always welcome to join us. So minor updates. We welcome Amim Naben from Broadcom as one of our new tech leads for CIG Windows. He worked a lot on Windows Operational Readiness, which is a standard of tests and things which should pass and work properly in any Windows environment to certify that indeed it's working as intended and properly. And he also worked for Windows DevTools, which are very useful to getting new contributors into Kubernetes space and especially for Windows. And we thank Jay Vias for all his contributions. He has been a great help for us, and he has been a very present presence at any conference and meetings and so on. One other important thing to note is that the Node.log query feature will be entering beta in 130. We have a couple of other updates. We've been improving the Windows Operational Readiness and Windows DevTools, and we have a few other things in the pipeline, so they're not yet ready to be announced, though. But let's go to the main thing for today, which is the Windows Image Building deep dive. So a little bit of context, what we're going to see here. We basically added a lot of Windows support for a lot of Kubernetes images, the POS image, the QPROX image, the CNI, CSI, and so on. So in most scenarios, we basically had to integrate those images into whatever build processes Kubernetes already had. So there were quite a few restrictions on how we were going to build the images. But we're going to go from scratch, from zero, from the most basic images that we could possibly build, to more complex, which are currently employed in Kubernetes image building processes and release processes. Okay. So one important thing to note from start is that there are two types of images for Windows, and they are going to be used very differently and be built very differently. First of all, we have host-processed container images and regular workload container images, your usual images, right? The idea for host-processed containers is that they are a special type of containers which are basically run on the host itself and has access to all the networking devices, storage devices, the PCI devices, so on, as any other type of service on the host itself would have. So that's very useful for things like deploying your CNIs, your CSIs, or different device packages, and so on. It also makes it a lot easier to upgrade because you simply just replace the demon sets for Kube proxy, for example, to a new version, and you're done. Since previously, you basically had to do that manually over every single node. So this also added a lot of quality of life when it comes to how you manage your Windows nodes. And of course, you can also do quite a lot of administrative tasks, like exacting into the Windows nodes themselves, applying security patches, collecting logs, and so on, so forth. So they are quite useful. You can think of them as quite privileged containers. But we're not going to talk too much about them. We have a lot of documentation written on the website for this, especially how you run them, how to use them, how to add them to your own workloads. Or even better, you can watch Marcus Zittis and James Stutter once talk at the previous KubeCon for this exact topic. It's a very nice topic, so I advise you to do so. But we're here to find out how to build those images. Well, basically you can use as a base image any other Windows image. There's no issue in that. But there is a special image which is only 7 or 8 kilobytes in size, which is a lot smaller than your typical Windows image, so why not use it? But you're going to have to use Docker build decks to build those images. But the build process itself is very straightforward. This is basically a sample taken from the KubeProxy host-based container. It's just your regular Chrome instruction, and you just add a few bits and pieces there. So that's not very complicated, as you can see. Then you just push it, and you can deploy it on any Windows machine, and it will run, as intended. There are a couple of things which you're going to have to take into account when deploying them, but you can simply check the quantization how to properly deploy host-based containers. Now for the regular container images. In the start, when we started adding support for Windows images, we started building our own images and pushing them to our registries before we could actually promote them to the regular Kubernetes registries. We started with some Windows build nodes. As an example here, we have Windows Server 2022 and the latest Docker version, or among the latest ones. And we used this node to build Windows images. So far, so good. It doesn't look amazing from any point of view, but things are going to be getting complicated very soon. So this simply works, and we can also build different images for different OS versions. We're built here for 2022, and we built here for 2019, because we might have users which have different Windows nodes. So until here, everything went as expected. But things start to go wrong when you try to do things which are a bit more complicated, when something is trying to run something. So what's going to happen if we try to build a Windows image for 2019 on a 2022 build node? We're going to see that it cannot run the command. It says that the container operating system does not match the host operating system. That was an issue. That basically meant that we could only build images for 2022 on 2022 nodes. And 2019 on 2019 nodes. And at one point, we had four different OS versions which were supported simultaneously. So it became quite a hassle to manage every single one of them. There must be a better way to do it. Of course, you could try to use Hyper-V aesthetic containers, but depending on the public cloud that you're using, that might actually be a larger flavor and extra money that you would pay for every single instance. Alternatively, there is a better solution for this. And that would be multi-stage Docker files. So this basically doubled in size. We see that's multi-stage because there are two from instructions over there. So two halves. Now the interesting part about this is the fact that the first half, you can consider it like a workbench. You spawn an image which matches the build nodes OS version. And you do all your things, preparing the binaries and whatever you need in that side. And then in the later stage, you just copy your build product, your build binary, into your final image and you publish that one. It's basically working on a workbench, taking your build application, putting it in a box, adding labels, all that fancy stuff, and you push the package. And you leave the workbench behind. Pretty much how this works. And we can see that we no longer have any issues building 2019 images on 2022 build nodes. So that was nice. But we still had to use Windows nodes to incorporate them into the Kubernetes image building processes, which was a bit more difficult. So we basically had to go back to the drawing board and try to find a solution to build Windows images from a Linux build node. We were told it was impossible. We said, bet, challenge accepted. And we tried to find a solution for that. We already kind of have an idea on how to do those. An idea would be to use multi-stage Docker files once again. But for this scenario, we're going to have to use Docker buildX or buildkit. So you're going to have to also bootstrap it. But there are a couple of things which are special when you try to build Windows images on Linux nodes. First of all, do keep in mind to use the latest versions. Otherwise, we're going to face quite a lot of hurdles, which we hit, trying to build those images. First of all, which was very interesting to find out, was the fact that when you were using Docker buildX on Linux, it would overwrite the Windows images path environment variable to a Linux one. So of course, nothing works afterwards. So that was fun. And of course, there were a couple of other issues. You couldn't really change the user or the work here in the Windows image. That has been fixed. And also, the OS version was not being included into the image that you were building, which is extremely important because that's how the continued runtimes know which image to pull from a manifest list. It's going to match the platform, the CPU architecture, and the OS version. But now it is included by default, so even better. You won't have as many issues trying to build Windows images. Next, there's an important thing to note. You cannot store Windows images into the Linux image database. Some Kubernetes image building processes were doing that. That's basically impossible. You effectively lose the image once the Docker buildX command finishes. So that was not useful. So you can solve that by simply pushing directly to the registry or saving it into a tar file or an OCI file, which you can later on upload manually if needed. But there was another fun discovery to see. You had to push it to actually not lose the image. And another thing which occurred more recently. Recently, Docker buildX would generate a manifest list by default, which can be problematic if you try to build multiple images and then finally create a manifest list with all your built images. This argument will prevent that issue, so another good thing to note. And this is a weird thing. It still happens to this day. Simulings do not work in the way you expect them to. And definitely you won't be able to use absolute paths for Simulings. But you could use relative paths for Simulings. The idea is that it pre-pans files backslash to any Simuling target. If you have an absolute path, it generates something like files, backslash C column backslash, that doesn't make any sense. But you could use relative paths as shown in example. And we use stuff like this in our case. Still hasn't been resolved, there's an issue open for this, so someday. So let's see the same image, how we would do it in Linux node. The second half is the same, that didn't change in any shape or form. The first stage changed a bit because now we're using a Linux image as our workbench to say so. So of course we cannot execute the busybox.exe binary, that's a Windows binary, so we cannot do anything. But we are essentially doing the same thing, that command we had before. We're doing this manually. Of course, you can also have cross-building steps. For example, you can simply build Windows binaries in our Linux node anyways, so you can include those steps there. And finally, you can just copy the binaries to the Windows stage. So there are quite a lot of ways to solve this issue, so it's not a difficult thing to do. So we see here an example. We can see we are using the flags that we mentioned. We have the provenance flag, we have the platform, and we have the output set to push to registry. And we have the arguments over there. So this works for a lot of images, not all of them, but most of them. There are a couple of cases in which this wasn't as easy to do. It wasn't quite feasible. For example, we had to change something very essential for the post image. And there was no pleasant way to do that. And we essentially had to build an intermediary image, which was then built periodically and published, and thus being used as a base for the post image. Essentially, for example, if you try to modify the registry keys in a Windows container, that's going to be a bit more difficult. You might want to use a Windows build node for that instead of publish an intermediary image instead. So there are certain cases which are not as pleasant to do. So there are a couple of things which we were using that technique. Next, manifest list, they're awesome. They're extremely useful to group your images into the same registry name and tag. So this is even more useful for the Windows images because you can bundle multiple Windows images for multiple versions. And then basically, the container runtime is going to pull the right image based on its own host OS version. So this makes deploying applications a lot easier and smoother for your users, essentially. Creating manifest lists is very straightforward. It's something that you've probably done before anyways. One thing to note is that you have to make sure that the OS version is included for the Windows images in the JSON, which is generated. It should be added if you're up to date. But if not, it's easy to add it manually with Docker Manifest Annotate command. Simply fetch the OS version and add it, and that's going to be enough and you'll be able to push it to the registry. So nano server-based images. I don't know if you noticed, but Windows Server Core images, which have been used in those examples, are quite large. They are something like 4.5 gigabytes in size, so not small, which basically means that your clients are going to spend more time pulling that first image for the first time, so they're going to be stuck in pending for longer. Additionally, one other thing to note is that Microsoft publishes updates for their images periodically, monthly, and they override the same tags. For example, LTSD 2022 is being updated monthly. This can cause some issues, especially if you're trying to build multiple images at different times, which basically means you're going to have duplicates of the same Windows Server Core image for different client images to say so. We have some issues regarding that, especially when it comes to the CI, which we use for Kubernetes in test grid. And of course, you also have more storage being used, but not all images require to have such a large image. You can simply use the nano server image for certain scenarios, and it's a lot smaller. It's something like 300 megabytes in size, which is sufficient. For example, the Pulse image is essentially a nano server image, so you don't have to wait as long to start your environment. And again, the issue that we faced in our CIs for C-Windows is the fact that we had the support for test images for the CIs in Kubernetes, but they were built at different times. They were built on pre-quest and published at a later date. So most of the times the images had different bases, and test would fail because it would time out putting the image every single time. So the CIs were quite unstable because of that. But after we switched to nano server images, we stopped having those issues. That's a lot more reliable and stable. So this cool thing that we had to discover, but there are a couple of restrictions when it comes to nano server images. They're not gonna work the same way as a server core, and there might be some caveats which you might have to know if you try to target this specific base. First of all, you cannot run 32-bit applications in nano server because the compatibility layer has been removed, so you don't have that anymore. That's one of the reasons why the image is so small. Additionally, some dealers are not present in the nano server image. Another thing which we had to discover, for example, NetApp32.dll doesn't exist, which basically meant applications like NGINX or Core DNS could not work since they required some networking DLLs. And we had to figure out what DLLs to include. We had to do a bit of digging. There's an awesome guide over there which teaches you very well how to hunt for those DLLs. It's using Procmon. Essentially, you spin up a container and try to run the applications and you can basically see what DLLs are being loaded by that container in that application. So essentially what we ended up doing is include those DLLs from server core images. You can still do that. And we still have nano server-based images for applications such as NGINX, for example. So we simplified those binaries or DLLs from the server core image. But again, we're basically pulling a 4.5 gigabytes image to simply copy a file. That's not efficient. And we used some small caches here and there to cache those binaries and DLLs in a scratch image which would be updated periodically with new versions. And we used that scratch image to get the DLLs. So again, instead of pulling 4.5 gigabytes of storage to get one file, multiply that by four because we were supporting four different OS versions of Windows, that ramped up the build time quite a lot. But this solved that issue. And basically the cache looks something like this. We are pulling the server core image and we're just storing it locally in a scratch image. If you want to see more actual ways that we build those images, you can simply check the pose image make file which already has all the paths for building the Windows images. You can check the Kubernetes Test Image Building process which already contains a lot of pointers regarding building Windows images which we have written. And you can also check the Kubernetes Test Image Building script which again builds a lot of images for including Linux, including Windows, and so on. And if you want to see a more complicated Docker file, we didn't see anything which is complicated, check out the busybox one which we use in the CI. It's a lot bigger and more complex. Another cool tool which we are using and want to talk about would be Crane. This is a very interesting tool. It allows you to do a lot of things without having to pull images. But one of the use cases that we had to use it for was the fact that building Qproxy images was a bit more difficult in the Kubernetes space because they were essentially storing the Qproxy image locally. And then they were trying to load the image into the Docker image database which of course you cannot do. You encounter that sort of error. So the question was, okay, we have the Windows image built here, but how do we push it? And we found Crane, which essentially just pushes the image without having to load it, so that solved our problem. But it has a few other use cases which I think are pretty nice and might be worth considering. For example, you can potentially use Crane to speed up your image building process because in most scenarios for Windows images you are just adding new stuff to a base image. You can simply use Crane Rebase to essentially pick up those new layers and apply them to a new base. Essentially something like Git Rebase works in the same manner and you can simply just generate new images with this command without having to pull anything because essentially what is an OCI image is just a bunch of JSONs with references to images. So why pull the entire image? And another thing which might be useful as I mentioned, Windows updates their images periodically with new security updates, which are very important. You could use Crane Rebase to essentially rebase your images to the latest security patch and also make sure that you have the same base for all your images. So you're not gonna use the same amount of storage anymore. So that was a quick deep dive into how we build Windows images. Basically we didn't try to rock the both too much. It was a lot easier to use whatever build processes they already had. So that also made it a lot more cost efficient because we didn't have to spin up any Windows machines to build Windows nodes, Windows images. Finally, I would like you to offer this window of opportunity to join CIG Windows. You can check our community page, follow up contributor guide, join our meetings, which they are every Tuesday at 12.30 EST and join our page sessions afterwards, especially if you are new to the community, if you need any help, especially with Windows build image building, I'll be there. Especially if you could use Windows, try it. Try to find some interesting use cases on the documentation. If you have any issues, please open up an issue and maybe send some pull requests if needed. You can reach out to us, to the chairs, to Arvind Puthia Parambil, sorry. Marco Zetti, Arvind, Anabin, myself and James Stuttervant. Join the CIG Windows Slack channel, join the mailing list. You can see the previous CIG Windows meetings over there. And here is the Zoom meeting link. We'd like to also thank our contributors, past and present, and most notably, yeah, Chang for the work for reloading for GMSA webhooks. Mateus Loscott for the work on CIG Windows developer tools. Tatenda Zifuzzi for CIG Windows operational readiness. Sorry. Vinicius Apollinario for the Windows MiniCube guide and presentation that he made. And Pedro Coutinho for the work for Windows Calico CNI. That was all. Thank you very much for the attention. We, if you have any questions, feel free to ask. Yes. One second. So one essential tool for package building in Linux is the Peca Manager and Windows recently or recently the last years. Also introduced Winged for clients. Is also what's useful for container building in Windows or is this just more client-focused? You mean Bilket or what? A Winged. Winged. Didn't use Winged, yeah. Or did you use any others like Choclack or? Yeah, we didn't. I mean, basically you would be running Windows commands in that scenario. So you would have to use a Windows stage for that. Essentially, Winged or Choclacky would essentially just install a couple of binaries in places. You would have to do those steps manually, especially if you would use a Linux build node. But essentially you could do the same work in the Linux build node. Or again, Windows multi-node. Especially since recently we have added support for Windows Bilket, so you can also use Windows Build X and Bilket on Windows. And it has a few other features which are not included in the Linux one. Especially when it comes to switching users and changing file permissions for different files that you would have on your Windows container image. But yeah, that's an interesting idea. Might look into it. So I haven't personally used the screen rebase command. But could you theoretically also rebase, for example, a LTSC 2019 image onto LTSC 2021 and basically compare them into the same manifest and? Yeah. Yes, that is quite right. And I think that's what we also did here. It's basically we built an image basically 2019 and we set the new base to be 2022. And added a new tag essentially. But yes, that's what we were mentioning. You can use to not have to build the same image twice essentially. Very useful, yeah. Yeah. So he's saying that it's only copying the top layer. Not necessarily. It copies, basically needs a reference to the old base. So it knows which layers to remove. And which, so basically it moves those old layers and you are left with the layers which are added newly to the image. And those layers are getting appended to the new base. So not only the top layer, any number of layers which have been added to the base. So essentially quite like get rebase because you also rebase a lot of commits in git, right? To take care of. It's okay. Of course, if you just copy a binary part of data here then it doesn't matter but there may be more complicated operations here. I don't think it's gonna work for... Changing other places, yeah. So I don't think it's gonna work for every single case but in most cases, all same edges are just a list of tar files which are overlaid one on top of each other. So essentially it just applies the same tar layers over a different base. So that can still work as intended. Now, I think if you try to manipulate something essential to the base layer itself, for example, if you try to change some registries, I don't think in that case it's gonna work with the crane rebase. So yes, there might be those possibilities. In any case, all images that you build should essentially be also tested and especially if you have some integration tests which you can run, those are still recommended in any case, even if you use traditional building images or Linux or use something like crane rebase or Docker builders and so on. But yes, there might be some case which is not feasible. Just a quick question. Throughout this pretty incredible process, have you ever looked at PowerShell Core between both operating systems? We did. Sorry, somewhat of a loaded question. It is, so PowerShell, huh? So we do include PowerShell in the test images that we build because we need it for some tests because there are some execs for some reason. And of course, none of the server doesn't have any PowerShells in it. But you could just pull them and add them to the non-server image so you could still have PowerShell. One other thing which kind of happens is that when PowerShell runs for the first time, it also pulls up its caches, which was another interesting thing because we had to use an intermediary Windows image which already had all those caches built and set. And we are using that cache image to essentially add PowerShell and all its caches to the test images that we use. So we do use PowerShell in our scenarios. I don't know if that was the question that you asked, though. You answered it. Oh, cool. Yes. For the server core and the nano core images, do you support the platform type of ARM with Windows? ARM? That's a good question. I'm not exactly sure. I did some digging some time ago and I inspected the images that were published by Microsoft and there were indeed manifest lists which also included ARM images. But I didn't use ARM images because I do not have any ARM to actually test it. But the image building process would be pretty much the same in any scenarios because one other thing which Billkit does and is being used is also emulating other platforms of CPUs. For example, Billkit is being used to build images for AMD64, S390X, ARM, ARM64, PPC, 64LE, something like that. So in all cases for Kubernetes in general, it builds a lot of images for a lot of platforms and in the end, we'll just create a manifest list which includes all the platforms in that manifest list. It essentially uses QMU to emulate those platforms which is another interesting issues that we face at one point and it was kind of hard to actually figure out this was happening was that sometimes the image building process would fail in Kubernetes and we didn't really know why. It suddenly says I cannot build an image anymore. I can discover the fact that what happens sometimes, the same image jobs would be scheduled to the same node at the same time but whenever QMU was being used to mock a different platform, it would essentially change the kernel. So flags in kernel which is the same as the host itself. So basically it was a risk condition between two building jobs and they were both overriding the same flags for the operating system. So of course, if you try to build an ARM64 and AMD64 image at the same time, one of them is going to fail. That was interesting to discover but again, it's possible to build any sort of image for any platform and yes, I saw some ARM64 images some time ago. I think we have to finish here but if you have any other questions, feel free to ask. I'll be around here either around or outside but thank you so much for your attendance.