 Hi everyone, I am Natalie. I am an engineer on the garden windows team And I'm Matt engineer on the garden windows team This is that team that we work with most of them are still back home in New York There they are with our old team name Bander there greenhouse Now we are two teams garden windows Bosch windows and a few other ancillary teams as well so one of the things that we mentioned this morning was the dotnet Renaissance and What our team does is enable the dotnet Renaissance? What do we mean by that? Well now that we have dotnet core and cloud native dotnet with steel toe and DevOps becoming a thing in the windows world We need a place to run all of that right and for Linux. That's great. We have a Linux story You can you can push with the dotnet core build pack you can Already have the Java experience with the Java build pack it all works just fine, but what about windows? What about windows applications that are tightly coupled to the windows operating system? So that's where we come in so there are some applications that are just Not suited to Run in dotnet core. They need to run somewhere That developers don't necessarily want to be operators. I personally don't want to write a docker file or think about anything other than my application Microsoft thinks developers want to think about docker files, and that's okay We'll let them focus on that and we focus on the CF push experience And that's what the cloud foundry application runtime brings and so we have this world where both of these Exist in harmony right you can push your docker files So you can push them to kubernetes that works fine, and you can also push them to the application runtime That's what Natalie and I work on so our team is garden windows We make Containers on windows, and we also maintain the root FS that you will use with Windows server containers in cloud foundry We contribute to the application lifecycle for windows and net so things like the hwc build pack and the build pack app lifecycle and We are a source of windows and some dotnet domain expertise for the cloud foundry community So what have we been working on since the last cf summit? We are keeping up with the latest releases of windows We have been continuing our effort towards pragmatic parody that Matt and Shanfan talked about this morning and Finally, we're working on enabling future platform features So really quickly just a recap if you're new to windows workloads on cloud foundry Here are the components that we maintain For context, we're showing the garden server It calls out to different plugins to do different things So we have the containerizer on Linux that would be run C which is a binary that knows how to do things with containers On the windows side, we have win C which also conforms to the OCI standard. So we fulfill the same API for networking, we have wink network and For image management, we have group windows So our components make use of the host compute service and the host network service Which are windows services that let us do things with containers and their networks So now Matt's gonna talk about how that's been changing with the different versions of windows Great. Thanks, handling so we work pretty hard at Keeping up the date with windows server releases Turns out it's not super simple. So right now we are targeting the windows semi-annual channel Windows server 2016 is a long-term support release. This means is when it's released it has Four years plus of support. However comes at the caveat. There are no new features or improvements brought to server 2016 So you may have noticed if you've deployed the windows stack. We called it windows 2016, but that's a bit of a misnomer it's really windows 1709 1803 and 2019 we'll get more into that later The six-month release cycle of the semi-annual channel is great because that previous bullet no new features or improvements Well, that doesn't apply to the semi-annual channel However, the downside of the semi-annual channel is it's only supported for two years But you cannot think about any of these problems when you're running cloud foundry, right? Because you see if pushed you're at you have a droplet and we can just take care of everything below the application So that's really where the value of the application runtime comes in We are always testing against the next release. So we did verify compatibility with windows server 2019 Before server 2019 disappeared from the internet this week. So it will be coming out soon As of see if deployment 3.5.0. We introduced support for windows 1803 Windows 1803 has a smaller root FS Networking improvements. We're now using what's known as h&s ACLs instead of windows firewall to enforce the net out policy Net out policy is all the rules from your application security groups The good thing about this. This is a security feature is that the container admin can no longer override your network policy previously in 1709 if you ever got to container admin, hopefully you couldn't you would actually be able to break out of our net out rules so 1803 is a huge improvement and And finally moving to 1803 enables new platform features, which we'll talk about shortly So fragmatic parody is another thing that we think about In the in the windows world. What do we mean by? Pragmatic parody. Well, this is sort of the diagram that we show to people when we talk about this There are features in the platform that makes sense for Java applications There there are things the Java applications could never do without And for us we kind of pick and choose what makes sense to bring over to dotnet and to windows So one of the things that we decided made sense last year was CF SSH, right? It's really helpful to be able to remote debug your applications. And so CF SSH enables that Of course, there are other features that we might want to bring and other features that don't make sense to bring The windows but some of the future features that we are thinking about our dynamic egress route integrity in Istio So we've been working on making the hwc build pack better With multi build pack support, so that has been around for some time with the other build packs And we just enabled it a couple weeks ago So what does that mean? That means as an application developer you can now have an extension build pack that will supply dependencies or application performance monitoring tools or debugging tools to your app Without have to without having to copy that into your source code So just to illustrate what that looks like you can now see if push Provide a path to your extension build pack along with hwc build pack. Those will compose together They will both run in the staging staging container and finally be packaged up in the droplet We are iterating on this and we're currently working to provide some more documentation and guidance to Build pack developers on how they can write their extension build packs We're also welcome any feedback that you might have on how we can kind of make improvements to To the build pack itself to better suit the needs of dotnet developers Also an hwc we enabled HTTP compression This is for both dynamic and static content. It's turned on by default. We have sensible defaults for each This is just illustrating that now, you know with hwc build pack when you hit your up your content will come back compressed So moving on to the future platform features that we're looking at Graceful shutdown. This is a popular one so Windows apps running in Cloud Foundry have always been killed which is too bad because Apps don't get a chance to run any sort of cleanup code This is different from the Linux experience where you get 10 seconds to clean up your app before it gets totally shut down However, we are looking into ways where we can make this possible in Windows Server 18.03 and above We've tried a couple different approaches to this the one that we're most favored right now is Application processes running in Windows Server containers do receive this control shutdown event when the containers are destroyed We are trying to understand how we can bring this or make this available to application developers in Cloud Foundry and As a Windows Server 19 2019 we expect that the amount of time that apps have to handle this event will be higher So that's why this is preferred for us Great So one of the things that we talked about previously was bringing container D to Windows So there there is currently only container D support for Linux. We were Hoping that maybe there would be a wink runtime for container D, but at this time we've currently paused that effort Microsoft has been working on bringing container D support to Windows and so there's a little bit of duplication of effort there We're going to wait to see that happen They also in the meantime introduced a library called run HCS It's actually very similar to WinC. However, we can't get rid of WinC at this time It doesn't have all the APIs that we need so it's the thing we're keeping an eye on for the future But at this point, we're still maintaining WinC. We're not yet using container D In terms of volume service is another thing that we hear quite quite a few requests for Due to operating system limitations Windows server containers cannot mount SMB shares by themselves But there is a workaround The good news is that inside a container an application can access a file server so long as you have the appropriate application security groups set up And so we've come up with a solution for Clifendry users with .NET apps can use a C sharp library that will actually talk directly To the file server and so usually if you need to talk to a file server you can achieve that by using a sharp sips library And you can expose all those credentials to your app using a custom user provided service Natalie talk about networking. Sure All right, so hopefully you had a chance to check out Amelia and Christian talking yesterday about dynamic egress But just in case you didn't Dynamic egress will let Operators specify net-out policy and have that be applied dynamically without having to restart your app This is a contrast to application security groups that today do require an app restart on to take effect so this Makes use of the VXLand policy agent from silk, which we actually we don't use silk on Windows So we are currently working to Extract that piece of functionality have it run on its own on a Windows cell and we're working closely with the CF networking team To see how we can get this all working on Windows. Oh Sorry one thing to point out just on Dynamic egress is that it does require Windows server version 1803 or higher because of the way we're specifying the network policy So more on networking Earlier this year we explored container to container networking using silk Which involves setting up an overlay network and then configuring policy to allow apps on that network to talk to each other We learned a lot about the Windows kernel in the process and we had high hopes But they were dashed because unfortunately we don't think the silk model will work on Windows So just to illustrate that in a little bit of a different way More detail we have two cells here in CF Each of those cells has its own subnet on the overlay and we have two apps that want to be able to communicate with each other East-west without going out through the go router So it turns out that setting up an overlay network is hard on Windows There are some serious performance considerations that we're gonna have to take into account if we decide to go with this approach But actually the network policy enforcement pieces We think is in impossible using the silk model. So just Seeing a network packet that originates from app a is destined for at be With silk we would be inspecting sorry we'd be Modifying that packet as it leaves the cell to have a tag in the GBP header field and then When it arrives at its destination, we'd want to inspect that Inspect that header and see like based on the contents of this of this field is this traffic allowed Unfortunately on Windows. We don't have a way to modify or inspect that That header so we think that we're out of luck in that area So more networking stuff. All right So hopefully you caught Shannon's talk yesterday. He talked a lot about Istio, but here is a quick primer So instead of implementing silk, we thought well, could we just leverage Istio? Well, why would we think that in an Istio model? we could rely on an envoy running in a sidecar container to Proxy all the application traffic out to some sort of shared router that lives in in the infrastructure And that shared router would have access to say an overlay network for Windows cells and an overlay network for Linux cells Not necessarily the same overlay network And so leveraging Istio an application has no idea that it's not on the same network And it doesn't matter because it can still communicate with everything that it needs to right this is very theoretical It's an idea that we have but we're thinking about all these other things that we could do instead of delivering a unified container network Why do we want to use Istio? Well There are a lot of benefits so Natalie mentioned a whole bunch of challenges in implementing silk on Windows and Instead of relying on that that tag that we can't get to on Windows We could offload policy enforcement to envoy All we'd need is some form of container overlay network And we already know we can connect containers to each other in some way just maybe not using soul We actually proved out Istio on Windows using a proof of concept an engine x proxy and In a dotnet application, but settings of magic registry keys We were able to force all dotnet application out through an engine x proxy that could approximate working with Istio Very early stages, but we're pretty excited about the the prospect here now Istio requires Envoy and there's a there's a problem there Envoy does not currently support Windows However, we've been working to get Windows support merged upstream Microsoft initially began porting Envoy to Windows back in 2017 around this time and Since then we've managed to get Bazel the build system for Envoy Merged with Windows support into the upstream repo We also have a whole bunch of PRs coming to make everything actually work on Windows Here's a little teaser We've got a concourse CI Running a subset of the Envoy test and so you can actually go and check that out You can't see the URL there, but it'll be in the in the slides So you can follow along at home So with Envoy we think we can enable route integrity which has been around for quite some time now in Diego as an experimental feature This of course leverages Envoy for router validation of the app instance identity And just to illustrate that right here The go router is using mutual TLS with the Envoy proxy to verify that the app request is Going to the app instance that it was destined for and we are going to Demo Envoy running on Windows in a bit So this solves another one of our Problems that we encountered earlier this year, which is that IPsec does not currently work with Windows server containers. This is due to A limitation of the operating system But route integrity provides us a way an alternative way that we could secure traffic inside cloud foundry and And Diego recently maybe you saw the email to CF dev. They will introduce a feature to tunnel all traffic in Including SSH through Envoy. So this would be A way for us to make sure that traffic inside of CF is totally secure and now we're going to demo All right, see how this goes live demos always goes seamlessly so Can't see that So I'm going to SSH into an instance of a application. I have deployed to a windows So so I'm going to CF SSH to Nora and I'm going to forward for it. It's 61 003 So here we go. I think I'm still on the Wi-Fi Okay, so here is that Envoy running on Windows on cloud foundry So pretty excited about this if I show you what's happening here We have Dynamic certificate rotation loading up here So you can see these are actually the instance identity credentials that are being presented to our windows container Just like on Linux being dynamically loaded in here into the Envoy proxy Surprisingly works pretty well And I can see if I look at my listeners We have two listeners here So Natalie mentioned that all traffic can be forwarded through the Envoy So the two listeners one of them the 61 001 that's actually mapped to port 8080 inside of my container and Port 61 002 is mapped to port 2 2 2 2 which is the Diego SSH so back over here inside my container if I look at the Process list you see that This is a little bit bigger There's no there's no Envoy process running inside this container There we go So you can see this is actually where my app is running hwc is my is my application So where's that Envoy running? Well? Let me just first show that the Envoy is in fact listening on this cell right up here is the 61 001 and 61 002 listening inside this container. Okay, there's some magic happening So let's get out of here So I'm going to fosh SSH to the Windows cell that this container is running on give it a second Once again launch a power shell Okay, let's look at all the Containers running on this cell so this is my application container and There's a second container here. It has a dash Envoy on it. Well, it turns out we have sidecar containers on Windows so this is pretty cool in Out here in wink. I Can wink exec into that container We have a little bit of a problem with line wrapping and and the open SSH server from Microsoft the the CF SSH server has no problem with Line wrapping so anyway now I'm inside of this sidecar container and I can take a look at that process lifts and Look at that. There's my Envoy So we have sidecar containers running an Envoy that we have managed to build connected to that that main container and Proxying traffic For Windows, so we are super excited about this. This is something we've been working on for the last three months or so and it's taken a lot of effort both inside and outside of Pivotal and We can't wait to see it That's our talk Pivotal is hiring come work with us We love pull requests. So please check out wink Guardian on GitHub We're in CF deployment use the Windows cell ops file and Find us in slack. We're always here to answer your questions So any questions to Natalie and Matthew Thanks a lot Matthew and Natalie and I've just a question about route integrity in our current Foundations we have enabled this feature for Windows cells and this also Changes the way the go routers work because they now depend on this that they do not have a timeout anymore To delete unused routes from their table. How is does this now contradict to Windows where this is currently as I understand it Not working. Yeah, so so the run integrity Implementation on Windows is like very very experimental however, it uses the same code path as as on Linux, so essentially what what happens is the the go router initiates that mutual TLS connection to the container directly to that envoy, right and instead of Doing a heartbeat every I'm not sure if it's 60 seconds or something like that The go router just knows about where all the application should be and then uses mutual TLS to ensure that it's communicating with the instance That's on the other end right now We're we're planning to introduce this as a very very experimental feature just to start gathering some feedback You might not have seen it when I was SSH'd into the Windows cell the the envoy had used a significant amount of CPU Currently it is not optimized at all. It is very much very much an experimental feature We have currently a route integrity enabled in our foundations. So the system currently relies on this feature and we even had a problem when We could figure on the other side to not check for SSL Consistency and so we had often throughout which one were never deleted in our foundations And this got us into a lot of trouble in the Linux world Now we have enabled SSL checking and now everything works and if you know Windows does not support a route integrity with mutual TLS then We don't know I understand that you might run into a problem here Yeah, we should we should dig into that a little bit more Especially with some of the other subject matter experts in this area Yeah, it's basically about mix and match Windows and Linux cells in that environment Right Around loading SSL certificates into Windows containers. I know this is something that you all have have struggled with We think we might have a fix for that that we came up with over this trip. So stay tuned Any more questions? Thank you very much to Natalie and Matthew