 Hello that's loud, okay good So we have Jeff Koffler and Andrew from Microsoft sharing their experiences with their work on Windows may just on Windows as well as a CMake build system. So I'm gonna head off to them and they can get started Thanks, so I was Andrew and this to my left is Jeff and we have some cohorts here Paul Allen, but not the Paul Allen Yes, as he pointed out is the only Paul Allen in our group and Then Lee Lee who's the who's our manager? All right, so as this is really loud this How's that? Okay, so we're here to talk about messes on Windows and our experiences with it so far so Why add Windows support to messes Well as you can see we have customers that are interested in Windows We're not fundamentally modifying messos in any way what we're doing is we are bringing up a messos agent on Windows, so if you have an existing messos cluster, it's the same messos cluster You're you're used to you can use it in the same ways you simply have the ability to add Windows nodes and Windows workloads to that Using Using messos you can Select Windows using the typical messos features You know OS CPU speeds other criteria And again, I wanted to emphasize it's the standard messos, you know, we're simply adding an agent for for Windows Little bit of history you can read the slides basically when we started it was started as a straight port So for example auto tools wouldn't work properly on Windows So they did a complete straight port of Basically what auto tools was doing Into CMake Since then we've come across and we've heavily modified of that modified that and made it much much cleaner and Annie You'll talk about that in a little in a little bit Where are we now As you can see we have the messes containerizer that supports job objects not quite the same level of isolation as You would get with Linux, but the best we can do on Windows. It supports Docker today. We're gonna demo that We have CMake and TFS long paths are working in a much less hacky way. You don't have to modify the registry any longer And we are there's a team of people that are actively working on messos every day I'm pushing changes into the master branch Add to that job object support So what I want to really make clear here is that we're trying to bring the best of both worlds like the expectations You would have using the messes containerizer, but also the expectations you have using Windows in general So the fact that we don't have C groups is not really a problem We're just doing the closest thing next to it which is sticking Processes into job objects so you could run a raw process on your Windows machine what that can give you as resource limitations I have that prototype right now You can set a hard cap on like your CPU and your memory usage But you don't get like the C group namespaces that you might have on Linux simply because that's not a thing on Windows We don't have namespaces in that way on the other hand the Docker containerizer totally works The Windows container support that the rest of Microsoft has managed to get working quite well is supported pretty natively here as well Okay So here's a brief demonstration or an example On the left we have a relatively standard marathon JSON file It's launching. I think you can maybe see okay. It's launching Microsoft IIS on a Windows system the window on the bottom right is IIS running and the window on the top right shows Docker running in An IIS container started by the messos agent Alright So here's a little more sophisticated example here. We have a hybrid DC OS cluster deployment So here when we're to trust when we're deploying services We use constraints like OS like Windows OS like Linux And that serves into specific services that are running The one that says Microsoft is obviously the run running Windows and the one that says Basic zero is a run that's running Linux in this example So here you can see we have nodes one one of the nodes of course is a Linux node one of the nodes is a Windows node and here's a perfect example of we haven't fundamentally changed messos in any way We're simply adding the ability to run a Windows workloads within your existing messos Linux clusters We're expanding that ecosystem Now we kind of had to squash two talks into one So one of this one of the things we've had to do for Windows was of course implement a new build system auto tools On Windows is not really a thing. It wouldn't be a proper build windows You generally use you know visual studio solutions or whatever you're targeting and this actually worked out really well because There's been a movement in the greater open-source community and in maysauce itself to kind of start replacing auto tools It's a great system. I do love auto tools It's gpl, but see make is a lot higher level. It's a meta build generator It lets you define at a really high level your build with not much code And then you generate the build files for the target platform you're on if you're on windows You get MS build solutions that you can open up in visual studio if you're on Linux You can use make files if you're on Linux You can also use the ninja build a system if you want if you're over on Mac This doesn't work quite yet But see make supports generating X code solutions there as well So you could open it up and your preferred IDE kind of just however you want and the nice key part is you don't have to care What your platform is see make has a nice little consistent? Interface to use I actually I really like this. I think it's on my next slide It is on my next slide. I do want to point out while see make is brings a lot of consistency to building There are still I don't want to call them hacks But there are still fundamental differences between some build systems, right? so debug and release configurations are something I recently enabled in see make and Unfortunately, although most things are consistent There is a difference between like single configuration generators and multi configuration generators in a make file build system You configure a folder for your build at configuration time You choose if you're going to have a debug or release build so you can set a variable Other platforms like visual studio you can open that up in a drop-down menu switch between debug and release at build time So while there's a lot of consistency these two commands are different for the debug and release configurations And I'm also going through and documenting this so that people as we switch over to see make Well, if you want to learn how to change your build I'll have the explanation of exactly what you need to do here other recent improvements that I've made We're supporting the Java bindings. So up until maybe a month ago if you built to see make master That may us master on Linux using see make you couldn't use marathon because we didn't have Java support whatsoever So I brought over the Java bindings just so all the projects that are using those are now supported the other things we've done I've really to Been tired even to speed up the build when I started on this December last year or so the windows build would take close to an hour to finish So we have gone through and we added pre-compiled header support because one thing that does happen a lot on Windows is it the compiler chokes on complex headers if you use stout if you use mess us You know about stout our giant header only library So our pre-compiled headers have cut down a lot of the build time just helped me out with that as well The nice thing with it is it uses a CMake module called co-tire. We don't actually change any source files It's just a configuration time header that's generated and included on the command line when it's built nice and automatic After going through and fixing up the CMake dependency graph, which was a mess Our link times are actually on par with Linux I did a fresh build from nothing earlier today on a reasonably fast machine. It was about 18 minutes I still want to get that down, but going from an hour to 18 minutes. I'm really really happy So to dive into some of this before you can move on I wanted to point out a couple things CMake allows you to build using MS build and Very typically projects that use MS build actually check-in VC projects that are tied to a Specifically to a specific version of Visual Studio One of the nice things that CMake gives you is it just builds with the video visual studio already installed on your machine So you don't really have to worry about it now in our particular case with messos We actually need the latest version of Visual Studio because we're using some compiler fixes in the very latest version But in the normal world, you know when we move beyond Visual Studio 2017 You know conceivably if we don't have to take the newest version Because of some dependency you can be a little more relaxed about what version you're running That's actually a great point. We're not doing this just for mess us But also for a lot of our dependencies zookeeper. I ran into a problem of on windows originally We just added a patch to mess us to patch zookeeper on the fly to add a hard-coded visual studio 2015 solution That's not maintainable because as soon as I wanted to go to 2017 What was I gonna do generate a new patch and if somebody wants to use go back to 2015? Do I have to tell them to check out old source? No instead. I deleted the solution entirely I pushed a new CMake build upstream to zookeeper. I worked with the thing It was Michael Hahn of the zookeeper project. It's integrated I got it back to the version of zookeeper we're using and will be Patchless soon as soon as we can just update our zookeeper bundle. So we're doing this from top to bottom We're trying to do it right So to go into some of the more Specifics a lot of people might question. Why why switch to CMake? Why use it at all? I touched on this a little bit earlier. It's consistent. It's easy to use when I go to Linux I don't have to remember make check. I can just run CMake dash build dot dashes target mess us tests that might be a bit longer than make check But that will also work on windows I can go to PowerShell run the exact same command and it will build the exact same target for me So as a developer that higher level abstraction makes working a lot easier Yeah, it's really quite nice. It's built for C and C plus plus And what I mean is it takes care of a lot of the Rote things that you might be used to an auditful is having to specify all of your headers. That's automatic Your compiler can look at your source files and deduce what headers you need to include on when you're building that object, right? Well CMake just runs your compiler and sees what you need to include So you don't have to specify all these dot HPP's all over the place. They're just gone. They're figured out for you It it's it's a beautiful little system I did yeah, I also supported ninja builds. There's a I'm not sure if he's a committer There's an active mess us community member named Benjamin Bannier who really wanted to use ninja It's an amazing little build system for Linux. It's like make but with a lot better dependency checking And it's really good at maximizing utilization of your processor without with automatic parallelization Without a lot of thrashing going on so ninjas supported as well Those are now these patches are all upstream in master branch of my sauce. You can use this today. It's I had a lot of fun with it so The other thing with CMake was if you looked at the code previously We touched on this a little early on the original port over to Windows They looked through the auto tools files and they kind of just wrote copied it when auto tools said include these directories They made a CMake file that said include these directories. Yes, that worked We had a working build, but it's not the way you would normally use CMake. They kind of Skip past all the high-level abstractions. You can use in CMake to generate a nice build system for yourself So I spent the last couple months Rewriting it. It's been completely refactored. We now actually use real CMake targets So but what I mean by that is stout is our header only library Usually with a header only library you need to remember that for every dependency that uses this header only library to specify in its Compilation flags like in your make file. Hey pound include not pound include dash I The stout folder you don't do that in CMake in CMake. There is a CMake list file for stout We say add library stout interface We have one more line that says target include directory the include folder for stout Anything else that needs to depend on stout can link to it like a normal library. You say target link libraries lib process stout It's picked up for you automatically It goes into the dependency graph when lib process needs when lib miss us needs to then link to lib process You don't specify it again. It's just picked up. There's a proper graph underneath it I also went through and did that for all the rest of our third-party dependencies as you know There's probably what 20? I think we counted different dependencies under a third party that we bundle or find on the system All those are properly imported as real targets that seems like it would add a lot of code But by doing that I was able to delete giant files Over a thousand lines of unnecessary code was deleted in my refactor and to make that number actually seem To give you perspective here. I added 200 lines So I replaced that like a five-to-one ratio of high-level build code It's well I have screenshots to show you what this looked like in the refactor The left the totally unreadable code was the totally unreadable CMake belt it worked It was a really really good first go it got us working on Windows. It started this whole process I couldn't have done what I did without the previous developers writing this But once we had it there, I noticed it was just unmaintainable people wanting to add a new third-party dependency We're giving up. I would go and help them. I try to explain how to do it. I'm like, you know what? No, I can make this easier. I know I can make this easier. This is the new code for the mesos agent That's it. That's the whole file. There's a Apache license header above it But if you go to the agent folder and source slave that is the file You'll find add executable messos agent main dot CBP That's the source file that you use to build the messos agent The only other bit of information you need to tell it is that it depends on the messos library All of the dependencies that lib messos has are picked up through the graph when you build messos agent It's just figured out and generated for you. I don't want to say magically, but it works like magic The add sub directory pieces are just the fact that CMake is kind of like auto tools The right way you do it is a CMake list file per folder per target that you need So that just tells you hey, there's three folders in here with more targets I included it because it was in the original Those are two different files actually both of which were required This middle file is completely deleted. It was the agent configure file It had a bunch of magic variables for the third-party information I just get rm'd it and all that information much of it was redundant the link directories The target link libraries the add dependencies. It just comes from the graph. They include directories That just comes from the graph it turned into those two lines. It was really something The other comparison I want to make is to auto tools, which is a nice little build system But like I said CMake is a really high-level Metabill generator it figures out your headers. So this is the Relevant parts of the make file that I could find for building the messos agent in auto tools Comparing it to the right-hand side over here. This is a lot easier to reason about and maintain those headers are just Figured out the proto libraries are part of the messos proto bus Which is the library that live messos links to why do you need to specify it again? You really don't why do you need to specify the messos CPP flags for the messos agent if those are project-wide flags? They're picked up for you. So CMake is a nice build system. It's far from perfect. There are hacks. There's always hacks in a build system I'm going through and documenting those they're commented in the code But part of my hackathon project on Wednesday here was to actually pull all of that out I'm also gonna have a CMake bag example to show you. Hey, you want to add a third-party library? Here's how to do it I Don't have a slide for third-party libraries, but I wanted to point out that a lot of what I did To fix the third-party libraries was to localize the information So for G log one of our third-party dependencies that takes up quite a bit of setup the setup used to be split among a Few different files in lip process in third-party in the source files Instead when done properly, it's a set of maybe 12 lines of code and third-party CMake lists that say hey We have this project called G log. It's a shared library. We build it in this way The source comes from here. We import it as a CMake target Stout later can just say target link libraries stout G log anything that links to stout again head or only library But in may a sauce that doesn't and CMake it doesn't matter Picks up the dependency for you and just knows how to link it and how to build it. It's fantastic Now CMake is really really cool, but it's obviously cool. I love it the other work that we've done to Improve the mess us build for all of us is to add a review bot for Windows in January I say every three or four days I pulled master sources and somebody committed a patch that didn't build on Windows and I had a drop what I was doing and fix it again And I don't blame anyone nobody was testing it for them and I can't make everybody test their stuff on Windows So instead we added a real review by it's like the Linux review bot you know and love for Linux It just does it on Windows for you as well It's been up and the amount of times I've had to fix a build break and master for Windows has gone down dramatically I think it's like once a month now We've also been working to get nightly builds of the missus binaries available for Windows So if you want to test this project, we'll have binaries available that you can just go and build Part of what I want to work with being noticed. We're getting Apache Software Foundation CI up to right for packages Eventually as this project goes along. We'll have packages you can use to install mess us to I Think that was my next slide continuous integration for Windows working with mess a sphere specifically Joe He also helped us set up CI for Windows as well. So we know when our tests are passing. Oh I think it's in the next slide, but Do you want to talk about the test that for you? I was gonna point out we have not as many tests on Windows as we have on Linux 630 versus 1400 messes that's messes test. Yeah, I think it's like 1400 on Linux So at like a rough glance it might look like 50% test coverage, but we're actually Still in the process of porting mess us to Windows as we port components like as we port over lib processes We port over the fetcher. We're porting all the tests with it So that 630 tests represents a much higher amount of coverage for the components that we have working on Windows Maybe 70 80 percent. I'd have to actually dig into numbers But don't fret if you only see 630 tests. We we're trying to keep our coverage pretty high I'm gonna pass this back over to Jeff to give you a case studying Okay, so one thing I missed mentioning by the way is we're very engaged with the messers community We work closely with them our changes are committed to mainline after a careful review and discussion We are active on the dev lists. We are active in architectural changes that that are being discussed And basically our goal is to work very closely with the open source community to make sure that what we're doing is Something that everybody is comfortable with Here we have a case study C trip international really the point of this slide is just to point out This stuff works. It's there today. We have customers using it under production today So it's there it works if you guys want to play with it and see See it in action. You can do that right now today. We wouldn't necessarily recommend running it in production, but there are people doing it Okay, I'm gonna do a demonstration of a hybrid cluster Alright, so here we have a Couple of different nodes and if we select one of the nodes Go to details. You can see under attributes the OS is Linux and if we select a different node Here you see under attributes the OS is windows so we have some services here and using the the constraint that I mentioned we have Some two services set up one is for is with a constraint that it has to run on windows and one is for engine X with a constraint That it has to run on Linux So basically these are typical DC OS Services you launch it with a JSON file just like you guys are comfortable with and here is a window That shows it running on Linux and to prove it's real I will refresh and they're refreshed It's not cashed in fact here. Let's go to engine X and Suspend it and then we'll go to IAS and suspend it as well So now we have two suspended services and If I go back to Linux and try and refresh Cannot open page Working live demo here Interestingly enough you see the progress bar is having problems loading it will eventually timeout But in the interest of time, I'm going to come back here and I'm going to say Resume engine X I'm going to come back here and refresh Oh, oh It's still deploying There you go. Yeah, good demo. So here and then you can see that That windows is still having problems loading if you look at the progress bar. So I'm going to go back and resume IAS Wait for it to deploy this time Okay, and then here it's still having problems loading. I'm going to kill that and refresh There you go And so yeah, this shows IAS running on on Windows So that's the demonstration Did want to point out go back here go back to nodes This this cluster just had you know, it started is with just Linux nodes and the windows Nodes were just added to it without any changes to the master Okay, so what's next? We're actively working on adding authentication support hasn't I don't think that's made it to master yet But it's it's tested and working and I think it's out for review patches are up for it Krammd5 is supported and we're working on additional authentication methods using plugins just like on Linux Some upcoming DC DC OS services we're looking at is metrics Barton and navstar and I'm working on getting the fetcher deployed on Windows patches are almost up for that and I did want to Emphasize what Andy said increase unit test coverage by porting more tests I always start like when I started with the fetcher the very very very first thing I did was ported all the tests see what failed and I'm and and as part of my work I'm fixing those and I have something like 30 tickets in JIRA to look at all the tests that we may have ported and forgot To close the tickets on so yeah, we're getting through it the other thing we're doing of course is Planning the eventual deprecation of auditions in CMake for in favor of CMake so that we don't have to maintain two build systems Simultaneously, so that'll come along as well. Yeah, that's something that we're working with the community for it's going to be a while But but it is something that I think not just Microsoft would appreciate but the developers would appreciate as well Okay, there's some resources that you could look at if you'd like Github.com slash Microsoft slash messos has some interesting pointers for Messos stuff. Here's some DC OS links that you could look at as well There the windows repository is available as well and the ACS engine allows you to very very quickly deploy DC OS clusters Just takes a few commands to do that That's work that Paul's been working to enable. Yes are Paul Allen. Yeah Okay, so where are we going from here Really, we're very very customer driven. We'd like the customer in the community to decide if a customer has very very specific Usage scenarios. We would really like to engage with you and hear what they are so we can meet your needs You can join us on the Messos Slack channel The windows is a common one, although we hang out on most of them Right after this This presentation we're going to the booth if you have any particular questions Please join us at the booth and we're happy to discuss stuff with you. Thank you for coming. Does anyone have any questions? Why don't you give him the mic so it's recorded So you said now you could build messos agent on Windows when is the master going to be ready or is it already ready? A lot of the master code I know builds because most of the unit tests require spinning up parts of the master to build The messos master executable right now I mean the CMake file is just if not Windows. I would like to try it. It's something I'd like to experiment with we don't have a huge customer demand to have a master building on Windows because most of these people Most of you lovely people out there really already have mess those clusters with your masters up and on Linux and deployed And just want Windows agents added to it So priority wise it may not be our highest thing But the big troublesome part about bringing the master over to Windows is its dependency on level DB Which we've looked at and I like oh that might need to be ported so Depends on where we get with that Does anyone have any other questions? I'm just wondering if Like I saw it during the demo demonstration that it had I think one CPU 128 makes a RAM Are those actually being honored right now Windows when that's running either through docker or through the kind of native Windows deployments? Yes, they're real numbers that is the actual docker container Docker being ported to Windows. We just kind of shell out to docker So any isolation or resource limitations you set for the docker containerizer work The job objects That's coming. I have a branch up that I've tested I ran a process that is Supposed to use as much RAM as it can and I set with my branch a hard limit of like nope stop at a One gigabyte of RAM totally stopped right there So that's in process and we'll come along Probably in the next month or so, but we're working to make sure all those resource limitations are real live reported correctly and Can be used You showed a client that was already running this in production Are they running down as your container services? Are they running that on-prem? Yeah, the case study that we show there is actually on-prem customer and then currently we're also working on some DCOS customers as well. I think that will come in pretty quickly next I would imagine that the timeline to really let them to use our product DCOS In their production might be some time, you know April or May time frame, but the lower lower layer like missiles Layer has been in the production already Any other questions? I was curious About the GPU support if you guys know how does that work on Windows? No GPU support yet I like that you asked that though because now I can make a ticket to look at it eventually So if you have a particular request or a requirement I will encourage you to follow up with us on either a select channel Or you know send the email to and in the Jeff and then We'll work with you closely to support the scenarios that you want to support We have a lot of things at least there that we want to target next But if you have you know more concrete requirements, and then it can change the power tail flower Since our list as well We are open to listen and then we actually like to work with customers to support those scenarios So I was just curious about what what are the big items for moving or dropping auto tools that are left I don't have Python bindings working yet. So that might be a bigger item to work on Other than that We're getting really close actually. There's obviously a lot of effort to go into making sure the whole community can use C make comfortably As part of why I'm here. I want to ask you all to try it out I want to make sure that it works as well as it can for you. I think it's a great build system I know what's far from perfect, and I know I'm really the As the one who rewrote it. I tested it for my scenarios I try to test as many other scenarios as I could but please try it out in the real world give it a whirl Let me know if something's missing that you need so I can go at it We do have an epic tracking the Technical issues that we found so far, but yeah, there is I wanted to add I mean we do have review about running obviously that uses C make every single day In Microsoft we obviously you see make every single day because that's all that works on Windows So, you know it is working It it's working for what we need it doesn't have everything that auto tools have And I don't know if we're ever gonna completely replace that because auto tools does a lot We just want to replace what's actively used So kind of tying into what you just said when when do you think we can get rid of auto tools completely? Or at the very least switch over all the docs on the on the miso's site to say hey You see make like don't even talk about auto tools The number that I've heard thrown out talking to some maintainers is probably on about a six-month time frame to do that to say Hey by default, please you see make and use it without you know, of course deleting the audit tool stuff Beyond that we kind of need to see once people start adopting it widespread How long it'll take to get rid of all the rest of replace all the rest of audit tools? So six months ish to default any other questions? Thanks very much for joining us