 Good morning. Good afternoon. Good evening and welcome to another episode of the open shift administrator office hours I am Chris short executive producer of open shift TV. I am joined by two of my favorite teammates These are the people I actually work with every day Christian Hernandez and Andrew Sullivan, but since Andrew since this is your show and you know You've got your whole format and everything. I'll let you take over. Thank you. Thank you. I feel special now You should Yes, welcome to the open shift administrators office hour So this is meant to be your time, right? This is a time where you can come Talk to us ask us questions about anything everything, right? It is meant to be very much like office hours that you experienced when you were when you were in school, right? It's of going to ask the professor for help on whatever topic it may be so We welcome your questions and feedback and whatever it is that you want to chat about at any point in time However in the absence of those questions, we we generally have a topic prepared and Today I have both the privilege and excitement to host Christian to talk about windows containers And Christian what time is it for you right now exactly? It's it's not too early. It's 8 a.m Oh, that's all I act. Yeah. Yeah. Well, I actually had to I actually had to change my workout schedule just to appear on the show. So Working out during this time. Yeah. Yeah, I have a I have a home gym and usually today is leg day I did it yesterday. So never skip leg day as they say so So yeah, I'm very excited to be here. I you know, I didn't want to miss it especially since I'll be talking so Or Or else I would have just called in and did it from the gym. I don't know how I would have done it But we're here. I would have seen you done it while you were doing legs, right? Like that Christian's in the squat rack with a he's got one of those like old-school data center like drag around with a keyboard A little cart right like I have a little heart, you know with my with my console cable. Yeah So Now I have a mental image of you doing like jazzer size and listening to With with the headband and yeah, yeah, so before we get started The first first thing that I want to talk about is for those of you who are regular watchers or listeners of the show You're going to start seeing some Some branding changes if you will and starting next week So thanks to some of the hard work by mr. Short's as well as some others and the organization and the other Both this show the OpenShift and the rest office hour as well as Christians reoccurring show the the get-ops happy hour are being Promoted isn't the right term right graduated graduated. Yeah, we'll use the CF terminology Yeah, we went from sandbox to no, no you were in incubating and Yeah, so the result of that is we're getting a lot of help and a lot of We're gonna be fancying up the place if you will thanks to our friends and Red Hat's Markham and branding and all that other stuff So part of that you will see some changes in the way that the stream looks We will also be changing the name. So instead of the OpenShift administrators office hour It will be the ask an admin office hour But yeah, I'm really looking forward to it. We've been working with the creative teams, etc. Christian and I They've been doing just phenomenal and amazing work. I'm really excited about the way it's gonna turn out. Yeah So, yeah, it's just don't be alarmed if it looks a little bit different next week if you hear a different name It might be a smidge fancier, but yeah, that's right. Yeah, which is a stretch when I'm involved, right? I am not a true Was the last time you saw me not wearing a hoodie So it's it's what do they say it's a different label same great taste, right? So it's there you go This is not new coke. Yeah, we're not new coke same great taste just label looks maybe a little different. Yeah So, okay, so again just heads up warning same great show same great taste Times I'm ever yeah, yeah, just fancied up a little bit starting next week So the the next first thing that I want to talk about Which is kind of my my usual style or the usual flow of things here following up from last week So last week if you weren't aware we talked about Controlling pod resources through things like limit ranges quotas, etc So I will actually I will dig up the blog post for that right now. Thank you If I can talk and type at the same time, we know you can't so don't try too hard Yeah, it's well. Yeah, well, well, Andrew's pulling that up actually this actually came up with one of the customers I was I was helping with with they're like hey, I I'm you know, I have plenty of resources, but like nothing scheduling and I'm like, oh you need to watch this This office hours here talking about resources quotas and you know, try trying to schedule some of these things because You know, if you're you know, if you're 90% idle and you can't schedule anything you have you have problems there Yeah So I found that blog post I just posted a link into the chat there So the reason I wanted to follow up is kind of at the tail end there I was covering how to modify the default project that's being used. Let me go ahead and share my screen here So in the documentation we cover that in this particular section of the documentation and I was able to go through and successfully reconfigure the project You know, it accepted all the changes and everything and then at the very end I was trying to provision another project and show that it created those objects So we modified that default project to create a limit range and a quota and Christian rightly pointed out in the chat that I just needed to wait and sure enough about 60 seconds after the stream ended I created a new project and it worked exactly as it was supposed to so this is how it goes sometimes Yeah If you're going through the process if you were following along or if you choose to follow the documentation here that I have up Or that I posted into the chat Just be aware it can take a couple of minutes for all of the stuff inside of OpenShift to reconfigure restart and then behave The way it's supposed to so I mean you're touching your quiet Christian We'll say especially and I think Chris you're gonna make the same point especially if you're making changes that will require the API to restart so things like You know changing configurations or changing the way projects get created like you always Always look to the API pods and see that they're you know recycling right after they're restarting in order to read a new configuration So that's kind of something to look out for and that's not unique to OpenShift. That's just Kubernetes in general, right? Like any time you touch the API server, you're gonna be waiting for a little bit. Yeah. Yeah. Yeah, you have to wait till that rollout happens Yeah cool So the second thing that I wanted to bring up So because you know again for those who watch regularly I like to bring up things that I have seen come up recently that have been recent issues great things to kind of warn you about So strangely this past week I have seen multiple email threads around at CD performance issues These come up occasionally, but it's unusual to see I think I've seen three or four threads all in the last week around it So I want to real quickly cover the topic But really I'm setting the stage for in I think three weeks March 10th I think I said Before we started the show we'll have a an episode dedicated to Etsy D Nice and kind of digging into what its purpose is how it functions right how to size For performance underneath there as well as how to verify and validate performance. So The ugly yeah, yeah, and Etsy D is if you're not familiar with what it does it's sort of the the heart of Basically every Kubernetes that uses it right it is the persistent data store It is what tracks the state of everything that is running inside of Kubernetes and OpenShift and all of that So, you know a performance issue in Etsy D is really a performance issue for the entire cluster Yeah, and it manifests itself in a lot of different and sometimes unexpected ways So the first thing I want to link here is a KCS and I'm gonna post this into the chat that thank you is intended to lay out kind of the baseline set of performance requirements and Really, it's summarized into two different things so one is The we specifically call out storage performance or storage latency because it's usually the most common issue with regard to Etsy D But really the important one is we want to have here We see applying our request should normally take fewer than if I could come on highlight should take fewer than 50 milliseconds So when we do an Etsy D operation, you know, hey at Etsy D set or Etsy D put I think Etsy D control puts operation we want that to take less than 50 milliseconds for it to be completed and Doing that involves a number of different operations, right at a minimum It's going to the Etsy D leader and then the leader sends it to the two Other member nodes. Everybody acknowledges that they got it. Everybody then commits it and then it returns back So there's at least four different network operations in there and at least two separate storage operations First on the leader second on the two member nodes So all of that latency and aggregate should be less than 50 milliseconds in order to ideally avoid performance issues So just be aware of that's right. There's if you look through this KCS article, there's a number of different links in here to other things including IBM folks created a Here we go down here. Yo Jules. Yeah. Yep. So FiO. There's an FiO test that is specifically tailored for Etsy D like workloads Doing f-sync rights and that type of stuff So pay attention to those right that kind of sets the baseline of is my infrastructure capable of hosting Etsy D the way that I want it to But that's just storage, right or just network latency We want to be conscious of a lot of other things that are happening in in the infrastructure So one of the Etsy D performance issues I was helping to troubleshoot earlier this week was they were running GPFS for the storage and Etsy D would start to suffer when there was other big data workloads happening against that So effectively it was a noisy neighbor problem, right? And it was completely unrelated to OpenShift to they were using Rev as the hypervisor right now completely unrelated all of that It's you know being aware of of what's happening elsewhere and protecting the open shift workload as the case may be So sort of the extension to this is so what happens if you know my Etsy D does have higher latency, right? What if it's above this 50 milliseconds, you know, what does that actually look like? So a couple of things can happen So one if it's really high what you'll start seeing is Etsy D go through leader elections and During a leader election. It's effectively unresponsive and you know, kind of none of the normal Etsy D operations are able to to happen and OpenShift gets unhappy If it is slow, but still functional effectively what you're doing is limiting scalability So Etsy D Basically the more objects you have in the cluster. So that's a combination of nodes and pods and containers and all that other stuff The more objects you have in the cluster the more it's going to rely on Etsy D to be able to do its thing, right? Every node has an object associated with it. Every time something changes in that node. It's updating the object in Etsy D Right so on and so forth. So all of those operations Basically with slower storage or a slower Etsy D instance means that I can't scale as high so But I will I'll stop with that, you know, I've spent five minutes or so on that now just to summarize and say March 10th We're having a show dedicated Etsy D where we'll talk about all of this stuff in depth We'll look at some examples stuff like that of the real world and Etsy D To me personally Etsy D is just like a fascinating Thing right like it's it's a key value store and it's simplest component and it's simplest form But like the fact that it is the thing that powers Kubernetes clusters it makes it very very important to protect and isolate not isolate but Just make sure it has its needed resources all the time agreed And there's you know the tangent to that is you know things like should I You know reserve my resources and the hypervisor and stuff like that. Yeah, which that's that's a whole another conversation exactly So the last thing I wanted to talk about this was an interesting one that I'm surprised have hasn't come up before Maybe it has and people just figured it out on their own But what happens if I delete the template virtual machine that's used for an ipi deployment on-prem So this is my my vcenter here and I just I didn't actually deploy the cluster I just started the installer so it would create the objects and then stopped it So with every ipi deployment and it doesn't matter if it's vSphere or rev or whatever the infrastructure happens to be You'll end up with a virtual machine image that is effectively the template for all of the worker nodes As well as the the control plate nodes, but that after deployment that's less relevant So what happens if I accidentally delete this? And essentially the answer is well Obviously, you wouldn't be able to scale up any of your machine sets because it no longer has a template But you would be able to scale down So you kind of have two options right so one would be to Reupload a new image let it be whatever name it is Create or modify your machine sets to point at that new template And then do a scale down scale up type of operation That would reset it On the other hand if you know if that template was named and you should because it's defined in the The machine set definition just recreate the VM right reupload the ova. Don't turn it on reupload the ova Name it the same thing as before and you should be good to go Christian and I actually we Tangentially touched on this a couple of weeks ago between he and I talking about machine sets and whether or not they use the Name of the template or the uid of the templates So now if you happen to create a machine set and you use the uid Then you would need to definitely update that machine set to use the new uid But if you use the name then it'll automatically detect it and use it Awesome Yeah, so uh, oh And then the last thing that I have and I know I'm running a couple minutes behind from where I wanted to So the last thing that I had was uh, we got asked again about Disconnected vSphere ipi. So can I do is it supported to do ipi? On vSphere at a disconnected network and the answer to that is yes, absolutely It is fully supported. It is tested the documentation is wrong and it is misleading So if you look in the documentation, there's I think there's one page that says like Disconnected is only supported with upi right that's wrong. There is a busy end to fix that Just stuff like that know that it is fully tested fully supported the one thing you will need to do if you choose to do that is You will need to specify so you'll have to pull down the ova and then host it internally on that That disconnected network on a web server. So if we do an open shift install explain Install config dot platform dot vSphere So in your install config dot yaml You would want to specify the cluster os image And point that at whatever url whatever web server Location you are hosting that ova at and then it knows to pull down the image from there nice All right, I'm done. I'm I'm through all of my opening material if you will so It is now time to talk about windows containers. Yeah, awesome Yeah, so this is something that is particularly interesting to me, you know, christian and I have had a number of conversations about it We've both been involved in a number of of meetings around like marketing and positioning and all this other stuff Although christian is definitely the technical expert. I'm just a guy with a big mouth So christian, right, I definitely want to kind of open here with You know, tell us about windows containers. What what is what is windows containers? How do we use windows containers? What are some of the use cases for windows containers? Yeah, so um windows containers As you would be surprised to know is um, you're running your windows workloads On a windows node Using a windows container, right? So you're it's not running like dot net or anything In a linux container, right? It's not it's not running like a windows container on a linux Vm no, it's actually specifically you're running you have a windows. You have a windows node running a windows container workload, right? So The the idea behind this is that you have like a single fabric, right? You have a single platform open shift platform that manages all your linux nodes and vms and containers and then you the same platform manages your windows nodes and Windows containers and then both east west Network communication works north south works all of that stuff, right? So, um There's there's a few caveats In terms of running windows containers, right? So the the the first caveat is that For for our purposes for open shift. We only support windows 2019 right and then that this is the the long term What I call the LTSC right long term support channel for windows And it's specific to a specific version, right and I'll I'll go over that in the When I want to go exploring a little bit here what specific version right and And these really the use case for this really is that if you have If you have windows containers Right in your in your in your network, right in your um in your environment And you want to Run it in using a single platform, right? So it's really it's really very much bring your own containers, right? So like You have windows containers already, right? Or you have a windows like legacy net application that Or is already containerized, right? Or you want to containerize and you want to bring that over and run it on open shift You can do that running windows containers, right? So it's a very finite use case Um, but a very powerful use case, right? So so just to clarify So we're deploying open shift as we would always deploy open shift. So core os control plane Two or more core os worker nodes to host kind of the the normal services associated with with open shift like logging routers, etc Yep, and then adding Windows server 2019 Compute nodes worker nodes to the cluster for those windows container based workloads That's correct. So yep. So, uh, you got that 100 correct. So, um, it's really It's really having a mesh I would say I guess over overload a term of Of uh, of linux nodes and windows worker nodes, right all managed by open shift. So, um, and And as you'll see you interact with the containers and with the node pretty much the same way Um, as you would a linux node. So a open shift administrator isn't too Um, isn't too lost, right? So, um, sully knows I was, um, you know, I was uh, when not when I was when I first took this on, right? I was it was kind of like, um You know, one two three not it right sort of in our team and When I first when I first took this on it was kind of it was kind of, um For me it was kind of a little bit overwhelming because like the linux guy that never used windows and the guy that, um, never used VMware all of a sudden got assigned windows containers on VMware. But um, But I was actually as I was going through it As a wind as a open shift admin I kind of felt already familiar with the whole thing, right? So it was kind of, um um, so the It was a pleasant surprise on my end whereas like you interact it, uh in in a similar way, right? So um in order to get windows containers up and running We have like an entry point, right? And let me actually start sharing my screen here. I'm always bad at this. Hold on So and and quickly button green. So there's there's two different types of dot net, right? There's dot net dot net and there's dot net core And dot net core is the version that will run on either windows or linux Very much like a mac os universal binary Whereas traditional dot net is windows only Hey, you're using a fancy. Yeah fancy, uh, search shortcut Yeah, I'm using the shortcut because it's just easier this way. Um, let me hear. Although I don't use duck to go Streaming box. Oh, did we? Oh, yeah, that's no good. So hang on Yeah, it's okay. It's okay. It's just a black screen. We're getting an hour back. Yep. We're back. Oh, hey, look, we're back Just like that. Yep. There we go. Look at that short knows the immediate fix. It's amazing. Um Well, it's kind of amazing because well because it's it's when this used to happen It used to be like a 10 minute blackout, but now it's like, well, now that's like automatic. Chris just knows like All right, I just need to press these couple buttons and we're back on. Um, so I was like super weird It was like the streaming our streaming rig that we're using right now actually sits in Azure and I was googling dot net core and like I had like this weird like internet hiccup And so did the streaming box and azure for some reason. Yeah, I don't know. There you go I guess we'll see maybe it's in the same region Who knows? Yeah, it's not dependent on texas power Yeah, hopefully not. Yeah It's in east u.s. So that might be why there you go. Yeah So yeah, so at first 4.6 we were uh, we supported the cloud, right? So it's like aws in azure 4.7, which i'm going to kind of preview today is um, we're doing vSphere ipi, right? So now we're bringing Windows containers to vSphere ipi The reason I say ipi specific is because ipi automatically Implies that using dh2p. So um I don't I don't know about you andrew I always get the the question Hey, can I use static ip's with Whatever ipi and it's like you're you're literally you're literally contradicting yourself when you say that Because as soon as you say uh static ip's you mean upi So just take ipi out of your out of your brain there. So um Yeah, dhcp is required for all ipi Yeah, so We even have a bite for that. So uh a bit of video a clip for that. So um And like like some of the things I say like you were on a windows server 2019 long-term uh support channel. So um And then but really the what we say the entry point to get all this is the windows machine config operator, right? so the windows machine config operator is an operator that is um It is a version of the machine config operator, but specific to windows, right? So it's kind of like a a um a version Of the mco That knows how to talk to open shift, right? Um, that knows how to talk to windows and open shift and it knows how To orchestrate all of this for you So um kind of the the workflow here is kind of outlined in this little diagram and you can um if you want I can just put in the chat, but it's it's part of our official docs. Anyway, um You install the operator and then the operator, you know goes to work, right? It'll log in to the windows virtual machine. It'll um It'll copy all over all the binaries and needs, right? Basically all the kubernetes specific binaries And it'll set up the kubelet to talk to open shift. It'll set up the the c and i Um, which we'll talk about in a little bit Um, it's it's a hybrid overlay that needs to happen and then um, the then it sets up the qproxy in order for it to configure all the networking stuff around it, right and then It'll register it to uh to open shift, right um in a little The the what we call it wmco, right the windows machine configure operator will Um approve the csr, right? So if if you've if you've ever done a um I was good. Yeah. Yeah. Well the the csr approval is for I'm andrew. I I always type this out now. I have trouble saying it for the uh platform agnostic Uh, upi non integrated installer formally known as the bare metal upi installer If you have ever done If you have ever done that you'll know that you need to accept the csr, right? So but the wmco does that for you Upi you have to accept csr's too. So yeah upi. Yeah Yeah, so and and so the the clarification there and we've talked about it a couple of times on the show is what we call bare metal upi is the More technically the non integrated platform agnostic installer. Yeah, right. So essentially it deploys open shift with no integration No awareness of the underlying platform And it can be as a result. It can be deployed to anywhere Is also the only way to deploy across multiple hypervisors or virtual and physical that type of stuff Yeah, yeah, so we have to be specific because it could you know, a lot of times we trip each other up Yeah, because there's a difference between bare metal What we commonly call bare metal and bare metal upi because bare metal upi does require actual physical servers and only supports physical servers Whereas bare metal or bare metal upi can be deployed anywhere It's not confusing at all. It's not confusing at all. We're working on it. Um And so these are kind of some of the things kind of what what andrew was kind of touching upon here this This this will give you the ability to have a kind of mixed windows and then linux workloads, right? Whether Whether you're running windows containers on like on actual windows Whether you're running like dot net core or Linux containers on on core os or if you're using red hat virtualization, right? the red hat open ship virtualization Installing a windows virtual machine y'all controlling that to the common The open shift container platform. So you're using that the entire platform there And this is kind of like our vision of how are you going to manage your your data center? Right? So you're going to have a common platform So and to be clear and when we look at this graphic that that you have there, right? There's two ways of doing windows with open shift, right? There's Or as of now, there's two ways or shortly. I guess because it's released with 4.7, right? Officially. Yep. Yep. Yeah. So shortly So you can add windows nodes to the cluster and then deploy windows containers directly to those So that of course depends on your windows application being containerized and server 2019 compatible, etc Or you can deploy windows virtual machines using open shift virtualization And of course that is predicated on, you know, it's still a virtual machine. You still have to manage the operating system It's probably a one to one ratio As well as your open shift cluster or whatever worker nodes are hosting those virtual machines need to be physical machines as well Whereas with windows containers on windows nodes, you would be able to have A completely virtual cluster if you wanted to Yep. Yeah. So This is kind of the the view of what we have, right? And so kind of just moving along here Like I was explaining before kind of the windows node services. We have the kubelet the cni the bootstrapper, right? Because it kind of makes it That that's kind of like the the magic that happens behind some of the stuff like you, you know Being kubernetes open shift admins, you would know things like the high the kubroxy and the kubelet, right? But uh, one of the things I'll talk about, um, you know, in the example is I'm going to talk about the hybrid overlay in the in the bootstrapper here. So um So what do you need? So first and foremost, what do you need in order to enable the? The container workload, right? So the the the wmco. So, um First and foremost, you'll need to configure hybrid networking, right and you do that as part of the install process So if you are So if you are on if you want to run this on vSphere, um out of the gate, you have to install from scratch, right? Because 4.6 we didn't support vSphere now 4.7 we do, right? Um, so you have to Out of the gate create a new install, right? So to answer the question of can I upgrade from 4.6 and 4.7 use windows containers? The answer is no because you're probably running a um different version of the vN and you probably don't have hybrid networking configured and you can only configure those on install So You have and the first step here right now. I'm going to kind of go through this is that you have to create an install config um You have to modify the install config and then you have to go and configure the um The hybrid networking, right? So let's let me drop down to a terminal here First you need and I did this before Okay, so um oc version you need 4.7. I'm using the release candidate obviously, right because 4.7 is due imminent, but not today so I have to use the um The the nightly bills, right the release candidate and you also need the um The shift installer 4.7, right, right So um, first foremost, I always love to create a um Uh, a directory where I'm going to install things, right? So this is kind of like a working directory if you've done an install of open shift before Uh, this should be familiar, right? So I'm going to do a open shift. Oops, open If I can if I can type open shift Welcome to my world install yeah create right so first um, you know, we're we're gonna do these you have to do the install in uh in um In sections, right? You can't just create the just run the automated installer, right? It's because we're doing network customizations And so I do a create install config Um, and this will create the installer, right? So I'm going to use Any ssh key, right? It's whatever the one you use and this step is um in these steps here This this prerequisite step is the same no matter where you are aws azure Um or a vSphere, so I'm going to choose vSphere because this is the world premiere the vSphere And then if you've done if you've done this installer um Before this should look familiar Oops, uh, there's a request to Increase your font size if you could please well this really this one. Okay. Yeah. I think it's because the uh We have the new lower third and it makes Yeah, we have smaller. We have kind of crunched the window a little. Yeah Yeah Let me know if that's better folks. Yeah Uh, and then yeah into your password. So if you um, you know The the prerequisites for the ipi is always is vSphere ipi is all the same, right? So you should know this Um this information beforehand and it's all in this data store. So virtual ip address for ipi. Which one am I using? Is a good question I'm going to do a dig on my my other window here. There you go There we go 102. Okay This shouldn't matter because I'm not going to actually do the install because since this is like a cooking show I already have a cluster up and running but and then for the ingress, right? um I'm going to do 103 base domain, right? This is like whatever Um, how does age and then ocp4 is the name of my cluster my pulse secret. I should I have it in another window Okay, so that's fun. I can't There we go. Got to dig through all those tabs. Yeah, I know Ignore the guy behind the curtain. Um, all right. So this creates a uh an install config um file here, so Uh, if you grep, I'm not going to open the file because it has like you said like my password and um But if you uh, open shift sdn, right the default for is the open shift sdn We're going to change that. We're going to use, um OVN kubernetes, right? So that's the um End of which is fully supported. So um, I'm going to run the set command But really you can just open the file changed out this one little I'm pointing at the screen change this one little section here To read oVN kubernetes, right? So as soon as I run that that said So let's touch on oVN kubernetes for a moment christian. Um, so open shift sdn is sort of the original open shift sdn is the name implies, right? It's uh OVS based it uses vxlan as the encapsulation, right? It's it's been around. It's been in open shift for ages So with 4.6 was a 4.5 or 4.6. I think it was 4.6 We released as generally available Uh, and fully supported the use of oVN kubernetes with open shift Yep, uh, so as the name implies it uses oVN And it also uses geneve as the encapsulation Yeah, and right. There's just a a number of other changes differences Compatibility is not the least of which is oVN kubernetes and oVN works on windows as well as on linux unlike Uh, uh open shift sdn and oVS Yep So the core of it and which one should you choose is ultimately Which one would you prefer right there in the docs? There are some It does list out some differences between them and which may be Specific features which you may need or may not need which can influence that decision Um, one of the driving factors behind that change is also Quite simply oVN kubernetes has a much larger community an open shift sdn is It was created for it is used by open shift oVN kubernetes is used by lots of things So that that helps us out there too. So and and also to drive your decision is that to use windows containers You have to use oVN kubernetes. So that may be a deciding factor. That is if you want to use Yeah, so you absolutely have to use it right Yeah, by the way And so once once you set the network typed oVN kubernetes, uh, you'll run a open shift Install create manifest right so from this install config we're gonna create the manifest And this will create Two directories a manifest and open shift right and these have various a tree this here various yaml files That is used to configure the cluster right because with open shift 4 we use kubernetes to install kubernetes So therefore everything's a yaml file. So um And so Here if you ls Let me ls the manifest and then do a cluster cluster network Right, there's two cluster network files um, we need to create a third one right and this one is um cluster network 3 and then so now you should have uh three of those right so A blank one's not going to help us so um So we're going to do this configuration right so let's so this configuration is should be the same No matter which platform you're on unless you made network customizations, but since you're not making network customizations because this is ipi This should be fine. Um So going back to the doc this this is all all these steps i'm running is in the documentation um So the one thing you do have to keep in mind right like to say if you are setting this up Is the cluster network right which this is the um the ip address range that the pods are going to get assigned Is different than the hybrid A cluster network right so what's what's happening here is that you're going to have a We're configuring an overlay network right so we have the the um The the network that the linux pods are going to use and then we're going to have a network separate That's the windows pods are going to use and we're going to have this overlay networking for the um for the west east east west whatever you call it uh traffic to go through right so if a Linux pod wants to communicate with the windows pod or vice versa. It'll use this overlay network here so um So once that's done Then you can actually go ahead and do an uh open shift Install create cluster right and this will actually go ahead and install the cluster once you made this network customizations So it's not hard. It's it's not like black magic or anything but you do have to do these steps and they're outlined in the documentation so if um And this is the same again. Like I said, no matter where you go. So um, I'm not going to run the installer because that'll take forever as slowly knows but um question Real quick. Is there is there any reason you would configure hybrid networking without windows nodes? Um, that's a good question. I believe you might have a reason to do that. Um And I don't know if we're planning on using hybrid networking when things like sub-mariner comes comes along I believe sub-mariner takes uh Submariner sub-mariner. I don't know how to pronounce it. But I believe yeah, I believe it it'll it'll handle that for you and I don't think you need to create the hybrid networking but um Uh, the answer is I don't know I don't know if you would for windows containers. You do need to I usually try not to ask questions. I don't know the answer for but that one. I don't know the answer to Yeah, so so we do have a question um from osama and I'm not going to try and pronounce your last name. My apologies So the cluster network is the one exposed to the customers Um, so christian, I don't know if you want to answer. No, okay. No Yeah Because no it's not exposed or no, you don't want to answer No Cluster network is the um Is is the network that the pods are going to get right? It's an internal network. Um only for uh for open shift I think it was Last week or the week before I talked about the on on this show The three different network types defined in the install config.yaml. So yeah cluster network You'll note in the definition. It is a subnet. So it's by default a slash 14 And then there's a host subnet section which by default is a slash 23 So essentially that is the block of ip's that's open shift kubernetes takes In slash 23's out of by the default configuration and assigns each one to the nodes a One per node for the pods to be connected to So if we were to And christian, you've got yours up. Um, are you connected to a cluster? Now I am. Yeah, see If you do like a you can look at the node ip's like you can do a oc get or excuse me the pod ip's if you do oc get pod dash a I think um, anyways, yeah One of the commands will show the pod ip's and you'll you can equate the subnet that it's on to the node that it's on So it is not It is visible, but it is not publicly exposed if that makes sense, right? You can't directly route to That cluster network. Yeah, I think if you do a error believe you do Anyways, I don't want to get a sidetracked here because there's a lot to go over. Um, yeah But yeah, I'll dig up a link to where I talked about that last week and I'll include that in the show notes So if I do it oc, uh, so I had I so to um I have like I said, if it was since it's a cooking show I I already pre-deployed a cluster And if I do a oc get network operator cluster at o yaml, right? Um, and I scroll up here. I can see my My overlay network is set up here. And like like you said, uh, solely the genie port is set there. Um So I'm using, you know, type ovn kubernetes with all my network customization, right? So Here this is this is just like a freshly installed cluster if I do oc get nodes I get, you know, the three masters three workers. If I do oc get nodes l kubernetes.io os equals linux I show them all my linux nodes if I do, um A oc version, right? It tells me server version 4.7, right? So I got all my my prerequisites Um, prerequisites out of the way here. So if going quickly over to the, um vSphere let me log in here Um Fun fact solely helped me get this cluster up and running when I first started with vSphere. So, um, oh So it's still up and running, man. Um, if you if you note here, let me make this a little bigger since, um so the installer creates a A directory Inside of vSphere, right? It'll have the the cluster id dash some unique identifier Um, if you notice here, it'll um I have, uh Like andrew said before I have my red hat core os default vm here and I also have Something called the windows Vm template It's not actually a vm template. It's like a vm, but you can use the template as well Um, and I'll go over kind of the the prerequisites there Sorry to interrupt you christian. Uh, so one thing to note there at the beginning I talked about that that image the core os image as well as the windows image So by default they're created as just standard vm's and then The machine sets or machine api will clone that vm Uh, it actually doesn't care if it's a A vm or a template inside of vcenter So if you prefer it to be a template convert it to a template And the reason why it's not converted to a template by the installer is simply because There is no reason to it it works just find the way it is. So if you want it to be a template because that Gives with your you know the way that you manage vcenter then absolutely convert it to a template and that won't hurt anything Yeah, yeah, and it'll work either way and the same for the windows, right? So, um So as I said before the um get routes the the entry point is the the windows machine config operator um console and so, um In order to do that you need to install the operator first, right? So let me go back here and then Let me open a new tab paste that guy So this is a fresh cluster as you can see um go back. No, I don't want to go. Um There we go I haven't even logged into the ui yet Go go go I should give me another certificate issue here is your v sphere up to the task Is my v sphere up to the desk? That's right. It's smoking right now. So I forget it's right beside you. So, yeah, it's right next to me wrong Um, I don't know the uh cluster Opinion for uh off Cube admin password. There we go You can do it. This this cluster is in public. So it's uh, yeah, you're safe. You can look you can look at my Should be Um All right, cool. So kind of the the um The the workflow to install the the operator is is dead simple. Um, you go to the operator hub you type in windows Right and uh, here I'm gonna use a community operator because um, you know, this isn't ga yet Like we're really close, right in the next couple weeks. You should see this But uh, what you won't need to look out for is you want to use a version 2.0 Right 2.0 is the one specific for v sphere if you're using version 1.0 That's for the cloud that won't help you on v sphere. So, um You know, you choose your channel Um specific namespace in the cluster pick specifics name space in the cluster because it'll install the operator It is pretty much automatic here. You don't really need to touch anything Um, the only thing you might consider is the approval strategy, right automatic or manual I always use automatic to keep my operators up to date, but if you want Uh to manually approve the updates you can set this to manual, but everything else should be fine um And this will go, you know, do the install, right, which you can actually watch here um Watch oc get pods Uh You can also check for the install plan. Yeah Yeah, and this will yeah, you can check the install plan. Um, I'm Basically waiting for the for the pods right here. Um So this this will take, you know, like a minute or two For it to install once this is installed. We need to do a couple of things Um One of which is to create a windows golden image vm, right? Um, I've heard this before Yes. Yeah, so you need to you basically need to create a Uh vm template, right? And then I have this have this little slide up here. I'll present Okay, there's one slide the world's smallest slide, right? Um Um So when you create a golden image, right? We're we're linux company. We're not going to tell you how to do this, right? Um, but we what we are going to tell you is what we expect to be in that golden image, right? So we expect windows server 2019 in the ltsc version Um This version all older. I'm not going to Read those numbers But we need that version or older It does work with other versions of windows server 2019. I've tested it with different versions But then you're we we don't officially support that, right? So if you if you want official support This is the version you're running. Um, and at you know security patches are always good But at a minimum you need this kb patch And this is specific for networking to work. So, um And the good folks at microsoft back ported some of this this fix to the older versions of windows So we need this kb patch installed. Um, again, if you're running the windows firewall We need a windows the tcp port Open, um, what is it 10 25 0? And this is just for container logs To go back and forth into the to the cluster, right? So ssh installed. Yes, it's kind of weird We're running controlling this machine with ssh on windows, but yes, we need ssh installed With key base authentication, right? So you need to create a ssh key Load that into the windows vm and make sure that you can ssh as administrator using that key So for your linux guys, this should be Very familiar to you for your windows guys Grab your local linux administrator to have them explain what's going on. So, um So you need the docker runtime installed on the windows vm We're going to switch over to container d As soon as windows server supports it, we're going to switch over to container d So this is only temporary to run docker runtime The docker requirement is because windows only supports docker run docker right now, right? Yeah, right now. Yeah You need the container images pre-pulled, right? So the reason I say this is because the container images for windows could be really big Like i'm talking like 10 gigs. I'm talking five gigs, right? And that'll cost timeouts So when you launch an application Windows application, it'll cause a timeout because it's just, you know, the docker pole is just taking, you know, long It takes a while to pull down 10 gigs from the internet. So pre-pulling those images on the server is very helpful to To to expedite, you know, the the deployment of your applications, right? And obviously vm where tools installed We use a vmware vSphere API to glean information from the about the vm And then you sysprep it right just kind of so for your linux guys grab your local windows admin nasty So you need to sysprep it right like this, you know, we're talking dev ops, right? Right where everyone's everyone's helping each other out here Um, you know, uh linux guys grab your windows guys and have them do a sysprep Um, and you need to sysprep it in such a way that it preserves all these changes you made I've ran into issues with sysprep where it starts deleting things Um, you know, because you're basically creating a template, right? And you know, it's like, okay Well, you don't need the ssh directory. I'm like, well actually yeah, we do. Um So grab your your windows guys And uh, have them do a sysprep and have them preserve all these changes you made, right? So Um, and again, I Go ahead. Just a couple of questions for you christian. Um, sure. So Is what what is it called? It's the Windows server with desktop experience. Is it is it required to install the gooey or can you use windows server core? Yeah, um, you can use windows server core. You can use both. Um, I'd use windows server core because um, that's what the engineers gave me so um, that's So I just work with that and I guess um And then sully has some some funny stories about me trying to work with windows without a gooey Which is weird because you think I'd be at home As a linux guy, but like since I don't know my rate my way around powershell that well It was kind of weird navigating, but after a while It was okay. Um, and then what about domain membership? So, um in this version, uh We don't support you joining to a domain. Um, that is on the roadmap the things we will be called by oh H, right bring your own host Then that's for the upi installer for all upi installers. Um You'll be able to join it to a domain and use uh, like a like a domain administrator in order to do all this So, um, but for for this version, um, it just needs to be like a standalone windows machine Okay. Yeah, I I wonder and you may not know the answer to this. I wonder if we can apply You know local Group policy, right? Oh, yeah, I was just thinking about her and yeah type of stuff Let's see. Well, um, I do want to give you, uh, sorry to interrupt you. Uh, just a five minute warning By the way, cool Yeah, um So, you know after doing that, right? I got the community operators is up and running I've because it's a cooking show. I created a windows vm already and sis prepped it and did all the the fun stuff Sully has funny stories about me trying to sysprep a windows machine as well, but I did do it um I did do it and then if I if you go over to the um, let me move this over here If you click on view operator and you scroll down, this actually gives you a lot of good information in the operator Uh, the engineers had a good job of providing as much documentation as possible here. Um One of the things is that you need to upload the ssh key, right? So the ssh key you used To sysprep the windows machine You need to give that to openshift, right? Because then openshift needs to know how to log in to the windows vm So, uh, to do that here, this is up and running Uh, to do that You need to change this here My key I have home Home directory ssh and I have uh windows I created a windows. This is my windows ssh key, right? I'm gonna upload the private key Um, because the public key is on the windows vm, right? So that's created cool So now I'm actually ready to create a windows machine set Right, so create a windows machine set. You'll need some information. Um, like your cluster id The windows vm name that you're gonna, you know, the this guy here this guy's name. You need to know that um and uh, you need to know things about your The v sphere cluster or vcenter cluster this information is easy to glean from if I do an oc get a machine sets From this machine set, right? Because it's the same it's in the existing one here Yeah, so you can glean that information. Um, Open shift. I'm gonna cd to the night least here. I have a simple script here um That creates the machine set for me. So I know someone's gonna Tell me can I have that script and it's like it's it really only works in my environment, but I guess so, right? Um, so But fine, right, um, but essentially what I'm doing is our work for me Yeah, so like it's essentially I'm just taking information that I'm gleaning from the the actual, um Server itself, right? You can take it from the other machine set. So I run this um, and came machine set This takes a little bit always because I'm doing oc gets and stuff. Um, so let's let's take a look at that real quick and so, um A few things I want to call out that are very important line six. I use wind worker um And the reason I use wind workers because it's nine characters The reason I use nine characters is because I have I can't have more than nine characters. This is um a limitation on how window, uh, how It's actually not a windows thing. It's uh, how open ship formulates these names plus The limitation on v spheres names for windows vms for whatever reason. So, um So that's one thing you need to call out here. Everything else is pretty much the same for a worker um A few more things I want to call out here the template. I'm actually using the whole path to the template You don't have to as Sully said it'll kind of find it yourself I have multiple windows templates And so, um, I've had the problem in the past where chose the wrong one And so, um, I just gave it the whole path And then this information like I said, you can glean from your standard, um Machine set here. So if I do an oc create of this machine set And then I do an oc get a machine sets If I could spell it right Um, it has it created this machine set and if I do an oc get machines Right, it's in the provisioning phase Right. So if I go to my ui some things that are happening here, um, it creates this machine set Sorry, this machine this vm In v sphere in that directory and if you go to the events here, you can see that this cloning process is starting um So essentially it's gonna it's gonna, um It clones this vm, right as soon as it's up the windows machine config operator will log in to this vm and it'll run the setup process here. So, um So this will take like a minute or two, uh, I don't know if we have many more time I don't know if you want to go over just a little bit or we can go over a little yeah Okay, I want to I want to have the big reveal, right as like you'll have a windows, um Any windows vm up and running. So do we have a end in video posted anywhere? I made a video It's not posted yet, right? So we're waiting for the g essentially we're waiting for the ga, uh of this. Oh, okay Um, so this is your sneak peek here of this. Uh, so there we go. So it started it up right and this will go through, um You know, it's sys prep E things I don't know if there's a windows admin on that can explain what's going on right now But this is doing sys this is doing the sys prep stuff, right? Um, I don't know solely if you know what's going on It's doing all bunch of stuff Not the least of which is configuring hardware. Basically the sys prep removes all the hardware configuration and user information and all that so it's Detecting the hardware setting up drivers adding the default user all that other type of stuff Yeah It's all good It's just uh It's just for well and also for us and for us. Yeah. Yeah. Yeah for this admin. I have no idea what's going on, um, right? so So i'm like, I just you know This is magic for me. I run sys prep and it did stuff Wait, was it was it uh solarius that had the Unconfig file it was like if you put a Like slash config or something like that. Oh, yeah effectively the same as like when the next boot it would say Oh, I need to un-configure everything and basically templatize myself. Yeah, that's that that came to windows Sorry windows at linux as well. Um, there's like a file you touch and then um, yeah, see look here. It's doing This is pretty much hands-off right like i'm not i'm not touching anything. It's doing Whatever it is it needs to do And so Yeah, so this is and what we're watching here is turning away the the windows node operator doing its thing essentially so and so like if we go here if you do an oc get pods in the open shift windows oops machine evaporator and I do oc logs of this guy um I noticed you have bash completion set up in there very nice. Yeah so, um, yeah, so here it's um, it's this initializing ssh connection Right, you'll see a bunch of errors at first if you're telling this this log because like it's just essentially waiting for machine to come up and so This will this essentially is configuring now as soly was saying is The machine config operator is copying the binaries setting it all up Doing the plumbing, you know, all that good all that goodness here. Um, Yeah, see here you can see It's copying over stuff to slash k I guess that's the standard location for property stuff. So Nice Well, it's the one that we're going to use regardless, right? Right It's the one we're gonna. Yeah, exactly. It's the one we're going to use regardless here. Um, so if I do oc get machines It's provisioned if I see oc get nodes not up yet. No not up yet. Um, Just tell that log here So, uh, I had so much to show you and like we're essentially running out of time I was going to show you uh storage and all that all that fun stuff that that happens a lot Yeah, and it's so yeah And we'll make sure So when 4.7 goes ga when you get that, uh, that video posted we'll make sure to to link that and show it And I think you've got a blog post planned as well. I have a I have a blog post. I have a video I have all kinds of content ready for you guys. So nice. There we go. So you get this boots Joe missing out on all the cortana conversation. No, no or not. I don't miss that at all In fact, if they completely did away with that, I would be very happy It would be funny in the logs. It says, hi, my name is Cortana. Like it would be In text, right? You'll see that would be hilarious. I am sentient now Yeah So if you do oc get nodes a kubernetes Um, i o os equals windows Right now you have a windows worker, right? Um, and if you Uh describe the node Uh describe the node will give you like which version, right? You notice that i'm using an unsupported version, right? 18 3683, right? Like I said, it works. I'm just unsupported. Um using docker runtime Coolit version, uh things like that Uh more importantly, if you do a grep Of the external ip you can actually ssh into this guy Right, um Or rdp if you were using uh If you set up rdp, yeah Or psremoting or any of those fun things Uh release date for 4.7 is soon soon. Yeah So i'm i'm hoping we can use next week's episode of the open shift administrator officer Sorry, the ask and admin office hour that started this week Um to talk about 4.7 So whether or not it's released, um, that's probably what we'll be talking about next week Yeah And so so here we go. So if you can ssh into it Do run this get process and then I don't know why they don't use grep, but here we go. Um, you use that You can see all the process You can um, see all the other processes processes that are that are on this right here And if you do a docker images You can see the images I pre-pulled and then docker ps, right? There's nothing running yet. Um, so I did go over time. Um, so it's okay There's there's so much to show We'll send you a bill Right Oh All right, there we go. So cool. Awesome. So yeah, um So christian, I know you've got some materials that'll be coming out Like I said a moment ago, we'll make sure to share those once they become available So that way we can really get an end to end And maybe if I can trouble you to uh to adjust your workout next week We can finish finish this demo. Wow. Yeah changing your leg days two weeks in a row. I know All right, that's right. Um, all of that being said, you know, thank you so much christian. Really appreciate you, uh You waking up early adjusting your workout and joining us today to talk about windows containers and windows nodes and open shift Another problem So for our audience really appreciate, uh, everybody joining, um, please look for the show notes Blog posts those have been going up Friday morning on open shift.com slash blog and we'll include links to all of the things that we talked about here Time stamps to the video all of that other stuff As well as any other materials that happen to become available between now and then Um, as always if you have any questions anything that came up for you during this stream If you're watching after this after the live stream and something comes to mind Please feel free to reach out on social media. You can reach me at practical andrew on twitter Or via email andrew.sullivan at redhat.com Uh christian alter you under the bus if you want to put out your contact information Yeah, so if you go to um on twitter i'm christian h814 Or you can you can always email me christian at redhat.com. So I got the cool email address. So it's nice And I am uh at chris short with two s's uh on twitter and just see short at redhat.com I should try and go get chris at redhat.com, but something tells me that's taken Yeah Chris right maybe yeah, I'm just thinking our cto might have grabbed that one already. Oh, maybe. Yeah, that's right Thank you all for for watching today Please next week same time same channel same host different name We will be here and looking forward to it. So thank you so much. It'll be great. Thank you all for tuning in and we'll Next up. Uh, just want to let you know 2 p.m. Eastern 1900 utc the scalable multiplayer game design with open shift the one and only eric jacob's will be Broadcasting that one and hosting that with a bunch of other folks. So definitely check that out I always find it fascinating what I learned during that show. So yeah Thank you all. Thank you christian for joining us earlier this morning and uh stay safe out there everybody Bye everyone