 Good morning. Good afternoon. Good evening and welcome to another episode of ask an open shift admin I am joined today by two of my fellow teammates here at Red Hat Andrew Sullivan and Christian Hernandez Andrew How are you doing? I am doing well It is a great a phenomenal a fantastic day here on the ask an open shift admins live stream So I'll get to that in just a moment, but not before a Christian, you know, welcome to the show yet again Yeah, yeah, I thank you. Thank you. I always love coming on here. I always say that you know my backgrounds in You know operations being an admin, so I always like to jump back jump back in get back to Well, my first love was before this whole weird get-upsy thing Now you're a double trader if you will right here. Yeah talk about windows and you're the get-ups guy It's it's the two sides. It's like a dyad right two sides You're off my Christmas card list. Yeah I'm off that so I won't get a picture like you have up there of the squirrel. Yeah So Yeah, it's I was actually talking with my wife earlier this morning talking about so the room is a Shade of green so I couldn't stew like a green screen in the squirrel picture But I could do blue and then I could just randomly phase it out for other things, you know using some sort of software Yeah, that would be fun but no so I was extremely exuberant here at the opening and that's because I realized about a week ago actually that Today or yesterday actually the 14th September 14th was the one-year anniversary of the ask-and-open shift admin office hour Oh, right. Yeah, like it's pretty wild, right? Like yeah in this for a year now and thank you all for watching and tuning in Yeah, very much so I I took the opportunity to collect some metrics around what we have done in that past year So if it's not obvious right from the title of this stream, this is episode 43 so effectively in 42 shows we have been averaging right around a thousand views per video Wow, which is pretty good pretty impressive, right? I mean, you know, this is a Highly technical show supposedly. Yeah. Yeah, and so for something that is both Technical as well as a live stream, which you know, the three of us have had lengthy Conversations around things like content discovery, right? How do we make sure that people can find this information afterwards? You know, how do we make sure that it is, you know relevant and evergreen, you know marketing people love love the term evergreen You know, so I feel I Am thoroughly humbled by our audience and yeah, your willingness to participate and our ability to you know Just get the information out there and hopefully it's helpful You know, you two know me really well. We've been on gosh countless meetings together of I tell me if I'm wrong Like I love it when you all call me out and like that's what we're here for so Yeah, it's um, it's been a lot of fun. So yeah, 40 It's a just shy or just over 41,000 views on YouTube alone So that doesn't include Twitch that doesn't include back when we started when we were doing What is the Twitter thing? Periscope, yeah Yeah, we we did Facebook for a little bit. So it's that's just YouTube, huh? So Uh over the last 90 days Because that's that's all we pulled Across the videos that are available. We have 1964 Hours of watch time. That's so sad. So that is the equivalent of Christian sitting in front of his computer for 82 solid days non-stop watching nothing but this So It's how I became And then just a couple more statistics around that so So a couple of top episodes so all-time highest views is episode 20 Well, where we had Peter Lauterbach on to talk about OpenShift virtualization So and that one has just over 3,200 views, I think The next post popular one was one that I only have participated in which I find hilarious And that that was episode three where Christian and Reese Oxenham And Reese will be joining us in a few weeks to talk about bare metal nodes But Christian and Reese hosted the first hour talking about the assisted installer Mm-hmm. And then no surprise the third most popular episode Was when Robbie Jerome joined us to talk about VMware OpenShift on VMware Which we've been talking about bringing the VMware folks back on so I don't know if Robbie will join us But perhaps one of the folks on his team will be able to join So lots of stuff happening there You got a question in chat today. Yeah, so you're done. Okay. I've got I've got two more statistics to cover So that the top couple of episodes from just a minutes watched So in the last 90 days again, that was all we pulled the watch. This is statistics for if I could speak this morning So the the most watched episode has been the high availability episode Yeah, and I think that one Yeah, that one had now I don't have it up anymore, but that one was the highest via like a 15% margin The next was compliance and security and then last but not least or in third place I should say was ACM advanced cluster manager So kind of expected topics there, right? It's I like looking at these things because it gives me an idea of what's interesting to you all so And always never please don't ever hesitate to send me Suggestions or requests for topics. Yeah, I love those. I'm going through doing show planning now I've got a framework of topics out through mid November and then it starts getting weird because of holidays and PTO and all of that other stuff, but we have entered what my wife calls the burrs The fun months with all the holidays. Yeah, the burrs Oh wait, September, October, December, December. Yeah, like So yeah, don't don't please don't ever hesitate. You can reach out on social media practical Andrew on Twitter or Andrew dot Sullivan at redhead.com via email. Don't ever hesitate to reach out with a request or a suggestion for a topic. And then the last one that I have is kind of a simple one. If if you don't know, we stream simultaneously to three different places. So we stream to twitch, of course We stream to the open shift YouTube channel and we stream to the red hat YouTube channel So long term because twitch, of course rolls videos off open shift the open shift YouTube channel accounts for about 64% of our traffic of our watch time. So It's you know, open shift folks love to talk about open shift or hear me talk about open shift anyways Well, yeah, it's who would have thought the open shift channel would be the open shift admin show would be popular on the open shift Yeah, yeah Which is not to say we don't we don't love and appreciate our our red hat YouTube channel watchers. So that's right. No, absolutely Jump on over. Yeah, and I want to watch is fine I will also for those who've been watching for a while. You remember back in the it's in the mid 20s episode 24 25 something like that was when We added that Added the red hat YouTube channel as a streaming destination and that team has been simply phenomenal Yeah, so all of the like that fancy intro graphic that you see all of the other stuff Like that's all them and they've done a just a great job of both the public-facing stuff as well as helping you know me and Chris with the organization and getting better at the The promotional side of things and all of that. So I can't thank them enough All right, I'm done rambling and talking about us and the stream So Chris you said that there was a question. There is a question, but I would like to say one other thing Congratulations to Christian last week was his seventh year here at Red Hat and that is Thank you Yeah, it feels just like yesterday. So yeah, no, I appreciate that. Thank you in five more months Red Hat would be my longest employer. So here we go Wow, how many of those years have you spent watching me? Anyway, one seven one seventh right? Well, congratulations Christian, thank you seriously, it's great to have you here at that. We really appreciate you So the question from our hope nine today is The only supported way to install quay is using the nuba operator using the OCS operator to get nuba installed Does this mean that this is a way to use nuba that is supported for other things than quay It's a really good question and I don't know if that's so the yes So the answer is that nuba supported as long as you're using OCS. Yeah So if you're using OCS, yeah, I use nuba it's object storage. It's oh, I guess ODF have we made that switch to ODF in No, and like marketing. Yes. Yeah. Okay. So we're we're halfway there So yeah, ODF If you're using ODF, yeah, you can use the nuba you can use that for You know for your applications if you'd like so Yeah, and just stand alone. I don't think so only in quay. I think yeah and just to clarify So ODF is OpenShift data foundation Which is the rebranded name for OpenShift container storage because it's more than just What was traditionally OCS, right? It has evolved to include a bunch of other things Not the least of which is of course nuba for object storage You'll be seeing OADP coming out before the end of this year is what their KCS says so Yeah, yeah, and that team The ODF team we have a show slot for them where it changes out storage admins and data scientist Every other week kind of deal. So it's it's a nice Rotation every Thursday morning at 9 a.m. So you can catch storage and people then using that storage to data science with it So yeah, pretty cool stuff. I just realized that I used a yet another acronym OADP OpenShift APIs for data protection There you go Thank you because I didn't I didn't want to ask Yeah, I'm gonna so I'm gonna post I said a KCS so I'm gonna post that KCS into the chat here where it talks about when that will be available And we'll of course Talk about that once it is available. Yeah, it's a hot topic. I just talked to one of my friends at a company that does API security and She was just telling me about like how their customers are pretty much blown away by all the stuff they find that's just like Open and available Yeah, I'm protected Well, it's it's funny because you know one of my responsibilities is to work with partners And I work with several storage partners and they ask me, you know, oh, you know, what is this OADP? What is this data protection? Is that something that is only red hat only OCS or is that something anybody can use? And it it's an open set of APIs. It's an open set of capabilities that You know, we would love for it to become kind of a standard if you will for data protection So keep an eye on that one. Like I said, we'll definitely talk about that once the time comes So yeah delayed due to the Valero CSI or ever, yeah, I know that there's been some complications there So kind of quickly in the interest of time, I'm gonna go through our top top of mind topics. Yes So please do keep in mind. This is one of our office hours series of streams So don't hesitate at any point in time to ask questions about whatever the topic happens to be So I Christian is on and Christian is usually one of my escalation points So all of the hard questions will get him to we'll get him to answer But yeah, don't hesitate regardless of what we're talking about on the stream. Don't hesitate to ask ask questions And as is tradition, I like to have a little segment if you will up front That talks about kind of top of mind Timely relevant things that I think are important for for you all for our audience. So I'm going to share my screen here Drag your little window of of heads around So the first thing that I want to talk about here and kind of all three of these today are really a continuation of something that we've been talking about for a Month and a half now two months now. When did 4.8 come out? It's been a while So it yeah, yeah, it's been a while. I want to say like two months so if you're not aware Upgrades in the stable channel from 4.7 to 4.8 are still not unblocked again We're on the cusp, but they are finishing out kind of the last set of blockers at least the blockers that weren't we're aware of So 4.11 and let me come to the customer. Nope. I want console So if we come to the console here, we go to open shifts and we go to releases so in 4.8 you can see in fast we've got 4.8 dot 11 and 4.7 dot 30 So these two versions incorporate a number of fixes for kind of outstanding Beesies outstanding issues that have been blocking upgrades So I want to highlight the three of those that have been the most significant and just point out that Hey, these are fixed if you're encountering some issues This could be why and you'll want to make sure to update as soon as is reasonable for your organization So the first one is this internal registry is rejecting rejecting the container creation due to a layer mismatch so this one is specific to The it is a rel kernel bug which remember CoroS uses rel the rel kernel CoroS is rel. So it is a kernel bug with Cephessa FS So specifically if you're using Cephessa CephFs, man, I can't talk I'm gonna write that down for you. No, and no, I'm not gonna say it like 10 times fast because that won't help It gets it gets worse It means I'm either under or over caffeinated so This one came up. You can see it was opened the end of October, excuse me the end of August So this one came up. They went they fixed it. We should be good on again. I think it's 4.8 dot 10 was the first one or 29 was the first one to include this But they're kind of lumping all of these into the 11 and 30 releases So if you encountered any issues with that Again, please be sure to update as soon as possible. The next one and this one was definitely much more broad Crio having a leak in the run FS as a slash run FS So this one was first found and in 4.9. It turned out to also be an issue in earlier versions So they back ported that So if we scroll this one has a long comment chain But if we end up all the way down here at the bottom you can see 4.10 4.11 is when it is fixed so Definitely be aware of cognitive of that you will probably have to if you have long-running nodes that meet the various conditions inside of here You may have to reboot those occasionally in order to reset it But if you update to 4.10 or later for that 8.10 not 4.10 for that 8.10 or later That one is fixed as well as whatever the equivalent version in 4.7 If I scroll up here slightly it looks like it may be 27 4.7.27 or later And last but not least We have a an issue that is near and dear to my heart Because this one was also or a very similar one was also an issue with the 4.6 to 4.7 upgrades in the early updates And that is a issue with vmx net 3 network adapters So essentially there's an offload issue that happens with later versions of And you can see here virtual machine hardware, which I always thought was a funny oxymoron So versions later than 14 or 14 and later so If you deploy by default with vSphere ipi and don't change anything the ova ships with hardware version 13 A lot of folks have changed that have updated the hardware version to 15 or later to be able to use the csi provisioner So if you have a default IPI install you may not be hitting this one if you're using if you've modified it or if you've done up Then you are you have a high chance of hitting this particular bug So the issue is effectively you'll see some packet loss happening inside of the cluster As before The resolution is to turn off the offload so the first time it was It was the It was the vx lan offload this time this solution is this ip generic transaction or transmission check some ip generic So this kcs And i'll link if we haven't been so i'll link all three of these that we're talking about yeah, no worries I'm just going to paste all three of these bz or two bz and and the kcs in there So if you look at the kcs it walks through exactly how to work around this If you don't want to wait for 4.8.11 to make it into stable But yeah, that's uh, that's all i got for today kind of quick and straightforward because i want to give christian Plenty of time Which chris you pointed out to me this morning that actually today's topic is a double header Yes, uh, yeah, yeah, so christian we're going to talk about it here In this context of open shift administration and then we will have A p.m. And an engineer on this afternoon just talk about bring your own windows hosts at 1 p.m. Eastern 1700 utc so Double header windows host That's right. You get you get the first you get the t.m.m. And then you get the the p.m. After so if you are If you're interested in hearing the the vision behind it stay tuned. Yeah, should we exactly make it a competition like Yeah, let's see who gets more more views. So I don't think that's fair because like literally anond was like Hey, I want to get this uh tomorrow. I was like, okay fine Who who shows it better? Who should know? Yeah, exactly I mean we've got to have some fun, you know, yeah When they're both done Let's gamification. Yeah, that's right. Yeah. Yeah. Well, that's the level up guys. They're uh, they're all about the internet points. Yeah Yes, sweet sweet internet points. Yeah, there's actually a tie for the lead there as of this morning. So kind of nice All right, well, so That concludes or that is all of the top of mind topics that I have so we will certainly move on to christian And today's topic which we've made we've not been shy about talking about windows nodes So the first thing I want to bring up here or or ask you christian is Windows nodes aren't new, uh, you know, we've supported windows nodes for Since at least four seven, um in a ga status. Yeah But it was limited. So what we're talking about now is an expansion of capabilities, right? It's it's less limited than it was before so Mm-hmm. Yeah. Yeah. So before It was really constrained to um At first it was just uh, the hype was the hyperscalers, right and it was just aws in azure that you were able to run windows node So it was even constrained even further was it always had to be an ipi installation um, and um, which Which is kind of the first step to everything right get, you know We want to get everything completely automated and make it making sure that it's up there with functionality with The linux nodes in terms of the ipi install um, and then there is also the um Later on in 4.7. We we did the v vSphere ipi. So um, so it's nothing new, right? We I came on the show before we talked about windows nodes and you know, all that works out. So um, we're getting excited for this next release of the Wmco, which is the windows machine config operator How he's pronounce it wimco in my head wimco. Yeah Now I will right Wimco so the wimco um The next the next release which is do um at the end of the month or beginning of october I mean we're always in In open in openshift land, right like we're always like it's probably will be released if everything's pretty good, which is kind of a a For our red hat customers. That's kind of like a weird thing because you know with with rel it was it's always been like The same cadence and when we say it'll be released. It's released, but um So the the the wimco Is the next release will be at the end of this month. So hopefully in a few weeks We're adding. Uh, what we call uh, bring your own host, right or um, so it's So this is the idea for the upi installations. So the upi specifically for the bare metal installs um, so we're we're bringing support for the bare metal Or as as more specific a platform agnostic Um, upi install right more specific right platform none upi install. So we're bringing that functionality to where um We you're able to add a windows node outside of machine um the machine configs right so the machine um api Because with the api it's all you need to make a machine set right in order to get a windows node now You know, we're we're doing the upi uh version of this So just to be clear. This is any upi. So I if you have a visor upi or a Maybe a rev upi deployment or just specifically to the platform agnostic non-integrated platform equals none upi So it is for um all upi, right? So it's for all all upi install. So not just the platform agnostic non-integrated upi install But also, um, what I'll be kind of just going over and demoing is the the vSphere upi So, um, which is kind of interesting. So, um, um, I think This version Breaks the mold a little bit of what we can say Um, because this is because this is technically a vSphere api install, but I'm adding a upi windows node Um into the into the mix which is which is perfectly valid by the way A lot of people don't realize that, um, you know, we've talked about it a couple of times here on the stream But you can deploy an api cluster and you can add nodes outside of machine sets I've what did I do that for? I I did a proof of concept for, um Oh Moving the or moving the ipi load balancer to an external load balancer So I I tested that extensively and it works quite well. Not the load balancer part the Adding adding adding nodes outside of machine sets The load balancer part does work, but it's not supported So, yeah, so that's um, so that's kind of like the the gist of Windows containers bring your own host So I will I prepared a kind of like a demo like I was I was I was selling Andrew, you know, we'll we'll walk through some of the steps, but I have um Like I said, since this is a cooking show I prepared some stuff like off-hand so that way we're not just watching screens go by Um, so let me um, let me first share my screen and I'll go I'll do So christian while you're bringing that up. Is there are there any other changes to the windows machine config operator or the capabilities kind of requirements of those windows nodes? um, like I assume we still need open shi or uh OVN kubernetes with hybrid overlay um Are we able to domain join windows nodes? No, so this uh, this that will actually be the first step In that so the first step was let me know if you can see my screen while as I move stuff around Full small, but yes, we can Oh, let me make that bigger. Thank you um so, um Yeah, so kind of the the prerequisites and that that sort of thing right so The prerequisites are the same right? This is kind of like an an incremental I would say update the focus was really on um bringing windows nodes that um Uh outside of the control of of of machine sets, right? So it's basically more more flexible more flexible. Yeah, the flexibility. Um We still haven't tested like, you know, I have a windows server. I want to join to the domain You know, I want to apply group policy to that We haven't tested any of that yet, right? So that that you know, that's that will be in subsequent. Um releases but You know, this is like the first step where it's like, okay, you know, I want to spin up a Window server, you know on my own terms so so to speak and um and and join it to uh Join it to the um other cluster, right? So um, let me put in This is the release notes here Oh, that looks like you already did that chris. Thank you. Yeah, sorry You're you're fast on the draw man Like it takes me forever to find some of these things and you're like pretty good at finding all I've gotten very good at finding things just Searching our box is something I'm an expert. You gotta have the short cut Yeah, you gotta get the yeah, that's it's pretty good. Um, so I mean, uh, currently with with wimco version two Um, you know, we support uh, like I said before aws azure vSphere with 3.0 version 3.0. It'll be um Uh It'll be the upi installs essentially, right? So it'll be version 3.0 and and above. Um, this will change as well And so, um This is actually a kubernetes thing. So let's look at It is a windows server I also want to point out something that I've seen and you and I christian have talked about this windows versions are confusing because We and like you and I have been on meetings with customers before and we say, oh, it's it's windows 2004 or 20 2004 2004, yeah, which is easy. It's easy to say but not to write because people get confused when you write it But it's also most of us think of like, you know, windows server 2003 not 2004 Right, which is really and and you can see in the kubernetes docs there, right? Which is really windows server 2019 version 20 h2 Just with a different. Yeah, it's so yeah, so it's it's uh um Yeah, so it's a little confusing, right? So, um You know here to four we've had we said, all right your windows server 2019 version 2004 Or when you write it 2004, they're like, what is that still supported? No, we mean We mean 2004 anyways, um, but now, um with with, you know kubernetes, um as they release here We're talking about it's windows server 2004 You know 20 h2, I guess so it's kind of uh, um I guess they changed it. They they know That was kind of hard running windows boxes for the streaming efforts I've kind of learned a little bit right like the h2 represents a Second half of the year have to part of updates. So yeah, like yeah, so one is not going to work If you have h2 it will kind of yeah, so 2004 is the old naming scheme. It was the april feature update for Year 20 whereas in the second half of 2020 they switch switch to the half thing So for example my desktop my windows desktop sitting over here to my left. I had to think about that. It's embarrassing is running Windows 10, I think it's 21 h1 or something like that Yeah, yeah, and um, yeah someone pointed out in chat windows server 2020 22 is a ga And we do a plan to support that in later releases because it includes patches that we desperately need To make some of the other functionalities that we that we're thinking about working. So, uh, but for now, um 2019 version 12 Either 2004 or 20 h2. I guess they call it. Um, I put this in the chat this is um This is what what what supported, right? So I so the reason I'm flipping back and forth here between the official docs and kind of like the upstream docs is because Um, I'm demoing I guess beta software So so the docs haven't haven't matched. Um, what? What we will will actually support in version three. So So for the windows versions, um 2019 Either 2004 or a 20 h2 will work. Um, let me go back to the doc. So now um, so now some of the networking, um information right so in order to The the prerequisites are the same right in order to run windows containers. You need to install ovian kubernetes and um And configure hybrid networking So in the docs if you go to enabling container workloads, um The prerequisites are listed here and one of them Has a link to how to configure hybrid networking. There's this big old red exclamation point here Please read this. Um Because if you're if you're preparing for windows containers and windows nodes You have to prepare at install time like there's no way to install the cluster and then switch it Like oh, hey, I want this cluster to run windows nodes Then you're out of luck if you didn't do these prerequisites. You have to it's essentially reinstall, right? So, um, so currently um Really two configurations need to be set here and then I'll switch over to so christian I had a question for you and this one came up internally recently, uh, sure if Let's say that Because we know The hybrid overlay needs to be done at install time and cannot be done afterwards Is there any harm in turning it on and deploying it from day one? Even if at this moment I have no intention of deploying windows nodes Yeah, so, um, there's no harm in turning it on because hybrid configuration It'll be it's basically it'll sit there, but it's not it, you know, it's not doing anything until you add something that supports hybrid overlay networking, um I think the only other thing is That that uses that I think it's the f5 plugin I'm not I'm not 100 percent sure don't don't quote me. There's something else that that uses the hybrid configuration But it'll basically sit there dormant. Um, and if you're using ove and kubernetes, that's going to be default anyway So I guess you'll be the ahead of the curve. So So here I have uh, like I said pre-installed, um Get the version here 8.2, right? I guess there's an update, but it works with 8.2. Um If you get nodes the kubernetes folks Slash os equals linux, right? It'll just this all the linux nodes. You can actually You see here. No funny business. Um So, uh, hopefully I don't pull a sully here. I'm gonna look at my install config. Um Yeah, I'm gonna fall here Well, at least you know how to Yeah So don't um, don't don't pull a sully, right? So here so um, so this here I'm going through like kind of the prerequisites and how it's installed Um, if I look at the install config Um, as you can see it takes that away from me your pull secret. Um Here, uh under networking, you'll have network types set to ove and kubernetes So that needs to be set in the install config file And then the other thing is um, let me get the network. I always have to copy and paste this command here So the um, I think you have to set hybrid uh overlay networking, right? Which is oh, that looks ugly. Where is it? There it goes. So, um There it is so hybrid overlay, uh networking It basically says that hey, um You know when when you're here's the ip address range for that hybrid overlay networking configuration This here is uh the vxland port this if you're on vSphere, it has to be changed, right? So you you you have to change it because of a bug or else Um containers won't be able to communicate with each other, which is probably what you want. Um So this needs to be changed. This is all called out in the documentation, right? It goes through the steps Here uh like this here it even calls out this vxland port um important to note that Uh, it must be set for vSphere, but it must not be set For others others, right? So it's kind of like if you're not on vSphere, it's not harmless It's harmful if you set this so so just keep that in mind I can't say this enough. Please read the docs Christian, I'm gonna pause you because we've got a couple of questions and I have one for you as well So first from alosadag alosadag, um, so Uh question from last week um about edge So I understand that schedulable control plane nodes are supported in compact clusters Why are they not schedulable in non-compact clusters? And is it not suggested to do so? so When you deploy a non-compact or a standard cluster We don't have schedule control plane nodes by default simply because We want to dedicate those resources. We assume that those are sized especially with an ipi deployment. It it sizes those According to the expected workload So of course if you are deploying say, uh bare metal And you have nodes that are you know massive and by bare metal. I mean bare metal to physical servers not bare metal the non-integrated method You can absolutely mark those control plane nodes as schedulable on day one or add install time And you can do that kind of two ways. Um, so one with ipi if you mark the And actually I don't know if it's supported to do it that way Although it does technically work and that would be to declare zero control plane nodes Alternatively, you can do a In your install config.yaml. So you have three control plane nodes three compute nodes And then you do an open shift install generate manifest And it will spit out a bunch of in two folders It will spit out a bunch of files and then one of those is a file where you can mark the control plane as schedulable And all of those just for kind of side information all of those files become what goes into the bootstraps Install config or ignition file rather So yes, you can absolutely use schedulable control plane nodes with a traditional deployment You can deploy standards, you know, six node ipi deployment go in day two mark the control plane node or the control plane is schedulable And then you can scale worker nodes down to zero if you want and then you know scale up if you need to Not absolutely nothing wrong with that whether or not it's suggested is Really completely up to you and and your environment your workload So what I mean by that is if you have Say a cluster that's 100 nodes or a cluster that has you know thousands of pods inside of it where xcd is going to be busy The api services are going to be busy You may want to make sure that you know, there's not just random workload running on the control plane So that way it can have access it has all of the resources it needs Because if the control plane and especially at cd is struggling for resources The whole cluster is going to have a bad time Um, but if you have light workload if you want to have kind of a scale the zero type of thing and scale up from there You know for burst ability or whatever it happens to be absolutely, you know use use Scheduled a little masters um Is ossm so open shift service mesh still slightly funky with ove and kubernetes. Yeah Um, I thought it was yes. I thought it was fixed already. Um I remember asking them about that. Um, Our hope nine. I think it was you who reached out to me And I asked the pms about that. So I thought it was fixed. Um, although it is a very recent fix Yeah It might be one of those things where it's fixed upstream right in um in the upstream. What is it called? madastra whatever that Whatever our upstream version of red hat service mesh. Yeah, and it might not come down. Oh, yeah here it goes Coming in 2.1. Okay. Um, and then uh, so so hykel my apologies for butchering names. Um, I'm I'm terrible at them. So Uh, can you please elaborate on hybrid network? And that is more or less the same question I was going to ask you christian, which is yeah, what does the hybrid network actually do? And I think it's important like just looking at the config it's important to highlight two things here So one ove and kubernetes uses geneve as the overlay technology Whereas the uh hybrids appears to use vxlan So I am now at the end of my knowledge of what hybrid So um, so from you know without without getting too deep, right because I don't think any of us are really deep networking Experts here dns. Yes, but dns. Yes, but but like, you know, once once you get below layer layer four kind of uses um, but so um high high level is that OpenShift has a software defined network right the sdn Currently we're using ovn kubernetes for that software defined network When you install windows node it installs its own Software defined network it has its own almost like it's um like it's isolated right from the from the uh from the rest So the hybrid configuration is basically creates a um a layer three I don't want to call it a bridge. I guess route. I guess would be would be the the best the best way to call it um gateway whatever I'm probably getting the terminology all wrong But it creates basically a layer three communication tunnel between this service network and this other service network So that way you can deploy Windows workloads that can communicate with the linux workloads over, you know, the the native kubernetes Methods right with service right so then you can refer to something as the service name You know the local resolver will resolve it right core dns will resolve it Yeah, and the networking is just automatically works right so just from a high level the hybrid cluster Is it that it's essentially that the hybrid right you have two software defined networks and the hybrid basically connects them together Yeah, I'm I'm curious and all I'm gonna take a note to uh ask the folks Is are we putting vx lan on top of geneve or are they running in parallel to each other and just windows nodes talk to the rest on You know vx lan and linux nodes talk to the rest on geneve or what does that actually look like? That's a good question. I would have to We'd have to ask the networking folks for that That's for sure Well, I I know a couple of those guys. So we'll we'll definitely reach out. We'll see if we can get an answer to that We might know a few people. Yeah, we might know a few so um all that all that configuration stuff, right? Like if you just um If you want to configure it is in the docs um, so again, you know set up a hybrid overlay networking for your north south east west uh networking to work so five easy steps Sorry, right. I hope nine says geneve is a tunnel protocol. So there you go question. There you go tunnel I thought it was a city in like switzerland or something. Anyways, okay Geneva. Oh, okay So, um, so once the So once the the clusters installed as you see here, um, you know, it'll it's a normal open shift Opuship v4 4.8 cluster So once you once you've installed that, um, you have to install the the wimco operator. So That's easy enough, right? If you go to operator hub Again, you're a cube admin And just type in windows, which feels really weird, right? This whole this whole on a on a red hat system type windows um How far we've grown um And then on the windows system go cat. That's eos release and see what happens. Yeah, yeah So here, uh, I had the community operator So for those of you who are using who i'm 4.8 today and who are using okd The community operator supports this right so I have that installed It it's that version here. So, um And then but when this ga is obviously used the ga pro release, but I'm using the community operator that's installed And then one last prerequisite right so once You install the cluster with the networking prerequisites once you install the wmco the wimco You need to um, let me see if I can find it here Let me go back Uh prerequisites here we go. So the last thing is um, you have to add You have to provide a an ssh key, right? So you get to create an ssh key pair Right private public key here. I'll I'll actually anchor link that here in the chat. Um So a couple of things uh to note, right? This is just a um Just a standard ssh key um, you have to upload the private key to um to open shift And uh, you need to use this key. You need to use the public key. You need to load that into your windows node Which I'll show in the second how You'll load that into the windows node um Last thing I want to call call out is that you need It's It's one key per cluster. So there's there's no way to Say, hey, this windows node has this key. This windows node has this other key You know, you can't have multiple keys. So um, currently it's one key per cluster I have asked about this because my first question when I was testing this is like, okay, well Can we have multiple keys, right? Because a lot of the times Um, I would imagine administrators would want to have separate keys Uh wanted to be able to rotate keys. So that that's all in you know backlog That'll be addressed right because I've I've created a couple RFPs around that so um So yeah, so now You know prerequisites install Install the wimco operator uploader ssh key. So um And if I do oh see get that gets uploaded in a secret secret uh open shift Windows machine config operator That gets it's that's the cloud private key, right? So that's the the ssh key I won't Expose that because That's an actual private key that I actually use in my actual day to day But trust that it's there. Um So, um So once you have those installed, um Really the the prerequisites for the open shift is is the easy part. I guess for me maybe because I'm a Linux guy Kubernetes guy, this this all was easy to follow. Uh, the hard part was actually make this a little bigger. Um Was actually preparing the windows node. So this actually took me a little while. Um, It is straight toward And then whatever my password is Okay No whammy's no whammy's all right cool. So then um, I always get these little exclamation points. Maybe maybe uh as Um, and you can explain these because I'm gonna reset these It's just it's okay. It's just using a bunch of cpu all of them are at eight and nine gigahertz So Doing what control plane does Um, and so uh, so here, um, I have a windows node Uh, so we need to prepare the windows node to become um a uh A an open ship node So as as you see here, I installed the um the vmware tools or whatever on here. You can go ahead and do that That's you know, normal workflow. I don't know how to make this any bigger. So sorry. Uh, oh there it goes Oh, I sign in So as you see here, I don't have the gooey because that's just like the iso that was given. Um I'm pretty sure pretty sure it's the same iso to to install the goo. Yeah. Yeah, I uh How do I bring up the um CLI here? I don't know. You gotta It just gives me this what? That's weird. I don't know. I'm not a server core user Me either Okay, let's do this here There we go. There you go Uh What was the password? Welcome administrator. Okay And then it Handily closes that window for you and then it just handily does close that window Uh Man, there has to be some sort of uh here. Let's do this It's always gonna laugh at me power off. Yes. Oh, okay Force off come on forcing off a windows node That's that always ends well, right? Well, wasn't it We we always we we always avoided forcing Linux nodes off because of file system corruption risk, but ntfs was always pretty resilient to that Well, they it because they always they say well the windows fixes to restart it right because um Yeah, yeah, because like it it probably got that like It probably needed to be resilient reboots. I would imagine. Yeah Something's wrong with my windows box. Well, have you rebooted it yet? Yeah, let me try turning it off and on again All right, let's see if you close it now There we go. Cool. Hey And then I always go into power shell. You weren't wrong I wasn't wrong, right So here, um, I'm not actually gonna go through I always hate doing this. So I go to the I think you posted the github page Um, you need to do a few things to prepare the node. So let's go There's a prerequisites here, right? So I'll open this window here. Um, make this a little bigger So, uh, first you need to um, this is a prerequisites for build your own host I'll put it in the chat here. This will actually eventually make it up to the official doc. So, um First you need to install docker So that to be clear What is this tech preview g a what level of release this is of uh, this is I guess beta it's community It's a community operator. So it's wimco Wimco 2 is g a wimco 3 is beta slash community, but will be g a shortly Yeah, so keep in mind if you're trying to do what's on screen right now before That 3o goes g a you're gonna have problems and if you call support They're not going to be able to help you or they'll try their best But they might not be able to solve the problem completely. So be careful That's all I need to say So I always go, um, I use the I always look at the scripts that, um The How do I say this in such a way? I always look at the scripts that the engineers use for their ci process However, these scripts aren't supported It's I know it sounds weird These are the scripts that red hat uses to test the thing that we are supporting But you using these scripts directly isn't supported. So And christian because I think this is a question you probably answer regularly Right now we use docker for windows. Is there any plan for a cryo or cryo for windows? Yeah, um, you know windows. So microsoft is planning on using container d So, um, it'll eventually move away from docker to container d so Um, so the best way is was the i-compliant so it should work. Yep. It should work just fine. Um, So this is something that solely showed me right while while testing this out. Um, if you enable, uh, what is it p s r Remoting This allows you to use, uh, power shell Remotely so that way you don't have to Connect to this Yeah, live in this little console world, right? So here What I like doing is I like doing, uh, potaman run A power shell. Where's it to my history, right? So, um, you can actually run power shell in a container And this is all um So then you all you need to do is have um IP config So this is uh 1.55, right? So let's just all remember that And then Yeah, let me copy and paste my notes here because I do have notes Okay You enter this which is equivalent to ssh. Um There we go Go on go go go go go Cool. I'm in right? Um So I'm in there and This is sell docker, right? So I go to the scripts So first you have to install docker. So let's go to docker Um Right and you can essentially just copy and paste that here So christian in in the interest of time, uh, because we got about five minutes Ah, I know. Okay. Do you do you want a cooking show this one? Yeah. Yeah. Okay. So you install docker uh docker, um ssh right because the wimco operator uses ssh Uh, you have to open a few firewall port rules um And you have to add the ssh key, right? So the ssh key you gotta make sure the ssh when you install ssh You gotta make sure the ssh key that you gave The operator is on the host and here this is something weird. Um for windows admins, right for us linux guys is like whatever, but um You could only have lowercase host names So what does that mean? That means that's the um, let me go back to So if I keep this as this uppercase, um, the it'll just barf, right? So, um You have to change the host name for this for this host Um to lowercase, right? It's whatever you want as long as it's lowercase. So Um, as soli said in the interest of time I have prepared a node and ahead of time With all the prerequisites already already loaded up, right? And so in order to, um Let me exit out of this here Um, so in order to add a windows node You have to create a um Bring your own host node. You have to create a config map Called a windows instances, right? So here Uh, you need yeah, it has to be named windows instances and it has to be in the same namespace as the operator And essentially you just give it, um ip address and the username, right? So um This is you know for future, right? Like if you have Oh, you know Windows admin right in your, uh active directory and that has to be the admin you can do that future proofing, right? um Here what is the ip address actually for my first windows node 92 Is it 82? It's on the uh, yeah, it's right here. I was like, how does how does he know? It's 92 Um, so you just give it, uh So there's currently a bug you should be able to put the fq dn, but right now that doesn't 100% work Um, so the ip address is what we have to use Right, let's see apply windows instances And then Open shift Windows machine, so there's that operator and then you can logs that guy And this this actually takes um doesn't take long at all It'll take it actually takes longer Uh for the machine the machine set way because you have the machine set actually creates the machine for you Whereas this is already up and running. Um getting a few go errors. This is probably something Um That Yeah, I mean that's usually it usually doesn't mean an error even though there's like a Yeah, that's just explaining like standard out or something. Yeah, it's probably because A machine operator unable to get okay. Yeah, that was an old config there It was barfing at an old config so um So then I usually just tell these logs and these logs will Um We'll tell you that it's installing here. So this will take I don't like two minutes or so Um, we've got set up the operator. So less than so So, uh, yeah, so here i'm pausing see if there's any any, uh Any questions So have have any of the other like do we still recommend that you pre pull the container base images to the to that windows node Ah, yeah, good question. Uh, yes. So here if I um and we never get that right if I do um Docker images I pre pulled the um The image specific to this version of windows. So For windows it matters Right. So like um, we've been kind of spoiled in the linux world where hey I'm just going to pull in a bun two image and run it on open shift and it runs fine The same is not true for windows if you if you try to run a I don't know server 2016 image On you know on on this windows windows, uh, 2019. It's not it just not flat out doesn't work And what's the reason for that and I asked because the reason when you told me it made perfect sense Yeah, yeah, well because the the kernel version right because no, I mean why we pre pull them. Sorry Oh, why you pre pull them. Okay. Yeah, if you pre pull them because if you see here This is uh 50 gigs inside 50 gigs five gigs in size Um, and it just in most environments that just takes a while to pull Uh, the default timeout Is two minutes, right? So anything that takes longer than two minutes To pull the scheduler will say error and then it'll try to pull again and error. It'll try to pull again it'll it'll basically just loop forever because Um, the image will never get pulled because of the timeout So maybe if you have really fast internet, it'd be fine and confidence in microsoft's image. Yeah or or uh, yeah Or uh, yeah, or you know, you have a local image, right? Oh local image registry or Uh, you know do what most of us do and just kind of just pre pull it. Um You know that makes the most sense So we're we're at the top of the hour, uh, if you can spend five minutes to answer the question from our productions or five seconds Rather, um, it how is rosa doing with windows nodes? Um, it's on the roadmap, right? So Right now because of the the prerequisites for the networking Um, rosa still uses open shipped sdn. So, um, I know it's on the roadmap. I know, um, they're, you know All the hosted right, um the open shipped, um hosted solutions are, um Are planning on eventually supporting this right this because it's a thing of like preparing, right? We need to have sres Right, we need to have support people on there Uh, to be able to we need to essentially ramp them up So we are uh now just now one minute over which means that we need to uh drop for the next uh next stream Yeah, um, we will have some more windows node goodness happening in an hour um, so Version two or or the next iteration if you will, um that being said, thank you so much for joining us today Thank you so much for a great first year of the ask and open shift admin office hour really appreciate everybody Um, thank you so much for all of the questions interaction Um, I hope that this is useful to you. Don't ever hesitate to send us feedback questions requests, etc You can reach me at andrew.sullivan at redhat.com or on twitter at practical andrew All one word just like you've seen me in the uh twitter chat there Uh, so join us next week next week. We will have Rob Zomsky on Along with some of the engineering folks to talk about api deprecations. So nice Yeah, favorite topic. I'm gonna get to talk about that. Yeah, honey. Good. All right. Well, thank you so much everyone Thank you See you soon folks. All right