 Good morning. Good afternoon. Good evening. Wherever you're handling from welcome to another edition of Ask an OpenShift admin today We are joined by a very special friend from VMware mr. Robbie Jerome If I said that wrong, I'm sorry, but Andrew the cutliest curmudgeon in the company. Would you please? Introduce, you know yourself and what we're talking about today. Absolutely. Thank you Chris So yes, welcome to the ask an OpenShift admin office hour You notice I had a glance over here. I still have to look at the title Yeah, because I still want to say the OpenShift administrator office hour I've decided to put the title in front of me from this point forward. Yeah. Yeah, so You know forgive me I'm if I stumble over the name of the show because we're still this is the second week third week Third week, I think yeah that we've that we've had the new name. So So yes, welcome. Thank you everybody for joining us today This is one of the office hour series of live streams, which means that we are here for you We're here and an ask me anything style of interaction You know, we encourage you to ask us well literally anything about OpenShift about administering OpenShift About anything that's top of mind in that respect and you can do that at any point in time across any of the platforms We have some fancy technology that that rebroadcasts rebroad class, okay Yeah, I know it's storage glasses. You've been dealing with those for the past few months. Actually, yes So if we broadcast everything across all the platforms so matter where you are We will we will be able to respond back and see that in the absence of those questions in the absence of things that are top of your mind We generally have a topic and today as Chris highlighted when he opened today That is OpenShift and VMware together and I'm really happy to have Robbie on today Robbie and I have been working together for I don't know two plus years now, which is kind of funny Yeah, it's kind of funny because I've I have spent so much time with you on the phone and video chats and all that other stuff I don't think we've ever actually met in person, which is kind of funny. So Yeah, so so Robbie if you don't mind, please introduce yourself. Yeah, so I'm Bobby Jaram I work at VMware and I talk about apps and platforms on on vSphere in the varying flavors And I've spent probably the best part of the last two years focusing quite heavily on OpenShift and working with Andrew and the engineering teams Yeah, and Robbie and I we work together quite a bit because we are kind of each other's interface into the respective companies So whenever VMware folks want to know about OpenShift things, you know He interacts with me and vice versa whenever I want to know about VMware things I go through Robbie. So it's been Really beneficial for for both of us to learn a lot about about the respective side therapy. We're okay. Yeah yeah So For anybody who is watching the stream, please feel free to ask questions I'm going to spend just a couple minutes here going over the the top of mind things. So for anybody who is A regular watcher of the stream, right? I tend to spend a few minutes up front talking about The things that Andrew was wrong about last week, which nobody sent me any Obscene emails telling me that I was horribly wrong. Oh, that's good. And yeah Every once in a while it happens And the other one is kind of things that have come up in the last week or so Things that I keep seeing internally and sometimes externally that I think are worth bringing up to you all to our audience So the first one is There was a pretty lengthy thread that happened this week about the difference between A open shift deployment that is mixed operating systems versus mixed architectures cpu architectures So with 4.7 we announced with vSphere vSphere ipi, right? We can do windows nodes inside of an open shift cluster So Coro s control plane windows worker nodes that are able to to do all the things that they do Absolutely fine fully supported. Nothing wrong with that works great What we can't do is mix for example a x86 deployment so x86 control plane with Power nodes or ibm z nodes, right? It's stuff like that And we get that question somewhat frequently, especially from of course our peers at ibm But unfortunately all of the nodes have to be on the same cpu architecture So just just be aware. I know that's a core kubernetes thing And there's not much That I think anybody can do about that at the moment, right? Like mix arch is hard period. Yes. So here I I'll post this into the chat. I have this I have this kubernetes issue bookmark because I use it at least once a week And this is the the kubernetes issue the upstream upstream issue where we can't mix different cloud providers essentially so When with cloud providers what we're really referring to is infrastructure types So I can't have some nodes that are deployed to vSphere and some nodes that are deployed to physical servers Unless I do a non-integrated open shift deployment, right? So what used to be called bare metal upi Yeah, so and that's just because it's a kubernetes cloud provider limitation It doesn't understand it doesn't recognize when those nodes can't all be managed the same. So it just doesn't allow them to join the cluster Um, so second one, uh node auto scaling This was a question that comes up somewhat frequently Um, you know once every week or two we get questions around can I have A cluster preemptively scale based off of some sort of criteria All right, so hey the cluster has hit 80 cpu threshold or you know, 70 percent memory or whatever that means Can we go ahead and provision some extra nodes using the ipi mechanism? The answer for that is unfortunately no um, but maybe So the cluster auto scale or the node auto scale mechanism doesn't work that way It it doesn't take action until pods fail to schedule So I just submitted a job that job needs, you know, 40 pods that have x amount of cpu ram and at pod number 26 I'm out of capacity, right? There is no more cpu available Right, it needs to take action. So it'll it'll start spinning up nodes to be able to schedule that So if you wanted to do some sort of preemptive scheduling or preemptive scaling rather There's a couple of ways you can do it. Um, they're all basically the same which is some sort of additional automation So maybe that's creating, uh A simple pod, right? It's some sort of automation that goes into a pod deployed to the cluster That is checking those metrics internally and just says hey, you know, does an oc scale or something like that against the machine set Uh, and basically the same thing but done externally right something your external monitoring system Whatever you happen to be using that is watching those metrics and then taking action for you So the last one that I have here is extracting ignition files. Um, this was a fun one Um, so somebody asked me, you know, hey, I uh, I accidentally Broke my or uh, deleted the installation folder for my cluster and I need to add some more nodes This is a upi install. How do I get the ignition files so that I can add those nodes, right? Because you know, you have to provide ignition Uh, so how do I get that out of my cluster? So there is a command that we can use. Let me see if I can share screens here You can do it. I believe Uh, you know, sometimes it takes a little faith. All right, here we go. Yes always So I'm uh, this is this is my cluster for today. Um, come on cluster. This is a vSphere cluster It is running on vSphere 6.7 u3, I believe So it happens to be ipi, but really doesn't matter in this instance So the command here is not that one Let me Pull this guy So the command here is this oc extract command So from the open shift machine api namespace We're going to pull this secrets slash worker dash user dash data And so what we get is The ignition right so this would be what we you know, if I'm doing, you know vSphere upi This is what I would attach is that machine property if I'm doing You know bare metal or something like that. This would be what's uh hosted on that web server that we pass into that And the same thing we can do for the control plane nodes just change this from for example Worker to I think it's master. Yep. So over to master. We pull out that one so on and so forth. So Super straightforward. I'll paste this command into the chat. Thank you Well Well, I say that And because I highlighted part of this over here, it uh, It decided to yeah, thank you macOS for being overly helpful. Yeah All right, so Yeah, that's that's those were the three things that I had Today or leading up into today. Uh, I do want to take a moment to highlight very very important tomorrow at 9 a.m. Chris 10 a.m. 10 a.m We will have the what's next presentation Streamed here on open shift dot tv across all the various platforms all the platforms. Yes, sir So what's next is the roadmap presentation for open shifts? So there's usually two of these that we do quarterly So one is a what's new which is what's coming in the next release And then the what's next which is what's on the roadmap for a little bit further out So I will be on the streams. I'm sure chris will be on the streams as well, right? Answering questions anybody who's interested. Please join We will be if we can't answer the questions we'll take those and we will re ask them internally. Yes, so and We have multiple ways of harassing the product managers to get those answers So don't hesitate to ask any questions that you have about what's coming in the roadmap for open shift All right, that's enough of me rambling. Yes, we already have a list of questions that I'm trying to pull back up Yeah, so Somebody and I did not know that you could do this So when we schedule these especially now that we are part of the red hats YouTube channel They schedule these like a week or two in advance and apparently you can come into that pending Stream and you can chat from the time that it's scheduled So somebody asked us a question Last week Which is pretty interesting to me So I think we'll start with that question and then I will Come down through the rest of the chat here and see what we can see what we can answer before we get started talking about other things So first what about storage fee motioning happen on an open shift worker node? Which is carrying vmdk's provisioned by the csi driver So essentially pvcs from the csi driver So some good news. We recently updated the documentation Let me find my right tab here And I will share this guy All right, so we recently updated the open shift documentation And if we scroll up here, I am on the open shift 4.7 docs Um, and this is just installing a cluster on vSphere. So we go to installing we go to installing on vSphere and take the first option And if we come down here, we have this Using an open shift container platform with vMotion So previously the guidance here ranged from Don't use vMotion vMotion is not supported to vMotion works and it's a support gray area. So you probably shouldn't do it But now it is officially tested. It works right. It's fully supported to do and you can see here compute only The emotion, right? So the compute only is the important part here. Um, essentially that's what's tested They make some comments in through here about migrating with pvs and stuff like that and how it might change the references I haven't verified this but I think that that is talking specifically about the entry provider So remember with an open shift deployment with an ipi deployment or upi deployment We will automatically configure And use the entry vSphere storage provisioner It's there out of the box. Um, and that's the one that is Fully supported by red hat The csi provisioner is different, right? You deploy that day two. It is supported by vmware, etc so My understanding is that this is true. This paragraph is true about we don't want you to do storage vMotion with entry pvcs Robbie and I believe that and robby, please speak up if i'm speaking at a turn believe that this should work with csi provisioned volumes however If you want to be absolutely sure and comfortable and not have any worries about it at all I would definitely recommend doing a cordon and drain And doing that storage vMotion while the node is not hosting any workload And if you really want to take it to the maximum, you know security or a comfort level Turn that note off right shut it down. So that way it's a cold migration Yeah, and it's it's as you say as you say andra with the with the new provider Um, if you've got that loaded in, you know, I I move I move I move nodes around all the time in my home lab and never have any kind of issue But if you want to be absolutely in line with the support statement in the docs That's that's what's in front of you right now um Yeah, your mileage may vary, but it does seem to work spot on the new driver has got a lot of changes and You know, it does have a different vcv version. So you need to check your version numbers If you're going to be running with the the new driver instead of the entry driver Yeah, and I should have asked you this before we started Robbie of checking to make sure that you have a cluster on vSphere 7 with the csi driver One of the things I'm hoping you'll show is some of that vCenter integration that comes with the csi driver Because I think we talk about that a lot, but we haven't often shown it Of you know, hey as the vCenter administrator I want to feel comfortable. I want to see the things that I have if you don't have that That's okay. Um, we'll we'll follow up with it and include it. Give me about two minutes. I can magically create one Magic magic got to love magic Uh jp dade Can we do a bare metal ipi vSphere installation of open chip 4.7 with windows worker nodes? So bare metal would imply that So or bare metal ipi would imply that we are doing something like a vbmc or something like that so No Because I think what you're asking is can I deploy some nodes to vSphere and some nodes that are maybe physical and some nodes that are windows maybe physical or virtual And we can't mix like that So you can do And I'll take it further as windows worker nodes are only supported either in azure or on vSphere 7 ipi right Those are the only two platforms or in the only two installation methods that windows worker nodes are supported at all jp They'd clarify these on a hpe synergy frame. It's all virtual okay, yeah, so Yeah, unfortunately or fortunately depending on your perspective. Yeah, it would need to be a vSphere ipi deployment Make sure you watch. I think it was two weeks ago. We had christian on maybe three weeks ago We had christian on the screen. Yeah, he talked about windows nodes. So there are some caveats there right make sure using ovn stn with Hybrid mode so that way it'll work across the different operating systems all that fun stuff What do I get from the vSphere cloud provider other than storage? So that's a great question And robby, please feel free to interject here as well so There's a couple of things. So the cloud provider which is responsible for Really managing nodes in an ipi deployment, but can also work with upi So what do I mean by that? so the machine api and the machine config operator etc Are designed or or they use to create and destroy and configure Nodes that are a part of the cluster and the cloud provider or what we And cloud provider is a bit of an overloaded term right depending on who you're talking to in red hat and certainly across other vendors in the industry we sometimes use it to mean different things but With open shift and with andrew So the cloud provider is that bit that talks to the underlying infrastructure provisioner and or provider rather and provisions manages those nodes it is available through Upi let me dig up the link here Too many bookmarks i'm having a really hard time searching for this episode. I'll just go All right, so I will paste that into chat and then I will bring it up over here So this kcs How to create a machine set for vmware and open shift 4.5? And this is if you deploy using upi And then you want to be able to do node scaling right using the the cloud provider right the machine set paradigm inside of that cluster So this will bring back or or introduce some of the ipi requirements to your upi cluster So for example with upi you can do static ip's right and you can you have to create the dns entries the load balancer all of that other stuff So when you introduce these machine sets when you start doing dynamic node provisioning, you're going to have to have dhcp right those nodes have to get their ip's from somewhere and you'll want to be careful about how in particular the routers land across those nodes and the reason I say that is because Well, we'll end up with you know, you could end up with a router That ends up on one of these new machine set provisioned nodes that hasn't been added to the load balancer Because remember it's upi the load balancer would be manually configured by you So either use a load balancer that has an operator for integration so that way it can dynamically update it Or create like an infrastructure machine set so that way you always know where the routers are You can always have the the load balancer configured to point at those But otherwise it worked great and it's really good for You know that dynamic provisioning that you want to write dynamically scaling the cluster up and down that you want to be able to do While also being able to take advantage of your external enterprise load balancer instead of you know, the ipi keep alive d thing that that we have there So please hopefully that answer your question. Um, I think that was well, yeah, yeah, I believe yeah, so If that didn't answer your question, please feel free to to follow up I see that you can you talk about machine api machines and machine set if we were using upi So yes very quickly so machine api machines and machine sets Effectively don't apply with upi Unless you're doing something like this right where you're creating that machine set after the fact So that doesn't exist for all providers. I think Actually, I don't know. That's a good question Um, I know we I think the only one that it's documented for because the only one that we've really been asked for that I'm aware of is with VMware Yeah, so yeah, but normally with upi without Day to configuring this machine set configuration those things wouldn't apply Essentially they'll show us there's a quick one. There's a quick one. Andrew back on the csi topic just come through a chat. Um Saying, you know, shouldn't shouldn't we use csi and forget the built-in driver? Um Taking taking taking out the built-in driver taking out the entry driver is is a bad idea. Um I don't Andrew whether you want to speak to that Uh, I will say it's not possible. Um, because the the cluster storage operator will ensure it's always there Yeah, it's it puts it back and um, I think it's tight. It's tight quite closely with the registry. So You can you can put them both in and you can change the default So you can change the default to the the new vSphere driver Which is going to give you the integrations into vSphere and vCenter And visibility into the provision vmbk's and and pv's Up into vCenter And but don't spend too much time trying to delete the existing csi because the operator's going to put it back And it's it's going to upset your cluster a little bit if you do successfully manage to remove it So Thanks default. Yes, don't delete the existing one that's that's in there Yeah, and I would I would say and robby, I think you'll agree with me I would encourage folks to use the csi driver Yeah, absolutely The integration with vCenter just the general improved capabilities, um, especially if you're also a vSAN user Right, just the general improved capabilities and reporting and everything there You know snapshots, um, the rwx of the file modes One of my what's the word i'm looking here for robby, uh with vSAN Yeah, the vSAN integration you can type storage policy so you can You can you can set your encryption levels and stuff like that And you can have a number of different policies that will let you have And different different numbers of copies on vSAN and stuff like that So you've got far more control if you're using the new driver um And then you've got other access to some of the new stuff if you're on on the vcf platform You can use some of the um, you know the the file access and stuff like that as well Which is which is quite new and I don't think we've spoken about before And so maybe we'll we'll do another session on that another time Um, yeah, so that was that was fine I have a whole line of vmware shows that I hope to get to someday Um, so I'm kind of going in order here So forgive me if if you've asked questions and we haven't gotten to them yet. I've got them all taken down here as no By the way, so so Daniel asked with a cluster installed using upi before static ip is easily available Can we replace dhc with static ip's? Uh Yes, but no, but yeah, so Yes, you can the problem is Doing it is messy And and by that I mean the way to do it inside of the cluster post deployment is to either Create a machine config for the nodes Right and then for each node right have a machine config that goes in and drop something into Like etsy's this config network dash scripts or into the uh network manager Directory right that tells it what its static ip is Um, that gets messy because now I have to have a machine config for each one of my nodes And now I have to you know potentially have a machine config pool for each one of my nodes and it just it's it's not nice Uh, so the other way as of 4.7 in tech preview you have the nm state operator So you could use nm state to go in and reconfigure that interface to give it a static ip address So the recommended way of doing it For better for worse is to reload the nodes. Um, you know, take take them down one at a time reload use the the The vm property, you know now that now that we can Use the vm property to add that static ip information and and do it that way So I know that that's not the uh, not the most friendly or the most ideal way. Um, but You know, sometimes I have to remind people that we treat coro s as an appliance and as disposable So when you want to make you know super low level config changes and the ip address for that first primary interface The one that's on the machine network subnet That that's one of those low level things that we really want to treat a little bit carefully with that type of stuff We don't want to Um, so I hope that answered your question dancel daniel. Excuse me. Man. I can't talk today Uh, while lead not just view motion. It's also dr s that was rescheduled vm space on utilization So I really I know this isn't a question It's a it's a statement But I really want to touch on something important here and robby and I have been talking about this for Well, basically two years and that is What is the relationship between? nodes deployed open shift nodes deployed to vmware or to vSphere with dr s with resource reservations and you know that that type of resource management type of principles So I'll I'll ask you robby, right? How should we configure those things? Um carefully no, it's it's It's it's it's actually fairly straightforward. So with esx, you know vSphere clusters um The system's looking after the cpu memory for you. So when you deploy vm onto the cluster It's going to give you the resources and balance them based on everything else that's running On that cluster or on that esx. I host If that host starts to become resource constrained constraints, you're using up too much memory or too much cpu Workloads will get get pulled back and you might get as much cpu or you might become memory constrained That will impact the performance of the application in this case, you know an ocp If it slows down too much ocp is going to get upset and think that there's a problem with a node or a master and And and it's going to try and resolve that So if you use a resource pool You can allocate a certain amount of dedicated resources to your To your masters for example or to a particular set of workers and Say absolutely for these these virtual machines I want to guarantee that they will always have at least 16 gigabytes of ram or 32 gigabytes of ram And that's going to have no impact whatsoever on resource management unless the ESXi host starts to become resource constrained In which point you're always going to have 16 gigs Allocated to the VMs that are in your pool And and you're going to be good with that Now the subset of that is there's a concept of shares Where you can actually sub allocate even further and say of that 16 these particular VMs have have shares Most people these days just just stick with reservations Just to guarantee a certain amount of resources so in a In a sort of deployment for open shift You would allocate Some dedicated resource a resource pool for your for your masters for example or your control plane and So if you do start to run out of resources, you know that's protected And then if you have particular Workers that you want to schedule particular tasks to and you know that you want to guarantee that that particular subset of workers has resources You can put those into a resource pool and allocate that yourself At the same time, you might want to tag one of those tag those nodes that it's it's a special kind of node So you can schedule workloads straight onto that node and if you are If you are filling up the cluster You can you can guarantee those resources Now that's for a single ESXi host across a number of hosts in a cluster and DRS, which is the dynamic dynamic resource scheduler is going to Vmotion live migrate the machines around to make sure you've got the best possible utilisation for that entire ESXi cluster So it's going to balance out those workloads for compute and memory And IO as you go across So if you've Yeah, if you're if your resource constrained in the entire cluster Then things will move around resource pools will kick in So you've you've got the flexibility to to over commit resources to say I need to use 300 gigabytes of the 300 300 gigabytes of the 250 gigabytes. I've gotten this ESXi host because I know that Actually, all of these VMs that are are my open shift cluster Aren't all going to be fully utilised all the time I just want them to think that they have the resources and the workloads will balance out appropriately So it's just a way of protecting critical parts And I protect some of my workers. I protect the masters by giving them resource pools But unless you are actually running out of memory, it's it's not going to kick in So I'll ask you and I think you and I might have slightly different perspectives on this So previously in multiple other streams, I've made the suggestion that over commit for Worker nodes specifically essentially you should never over commit the control plane nodes, but for worker nodes Should be handled as close to the application as possible So if if I'm wanting to over commit oversubscribe my cpu and memory resources so putting Applications that have or want more cpu more ram than is available in the worker nodes That should be done at the kubernetes at the open shift level And I'll and reserve or essentially protect the resources at the hypervisor level But what you're saying is essentially let vSphere do what vSphere does Yeah so and it depends on your workload and How dynamic you want to be and how you know whether we have for a development cluster, for example over commit would make sense If you're in a production cluster and you know the workload you understand it then maybe you don't over commit but With kubernetes when it schedules a workload it says, you know, have I got 50 gigs of ram and this much cpu and kubernetes says, yes, I could schedule this pod here and off you go It doesn't go back and check You know, that's that's just running that resources allocated to that workload And if that workload doesn't actually use what you asked for so it it uses one gigabyte of ram and almost no cpu Those resources are wasted um Now vSphere ESX is constantly monitoring. It's constantly looking at memory utilization and cpu utilization So it's going to rebalance It's going to do some clever things with things like memory compression to make sure you get the most out of Whatever workload is running on that platform So you can safely over commit memory. You can safely over commit cpu um and Open shift is completely unaware. It's quite happy um, the only Point you get to when it becomes interesting is when you do start to become resource constrained ESX will Will do things like reduce the amount of cpu you have so the the vm will Actually slow down which will slow down the processing of your of your of your workload um, and linux And kubernetes is completely unaware of that. It just it just starts to run a little bit slower it doesn't time shifting on it to to make sure every Every virtual machine gets a cpu cycle Um, and it does the same with hyper threading as well. You know hyper threading is effectively half a cycle um on ESX You don't get half a cycle It will that that thread will get scheduled twice if it lands on a hyper thread So it does some really clever stuff. Um Yeah, you can really get into it and get super nerdy if you if you want to get into cpu cycles and numerous awareness and stuff like that So you can really tune the the performance of the cluster Or you can just over commit a little bit and monitor it just here. It's going but It can drop off very very quickly so To your point andra about being cautious and saying look don't over commit resources if your workload is You know, if you are actually using 32 because ram you are using all that cpu all the time Then there's no benefit to over commit um And the ESX i schedule is still going to try and deliver that capability So if you know that workload is always going to be busy Don't over commit it. Just say right. This is a prod workload. This is how much it needs put it in a resource pool and It's fine um But for dev clusters or build clusters where you've you've got this You know fairly rapidly moving memory and cpu requirement as things get built and destroyed You can get more more bang for your buck on the hardware issue if you go with the over commit Yeah, and I'll say, you know, I want to get get all nerdy and down in the weeds on that I don't know if we have enough time especially because we're continuing to get more questions Thank you everybody for questions. We're we're going to address those uh But I know you and I have talked about you know creating some sort of You know usually it's referred to as like a best practices guide or something like that but more of uh I don't know if we want to call it best practices so much as here's what you need to know about open shift in vmware and stuff like that So that's that's one of our background tasks that hopefully we'll get around We should create a little video of over committing cpu and watching what it does Yeah, that'd be fun Yeah, so uh jpdade asks and this question for you robbie. Can we use vcf vcloud foundation with open shift 4.7? Uh, that's easy. Yes Uh Yeah, um Yeah, vcvcf is one of the supported platforms. You can create a workload domain for Before your open shift clusters Drop drop drop that into a workload domain and just find me some docs. There we go Um And and spin that up as you would any other other workload domain um Use the use the cloud native storage driver to integrate your vsan And if you want to do some stuff with nsx and plug that into open shifts, you can do that as well Uh, depending on what version of vcf you're running on Um, so yeah, it integrates really nicely. Um, andrew's kind of yawk Demonstrated that we have we even have documentation around how you do it and and some best practices on that I was here to talk about it. Can yeah Um, so I see a question. I don't know if this one is vmware specific, uh Playing to buy in a rise in 5 000 series cpu with bare metal coro s. Can I install open shift bare metal? Yes I actually use rise in 2000 series, um in my uh in my homeland Uh, so yes, coro s does support those cpus. Um, and then the kernel version It it is the same as rel So the simplest way to get the actual real current running kernel version And let me switch what i'm sharing here Okay, so I can do a oc get node Come on and do an oc debug against one of the nodes We'll just pick one of these one of these guys. It'll pull down that debug pod here in just a moment And once that comes up, we'll do a just as it says here. We'll do a change root to slash host And then I can do like a simple uname dash a So you can see this brand new 4.7.2 Node is running kernel 4.18.0 dash 240 So pretty easy to find out the real actual running kernel If it's not a running node if it's not a running host that you can that you can ask It might be in the release notes. I'm not sure. Um, so we'll we would just have to check there um, let's see I see I think you're you're working with you want me to Yeah, I got I got roga more handled. I think just pointing them to the container catalog was a huge help So if you're trying to run containers as root in open shift, you're gonna have problems and that's okay We want you to have those problems because running everything as root is not a healthy practice Yep. Yep. Yeah, it's uh You know the whole security thing that we we forcefully place upon you whether you want it or not Our info nodes declarable in the install config.yaml Not an install config you can create them in the manifest and have them deployed when the cluster is Set up, but you still have to day two go in and modify the workloads to actually schedule onto them Uh, christian hernandez did a great stream on this um Actually, I think he did one by himself and I think he and I did one together But it was last year. We'll dig out the links for those and I'll put them into the show notes Thank you. Um, she was here holy smokes. Yeah. Yeah, it was uh, I want to say it was Late summer early fall last year. Um, so and for anybody who doesn't know we do have a weekly blog post on open shift.com slash blog Goes out Friday morning where we recap everything that was in the show We provide links to all the stuff that's here. So if you miss the links in the chat or anything like that or If you if you didn't see something that I typed you can always go back into that Blog post and and find all that information again. It it publishes Thus far it's always published Friday morning early Friday morning I say early Friday morning. It would actually be lunchtime for ravi because you're you're over in the uk I've given up working out time zones. I just I just sit here forever I know I I I always feel bad because I talk to you a lot in like what is effectively mid-afternoon for me And I know that's like early evening, you know time for you and you always brush it off as oh But I have to deal with you know the west coast time zone because that's you know vmware stuff and I still feel bad Yeah, be honest Right, that's more questions around uh, dr s here. Yeah, please. Oh Yeah, okay. So there's there's I'm I think I'm getting the questions in a different order to to you guys But there's there's one one I wanted to call out here around affinity rules. Um So what should we do around how we manage affinity for control planes? um Now when you when you do a default install or ipi upi um op shift doesn't create any affinity rules at the v sphere level Um, so the vm's get get spun up, but there's no affinity rules If you want to start building a Resiliency into it and making sure that your your control planes are spread across physical sxi hosts Um, you need to go into virtual center. You need to to tag the vm and say right This is my first control plane. This is my second. This is my third None of these folks can exist on the same sxi host um At which point if they're already on an sxi host v sphere will just v motion them off and they'll get put on physically different servers um, so if you do lose a You know a physical piece of tin you're you're only going to lose one of those control nodes at a time Um, you can do exactly the same thing for workers So if you have a bunch of different workers that you've you've you've tagged with different workloads um, you can use affinity rules to spread those out across ESXi clusters and depending on the size of your environment that may also extend out to physical racks in the data center So you can have a policy that will let you say right actually I'm only going to have one control node um control node vm per rack in my data center And I'm going to put these workers across the sxi hosts So you you can be get quite sophisticated with your rule set as to where you want parts of open shift to be deployed Either in vcf or in or in just a homegrown home built v sphere environment um And v sphere is going to take care of that. Um open shift is not going to not going to care The the only thing you need to be aware of of course is if you provision up a new open shift node And or you or you recreate a control plane that's that's that's gone That doesn't have an affinity tag by default So you need to go in and recreate or use an automation to To do that for you. I hear power is really cool Yeah, it's it's really um the other the other things from Which is a nice feature Is you know, we've got a little tick box that says highly available tick that tick box for the control plane nodes Or you know any of the nodes really um that way if uh if we do lose a node Because of a hardware failure or it just stops responding for whatever reason um The v sphere will spin it up somewhere else immediately And and it's it's really really quick at doing that so if you lose a I mean if you lose a worker because of a hardware failure v sphere can often put the worker back Before open shift has realized that it's it's lost that node to start with um And yeah, and then the kubernetes layer will will schedule pods and handle all of its internal workload magic as well So they work really well together Um, but out of the box those settings aren't aren't turned on So high availability for control plane affinity rules to make sure you're not landing on particular physical hosts if you want to want that level of control And the only thing I'll add is I always recommend soft affinity rules Just in case especially if you have relatively small number of nodes in the cluster You know if if you have a a cluster that's You know only three nodes drs cluster vmware cluster That's only three nodes and you set hard affinity rules for Like the three control plane vms if one of those physical nodes goes down Then it won't restart that vm that control plane vm because it would violate that heart of anti affinity rule If it's a soft anti affinity rule it'll restart it and then when that physical capacity comes back it'll it'll automatically move it Um, so in the interest of time i'm gonna i'm gonna rapid fire some questions here Um, so daniel and this is in referring to the csi provisioner the primary benefits come if you're using v sand then, right? So I will say my perspective is no There's if you're using traditional data stores coming from whatever storage you're using you still get a tremendous amount of Visibility in vCenter as to what's going on with those Uh pvcs pvs right that are inside of there. Robbie Yep. No, absolutely. Um, if it's a data store in in vSphere Um, you can apply the same storage policies that you can with your your traditional data stores Um, as you can with vSan we we talk about vSan obviously because it's it's our thing but Yeah, anything that's a data store in vSphere cns is going to take care of it for you and you can you get that visibility at the At the vmware level, so it's not just tied down to vSan And you you said cns there, which is I don't think we I don't think we've explained that What what cns is and how it relates? well, um, yes a coordinated storage versus the They're just a driver Yeah, and cloud cloud native storage or cns is what vmware calls the csi driver, right? Yeah Yeah, so it's it's it's we have lots of fun with terminology. Let's talk about clusters, Andrew Is it possible to change cloud providers of an existing cluster? Unfortunately, no I wish that were possible. It would solve a lot of support headaches for us Um, why such hate for dhcp? Um, you know, I don't I don't know. I wish I knew I Andrew's opinion is there's a lot of Old old school think about dhcp, right? It was Never used in the data center for security reasons Right, you know, oh anybody can just go in and plug in and get an ip address and stuff like that But that being said, that's you know, just my opinion I find dhcp to be very helpful But then again these days I just run a lab. I don't run a data center anymore. So, right Uh, I got one here for you. Are you clearable in the install dash config yaml yet? Uh, oh, we we addressed that one. So unfortunately, no, we did. No, yeah So you can create them at install time just to just quick revisit you can create them at install time by putting the machine set yamls into the manifest file. So you would do a open shift install creating Manifests and then dump them in there So provision them, but you would have to go in after the fact and modify the services to actually deploy over there Gp data see that I debug into the node. Um, I've used ssh to access the nodes So my preferred so what is your preferred way of accessing the nodes? It's one more dangerous than the other So I tend to use the debug simply because it's easier. Um, you know, I don't have to, you know, use a specific or provide the specific SSH ID I don't have to constantly go in and edit my known hosts because I deploy clusters three or four times a day sometimes And if I'm sshing into those nodes, I'm constantly having to go in and remove them Um, so generally speaking, that's I use oc debug into the nodes Probably 95 of the time there are occasionally some things like if you need to access devices Um, I can find if I need to check and see whether or not it detected a usb device that I plugged in A lot of times that type of stuff. I will have to ssh in Um, and I see christian answered that question as well. Um Is there a way to control different cluster architectures in one panel? For example, I have three clusters and I think that we have ACM Sorry, I started to read the question and then it broke up a little I think so I think you might have asked Or might have helped answer that one chris ACM would be the yeah, I mean acm would be the red hat answer to that. Yeah um But like if it's like you need to give developers a view You know into all the clusters versus like the admin side where it's you know, the ops side Yeah, it kind of depends on what you're trying to do um Yeah, I think this is and andrew needs to get more familiar with with acm But I think what you're thinking is like, uh, where's my vcenter? It's like in vcenter, right? I can have multiple clusters like could I have you know cluster one in here and then this is my You know Open shift admin console or whatever for that cluster and go down to cluster two and see that one and so one interface Where I can join all of my clusters to be able to manage them Or same thing with cockpit right cockpit. I can have one host That is my cockpit host that connects to other hosts and is able to do those things. So I'm showing you this and I just realized that I'm not sharing that that window Ah, lovely. Thank you. Um, you know So anyways, so no unfortunately acm is the closest thing I know of but again I'm not super familiar with acm so that might not meet the needs in that So christian says, uh, my answer to everything is hard with cd um Get ops does solve a lot of problems along that route and you can definitely have that kind of uh All-in-one view with argo Uh across multiple nodes. I feel like not sure Um, um, so I see a question and robby. I don't know if you know of anything here Are there any kubernetes enhancements that will have insight into the fact or danger that The kubernetes nodes are running on the same physical host Oh Excellent question. Um, I know there's a lot of talk on the sig about it, but I don't know if anything's actually, uh What would be affinity rules, right like in kubernetes itself, but That would be an interesting rfe for the cloud provider. Maybe you know having the cloud provider Yeah able to either report that information or even configure affinity rules So the latest blog post on this is from kubernetes side Well this documentation, but um, I'll just link it here just to you know for Posterity, but they're kind of going through like how would you do this currently? And you know node isolation restrictions affinity and anti affinity rules are all in play there Yeah, um, so we'll lead I see ha fault tolerance. Um, so yeah ha and ft are not necessarily the same thing fault tolerance isn't it? Yeah fault tolerance is um It is the same vm Executing on two separate hosts at the same time So if one of the hosts fails the other one picks it up and runs with it immediately And to your point while lead there is a Um, a limit of eight cpus per vm for that and it that that uh using ft Well, it's fine if you need it. Um, be aware of two things one. Um, it can It uses a lot of network throughput right to replicate all of those That's from very specific requirements on On ft. Yeah, and it is super latency sensitive Yeah, I mean I I don't think I've ever tested it with With open shift or even fact with kubernetes because ha typically puts things back and before kubernetes cares um, yeah uh, suggest any tools To deploy an open shift cluster in one click apart from ansible Yeah, i'm just answering that right now, right? Like i'm thinking of a small lightweight kind of like Argo cd cluster like a kubernetes and docker kind of scenario actually robbie Is there anything in the v realize realm that can yeah? Um, yeah, I mean we can use V realize automation to do the one click button That kicks off the deployment for you um There's some stuff up front that you obviously have to provide with the configuration. I mean andrew we messed around with it with the um, upi installer And creating ignition files and automated that stuff Um ipi, I think it's it's fairly trivial to do actually. Yeah, ipi It isn't one click, but it's you know one command and then you fill out like you're What's your vcenter? What's your credentials and what ipis do you want to use and it kind of goes? Yeah, and and v realize will let you will will automatically provide all that stuff so Yeah, we don't have anything out of the box because v realize he's mostly around what is your environment look like How do you want to automate it? But we have the customers that have that one click Um, normally we we use it with some governance wrapped around it So the one click deployment is is more around are you allowed to deploy that 32 node openshift cluster? Or uh, perhaps do you want to get someone's permission first? Yeah So we've only got about five minutes left. Um, robbie one of the questions I wanted to ask you is We know that there's like Or we generally encourage folks to use csi You know for for customers who have nsx right? We generally encourage using nsx integrations. Are there other things that? Maybe in the vmware portfolio or in the red hat portfolio right ways that those two things can integrate and get kind of more value more benefit Um, yeah, I mean we spend a lot of time talking to customers that have A lot of the other solutions in the portfolio that are Normally, you know vm based But we've been moving everything to kind of be a more container focused over the last last few years So I'm going to take the opportunity to share share a slide and share a screen. Um, so give me a sec and I will so Our anti-affinity Going back to questions here after you can handle this one our anti-affinity rules recommended on the open shift you know playing the entire open shift cluster or Should you do it with vms and vmware? Right like What's the right answer there? So I would say Andrew's opinion is that anti-affinity rules Soft anti-affinity rules are strongly recommended for the control plane nodes Probably for infrastructure nodes if you're using those definitely ones that are running routers Um, and then worker nodes, it's really going to depend on you and your applications and a number of other things You know one of the things I I mean a conversation for another time is you know the philosophy of Do I you know if I'm deploying open shift to v sphere or really any virtual infrastructure, right? Do I have Fewer larger nodes, you know, maybe one vm per physical host Or do I have more smaller nodes where maybe I end up with three or four or five Open shift worker nodes per physical host and kind of the benefits and stuff there So but I don't want to interrupt from from what Robbie was gonna say. I mean, that's that's another great topic, right? I mean I I'm gonna just if you got can you see that is that working? Yes Perfect. So, I mean, we've we've been talking mostly around the just the v sphere cloud provider stuff Which is the stuff at the bottom here where You know, it's v sphere and vms, right? I mean, we talked a bit about cns class and csi drivers for the vSAN and storage um NSXT integration gives us the sort of full view Into the into the pod networks as it does the vm networks so the the bottom part is very much about giving you the You know the view that you have with vms, but at the open shift level and making everything kind of have a level playing field but the The other thing we've been working on a lot is You know, how do you do day two and how do you see? The open shift workloads the applications are running open shift alongside all of your other workloads So feeding feeding all the logs into something like log in the site a login site cloud Along with esx logs along with vm workload logs And getting that end-to-end view so you can see if the database that's sitting in oracle somewhere that you've never touched for the last decade Is causing you a problem with your with your microservices in open shift Example is where we've been focusing some of these integrations so Observe ability a wave front was was one of the first operators that we we got certified on operator hub actually andrew so You can you can go into open shift go to operator hub and say Give me wave front and that's going to give you the Dashboards a real-time view on open shift alongside vcenter and vSphere and vSAN And any other vm is that you've got in the environment So, you know, we focus very much on making sure open shift works around everything And plugs in the way you'd expect it to So, I mean, we don't really have time to go into any of this stuff But it you know, there's there's lots of other things beyond Just the cloud provider interface who we've been working on Yeah, I know, you know having and at the bottom of your slide there you had the Open shift on vmc on vmware cloud on aws That one's near and dear to my heart because you and I have worked on that one together and publish that So anybody who's a vmc customer and they want to deploy open shift on to vmc That's now a fully supported platform. Robbie and I went through and did the deployment guide for that and You know, I that was where I got Re familiarized with like log insight, you know, the last time I used log insight was see this is 2021. So, uh nine years ago And it was a great product then I can only assume it's only gotten better, you know in many ways But it was one of those like I've forgotten it existed So and you can absolutely configure, you know open shift To if you don't want to use the open shift logging service for whatever reason, you know If you've got log insight, you've already got one. Do you want to use two? Maybe maybe not But you can absolutely configure open shift to only forward to log insight or to Go to both right to the open shift cluster logging service and to log insight so on and so forth. So And it comes off sometimes where you you just want the operations folks to have Their logs in one place and then the dev guys Have their logs somewhere else And everyone has their favorite So I think we are at the top of the hour now So thank you everybody for all of your really phenomenal questions Great session. I'm going to take as many of these as I can. I'm going to put them into the blog post I'll I'll try and link to where we answer them in the video as best as I can If you have any additional questions if there's anything we didn't get to you can always reach out to me or chris So social media. I am at practical andrew on twitter Chris is at chris short. You can also send an email Andrew dot Sullivan at redhat.com Robbie is there you're on twitter as well. Yeah, I'm twitter and hit me up on ruby j on twitter And and I'll I'll work around you to answer as many questions as we can This was this was a lot of fun. Thank you for having me on. Yeah, we'll have to do it again So thank you very much. Robbie. Appreciate you joining us And to our audience. Thank you and we will see you tomorrow for the what's next presentation and next week at the same Same great team same great time same great channel. Yes, and I would like to remind everybody that red hat summit is coming So, uh, please register for that if you are interested in attending. Thank you And without further ado, I will turn everything off Thank you, Robbie for joining us. Really appreciate it buddy. Thank you very much