 Good morning. Good afternoon. Good evening wherever you're hailing from welcome to another edition of ask and open shift admin I'm Kershort executive producer of open shift TV. I am joined by the one and only Andrew Sullivan Andrew How are you doing today sir? I'm I'm doing well You know the the pollen storm here in Raleigh is starting to subside you lived here for a number of years Yeah, oh, that's so it's I think if we get another you know Good rainstorm to come through that'll do away with a lot of the pine pollen that's left and we'll be we'll be back to normal. So JP Dade, thank you for the The like on the intro animation Kersh and I were actually just saying how we've never actually listened to the music that goes along with it Like we know there's music we know this sound they told us there is but neither one of us We've always just watched it through the stream, which I mean, you know What's funny is it's like they just gave me the video and I was like oh, it's like six seconds Okay, no problem. Let me just jump it and throw it in like throw it in OBS and I was like this morning I realized I never had listened to it before Yeah, yeah funny how that is But yeah, but no the the creative team has done just phenomenal work with the move to you know From open shift admin office hour to ask an open shift admin and yeah all the artwork and the reoccurring stuff that they do to Keep all that up. So can't thank them enough. It's really been great. Yes, exactly So I know we we got distracted there and went a little off the rails But yeah, I am Andrew Sullivan technical marketing manager with the red hat cloud platforms business units And this is one of our office hours series of live streams So the goal here is for you our audience to ask us literally anything that's on your mind Whatever it is that's bothering you that you got questions about Right, it's anything that happens to come up that you want an answer to with regard to open shift We are here to help So Chris and I both have administrative backgrounds. We were both admins for a number of years and various roles and various organizations So we're happy to help with those If you have a developer question, we'll do our best But I can't guarantee that that will be able to answer that But that doesn't mean that we won't give up Regardless of whether of what question it is if we don't know the answer We'll be sure to go and find it and if we do give you an answer and it happens to be wrong Let us know. I love to be proven wrong. I was just having a conversation with one of our support guys this morning You know lit prove me wrong. I love it so With that being said if you all have any questions Don't hesitate any point in time put those into chat across any of the various streams or any of the various platforms that we're Streaming on we'll be sure to get those will answer those If you don't have any questions or in the absence of questions from you all are loving and adoring audience we do have a topic and I try to keep or try to have something to talk about in that respect and hopefully it triggers some questions for you as well and Today that topic is day two operations So red hats uses the nomenclature or we have this kind of day numbering scheme a day zero day one day two Day zero often refers to and I don't know how ubiquitous across the industry. This is I've heard very sometimes different numbers. I've heard different phases stuff like that. So Phase zero is often the architecture and design phase phase one is deployment and then phase two is essentially or Day two is essentially everything after that right day two until Decommissioning effectively which could be day ten thousand Well, that would be a long time. It's what day ten thousand. That's like 30 years. Yeah All right, maybe that one's a little long. We okay. Yeah, we don't want we don't want your day ten thousand operations to be honest Hopefully it's lots of updates in between, right? So yeah today I want to talk about day two and a lot of this topic Spurred from a series of blog posts that I did back in the gosh four or three four four days and it was four three That turned into a section of the documentation and since it's been a while since we talked about it And by that I mean I don't think we've ever talked about it on the stream It was just in the blog post. I wanted to revisit some of that and importantly I want to get your Perspective your feedback. What are the things that you're doing that you feel aren't well organized in the documents? What are the things that you feel might be missing or what are the things that are important to you as our audience from a day two perspective? However before we get started and I see some chat coming in yeah, I see your question there I answered it. We use hive. We don't use cluster API folks Pive is awesome take a look at it. There's a lot of fun cool stuff coming out of it So it's it's a project. I like a lot cluster API has its, you know use cases, but we use hive Yep, and then Catherine talked about that last week on the stream. We talked about the installation methods exactly so Next question is is there a plan for the oh 80? OADP operator Openship API for data protection to be supported by red hat for enterprise use is currently community only That I don't know So go check. I believe the answer to that is yes I don't know the time frame for that. So we do know the PM on that side So I can reach out to him and Chris. I'll chat that to you on our side. Yeah We'll get the answer to that But eventually I think is the answer. We just don't know yet when yep Yeah, so I strongly suspect that that will be, you know GA right a fully supported thing coming from red hat at some point in the future There will be an announcement of some sort keep your eye on the blog for sure. Yep And I would also suggest I don't know if tomorrow is one of the days So the the OCS team or I guess now their ODF The OpenShift data foundation team has a bi-weekly every other week live stream as well So that would be a good group to ask that question to as well, which is we we're I think we had to skip last week for some reason to be honest with you, but Yeah Next week we can talk about that for sure. Absolutely. Yeah We can bring that up with Michelle. So yeah, absolutely. Arvin asks, how does the V3 cluster administration differ from the four? And that is a big question. Yeah, that it's it's both It's deceivingly simple and incredibly complex, right? So the short answer is on the surface, not substantially, right? It's still OpenShift. It's still Kubernetes. You're still interacting with it. Yeah, I'm doing it. But when you step down just one step, it's hugely and dramatically different. Yes. And we've talked about that over a couple of shows. I'll see if I can dig up some of those. But effectively, the trigger or the reason why we went from OpenShift three to OpenShift four was the introduction and widespread adoption of operators for all things OpenShift. So at the core, this changes how we administer OpenShift. So, for example, with OpenShift three, the install was done via Ansible Playbooks, whenever I needed to change something. Yeah, lots. So whenever I needed to change something, whenever I needed to administer those nodes, the preferred way was to do it through an Ansible Playbook, right? If it was something that's red hatched, like we wanted you to do in a controlled manner or something like that, there was a playbook. If it was something that still supported still possible, but maybe I'm not going to say outside of the norm, but just not something that we had a playbook for, it was sure SSH into the node and do what you need to do or use your own Ansible Playbooks to configure something. But ultimately, it was managed outside of the cluster, right? Managing things like the underlying operating system, the underlying Kubernetes and KubeLit and all that other stuff was done externally. The move to operators brought all of that stuff inside. And instead of using an Ansible Playbook to go and say, you know, I'm going to pick on crony, right? Because that's one of the things I want to talk about today. Yeah, right. Configuring time synchronization instead of running an Ansible Playbook that says go and start the service on all of the nodes and put this configuration file in place and all those other things. Instead, we do it through what's now the machine config operator, right? Where I use the machine config operator to push configuration out to all of those nodes, right? The registry similarly, right? The registry instead of deploying those pods and providing that configuration as a config map, or I think it was a config map back in four or three days before you get too deep. What is the benefit of operators? Right? Like, I think that's the point that we're trying to get across here, right? Yeah, so that's a really good point. And I know that we have, and my tabs got scattered because I had to log out and log back in. So somewhere I've got a link to an episode that we did a while ago that was effectively what are operators for OpenShift administrators? And effectively what it does is it takes all of that knowledge, all of those actions, all of those things that were previously in an Ansible Playbook. Codifies them and then makes them available directly in the cluster. So before, right, if Red Hat created an Ansible Playbook, you could assume that it was Red Hat applying, you know, Red Hat engineering, applying our knowledge, our expertise around the system to say, this is how you should configure it, right? When I want to configure X, we're going to do these things. We're just putting that into the Kubernetes cluster itself and then executing that using Kubernetes native paradigms. So that's the big thing. And operators are a huge topic, right? There's literal, you know, weeks, long courses around operators and all of the possibilities that they have there and everything else. But the goal is, the purpose is, as an administrator, we get out of the business of going and managing individual nodes, right, SSH and connecting Ansible, whatever it is, to instead I use an operator and I configure the operator to how I want the cluster to look. And then the operator makes sure the cluster looks that way. And the best part about operators are you can use Helm charts, you can write them with Ansible, or you could do, you know, native Go to write your operators. And there's also a bunch of other projects. You can even write operators in Bash. Shh, don't tell anybody. Andy Block just had a shiver down his spine. I know, right? Yeah, yeah, yeah. So, yeah, and I also want to make it clear, you don't have to write any operators, right? No, you don't. As an administrator, yeah, you don't have to worry about, you know, I would say 98% of the things you want to do are probably already covered by an operator or can be done by an operator today. It really just depends on how much customization you're doing. Let's see. I'm just looking over the questions real quick here or the chat, rather, before I get into them. It's going to drop it in a bunch of links after Arvin's question. And JP Day saying operators are awesome because they can upgrade themselves, basically, which is entirely true. Including the cluster itself. Yeah, they can upgrade the whole cluster for you. So you've got some stuff that's top of mind for day two. I do, yeah. So first, for those who regularly tune into this particular live stream, you know, that I tend to start the show off talking about some things that have come up in the last week or kind of since the last episode that I feel are important to you, our audience, to administrators for OpenShift that we can bring up and we can talk about. So a lot of times these come from both you, the audience, right, if you reach out and ask us questions to help clarify things or point out, Andrew, you were full of crap, you were wrong. Again, I enjoy issuing retractions and corrections, so please don't hesitate to do that at any time. But it also comes internally, right, when our field folks on our engineering folks ask and answer questions and stuff like that. I take that knowledge, I swirl it away, and then provide those pearls of wisdom to you all. So the first one that I want to talk about is I've seen this question come up, and at first I thought it was a fluke. And what I'm trying to figure out now is, is this just a, is there an easily misinterpreted phrase in the documents or something like that? But it's regarding subnets used by OpenShift clusters. Oh yeah, that's a good one, yeah. So it's come up a few times, so I figured I should go ahead and discuss it. And essentially it comes down to, if I'm deploying OpenShift, what is Red Hat's recommendation or guidance for how I should configure my underlying subnetting? So first I'm going to take a slight detour and say, when you look at an install config, remember that there is three different subnets that it will ask for. Let me see if I have one available that I can use. And GKUMAR, I see your question, we'll get to it in a second. Because I think I have, yeah I do. All right, so I'm going to share my screen real quick here. Are you the one I want? Yes, yes you are the one I want. All right, so please let us know in chat, if you can't read this, hopefully it's big enough. So this is just an install config that I have, right, I ended this one in .clean because it has no pull secret inside of it, so or SSH key inside of it. So if we look at the networking section here, there are three network things that it's asking for. So the first one being the cluster network, this is effectively what subnet is it going to use for in the case of OpenShift SDN for each pod network on each host? So what that really means is it's going to take this slash 14, it's going to carve it up into a 23 for each one of the nodes in the cluster. And then when a pod is instantiated on that node, it gets assigned an IP address from that slash 23. That's all it means, right. And then OpenShift SDN and IP tables does effectively masquerade, right, Nat, in order to access those across the various nodes. So the second one, I'm going to jump down here to the bottom service network. This is just the range of IP addresses that when you create a service object in Kubernetes and OpenShift, this is the IP address range that it will be assigned from. So the third one is the important one here. And this machine network.sider is we're telling Kubernetes, we're telling OpenShift to look for a interface, a physical interface on the nodes on this network. And when you find that interface, use that one to instantiate the SDN and do a lot of other things inside of the cluster. So this is, if you have all of your worker nodes or all of your nodes in the cluster, I should say, have just a single interface, it's not critical that you configure this correctly. Basically, the cluster will probably work just fine. You'll never notice it the majority of the time, unless you're using a proxy or there's a couple of other scenarios in there. So for example, this network automatically gets added to the no proxy list. So if that happens to be something that you're using, you want to definitely make sure that it is configured correctly. Likewise, if you have more than one interface, make sure that that value is configured correctly. Because if it doesn't see an interface that has an IP in this range, it basically takes the first network adapter. And particularly if you're doing a non-integrated install and it's across different platforms, I've got some nodes that are in vSphere, I've got some nodes that are right at virtualization, I've got some nodes that are quote unquote bare metal, that are physical servers. Those interfaces could be in a different order. What is, maybe this machine network is the first interface in vSphere, but it's the fourth interface on my bare metal nodes, on my physical nodes. And that can cause a lot of problems. So the question is for this machine network, should I assign a dedicated subnet for every cluster that I create? Cluster one gets 14.0 slash 24, cluster two gets 15.0 slash 24, cluster three gets 16.0, so on and so forth. The answer to that is no, you don't need dedicated subnets. It is sometimes handy just as an administrator, because simple is what's the KISS principle, keep it simple, silly. I'm like that now. Yeah, so sometimes that is good, but it is not a requirement from a Red Hat perspective. So the next iteration on that is, well, should I have a dedicated subnet that only hosts OpenShift clusters? And the answer there is no, you don't need to do that, but there are some things to be aware of if you have multiple OpenShift clusters in the same subnet. And in particular, if you're using IPI. So if you're using on-prem IPI, so that would be bare metal IPI, right, physical IPI, vSphere, Red Hat virtualization, open stack in some circumstances, so on and so forth. Any of those on-prem IPIs, they use a keep alive D instance to manage the virtual IPs associated with API and Ingress, start on apps. So being keep alive D, it uses VRRP, which I don't remember what VRRP stands for now, virtual routing and recovery. No, I think it's fine. I forget that matter, yeah. Virtual routing, our protocol. Yeah. I don't know. So that, but yes. I'll look it up. Anyways, so those VRRP IDs need to be unique across the entire subnet or across the entire domain. Yeah, there we go. Thank you. There you go. So it uses multicast across the entire subnet to basically advertise, hey, where's my peers with this same VRRP domain ID? So the function that generates those domain IPs with an open shift IPI on-prem instance bases it off of the cluster name. So if you, and it is entirely possible to have a duplicate of those names or of those domain IDs, if that happens, it can be bad, right? Basically it'll think that there's nodes that shouldn't be there and it'll get confused and you won't be able to stand up, for example, the new cluster. The existing cluster will probably just have some, you know, random errors that show up. It won't accept those new nodes because it won't recognize them, but the new cluster wouldn't stand up. So just be aware of that. Other than that, no real issues, nothing to be concerned about. Likewise, it's perfectly fine to have a subnet that is shared between open shift and other things. All right, so you can, you know, put your, I don't know, domain controller, DNS server, whatever on the same subnet as open shift, you won't have any harm come from that whether it's IPI or UPI. All right, I'm going to pause because I think there's some questions. Well, one question has been answered. The Frankenholm question has been answered in chat. So Frankenholm, if you see, if you don't see it in chat, just keep scrolling down. It's right there at the very bottom. Emporium gave us the good name there. G. Kumar asked, is there any Red Hat certified egress IP operator in the works? There are only community versions for egress IPAM operators available, which are not fully supported by managed clusters like ARO. That I can't tell you. Yeah, that's a good one though. So I thought OVN Kubernetes could do egress IPs, but I don't know if it does the IPAM side. Yeah, it doesn't now. So the whereabouts CNI might be needed to do the IPAM part of it. But for a managed service, that's a whole another bag of door knobs. So the good news is, so one, we'll track down that. I've also reached out to our product manager for the managed services. So hopefully we'll be having an episode about managed services coming up in the not too distant future. So Chris, are you are you researching that one or reaching out on that one? You know, I'm looking through my list of people trying to figure out who to reach out to, to be honest with you. Okay, I'll, we'll take that one for a note. We'll follow up on that one. Yeah, the, that's a good one. Who is the operator hub guru? JP Dade, OpenShift SDN versus OVN Kubernetes. So we talked about that with Mark Curry a couple of weeks ago. And in general, there's not a substantial difference between them. OVN Kubernetes will move to being the default SDN in the OpenShift 4.9 timeframe, I believe you said. For now, it really depends on what, on whether or not you need the additional features or capabilities of OVN Kubernetes. So the big ones there are being cross operating system. So if you're using Windows nodes, for example, you need to be able to do that. If you want to use, I don't want to say more modern overlay technology, but newer overlay technology, Geneve versus VXLAN, that type of stuff. There's also a docs link that compares the two of them and gives you that option. And I'll see if I can find that at some point. Oh my gosh, but. SJBYLO, I don't even know what it is. SJBYLO, I don't even know how to say that, just shared a operator they wrote in Bash. Nice. Thank you for that. Yeah, that's cool. Example operator written in Bash, that's nice. I'm going to look at this later and maybe break it a lot. All right. So the second thing I wanted to talk about, this one is pretty straightforward, which is if you're familiar with, and I don't know when we started doing them, I think four or five quick starts. Yes. Four, five, and now more so in four, seven, yeah. Yeah. So quick starts are a way that we offer basically directly in the OpenShift interface, the ability to have a guided tour through some action. So the good news is, and I'm going to share my screen again, but this time I'm going to share the browser. So the good news is you can create these yourself. So for us administrators who, particularly like the senior administrators, et cetera, who are working on basically making things as easy and simple and consumable as possible for our junior peers, as well as maybe as administrators for developers or other people, you can write these little help scripts and it shows up inside of the cluster. Do I have one of these? I do. So it shows up, like we have this, brings up this little thing on the side and it says, okay, let me start doing this. What do I need to do? So click on the perspective switcher and it highlights it over here. I want to go to developer. I don't need the tour here, right? In the navigation menu, click add. So it's kind of this, right? You can step through, you can provide as much help or as much detail inside of your quick starts as you would like. The goal being provide that help, right? If you've got users who are constantly asking you the same questions over and over and over again, you can create these quick starts, deployment of the cluster, and then let them self-help. I won't make any comments on how prone users are to actually reading those things. We all know that KCSs are great in, our knowledge bases are great in principle, but not everybody reads those, but it's an attempt at making that easier anyways. An important one. So I think it was, gosh, it was a month ago we talked about, maybe even longer, like five weeks, we talked about OpenShift 4.7 was released. I am sad to say, I think is the feeling I'm going to choose that updates from 4.6 to 4.7 are still blocked. So this is the same issue that we've been talking about for a few weeks now. Effectively there is, so VMware with, VMware or hardware versions 14 and later enabled hardware offload for VXLand. RHEL kernel, or the kernel in RHEL 8.3 basically did the same thing or had something similar inside of there. So RHEL 8.3 is the RHEL backing OpenShift 4.7 and there's some sort of conflict that's happening inside of there. The result is packet loss. So anywhere from a few percentage points upwards, packet loss is of course bad. So we have a, we'll pile up and we have a BZ if you want to follow this. So you can see inside of here and I'm going to scroll down because if you are really desperate to get a 4.7 cluster up on vSphere and all that, there are some workarounds available inside of here. Right. So here's someone who has gone through and disabled all of the offload, all of the checksum, all of these things inside of here and they were able to get this up and running. So there's the potential for some workarounds but there is nothing that we have. But there's trade off there. Yeah. Yeah. Like doing everything in software not as good as doing it in hardware. So keep that in mind. Yeah. So there appears to be some workarounds. I haven't tested any of these. I don't know what our official stance on any of them is but just be aware it is a blocker. It is preventing those upgrades. I know I sympathize. Even though... I empathize. vSphere is a big platform for OpenShift. It is not the only platform and I understand it's frustrating if you're not using vSphere but it's blocking upgrades for you as well. So just know that we're working on it. I heard from engineering and I'm on the threads directly between OpenShift Engineering, REL, Kernel Engineering and VMware Engineering and seeing all the things that's going on. So we are working on it and are very cognizant of it. Yeah. Very, very cognizant. Just ask for patience. That's all. Let's see what was next. Oh. So I'm going to use the release of CodeReady Containers 1.25. So if you weren't aware, CodeReady Containers 1.25 released either yesterday or this morning. So it's available. It's an OpenShift 4.5.7 based. So if you want to get up and testing and experimenting with 4.5.7, that's one of the ways to do it. But I also want to use this opportunity to talk about what are some other options if I want to kind of have a test cluster or I want to experiment or I just want to, you know, what's the minimum amount of resources I can use or I can have available to me to get OpenShift up and running. And aside from CodeReady Containers, which remember deploys to your workstation, it creates a virtual machine. So whether you're Mac OS Windows or Linux-based, creates a virtual machine inside of there, and then installs a single node OpenShift cluster. There are some other options. So first, the CEE folks, the support folks, hopefully created this KCS, which basically explains that today, right now, the smallest cluster that you can deploy is three nodes. So I would suggest keeping an eye on this, right, you can click the little watch button up here. I think it's this one, follow. So you can be alerted when they update this because this will be changing in the future. And in particular, if you watch the what's next presentation, which was a little over a month ago now, and actually we can do the OpenShift.com new and next, I think is the shortcut link. Something like that. Next to new. New. Next to new. There we go. Next to new. There you go. So that's the shortcut to always get to those two presentations, the what's next and what's new. Anyway, so if you watch that, you remember that we are hoping to have single node clusters and tech preview with OpenShift 4.8. So 4.8 is still a ways away. But remember that you can access nightly builds. So in the not too distant future, and I haven't checked this, I haven't looked, when nightlies start to become available for 4.8, it may be entirely possible to do a single node cluster in that like super early nightly tech preview, tech or dev preview. I think it's a, would that be an alpha of an alpha? Anyway, pulling it from source basically at that point. Yeah. So anyways, we are on the cusp of having those, single node clusters available to do that testing, to do that validation, to just get hands on and start learning about OpenShift in a minimal environment that isn't your local laptop or desktop. So JP data asked real quick, can you install 4.7 on like fresh on VMware? Or is it still a problem there? So from what I have heard, I'm assuming no. It's 90% yes, but only under one scenario. And that is if you use VM hardware version 13. So I have heard mixed things about, because basically only VM version or VM hardware version 14 plus is affected. But I don't know if that's because we haven't, we just haven't heard about it yet with VM 13, VM hardware version 13. Sorry, I'm having, in my head every time I say virtual hardware, I just giggle a little bit because virtual. Virtual hardware, yeah. Anyways, so one of those oxymorons. So anyways, maybe, yeah, maybe with VM hardware version 13, it would work. Just keep in mind that VM hardware version 13 limits you and other areas, most especially the VMware CSI provisioner. So their provision requires VM hardware version 15 or later. So just be aware of that. It may be possible. I would definitely recommend testing it, not using it in production until you have fully and thoroughly tested that you're not going to experience any issues associated with that. Okay, so go ahead. Well, we'll lead was asking about is cluster API covered by one of the operators and he's on his phone. So he can't really type his questions very well at the moment, which I can't blame him. But Hive replaces the functionality of cluster API essentially. Yeah. So this is something of operator, right? Yeah. And this is something that's a bit strange with Red Hat and with OpenShift specifically because we sometimes mistakenly use the term cluster API when we mean cloud API, when we really mean machine API. So first let's be careful with the nomenclature and the naming that we use. So I want to take a step back. So cluster API is the set of APIs that are being driven by VMware and I think Google for creating new clusters programmatically. So basically you submit an object, right a Kubernetes object that says spin me up a new cluster that is X and Y and Z parameters. And cluster API is that API that does something with that to instantiate the new cluster. So as Chris pointed out earlier, OpenShift, Red Hat, rather than using cluster API, we use a thing called Hive. So if we go to github.com slash OpenShift and we do a search for Hive here, you can see we have this Hive. And Chris, I know you linked this earlier in the chat. So from here you can look at Hive, you can look kind of across it and it goes through and kind of explains what it does, how to deploy it, how to take advantage of it. I think, and this is one of those, like Catherine brought it up last week and it reminded me that Hive has been on my agenda to look at for the better part of a year and a half and I just haven't gotten there. So I believe that there is an operator so you can deploy kind of a, you can deploy Hive into a cluster to deploy clusters from there, but I don't know for sure. So let me log into my console real quick and look. Since it just logged me out, that'll take a second. Thanks, cluster. I'm going to say yes, because just looking at the installation instructions, there's a CRD. So yeah, so there you go. It's got to be something. Yeah, there's something there. Hive for Red Hat OpenShift, it's a community operator right now. I would assume that that's going to be better supported in the future, just because it is provided by Red Hat. So yeah, where's my cluster? There's that functionality, I think change. And I can find the link to the operator thing. I'm a jiggy. So according to that, oh yeah, that's right there. There we go. Just go to operator hub inside of the cluster. Yeah, right there. Search for Hive. And it actually, the repo it links to is the Hive repo you were just showing. So there you go. Yeah, well, that was easy. Okay, done. Moving along. All I had to do was log into my cluster and look for it. So SOGUT, see you? I'm not even sure who, I'm not, I can't. Sorry, today's not my day for words. What's the motivation for single node OpenShift versus code ready containers? So as a product, single node OpenShift is meant for edge type scenarios, right? I'm out at a cell phone tower. I'm at a remote office, right? Something like that where I need a- Your car. Yeah, right. Whatever that happens to be, I need a very small deployment where I have the supports, the features, the capabilities of OpenShift without necessarily all of the resiliency of a three node cluster, right? Because remember, it's only one node. If that node fails, it's nothing to fail over to, right? So code ready containers is targeted specifically at developers, although it's also applicable to us administrators. The goal being on my laptop or my desktop, I use code ready containers to instantiate OpenShift and then I can start writing my application and I can deploy it locally. I can take advantage of it. It's kind of the local version of code ready workspaces. So code ready workspaces being in a full OpenShift cluster, that development environment, that test environment, that I as a developer can interact with. So different use cases for us as administrators, at least my previous job, I didn't have a desktop. I didn't even have a laptop. I had a VDI. I had a virtual desktop. So using something like code ready containers just wasn't an option. I had a thin client and that was all I was given and all I was allowed to use to do any of my stuff. But ironically and strangely, I was an administrator so I had access to vCenter. And so I could go in and I could deploy VMs all day long. It was just, I couldn't do it locally. So single node cluster as an administrator means that I can deploy a cluster. I can do all that testing and validation that I would do normally. Nice. My lights are dying on me here so I got to make a quick change. Here we go. Oh no. Does it mean you're gonna get all RGB on us? No, I just, I will look like I'm sitting in the dark, basically. That's all. So I know we're, time flies, you know, when you're having fun. And I know that we've gone through quite a bit of time already. So I have a series of topics or a series of things that I want to talk about with regard to the day two stuff that we haven't already talked about. So the good news is of the, gosh, Chris, how many bullets do I have here? 20? 20, 30 now. Yeah. So there's no way we're gonna cover all of these things in the 20 or so minutes that we have remaining. So just make sure you look for the blog post on Friday. I'll have this whole list inside of there with links to all of the relevant stuff. So that way you can kind of follow along at home in your own time at your own pace as the case may be. I'll also say, so our contact information is always in those blog posts where you can reach out to us on social media on Discord, et cetera, to ask questions. Don't hesitate, you know, at any point in time to reach out and, you know, Andrew, why are you saying this or why should I care about this or I don't care about this? I really care about that type of thing. Don't hesitate at any point in time to reach out with any of that stuff. So day two. So remember at the beginning I said, you know, day one is kind of the installation process. So if you read the little abstract that we create for each one of these episodes, right, I think this one opens with like congratulations, you just deployed your cluster. You know, the OpenShift install just finished. Now what? And so that was the impetus where we started with this topic. And I broke this up into two different episodes. The first one kind of focusing on the low level cluster and node configuration type things. The second episode, which will be next week so far as I know, focusing on, I'll focus a little bit on storage, right? So things like CSI entry provisioners, networking config, if you need to do networking, day two networking stuff, and also user config. Basically, what do I need to do to prepare myself and my cluster to host, you know, users to host those developers and the applications that they're creating and turning through and that type stuff? You know, one of those important ones, like how do I protect the cluster? So today, what I want to talk about is, again, those kind of low level node and cluster level operations that are important, or at least from my perspective, important. And again, these may not apply to everyone. There may be things that you do care about that I don't list here. That doesn't mean that they're not important. Just be aware. All right. So the first thing I want to highlight here, aha. So I'm in the OpenShift 4.7 documentation page here and all I've done. So at the top level here, I've got this post installation configuration section. So this section of documentation is meant to be a collection of or an attempt at a collection of these types of operations. So this is always a good starting point for basically, what do I do now that my cluster has been deployed? What do I need to do next? So if you don't know of anywhere else to go, this is a good place to start. I'll also offer, if you scroll down here, to the stability and performance section, there's a number of things inside of this section of the documentation as well that are important. Just look through here. Look for things that, you know, check through each one of these like CPU manager, right? Do I care about, you know, using CPU manager for any of my applications? Do any of them need the features that are offered by this? If so, I'll deploy the operator and, you know, configure it to take advantage of or to make available those features to those applications. So let's come back up here to our, what am I looking for here? I want cluster tests is where I'm going to start. Actually, no, I'm going to start a machine test. So the first one that I'll talk about here is, remember, particularly if you, you know, just follow the default documentation or if you just did a default IPI install, you're only going to have two or three worker nodes. So depending on your sizing, you may need to deploy additional nodes, right? And this is where things get, and we've talked about sizing as a topic before, sizing can be difficult. You know, what's that saying about, you know, predictions are hard, especially about the future. Yeah. So we can do all of the sizing that we want. And there's documentation on how to do that. Come on, window. I think it's inside of here. So control plane node sizing, you know, all of that type of stuff. There's also, if we come down here, I'm not going to scroll anymore, but there's also infrastructure node sizing. Basically, the goal is to make a best effort estimation of what those sizes are going to be. And sorry, what was that? Click, it's the, nope. Yeah. Sorry, go back. And third link down control plane node sizing. Yeah. You mean post that one? That's what you showed earlier. Okay. Sorry. Yeah. Yeah. My bad. Post that one into the chat here. Oh, thanks. So, you know, if, like most things, open shift related, you don't have to get it right day one, but it's definitely easiest if you get it right day one. So let's say that, you know, you looked at this, you said, oh, I'll never have more than a hundred worker nodes, you know, or I'll never have more than 25 worker nodes, four CPUs and 16 gigabytes of memory for the control plane nodes is great. So what happens when I hit 60 worker nodes and my control plane just can't, can't keep up, you can change that day two, right? It's a little bit of a hassle isn't the right word, but it's definitely some work compared to like worker nodes, right? Effectively the safest way of doing it is to fail the control plane nodes one at a time and replace them with the new resources. Except with non-integrated installs, with non-integrated installs, I think you can turn off the nodes, adjust the resources and then turn them back on. But it's one of those like, yeah, it's always best to do it or to hopefully get our guesstimate as close as possible day one. But for worker nodes, maybe you deploy the first set of worker nodes using the defaults, which are relatively small. I think with most cloud providers, it's like two CPUs and eight gigs of RAM, something like that. If you followed the documentation for the install, pick on vSphere down here. And I'm going to choose user provision infrastructure and I'm going to choose minimum network or minimum resource requirements. You can see if you just go with the minimum here, it's two CPUs and eight gigabytes of RAM. So if that's done enough for your workloads, rather than spinning up, you know, dozens of nodes or something like that, you can resize these by effectively with IPI creating a new machine set and specifying a different CPU and memory count. Maybe I need to change to a different storage domain or data store, right? Create a new machine set that's pointed at the new data store and then scale up the new machine set and scale down the original one. And you'll end up with those resources moving around and getting to the right nodes, ideally before you have any applications hosted inside of the cluster. You can also create machines of different types and different sizes. And this is importantly where things like taints and node labels come into play, which are controlled through machine sets and IPI or you can manually label them with UPI and non-integrated. So that is important for things like maybe I have some nodes that have GPUs, maybe I have nodes that have, you know, different networking connectivity or something like that for specific applications. Maybe they have different local storage. I've seen that happen before with some customers where they use local storage on the host for very high performance applications like databases and stuff like that. So I want a subset of nodes that have these, you know, massive and expensive NVMe devices. I only want my database on those. Masses, aka this big. Figuratively, you know, capacity-wise. So, you know, having that or doing that additional connectivity beyond just the initial connectivity. Node configuration and layout beyond the initial deployment is really super important. I'll also say that, you know, yes, machine autoscaler, cluster autoscaler is there to help with some of those actions, but they still have to be configured too. As we heard Catherine say last week, you absolutely can use machine sets to do node autoscaling with a UPI deployment. And likewise, you can absolutely manually add nodes outside of a machine set with an IPI deployment. Nothing wrong with that, perfectly supported, and it allows you that flexibility to really tune and adjust the node configuration, the node layout to what's best for your infrastructure. I will say because this comes up, or I'll paste this in the chat because it comes up often, if we're talking like, you know, day, I don't know, 372. So, you know, it's been a minute since we did an install and oh no, I've completely lost track of things like the ignition files that I used to spin up nodes originally. So you can retrieve that file using that command right there, OC extract. So that pulls out that ignition and we use that. We point the nodes with UPI or bare metal install. We point the nodes at that file that's hosted somewhere on an HTTP server or depending on the install method like vSphere UPI, I can attach it directly to the virtual machine definition. All right. Sorry, dealing with SickDog. No, no, you're fine. I hope she's feeling better. I know you were telling me that she wasn't doing, wasn't feeling great before we started, so. Yeah, no, she's at a rough night and now she wants something and I can't read her new tones and signals and nuances and stuff, so I don't know what's going on. It's fun. Anyways. Yeah, well, I hope she's, I hope she gets better. Me too. That's rough. Yeah. No pun intended. Yeah. That was, yeah, sorry. So in that same line of thought, if you are choosing to use infrastructure nodes, remember they're not required, but you'll find that most Red Haters encourage you to use them because effectively infra nodes and the workload designed for infra, so things like the logging service, metric service, all of that other stuff. Remember, you don't have to entitle those. Your entitlements are reserved or only necessary for nodes that have your applications on them. So, unless they're co-hosted, in which case now the whole node has to be licensed. So, infra nodes are not a bad thing. Definitely encourage you to set those up and I do have a link in the documentation here. Let me copy that out. Well, I said I had a link, but I know where it's up. If we go down here to the nodes section and we look for, maybe it's machine management, here we go, creating infrastructure machine sets. So, this walks through how to create a machine set with IPI that will deploy, we can scale it up, it will deploy those new nodes that automatically or already have that infrastructure tag. Conversely, if you're using a UPI or a bare metal type of installation, non-integrated type of installation, you can add those in after the fact. So, you can go in and relabel or add a label to your machines, your nodes. And then machine config operator will take any action if it needs to. To add additional configuration or roles. Importantly, when you're creating an infrastructure machine set, you need to configure each one of the services to be deployed to that infrastructure machine. Let me paste this link inside of there. So, if I come down here, for example, to logging, and I go to configuring logging, what we have inside of here is, let's see, logging storage, using tolerations to control OpenShiftLoggingPOD placement. So, this is effectively how we go through and we configure, for example, Elasticsearch, which is a component of the logging service, so that it will run on our specific and only on our specific infrastructure nodes. So, using a combination of a toleration for the infra taint, as well as a node selector for those infer nodes. And I'm going to take a look at chat here. Yeah, Penguin Whisperer, Baconfork. I know Chris and I have been chatting about it. His dog got sick a few days ago. But what he told me before, and I'll let him fill in any details as he has an opportunity, is it's just a temporary thing. She should be getting better. So, it's just a rough few days. And I definitely sympathize with that. Next on my list here, I see you're back. I am back, sorry about that, folks. I have no idea what she needs, but she needs something. So, I tried to fill everything, and we'll go from there. Yep, yep, well, you know, because you and I are on the same team, and we have our Slack chat. You can always reach out if you need any help with anything. Yeah, thank you. So, the next one that I will talk about here is things that modify node configuration or that need node configuration modified. So, the example I'm going to use here is NetApp Trident. So, if you are a NetApp customer, you want to use Trident as your CSI provisioner for storage volumes. And specifically, you want to use iSCSI volumes. They recommend in their documentation a series of worker configuration things for the nodes. Here, I'll copy this guy out and bring it up. So, we can see it down here. If I'm using iSCSI with RHEL, then I want to do, right, make sure these packages are installed. I want to set the session scanning to manual, so on and so forth. So, that still applies with OpenShift and with CoreOS. We just have to go about doing it in a slightly different way. So, I'm going to copy and paste this gist link into the chat here. And effectively, what this gist is doing is two things. So, first, it takes a little bash script that applies NetApp's recommended settings. You can do it. And then the second part is, it takes that script and it puts it onto the node using a machine config. Right, so we're creating a machine configuration. Right, here's the name. It's going to apply to any worker node or any node that has a role of worker. And we want it to put the contents of that script up above onto this path. And then we're going to create a systemD unit that just executes that script. Basically, make sure every time the node starts that we start the Iskazi daemon, that we have multi-path configured, all of these other things inside of it. We also want to make sure that those services are turned on as needed. So, this is just one example. I happened to have recently helped a co or a joint customer, NetApp OpenShift customer with exactly this problem, which is why I use this example. But there are others out there. Right, there is, you know, I'm sure that's Dell EMC, I'm sure Hitachi Fujitsu, many of our other certified partners make similar recommendations. So, this is one of the ways that you can apply that. One thing, and you've probably heard us say this before, is you can't install RPMs. I mean, I think you technically can, but they don't persist inside of CorOS. So, if there's something that needs to be added to the host, that's like, are you going to be able to do that? So, this is one of the ways that we're going to do that. So, if there's something that needs to be added to the host, that's like, RPM install, that probably won't work. But that's a limitation of CorOS. Let's switch over and make sure that there's no questions or anything. Again, we've only got a couple of minutes left. So, any last minute questions, please don't hesitate to reach out with those. Throw them in now. And then I think the last thing that I will talk about today, and I'll follow on with a couple more of these next week as well as putting them all into the blog post. I mentioned before, configuring time synchronization. So, this link I'll paste. Yeah, so, you know, I'm an old school vSphere admin. You know, and time synchronization was always a big deal. Yeah. So, it's like the first thing I always look for, you know, and especially it's also important with OpenShift, because things like, you know, certificates depend on time being accurate, logging depends on time being accurate, all kinds of other things. Time is everything is what I used to tell people. Yeah, so, you know, configuring time synchronization, super important is one of the first things I do with all of my clusters. So, I'll use that as the last one I talked about today. But I also want to highlight that if you leverage GitOps and the Argo CD operator, you can make this super easy to reproduce across all of your clusters. Definitely. So, if you're not familiar with GitOps, if you haven't started to look into that and check and see whether or not it's a viable option, definitely, definitely check out Christian Hernandez live stream. So, his happens every other Thursday. So, not tomorrow, but next week I believe is the next one. Yes. So, definitely check out his live stream. He goes through a lot of great stuff around how to, oftentimes it's from a developer perspective, right? How to deploy applications. But the same exact stuff applies to administrators and configuring the cluster. He's also been doing, and I've been helping to review some of the blog posts he's been putting on OpenShift.com. So, some really great blog posts there as well about how to, like I need to do these things in this specific order. I need to create a meta application of multiple applications, right? Or multiple. I'm using application here because that's the GitOps term, but really it's any Kubernetes configuration that you want to apply, you know, like a machine config operation. Three minutes. Yep. Two minutes. Sorry. So, maybe that's a good topic for us to highlight at some point. We can do a demo of using GitOps to administer the cluster, not just to deploy applications. That's a very good idea. I know. Any excuse to bring Christian on and interrupt his workout, you know? Yeah. You know, I'm surprised he's not on right now listening to us. He may be. He's probably, you know, cursing us as he's listening like a podcast. Okay. So, as you said, we're basically down to the wire here because we have a hard stop at noon Eastern time, which is in less than two minutes now. So, first thing I want to say is thank you so much to our audience today. Really appreciate everybody who has been on, who has asked questions, who's interacted with us. Please, please, please. If you have any additional questions, if there's anything we didn't get to, if there's anything you're curious about, you can reach out. Andrew dot Sullivan at red hat.com. If email is your preferred method, there's also discord, which I'm sure Chris is posting the discord link right now. There is also Twitter. You can reach out to me at practical Andrew, which is like my username there in Twitch, which gets rebroadcast across the others. You can reach out to me at any point in time. Happy to answer those questions. And yeah, that's, that's all I got. So you got. All right. Without further ado. Thank you all for joining us today. Really appreciate it. Sorry. I had to duck out for a little bit. But we will find answers to those questions that we said we were going to. I'm already working on a couple of them right now. But coming up next, OpenShift Commons with one of my very good friends, Baruch from JFrog. I can't say his last name and he knows that. So that's okay. All right, folks. See you soon. Bye now.