 Good morning. Good afternoon. Good evening. Welcome to another ask an open shift admin office hours. I am Joe I am Chris short's executive producer of open shift TV. I am joined by the one and only Andrew Sullivan You know, what do we want to call you like vert open shift? Star to the gods what I mean the cold guy. Yeah, you know that That guy that says this is documented just do it like that Where's Chris Christian calls me the cuddly curmudgeon, right the cuddly curmudgeon exactly right like very approachable, but like don't Don't ask him dumb questions. No on this show. There's no such thing as dumb questions. So no, it's uh, I'm I think Christian calls me a curmudgeon because I tend to be very conservative in my approach to things that's a It's a Admittedly a learned behavior, right? You know, I was a an administrator for a long time so, you know, the the first big-boy job if you will that I had was as a an Administrator for a production v-sphere environments and our our storage admin decided one day that nope I'm just I'm done with this. I don't want to do this anymore. So I was also the storage admin and You know learned the hard way and learned through trial and error things like You know, don't accidentally delete the NFS data store from underneath, you know v-sphere. That's bad You know, don't don't trust the network guys when they say, oh, we'll just reboot the router It won't affect anything because no no all of that IP based storage you're using is traversing the router God So yeah, it it's one of those learned behaviors of trust but verify, right? Like you've just reminded me of a story about a reboot the router story where I had to like literally stop somebody's hand from touching the power switch before right like No, don't do that Yeah, it's Yeah, and my favorite one was I was working with the government and we had a junior enlisted person who was responsible for vacuuming the data center and they Were traversing through doors and the vacuum hose caught the lid for the EPO right the emergency power off button and of course when the lid lift ups lifts up it sets off an alarm right and The immediate reaction was well if I mash this button the alarm will stop Which was true It's not the alarm along with the rest of the data center. So wow that was that was a fun evening Just wow. Yeah, so anyways cuddly curmudgeon You know, it's less about you know asking questions or you know I try to never take the approach of you should already know this right? Why don't you know right? No, because you know, what's what's that xkcd? Let me dig up the xkc. Well, it's not yeah, it's the it's the you know, don't I use Julia Evans like don't feign surprise right like treated as an opportunity to learn I don't have a printed out, but I'm pretty sure it's still hanging up on my basement My basement office that is kind of a construction zone right now But it's you know treated as an opportunity to teach somebody new and exciting to them not you know Like oh, why don't you know this already right like a man page? No, we don't want people like that in the industry right like we want people that are enthusiastic about teaching Yeah, and it's one of those like When I was an administrator when I you know I eventually became an architect and all that other stuff and it became one of those the more you know the less You have to ask me right want to be a single point of failure. I got other stuff to do I don't need to do your job and my job. So it very much becomes one of those Education teaching helping rates bringing everybody Along and up and that's one of the things I love about red hat right red hat has that culture just ingrained inside of red hat I've never sent an email and you know asking a what is potentially a dumb question and gotten a response other than Absolutely, let me show you how that works, right? JP Dade that's just about the same. Yeah, I Haven't seen an update recently as far as I know they are still blocked Let's check and see Here, I'll show I'll share my share I want to get the link real quick. So what are you sharing the Cincinnati graph? Yes, so Get hub.com slash open shifts and if we go to Cincinnati, I can never remember how to spell Cincinnati I just remember this. I only know it because my daughter went to school there If we go to blocked edges and we check way down here on four dot seven four It's still got Blocked here and with a link out to the same BZ that we talked about last week This is only impacting these fear Yeah, so this is one of those Releases are done holistically across all platforms. So there isn't a an upgrade or there isn't a It's not I can't target a release that says you can upgrade all platforms except these fear It's either all or none in that case. Yeah So in this instance, yes all upgrades from or to open shift four dot seven are blocked as a result of a bug That we found in vSphere So for those customers who aren't using vSphere, right? Yeah. Yes. I I know and even for those who are using via vSphere, right? It's frustrating. Yeah, we're two months past now We feel your pain trust. Yeah Trust me We feel it both on a personal level of I'm constantly trying to deploy and troubleshoot and work with these things and and you know help you all our audience our customers You know with these problems, so I feel it as well as I know there's a lot of pressure You know coming into you know red hats into our organization around figuring and fixing that so Keep an eye on this BZ. I'll post the link to the BZ and I got it right here. Okay. Thank you No problem, but yeah, unfortunately Yeah, there's there's still blocked and I know that they are working diligently to get that figured out and again I see those emails between you know our team and the VMware team and there's kernel developers involved and all kinds of other stuff There's yeah, there's a lot going on behind the scenes. Yeah, absolutely So I know we got a little distracted there with storytelling But just to reiterate or just to remind everybody this is the OpenShift administrator office hour Or excuse me. This is the ask an OpenShift admin Man, I reverted back to the wow We've had this title for how long well I was I was doing work yesterday to go, you know to document and retitle the original episode Making the list for me. Yeah, sorry about that. Hmm. So what that means is that our goal here our Our purpose the reason why we have the stream is for you to ask us questions doesn't matter what it's about doesn't matter You know whether or not it's something that we can help with and what I mean by that is if we don't know the answer We'll find the answer right and we'll get back to you Yeah, it's one of my favorite things as I was just saying is just finding the right people inside a red hat and Getting those answers, you know a couple weeks ago. We had Mark Curry on with the we were talking about Moltis and and CNI and all that other stuff and You know Doug and and Oh right now told me Yes, both in engineering on you know and working in Moltis I have learned so much from them and seeing the email chains that have been generated out of that So you all our audience There was a couple people who sent me emails and basically said hey How does this work or how can I do this and I've you know connected the dots there or connected the people there And I've already learned just a tremendous amount about how all that stuff works So to me it's it's really exciting to see all that stuff Nice, so yeah at any point in time, please feel free ask questions Whatever it is that is at the top of your mind something that's bothering you some problem or issue that you're having Architecture questions functionality questions all that stuff. Please. Please. Please. Please don't hesitate to ask those questions Yeah, we encourage it and if for whatever reason we can't get to your question We can't answer it here on the stream. I always follow up each one of our episodes with a blog post So if we switch over to this tab here, this is the blog post from last week I'll post that into the chat here And you can see that we essentially do a synopsis So coming down here remember last week. I said, hey, you know We we don't have enough time to cover all of these items all of these different parts and components But I'll provide links to all of those inside of the blog post. Well, here's the Yeah So you can absolutely find all that we try to get those published every Friday so far. We've been pretty consistent with that So check OpenShift comm slash blog on Friday mornings Usually and you can see this week's blog post including links to we come all the way down here to the bottom You can see we have links directly to in the Stream where we answer those questions, right? So any questions that you happen to bring up? So with that being said or I guess the next Parts or the next thing that we tend to do here and I'm sorry I keep looking over this way that your little window is was way over here. So when I sorry when I go to look at you I'm actually looking over there. So remember. I'm the camera look at the camera. That's what I do I put look at me look at me So the producer now Not just now just always So one of the things I like to do here is Talk about the things that have come to my attention the things that are tough of mind for me that I think are important to relevant for you all So this week, there's a couple of things that I want to to bring up in that respect So first I've had quite a few conversations over the last Six or eight or nine months around the load balancer that's used with IPI So if you're not familiar with this effectively with on-premises IPI deployments So these fear redhead virtualization open stack and bare metal IPI We deploy a built-in load balancer So what that means is when you go through the installation process, right? It asks you for things like what is the apps virtual IP address? So star dot apps the ingress IP address and what is the API so the API dot cluster name dot domain name IP address and those two IP addresses are Managed by keep alive D. So keep alive D puts that IP address onto an eligible host DNS directs all of that traffic under the host and You get your fully functional cluster So the problem with that is particularly for the apps for the in the star dot apps ingress point It's limited to a single node in the cluster It's one IP address. It's that IP address is hosted on a node that has a router instance right the HA proxy OpenShift router instance And it doesn't do any real load balancing behind the scenes So if you have you know, if you scale your router instances from two to three to five to 12 or however many you have All of that traffic still goes into one those additional instances are just there for high availability Or failover capability So the question is Can I use an external load balancer with on-prem IPI and The answer here there's really kind of three answers that or three possibilities that you have here so the first one is more or less to Replace the ingress load balancer. So what do I mean by that you could for example? And let me see if I have the case. Yes handy here Let's see I Have to I have to search in one window where I'm actually logged in and oh, yeah, I feel that yeah Here we go. Well, I said that and now I clicked the wrong thing Anyway, so for example usually the example I use here is f5 So the f5 folks have a certified operators if I go to catalog redhead.com and I come up here and look for OpenShift operators and I search for f5 We get back this f5 controller for OpenShift and Kubernetes So for you if you were to use this operator and they aren't the only ones Citrix has one Cisco ACI has one right. There's a number of those Effectively what it does is it deploys the f5 ingress controller So instead of using the default red hats OpenShift provided ingress controller that is based off of HA proxy instead You use the f5 ingress controller which coordinates with an external f5 appliance Same thing for Citrix and their ADC controller so on and so forth So that's one option right just don't use the default one right instead use the one You know use a different ingress controller with your cluster and then you can absolutely take advantage of kind of whatever You know, whichever of the load balancers you want to Great. So the other option and this is something that as far as I know is only documented by OpenStack So let's go to docs So if I go to the documentation and scroll down to networking and scroll down to load balancing on OpenStack They have this configuring an external load balancer section, and I'll copy and paste this into the chat and effectively what this is documenting is for on-prem IPI with OpenStack Here's how you can configure an external load balancer and then effectively swing that ingress and API endpoints over to them So unfortunately, this isn't tested or documented with any of the other deployment methods So therefore we can't authoritatively say yes, it will work and therefore it falls into kind of a support gray area So I haven't been able to get an answer out of the support folks as to yes It's documented for OpenStack, but does that mean it's also supported with vSphere for example Even though the infrastructure isn't really material in that instance So just FYI this is here if you happen to be using OpenStack and you want to migrate off of that Default built-in load balancer and you're not using, you know, the courier or the Octavia integration This is one option that you have available So the third option is kind of Effectively replacing that but using something, you know unofficial and that could be as simple as You know following the documentation here and basically I'm not going to talk about it because it's it's definitely not supported And I don't want to encourage that at all So really the two options are use a difference ingress controller or Use an external load balancer following this OpenStack Configuration here, so there is some stuff going on here, you know behind the scenes around how can we make this better? How can we configure this? How can we make it more configurable? So just to keep an eye out I do have somewhere in my multitude of tabs. I do have an RFE that I can dig up for this that way we can Take it or that way we can track that progress and If it's something you any of our any of our audience right if you raise that with your account team with them No, they can just reach out to me and then we can kind of associate account information with that which helps product management with prioritization Basically, they they they use your input as customers as a huge contributor to what's important and what should be prioritized from an engineering perspective. So So JP data asks can't you spread the ingress controller over a set of info nodes, right? Yes, yeah, yeah, so with UPI absolutely Or even the non-integrated install, right? I I can it can be info nodes. It can be, you know Router nodes you can tag them whatever you want, but essentially you can scale that ingress controller to you know 25810 however many instances and Normally with IPI excuse me with UPI or a non-integrated install You have an external load balancer that is pointed at each one of your Nodes that's eligible or that's hosting a router instance, right? A HA proxy ingress instance So incoming traffic comes in it hits that external load balancer The external load balancer looks at all of the nodes that it has available to it so all the ones that are running that HA proxy ingress controller instance and then it routes that you know external session over to one of those and You know assuming and I think the documentation by default says, you know configure all worker nodes, you know inside of that external load balancer So if I have 12 ingress instances 12 ingress HA proxy instances, then it would be load balanced across all 12 the problem with IPI on-prem IPI is That external load balancer basically doesn't exist So it's just a single virtual IP address right the start out apps the one associated with the start out apps DNS name And it just moves from node to node You know with and only on nodes that have an HA proxy ingress instance on it So it would be the same effect as if you had an external load balancer that only had one node configured on the back end Yeah, not so smart. Yeah, but anyway I'm sorry meaning multiple you don't want a single point of failure spread it out. Yeah Yeah, and you know the keep alive the solution rate the the on-prem IPI solution does that rate keep alive D Maintains that virtual IP address. So if the node that's hosting it goes down for any reason it moves to a different node So you only have a slight blip Anecdotally what I hear is somewhere between one and five seconds So you have a slight blip in that ingress traffic as the IP moves and then a reverse ARP goes out to let the infrastructure Know that it moved and then things returned to normal. So Yeah, that's where we stand today. If you have any questions about that if it's something that is an issue for you Just let me know Andrew dot Solomon at redhead calm and then we can kind of work through that and look at what those different options are as well As work with product management engineering So, yeah Yeah, you're gonna love this one DHCP reservations DHCP it's it's a reoccurring theme. I think we have talked about DHCP in more episodes than we haven't talked about DHCP That's probably accurate now so a reoccurring question that we get is if you look in the documentation and I'll see if I can find it here real quick So let's go to installing. I'll pick on vSphere this time And we'll just go with this we'll go with this one installing a cluster So if I do a quick search for DHCP and Reservation no reservation. No Somewhere in here we but we more or less tell you, you know, yes with IP I DHCP is required But requirements, yeah So We tell you Blah blah blah Anyways, the documentation says DHCP is required for IP I because it is right. We require you to The nodes to be able to dynamically get their IP addresses but then we also say static DHCP reservations are encouraged for worker nodes and should be used for Control play notes So the rationale here is and if I come back over here, let me see if I can search the KCS this time Let's just go here. Well, thank you. That was not what I wanted Come on, so if I search for Open shift DHCP Reservation I should have had this link prepared right probably Here we go. So let me post this link in here and Effectively what this says is I guess I could so effectively what this is going to say is that When you use DHCP You want to make sure that your particularly control nodes always get the same IP address Very important. Yeah. And so what we're saying is, you know, somebody will shut off a cluster and It'll be down for, you know, an hour a day a week, whatever turn it back on and the nodes get a different DHCP address You know, this can particularly be an issue if you have a short lease time or if your DHCP pool is very busy, right? Basically The node goes down, right? Basically it does a release and then Something else immediately is requesting an IP address and there's so few available that it immediately starts reusing the ones that are there That stuff So we make this recommendation of use static, you know, you know, DHCP leases DHCP reservations for your control plane nodes the rationale being if The control plane node IPs change at CD can't reform itself Right and if at CD can't come up then the rest of the cluster can't come up Normally if only one node at a time changes the at CD operator cluster at CD operator, we'll be able to recover But if all three change then it can't reach its own quorum and that's bad So For that reason we encourage you to use DHCP reservations. The question is but it's IPI I don't know the MAC addresses beforehand. The installer creates the control plane nodes And machine API creates worker nodes. So how can I create those? And the answer is both simple and frustrating and and that is more or less After provisioning use the tools of your DHCP server to convert it to a reservation I Have reached out to some VMware peers I've reached out to some others and to figure out things like, you know VMware has this concept of a MAC address pool Can I create a pool of known addresses and then associate that with my open shift VMs so that they only, you know Whenever it provisions a new node, it only comes from this pool and stuff like that More or less what I got told is well, it doesn't work that way So I'm hoping we can make this easier in the future But honestly, I don't I don't have any and I have not seen any information with that regard So just FYI When you see that we Suggest and encourage DHCP reservations, especially for control plane nodes, you just have to do that day two, right? So after the after the node is deployed change that over to a DHCP reservation I will note that if you If you're deploying physical IPI bare metal IPI and your DHCP lease time is set to infinite Then it will automatically convert it will automatically apply a A bit of automation and convert that to a static IP assignment or a static IP configuration On the physical node only so remember this is bare metal physical IPI not the other installation methods Let's see Suresh has a question if you have a second. Yep, and and Suresh We're going to probably need some more details about your configuration specifically, but Um, is it mandatory to have DNS records of kates nodes an external DNS server? No, it's not mandatory, but it depends on how you set things up to like Yeah, so exposing the control plane to the public Yeah, so with IPI No with IPI it will deploy an mdns instance And all of the nodes will register themselves with mdns. So it will do internal resolution So effectively control plane nodes can find worker nodes and vice versa The only time that would be an issue is if you want to connect directly to a node So if you're trying to for example ssh from your workstation into you know worker 17 dot cluster name dot domain name That dns name wouldn't be externally visible Unless you're doing like a DHCP dynamic dns update right in which case it would be visible Uh with UPI I don't believe they're required No, I don't think they are either brings in my mind. It brings into question whether or so host names are set a couple of different ways and and the most prominent and and the way that will take precedence is Reverse lookups So if there is no reverse entry I don't know how it will react to that because normally it would want to use the node name Like if you do an oc get Node, right? It'll show the node name inside of there if it can't look that up If it doesn't understand how to talk to that I don't know how that would work if it would just return back an IP address or something like that Um, honestly, it's not something I've tested. So I'm not sure. I know in the documentation We say that you know here if we look at the So this is ipi Switch over to Installing a cluster on vSphere with user provision information or infrastructure Words are hard If we come down here to networking requirements I believe it says that dns We we basically say dns is required and I know yeah here user provision dns requirements um So I I know that we say that it is required here So yeah your control plane nodes worker nodes um, and and that's just To avoid I if nothing else that is to avoid a lot of potential issues So I would say follow the docs. They are required with Uh, upi with ipi It's not on the docs. It'll take care of itself Keep alive ds poor man's f5. I wouldn't even go that far. I wouldn't even go that far It's uh, it's just to keep the ip address accessible Right, it doesn't do any kind of load balancing any kind of you know anything like that. So um So the static ip question I read somewhere haven't tested that if we offer a dhcp ip address with an infinite least Red Hat core oest will configure the ip address as a static ip address Yes, and that's what I was talking about with It's only with bare metal Right, so we can see if we go to github Type in the window here So if I go back to open shift github and this time I went to go to the machine config operator And I'm going to switch the branch here to I'll go to release four seven And in here we're looking for the templates directory and then we go into I think it's common And then Bare metal and files Yeah, so we have this network manager static dhcp.yaml And effectively what this is doing is exactly what we just described Right, it's it says is this an infinite lease if so i'm going to apply the ip address as static Um, but you'll note it's only with bare metal Right and and specifically bare metal ipi So if you're using you know any of the other ipis on prem, it doesn't take effect if you're just doing a A bare metal upi or a non integrated install. It doesn't take effect, right? It's only bare metal ipi So yeah, you can see exactly where that logic lives and what triggers it inside of here because I think if we go up a window or up One of these here is the one that basically says If it has the template for the if statements where it detects the installation type An installation method if infrastructure installation infrastructure Um jp did on my upi installed to configure dns and dhcp Um Yep, a whole lot of settings and dhcp to set hostname and ntp. Yes Yeah, yeah, um, so the ntp if your dhcp server is serving ntp. That's a nice Convenient way to get everything configured correctly. If not, we talked about this last week In the day two stuff of figuring No, I don't know where it went inside of here, but somewhere inside of here is configuring crony Yes Now so The link directly to the documentation inside of here for configuring the time synchronization service using machine config in this instance, right? I mean is everything folks Especially in kubernetes because kubernetes makes great use of certificates and certificates are very finicky about Is this valid yes or no? Wait why is this time offset different from this box in this box? It will pick stuff up like that and you'll see some weird errors as a result So yeah, please keep the questions coming in. Um, so don't don't hesitate You know with great power comes more settings. Yes. Yes. Yes. Absolutely true. Uh, so The last thing I wanted to talk about is if you missed this blog post on openshift.com August ciminelli who is one of our peers? From down under he lives out in Australia He released an update to this tested Solution for open shift on open stack. So essentially he Went through it's it's not a reference architecture But it's a tested architecture tested and validated architecture for deploying Open shift on open stack. So Great resource if you happen to be doing open shift on open stack Be sure to check that out. I know he put a lot of time efforts Blood sweat and tears into that and it's a a hugely hugely beneficial. Yeah Thing if you happen to be using that architecture So what is the rationale behind having this behavior? Infinite lease to static ip limited to bare metal ipi only why not for other setups too? So I will say andrew's yeah So andrew's opinion is that or my my hope maybe is that we will expand that to other infrastructures as well Yeah, but I don't know that for certain. I would have to dig through the rfe's And see if we can find where that's documented at or yeah, which ones goes first kind of deal And everything else. Yeah So, you know the logic with bare metal with physical servers is pretty straightforward You know one it is the only time with an ipi where we're going to know the mac addresses ahead of time There's that you know because you can Mac physically look at it, you know, it's it's not actually, you know creating machines on the fly. It's just Configuring them on the fly. So I think that's one of the things that leads to it Um, I I hope that we do add it for the others simply because I know dhcp is a point of contention And this is something that would actually be really interesting to me for anybody who's listening in Why is dhcp so contentious right and and we've heard this basically since IPI on prem was a thing. Yeah, which is what open shifts 4.2 was when it was released with open stack You know of you know, hey, I can't do dhcp in my data center You know, can I do ipi without dhcp and a lot of times we get back this kind of hand wavy Well, the securities team won't let us And right while I can make a lot of assumptions there it would be interesting to know like what's the details there What's what's going on behind the scenes? Why won't they let us do it? if you will And you know in my experience it and please audience why won't your people you know Run dhcp in the data center. Please tell us why in my experience. It's because there is Anything that's not using dhcp that you would then have to configure and that anything could be of varying sizes between, you know four to four hundred, you know instances of something or Entire ranges of ip classes like slash 16s and so off So, yeah Maybe cisco proprietary os well if you're using your switches and routers as gateways and dhcp servers and everything else Yeah, there's some limitations there potentially Like hard to configure and limitations I mean right like not ubiquitous right like i'm not going to let my You know windows dns admin jump into the cisco router and start configuring dhcp. There's there's a learning curve there right so Yeah, I know You know when I was an administrator our security team wouldn't let us do dhcp and it was largely because their fear particularly in the land of virtualization was Right anybody could stand up something and you know, we don't know what it is and it'll just be an ip address You know kind of the old saying about you know, will anybody can walk through the data center and plug their laptop into a network port? Well, no, I mean technically yes, but that's why we have things like port security right So yeah, the jpd points out plugging in a foreign device under the data center is frowned on dhcp is good for workstations only Uh to which apostola says it's old school network security thinking to which I agree It is kind of an old school methodology, right like dhcp has been around a really long time and lots of people are using it in their You know data centers. Yeah I'll also I think some of it comes from um pre I'm gonna use the term cloud native here but yeah Kind of the the current and even really the last generation of applications You know when we think back to like the 2000s, you know 2010 and earlier Applications didn't change a lot right, especially when everything you know before virtualization really took holds You know servers were put into a rack. They were given a static ip address and then All of the things that needed to connect to it would a lot of times just use the ip address Right, they wouldn't use a dns name. Yeah, because no one wants to muck with dns because dns is dns What happens if dns goes down? What happens if you know, so, you know, my database server is that you know x.y.z.a and you know all of the Applications that connect to that database connect on x.y.z.a So yeah, that's an interesting one. Um, I've been Wondering that for a while as to why we don't see dhcp. Um, you know more widespread in data centers Cluster dhcp servers. Yes. Um, that is something that's um, absolutely if you were going to be using, you know dhcp in your data center, you know Something like open shifts which are ipi which requires dhcp make sure it is an enterprise grade service That was one of the things I learned from our field early on is that a lot of customers Even if they have dhcp in the data center, it's running off of like one VM and it's not You know, it's not treated with the same level of resiliency and availability that we see with other data center services So yeah, gpdade very much to your point to make sure that it is Treated and and managed appropriately for the level of importance that it holds Yeah, and I can't say I mean, I I'm sorry if I butcher your screen name my bad But in my experience most of the enterprise network admins setups don't like any much addressing because a conventional data center is everything For the VMs are predetermined with firewall rules and IPs and everything else. So yeah Um, so I know we're uh, just like last week We're pretty far in at this point and we've we've kind of brushed against several things that I wanted to talk about with regard today's topic, which is As the title suggests day two operations. So effectively, you know, hey, we just finished deploying our cluster Now what you know, what else do I need to do to get ready to actually use that cluster? So last week as you saw, you know, if you look at these headings here We talked about kind of node level or machine level operations that need to happen Or could happen Cluster level operations and then kind of some other things just to be aware of You know, many of these could probably be lumped up into the cluster side of things, but So this week I want to focus on network storage and Kind of preparing for users So network this almost always comes down to How do I do or how do I configure or how do I manage? The network configuration after deployment and especially and we're seeing this more and more now for people who want to have a secondary or tertiary or however many interfaces for additional network traffic or additional traffic types, I should say the most common one by far is A isolated non-routable storage network right I need to connect to nfs pvcs or iSCSI pvcs or whatever that happens to be And the storage team has a dedicated, you know, ip network specifically for that You know go back to my comments about routing and all of that other stuff and we'll just reboot the router. It's fine. It's good so What's the best way to do that? And I will say that and do I have it up here? Let's come up here to I want networking and we're going to Where is it in m state? um, so i'll say that there's really Two so there's three answers of which one of them. I don't consider to be a valid option. Uh, so The first one is you should configure those interfaces at install time So with upi or with a non-integrated installation That means either passing ip equals configuration parameters You're gonna multiple ip equals. So maybe if you're first interface, right? So i'm gonna, you know, eth0 is the one that is used for Uh, sdn, right? It is the primary interface. Maybe it uses dhcp. Great ip equals e0 colon dhcp And then your secondary interface needs a static ip address You can absolutely do that and you can use Uh, the standards right a second ip equals and then follow the normal nomenclature for doing that So i'll just to grab an example of that i'll come down here to installing on any platform quick search for ip equals And we can see an example here right basically exactly like this so Right here in the documentation We can see we can just append as many of those as we have interfaces And that includes things like creating bonds. So if we scroll down a little bit further here So we have this bond equals and then we can create bond zero that has two interfaces We provide the bonding mode and then I can then assign an ip address to that bond We do the same thing with vlands Uh, somewhere in here is the vlan documentation on how to create a vlan on top of that So this is one way that you can do that. It becomes persistent It is then controlled right throughout the normal open shift management processes You're going to do effectively the exact same thing using coro s installer and the live iso method so Boot to the iso it drops you into a command line prompt you can use nmcl i And then with nmcl i configure the network however it needs to be configured And then use the copy network. Here we go the copy network option When you execute coro s installer And that will take the currently configured network config in the live iso or the live environment And copy it to the Sorry for bumping the microphone copy it into the uh installed os on the host So that is with upi that is the suggested method of doing it Right with ipi it's going to be a little bit different In particular and i'm going to pick on vmware again So if we look at vmware with network customizations Actually, I don't want that I want But where is it machine sets So sarash points out. Yeah, I wanted to evaluate the nm state operator, but I'm waiting on the ocp 47 release for vsphere Yeah, yeah Sorry, buddy Yeah, um, uh, we all are kind of right so you you can So if you look at nm states, uh, and if you saw the tab that I have here You see this big red it's in technology preview only So if you deploy nm state or utilize nm state standalone It is tech preview if you use it with open ship virtualization. It is fully supported So if you happen to be running on physical servers, if you have bare metal available You can deploy the open ship virtualization operator Even if you don't actually deploy any virtual machines, you'll get the nm state operator It'll be fully supported and that goes back to 4.5. I believe I so yeah, you can you can use it there So to complete out my other thought around, uh, ipi and networking So if you're deploying for example with vsphere ipi you want to create a machine sets that will have multiple network device definitions here Right. So for each network that you need to connect to you create a new network name device here You add that in When the machine set goes to deploy that node it will ensure that each one of those is there So the first device presumably the one that is on the primary network that has dhcp enabled remember ipi requires dhcp It'll boot it'll get its ip address. So it'll be fine the second device You'll probably see some errors right. It'll probably complain and then eventually take a link local ip address That's fine Might delay the boot process by a few seconds that type of stuff Once once it's deployed once it's joined to the cluster then you can use What I would suggest is nm state So this brings me to the the third option that I don't really consider an option So yes, it is technically possible to use machine config Right, you can go in you can create a machine config Let's see here user provision deleting a machine We'll find it eventually Even if I have to search for it Somewhere in here Machine config pool machine config. Yeah, we'll go with this one So you can use a machine config And you can use the machine config to push that yeah here here's an example of one You can use that to push an ignition config that configures that network interface The problem is now you have to have a machine config per node Because that node right each node has a different static IP address and that machine config You can't just say, you know, here's I have 10 nodes in my cluster that all need static IPs I'm going to create 10 machine configs And then I'm going to put them all in the same machine config pool Because that'll push out the same config to all machines in the pool and right Well, they don't all have the same static IP address So now you have to have 10 machine config pools one for each one of the nodes in the cluster that inherits that chain Right, so it would be for example The worker and then the node specific and then you end up with a machine Machine config specific to that pool to configure the worker's static IP address Technically possible nothing wrong with that. Absolutely fully supported a management nightmare Like and and I mean to put it mildly in many different ways because It bleeds over into things. Well now I have a one node machine config pool When we think about things like doing updates in a cluster It does those updates on a machine, right? It learns things or it inherits things like how many nodes can be down at a time from the machine config pool Great, so it when it needs to do something like apply updates It says well machine config pool how many nodes can be down at a time. Oh only one. Okay Well, I will sequentially go through and reboot, you know, apply updates and reboot the nodes one at a time Or you could say maybe it's 25 so in a pool with You know 10 nodes I can take down up to 25 of them at any one time to apply updates or do whatever machine config needs to do So now with a single node machine config pool you effectively have to disable those so now The cluster can get into the state where You're relying fully on things like pod disruption budget to control how many nodes can be down at any point in time So it's definitely A management overhead Issue as well as introduces a lot of other kind of unforeseen risk that's inside of there And that's just the one that comes to off the top of my head so yes It seems like machine config is a great option for applying static ip It's really really not The other option which you can kind of squint at And maybe be okay would be to do something like writes a bash script. I'll I'll say bash And have it Be so write a bash script that detects which nodes it in which node it's on and then applies the static ip configuration that way Right, so then you have one machine config For all of the nodes in the machine config pool and that script just says hey, I'm on node 12 I need to give it this ip address I don't know how Should work, you know should be fine You know assuming everything goes according to plan right nothing goes haywire or anything like that That is an option. So Maybe consider that with the others, but again, it can you know doing updates or changes, you know Hey, I added a new node to the pool. I went from 12 nodes to 14 nodes When I have to recreate that script when you recreate that script and update the machine config It's going to trigger a reconfigure of all the nodes in the machine config pool So they'll all go through get the new file and then have to go through a reboot and all that other stuff So still not a panacea if you will Um, do we have any questions chris? I haven't seen no there's another court on my mind. Uh, come in Okay So we've only got just a few minutes left. Maybe six or seven minutes left So I want to make sure if you have any questions, please don't hesitate to ask those Um, just as with last week, uh, I I've got I don't know 20 or 30 different links here that I'll put all of these into the blog post Look for that on monday morning Or excuse me friday morning friday morning. Um, so you can find all of the stuff that we Would talk about you can of course always reach out at any point in time streaming or not through social media So on twitter, I am practical andrew So if you saw me chatting inside of the twitch chat or any of the places where it gets rebroadcast Um, same username practical andrew, um, you know all one word on twitter You can also reach out to me directly through redhead email andrew.sullivan at redhead.com You can reach out anytime happy to field those questions make the connections and get those questions answered for you Um, so I think the other thing that I want to talk about here While we are, um, you know During the stream while I have your attention Is something that has Come up that is kind of floating in the back of my head is one of those top of mind things But it hasn't quite reached that point yet, but I think I am going to talk about it and that is The concept of and and I'm combining a few things here and that is How do I do resource management inside of the cluster? And ultimately and I think most of us probably know You know clusters aren't static Especially if we're thinking like ipi and we're doing machine sets and auto scaling and all this other stuff Yeah, you know the nodes are constantly changing but I also want to remind us that We can change the size of the nodes for example, and we can balance the resources So cpu memory storage, etc according to what the applications are actually using And additionally, you know the metric service gives us things like Well, the pods are making requests remember a quest is Effectively a reservation, but they're only using you know, maybe 10% of that So working with our applications peers to basically say hey, why are you requesting 10 gigabytes of ram? And you know only using one that's nine gigabytes of memory that I can't use for other applications in the cluster And maybe their answer is Well, you know, we do have a batch job that runs, you know twice a month that does require that much memory, you know Okay, well, let's see if we can find A better You know a better way of meeting those needs And well, I know the you know the open shift metric service has a lot of data a lot of information in it um our partners, you know, so partners like datadog and Turbonomic and Gosh, there's like fire so many. Yeah On and on and on of, you know in all of these metrics in the metric space They take the same information, right? Sometimes they take it directly from metrics. Sometimes they collect their own And they have tools inside of their portfolio to be able to automatically take action as well I recently saw the Turbonomic folks give a demo where They were able to predictively move pods to free up resources for things that were happening and it was pretty cool stuff. So Definitely always be cognizant of, you know, real resource utilization versus reserved resource utilization How can I best optimize the cluster? You know, to take advantage of those resources if you're deploying in a hyperscaler, right? You're paying for those instances If they're, you know deployed whether or not you're actually using those resources So making sure that you're actually pushing that cpu utilization to real usage of, you know, 80 90% Is a cost-saving measure Instead of, you know Years ago They were I remember having conversations around our virtualization deployment You know was only at like 30 percent utilization or something like that And it's one of those like the higher you drive that utilization the better value you get out of it Which is definitely true with the hyperscalers So we got a question From Don Weeks, are there instructions for installing open source calico as the sdn and open shift? Can that only be done by upi install? So the second part I believe is yes. Yeah, so so the sdn can only be deployed at install time So it doesn't matter if it's ipi or upi or even non-integrated It'll absolutely work and you can if you haven't seen calico's docs Um, you can look at their docs here about how to deploy inside of open shift More or less the the principle is the same regardless of the installation method So you see this create manifest and then you drop all of their manifests into the correct place And then continue with the rest of the install. Uh, they don't make any differentiation here as to whether it's ipi or upi Yeah Yep, so, uh, how often do we need to update the rel? core oba file on vmware so My suggestion would be at least once every major or y release So if yeah, so when you go 4.4, yeah, exactly so 4.6 to 4.7 update the oba Within the same release Uh, it will and even with it between the major releases so long as it's an earlier version It should update coro s to the right version as a part of the installation process Right, so essentially, you know, it'll clone that ova for the new worker node It'll reach out to the control plane to the machine config operator And it'll say give me my ignition and part of that is hey pull down this new rpmos tree image and switch over so That should work You might be able to slightly speed up the node provisioning node joining process if you're on the same release that it wants um, but yeah So Question those manif like I have no idea on the answer to this one those manifests are for enterprise calico, right? And I would I don't know um I don't think so. This is the project calico website. Yeah, I mean, this isn't the project site. Yeah. Yeah, exactly. So Yeah, it's talking about tigera status Tigera operator. See that. Yeah, here's the tigera operator. Market place. I mean I have always assumed that this is the open source version. Yeah, not the enterprise version That'd be and I've deployed this a couple of times. So Yeah, I would be interested. Yeah, you haven't swept your credit card to deploy it. Have you? Um, no, I'm aware of right Um Yeah, that's interesting. Yeah, so, um, don't please feel free to reach out. Um, and yeah, I'll I'll also poke some of our calico folks. Yeah, go ahead. You poke So, yeah, please feel free to reach out andrew.sullivan at redhat.com. Um, yeah, okay. So That's really all I think we've got time to get through today. Um, as I said, I have a whole list of Um, of stuff that I will include in the blog post. Um, we can talk about that during the next show as well Um, so which brings me to my next point, which is we won't be back for three weeks Right. There's stuff happening. Yes. Uh, so next week is red hat summit If you aren't registered for red hat summit, I definitely encourage you to do that. It's free It is free So yeah, absolutely registered for red hat summits. Uh, it is a two part event So the first one is next week. The second part will be in june. Um, you'll get all the Announcements and alerts and emails and everything about both of those events Um, so definitely encourage you to do that If you haven't already the following week. So the first week of may is kubcon So we will be off air. Um, so the this show will be off air for both of those weeks But chris will definitely be on air doing Tons and tons and tons of streaming with all the stuff that we got going on there a lot of office hours a lot of Interesting fun things are going to be happening. So yeah, please tune in So we will be back. I believe it is may 12th The currently scheduled topic will be red hat enterprise linux coro s or as we lovingly know it our costs With mark russell who's a product product manager for coro s But yeah, we can absolutely bleed over some of these topics again Please don't hesitate to reach out if you have any questions social media practical andrew at twitter Or andrew dot sullivan at redhead.com and thank you so much for joining us today Thank you all have a good one and coming up next we'll be talking about Uh three critical areas of container resource management with the folks at densify. So please stay tuned. I'll be queuing that up now