 Good morning. Good afternoon. Good evening. Wherever you're handling from welcome to another episode of the open shift administrator office hours I am Chris short the executive producer of here of open shift TV Joined by the one and only Andrew Sullivan and special guest Peter ladder back today Andrew you want to tell us what we're talking about today Yeah, thankfully, there's only one and only one of me. Yeah sure that my my wife and kids all agree So, yeah, this is the open shift administrators office hour Which means that like all of the other office hour shows that we have here on open shift TV It is meant to be and ask me anything style of interaction Right. We we live on we thrive on we want your questions. Whatever the topic may be on those It doesn't have to be relevant to what we're talking about Chris and I or Peter today Instead, it's just whatever is the top of your mind That being said, yeah, if you got a problem, let us know. Yeah find the answer Yeah, either if we don't know it, we will track it down But that being said in the absence of questions from you all from our loving audience We do have a topic. We do typically have something that we like to talk about as well Just to fill in the time if you will and today that is going to be a couple of different things So first I have some follow-ups from last week And second and the reason Peter is with us today is because we want to talk a little bit about rev red hat virtualization and specifically What's changed with open shift on red hat virtualization here in the most recent versions? So we'll be talking about rev UPI. We'll be talking a little bit about rev IPI CSI with rev friend. So it's a it's an all rev all the time except for when it's not day So fear not if you are not using rev for your underlying infrastructure I do have and in Chris knows this I spent some time over the last week or so and came up with the Episodes for the next roughly four months. So during that time We will be covering basically all of the different infrastructures and talking about Different features functions changes, etc. That have happened across all of those So keep an eye on the calendar if you're not subscribed to the streaming calendar Which if you're watching in twitch is down below the video interface you can see a link to that We'll be updating all of the episodes with the topics and any guess that we may have over time Oh, thank you. You have a yeah, I have a thing short cut for that spam the crap out of us all of a sudden Yeah, thanks. All right, so without further ado and again, please do not hesitate to ask your questions in the In the chat there. I do see one from art are yeah, so I will address that just as soon as we let Peter do his introduction Hey guys, yeah, this is Peter Lawnarack coming to you from snowy Boston. So I've been pretty chilly up here I've actually been in high tech for quite a while now both in storage and performance and virtualization So I'm one of the product managers in the cloud platforms BU Focused on red hat virtualization and over which is our upstream and also open ship virtualization, which is the the new cool stuff That is part of open shift and is upstream in Cuba, which I think I got a very nice shirt You do have a very nice shirt. Yes so and and Peter and I also chat quite frequently about our home labs and all of the various Machinations that are happening in there my my most recent drama is in my my One storage server. I had to remove the video card because I needed the PCI lanes for NVMe so nice I It's now running without a video card and it seems to do well right up until something goes horribly wrong And then I'll have to adjust things but for now it seems to be functioning. So that's fun You have to go find the are in the nine pin RS 232 to plug it in the back of the serial port It's so 1990s I have one of those cables sticking around just in case I have a whole box of them in my basement my wife wishes I get rid of Michigan is in Boston both are a long way from Raleigh So arts I am new with red hats Welcome if you're an employee and welcome if you are I say merely if you are one of our esteemed customers So regardless of which new with red hat you are Referring to there. Thank you Should I learn the new version or the old version? So I'm going to assume and please clarify if I assume incorrectly that you're referring to open shift 3 versus open shift 4 right So I would definitely highly recommend Open shift 4 Right, so not not the least of which because open shift 3 so 3 dot 11 is the only version Remaining if you will it's the only one that is currently under support and that support expires in I want to say roughly a year maybe a year and a half But it's essentially 22. Yeah. Yeah, and essentially it is it is in an extended update status There are no new features being added right that type stuff all the new stuff all of the improvements All of the changes that aren't just bug fixes and security fixes are going into the open shift 4 line So that is definitely the way that you would want to go Chris Peter anything to add there I'm actually trying to pull up the release dates for open shift right now because I know we have that someplace But yeah, most of the most of the time You know the large large massive clusters, you know an application that first got deployed are an open shift 3 and we've actually got a Process for folks to get from open shift 3 to open shift 4, but yeah all the in fact open shift 4 has been out for 18 months almost two years now so Yeah, it's it's I wouldn't even call it the new stuff. It's it's been out for two years. It's it's pretty stable We're and we're just on for 4.6 right so right that's a good point The two-year anniversary of Open Shift 4 is in May. Yeah, I didn't I didn't realize that it's been that long It's gone by fast. Yeah, and Kubernetes 6 year anniversary is in May as well. Mm-hmm Yeah, so pretty lots lots of it's a busy month for for the Kubernetes world all right, so Lol, I apologize if I butcher any names Working with those 3 out 11 and 4. Yeah, that's very common Yeah, there's a lot of folks out there who are still on Open Shift 3 out 11, right? I don't know if anybody knows this or not 2020 was kind of a rough year, right? There was a lot of going on. They were yeah, and It is you know broadly speaking a lot of a lot of organizations You know, they quite frankly had other things That were priorities then updating, you know things that are running great and running fine And that's why you saw us extend supports I think support for 3 out 11 originally ended roughly the middle of this year But we extended it out by nine months or 12 months something like that to help with that So yeah, it's definitely common as Peter alluded to if you need help with that if there's if you're in the process of migrating from 3 to 4 Open Shift 3 out 11 to one one of the four versions the Migration toolkit for containers as it is now called is the tool for that so There's a it is in the documentation. I can see if we if you don't have a link I'm pulling Okay, yeah, if you can drop a link to that. It's a great way to help migrate those applications And then of course if you're learning about Open Shift 4 There's a huge number of resources not the least of which is of course OpenShift TV where we have dozens and dozens and dozens of Livestreams both catalogued as well as on the calendar coming up all around Open Shift 4 So and please feel free to ask questions as well. Yeah Yeah, the other I don't know if I'm allowed to plug our stuff here, but The OpenShift workshop it was like OpenShift 101 or something like that I think that happens regularly and I happened across to and literally in three hours It's a pre-built workshop you go in and and everything's all set up for you But you actually go through the process of understanding OpenShift and Kubernetes and containers and then you immediately dive into Service mesh and developer pipelines and the coolest part for me, which I I'm a product-minded, right? I'm not a developer, but you go in here you change some Java code and A build pipeline fires off and then a new application shows up here like five minutes later. You're like that's crazy That was a lot of fun magic. There was a lot of magic happening in that hour All right, so I will again Please feel free to ask any questions that you may have but I will go ahead and move on to catching up from last week So JP Dade, which I don't know if he is tuned in today or not. He's one of our regular viewers He we were discussing that One of the updates had been blocked so the update to OpenShift 4.6.9 that edge was blocked and he had posted a Support case that was opened on Or opened by him or his team or on their behalf and basically said, you know Hey, is this one of the or could this you know have been one of the reasons and it turns out that yes It absolutely was one of the reasons. So the what they discovered was that an update to 4.6.9 for customers who are using OVN Kubernetes as their SDN was causing issues and I Don't remember which customer who he works for not that I would say it publicly anyways But they happen to encounter that issue or that area So one thing and I tried to show this while I was while we were streaming and I failed miserably was How do we where does that information come from? and it was at the top of my mind because last week remember I was talking about disconnected OpenShift and At the very end of the show we talked about the the OpenShift updates manager or the for what we call it the OpenShift update operator right where you can in an offline environment basically pull in the Cincinnati data and Have it offer you that same, you know update experience so you can see this cluster, which I deployed yesterday Is running 4.6.9 and since yesterday we now have a stable update to Version 4.6.12 I'm actually in the process of upgrading my cluster to that. So very nice So a couple of things to note here So 4.6.12 has actually been available for a couple of weeks now if not maybe the better part of three weeks But you notice that just today it became available as an update So there is a difference between The different channels as well as if you were to go and do a new install right now So remember stable is always when we feel comfortable having the vast majority of folks who have OpenShift deployed Updating right typically that means that it is going to be a couple of weeks after it hits the fast channel So if I were to go here too fast If the fast channel is generally going to have Newer versions available than what we see with and right now they happen to align But it's going to have newer versions available that might not have had or there might be an outstanding edge case or something like that Where it's it's not available for everybody But it's not the same thing as a new deployment This is just for updates So when you if you saw over the last let's say it's two weeks if you went to you know the Cloud out red hat calm, you know slash OpenShift slash install and you pulled down the installer It was probably giving you version 4.6.12. There's nothing wrong with that. That's expected for new installs There's nothing wrong with going directly to a version that is newer than what you see in the update channel It's just a practice of what has been fully tested and Are we seeing any breaking right things that you should be alarmed about? We don't want you updating to type of bugs, right? So the other thing that I tried to look at and fail that while we were on stream last week was where the update data is stored at So I browse to this github repo. This is github.com slash openshift slash Cincinnati graph data So this is where we publish all of those updates and all of you know This is how the system how OpenShift determines which channels or where I can update to and from and which ones I can't update to or from So last week I looked at I went into the channels here and I saw so stable 4.6 What's available and you can see that 4.6.9 is technically here Right, but we knew it was blocked. We knew it wasn't available and it turns out that's because it's actually in this blocked edges So if I scroll down here to 4.6.9, we can see that Going to 4.6.9 from anywhere is blocked due to these two BZs So that's one way that you can find out when there are specific issues Probably the easiest way to see those is to quite frankly follow the PRs that are against this So if I get rid of this is open we can see for example Scrolling down and it's been a couple of weeks now Your block update upgrades to 4.6.9 So we can see precisely when that was put in we can see Rates that same information, etc So this is one way that you can look and kind of track when we block things when we add updates when we add upgrades that are available Inside of OpenShift and oftentimes they do provide that extra information so you can find out why it was blocked So just filling in information there So the last thing that I have to talk about Which is kind of an extension of what we talked about here I've also talked about updates and upgrades more thoroughly on a previous show We'll dig up that show and I will link it in what I am hoping to start as the blog posts following these shows is the upgrade process and This came from an email and an internal email on our SME list where Somebody was asking more or less, you know, my customer wants to know more about the upgrade process and You know a lot of times we market it and even when we demo it, right? We just show, you know Hey, come here to the cluster settings, you know, make sure you're on the right channel first and then you hit the the big update button and tell it to go and some stuff happens in the background and What what is that stuff? How do I know what's going on? You know that so and most importantly, how do I know when it's done? So there's a couple of resources available for that. So first let's talk about the upgrade process a little bit So when I click this button and I think it should ask for confirmation Yep, so when I click this button and I select this version and tell it to go. What's actually happening here? So if I were to hit the update button and I'm not going to because I don't want to have this cluster be out of commission for a few minutes Or during the show here What we're doing is updating the cluster version operator CVO is how we often refer to it. So cluster version operator you can think of as an operator of operators So which operators does it operate? So that would be all of the cluster operators. So let's switch over to My terminal here if I do an OC get co cluster operators What we'll get here is this big list of things these are all of the services that OpenShift provides to Kubernetes This is what makes up an OpenShift cluster So when I update that cluster version operator and go to 4.6.12 One of the things that I'll have in there is a set of manifests for each one of these child operators, if you will and It will tell each one of those operators to go to whatever version is defined in that set of manifests So essentially it'll say console go to a version from version 2 to version 3 you know at CD go from version 3 to version 4 so on and so forth and effectively it Steps back at that point We let operators do what operators do so this means that Operators and OLM have the logic baked into them automatically that says, okay Well, they and I'm just gonna make this up here, right? So the monitoring operator is dependent on the node tuning operator being updated first so it will determine the order that those need to happen in as well as Monitoring it may kick off and do steps one two and three But then it has to wait for this other one to finish before it'll do four five and six So effectively once you see all of these go from progressing true to progress or available true and progressing false That means that the cluster has been updated So that being said typically speaking, there's a couple of these that are going to take longer than others usually by far the longest one is going to be the machine config operator so machine config is responsible for For updating coro s itself So remember coro s is our PMOS tree based which means that it is going to have a separate image Basically, it pushes that new image It tells it to flip over to the new one once it reboots and then it triggers a reboot of the nodes So if you have you know five ten fifty five hundred nodes in your cluster It's going to take however many node reboots that is for that machine config operator to finish the update process So how can I make that go faster? So a couple of things to be aware of here. So one by default the machine config operator Will only apply to one Machine at a time. So how do we know that? So that comes from machine config pools. So let's take a look at our At our compute resources here I have two machine config pools and these represent a couple of different things. So one is Which configuration and apply am I applying to any nodes that match my machine config pool? So in this instance We'll scroll past all the noise here It's going to say so machine config selector. So it's going to say any machine That is a worker. Excuse me machine config that has the role of worker as well as any node that has a role of worker associated with it And we don't have it defined there is no value defined here But it is going to apply this to one node at a time by default so if I have a config pool if I have a machine pool that is 40 nodes And if we're talking physical nodes that could be you know four or five eight minutes for a node to reboot So five minutes times 40 nodes is a long time to wait for an update process to finish So you can adjust this using a an option Let's clear that screen. So if I do an oc explain machine config pool dot spec Inside of here we have this max unavailable value So I can use this value to set a hard number I don't want you to do any more than five nodes at a time or 10 nodes at a time or a percentage Only do up to 17 percent of the nodes at any point in time, you know, or 13 percent I want one eighth of them to go down So this is how we can potentially expedite that update process Just know that like any node drain node cordon operation. It's going to require pods to be terminated So make sure things like your pod disruption budgets for all of that other stuff is appropriately configured that should go without saying regardless of whether this is set to one or 100 You want to make sure that those values are set so that your application is not affected for node operations So that's kind of that process in a in a nutshell, right? When is it done? It's done when all of the child operators say that they're done Kind of straightforward but a little bit a little bit opaque So chris, I think there's a question actually So when you set this max when you set this max on available It'll take care of all the handling of making sure that your application Behaves appropriately, right? And as long as you don't pick a number that's too high and drives everything offline, right? Yes, but indirectly So what do I mean by that? Effectively what this number says is how many simultaneous nodes will be set to a cordon and drain status And be unavailable for scheduling Now when that drain actually happens, it's up to the scheduler to determine. What's the best way to make that happen? So it could be that Doing the actual drain operation takes multiple more minutes because maybe it needs to Hey, I need to turn off this node. I need to drain this node So I need to spin up another pod somewhere else and that takes some time for it to pull an image and spin up And for everything to be in a healthy status remember health checks so on and so forth so Yes, it controls how many nodes are unavailable But it's really up to the kubernetes scheduler and the rest of the kubernetes mechanisms to make sure that Within the parameters that have been defined The application stays available and healthy according to those definitions Yeah, and if you have if you're in a virtual machines in there as well, it'll live migrate those over to other nodes in the cluster right So the question we have in chat right now is does the six four six e us channel Will be available to premium subscriptions and is there any difference between four six stable and four six e us That's kind of a nuanced question. I feel like so. Yeah, so I I think there's two questions there right um So I don't know about subscription levels Um, let me rephrase that. I don't know authoritatively right. Um, so we'll we'll have to check on that one to determine which subscription level it is Will automatically have access to that us level after the fact Between you me and the wall, right? I strongly suspect that like most things in open shift and at red hat for that matter There won't be anything that technically prevents you from choosing that channel at any point in time The question is if I pick up the phone and say hey red hat, please help me with this They're gonna say Do you have the right subscription for us to help you with four dot six us or do you need to update? And I have the answer to that uh to quote the doc which i'll paste in chat here shortly E us is provided only with red hat open shift premium subscriptions There we go The answer to that very very clear and precise. Thank you So is there any difference between stable and e us? So today I would expect the answer to be no And we will confirm this. This is based on my assumptions and based off of some Ancillary email threads and other conversations that I've seen But we'll make sure to confirm this and correct it next week if we need to so today I would say no where that will come into play is more or less once we get out to So the support policy is current minus two Right, so that means that once open shift four dot nine is released four dot six would no longer fall into the regular support cycle So if you want to continue to receive updates to four dot six You would then need to be subscribed right have that premium subscription with the e us channel for for what you're doing So until then they should be basically the same as far as I know But we'll go ahead and confirm that Right, but the e us does only means we Backport like security things and you know, right relevant relevant cvs, right? It's not we're going to backport everything that goes into newer versions of open shift Yeah, yeah, it's the standard us definition Which as langdon pointed out the other day. There's also an lts Not for open shift, but for rel which is something completely different. So yes So I there's a link to the ocp four phases Um, I just dropped that in the chat for everybody to check out And then there is that e us Oh, and it has all the that link. I just dropped has the Timelines and everything laid out there in a nice life cycle dates kind of table. So there you go All right, so again, um, please don't hesitate to ask questions We do look forward to those But peter has been exceedingly patient and yes, we do greatly appreciate that. So I'm I'm going to lob the softball to you peter of let's talk about rev And open shift 4.6. I know that there was a Pretty substantial number of changes improvements expansions that happen there. So can you give us a breakdown? I can there's actually a bunch of things Just to give you a little bit of history right without hopefully boring everybody right open shift People have been running open shift on rev for quite some time Even before it was officially blessed and sanctioned You know with an ipi installer. So we came out in Open shift 4.4 And supported that infrastructure But it wasn't entirely complete right so things were missing like a storage provision or and stuff like that So you kind of you kind of could get further down the road, but there were some pretty key things that were missing We started working out we've been working on those immediately Some of them didn't make 4.5 and since we're actually now on the You know the release cadence for rev is much longer. It's about 18 months Open trips coming out every quarter right so Basically what ended up happening is a pile of stuff dropped in 4.6 that included The user provision infrastructure The csi storage provisioner Auto scaling was working and proper and literally all the things that people were waiting for kind of showed up in open shift 4.6 So if you're if you're looking for those types of capabilities In your on an earlier version of open shift on rev You probably want to consider at least upgrading to 4.6 To get a lot of these new capabilities. So let let me poke at that real quick because I know that there was some We had to be very clear and very precise about upgrade paths. So can you can you elaborate? So I don't know if we have a good picture for this you might so so there's really and we need to be careful because We at red hat tend to use shorthand and we say oh 4.6 and 4.3 Well, since I worked in both worlds right where I've got red hat virtualization or rev 4.3 and 4.4 And then open shift, you know 4.4 4.5 4.6. I actually had to insist when people talk to me like Use the product name space and then the then the actual version because otherwise we're not going to have a productive conversation so rev 4.3, which has been around for quite some time it actually released Right after I started back. It's about two years ago now Is based on rel 7 That's got a life cycle of its own rel 8's been out for a while Rev 4.4, which released last summer is based on rel 8. And so we we're now trying to keep up with The current modern operating system that I mean rel 8's more performant more stable It's just a better infrastructure to build your a much better operating system to build your virtual infrastructure on But we know customers don't move right away. So what we've had to do is support open shift on earlier versions like so open shift 4.3 and 4.5 on Rev 4.3 all right, the problem is is that now that we Are moving further ahead and kind of the breaking point or the The tipping point was red 4.4 has been out for a while now And it's going to continue, you know We're now in the us on rev. So if there's any changes that come They're only going to happen in the hypervisor right and what that means is Most of the interaction in the ipi is done through the rev manager. So if there's actually a defect or something that we need to fix We actually don't have a way to ship Fix for you, right? It's basically the the fix is upgrade to rev 4.4 So what we've been saying to customers is If you want to go from oh, you know, you're now running. Well, I wish I had a picture. You're now running open shift You know open shift 4.5 On rev 4.3 and you want to go to open shift 4.6 You should really upgrade your rev infrastructure first To 4.4 and then upgrade your open shift cluster on top of it, right? We've got that's the that's the route most customers are taking It will most likely work if you just put open shift 4.6 on rev 4.3 But it's actually not a tested combination, right? And there's actually a I can go dig up. There's a separate document we have that says These are the tests because we actually have this problem on every virtual platform, right? So on vmware and open stack So there are certain things that we test as part of ci and there's certain things we do regression testing on but Now that you're trying to test multiple versions of the virtual infrastructure plus multiple versions of open shift Uh that matrix blows up very quickly And I just pointed that or uh pasted that kcs into the chat here. Perfect. Thank you Yeah, so the tldr is if you want to run the latest version of open shift, you should be running on rev 4.4 Fantastic and the upgrade path so remember Today open shifts 4.6 4.5 4.4 are all fully supported If you are running an open shift 4.4 on rev 4.3 If you want to go to open shift 4.6, you first need to update Either rev to 4.4 or open shift to 4.5 and then update the other one And then move to open shift 4.6 So essentially all of those need to be updated They don't have to be done simultaneously, right? Or they would be done sequentially in whatever order you you happen to choose to do those in Um, so Open shift 4.4 was The first rev ipi As you mentioned straight that brought with it's the automated deployments We've talked about on the show before If there's any questions feel free to ask them the load balancers that are used how that's done with ipi not using an external one 4.5 we had some minor expansion and capabilities and then 4.6 we had kind of explosion as you as you said So big ones rather good stuff landed. Yeah, so so the big ones from my perspective ipi related So we added the ability to do auto scaling So we'll we'll talk more about that in just a second because you may be thinking But I thought if you could deploy nodes with a machine api, I can automatically do auto scaling No, that turns out that's not the case CSI provisioner Which I think is is an important one And then upi was added And you'll you'll notes for Those who are paying attention or have been paying attention for many versions of open shift Typically, it's not unusual for ipi to come first and then upi seconds In fact, that's almost always the way it has gone with the exception of v sphere right So and actually since we're talking about upi I'd like to give a shout out to A team of red hat consultants and I don't remember they're in the northeast I think that actually wrote a lot of the ansible stuff that we based our work on Because it was exactly like you said is you know, upi has a specific set of steps that You really need to and and on some of the platforms that's just here's a document and Your automation is you know, joe engineer But these guys actually said look we do this often enough for customers Um, let's actually turn it into an automation The set of playbooks that can just do the job automatically Yeah, so I actually want to talk about that a little bit. So Let's come over here to Which documentation page am I on? So I am this is just the standard open shift 4.6 documentation You see I've browsed to installing installing on red hat virtualization And then installing a cluster on red hat virtualization with user provisioned infrastructure And one thing that you'll note if you're comparing this to rev ipi or any of the vSphere deployment methods is this one is pretty substantially different As peter said, we rely on ansible to do Basically all of the work for you There is some some setup up front that you need to do And and I chatted with peter about this yesterday as I was going through testing it out and validating things prior to the show It's sometimes it's a little confusing In my opinion But I wanted to look at what this looks like and kind of go through the process a little bit So if you're familiar with me rates for for our audience, you know that I like to follow the documentation It gives me a great chance to provide feedback Likewise for any of you watching Providing feedback providing Suggestions on our documentation is super easy. Click this big open an issue button up here Takes you right to the github repo where you can submit a hey, there's a problem with this every time I've done it I've gotten a response within a couple of hours at most So if you have problems issues concerns with the docs, definitely take advantage of that button It's really helpful. I'm pointing over here at this screen like you all can see me Which way do you go george? Hi, where do you go? Yeah, the back team is awesome because they're actually Geographically dispersed and to follow this on model. So I'll I'll work with some guys on the east coast. They'll make changes. They'll get approved overnight by the folks in australia And it shows up, you know next day when I wake up. It's actually pretty cool. Yeah Yeah, it's um, I'm really impressed with not just their responsiveness, but The level of care that they put into the docs, right? They really do care and I've even gone to them with suggestions of like major reorganizations and It doesn't always come out the way that I envisioned it, but they're always willing to entertain those things So kudos to the docs team So a couple of things, um, it is ansible based which means that you need ansible So the docs which is a little bit unique For the installation methods, right? So you can see here the docs tell you how to set up how to install Ansible into your environment how to deploy The various ansible modules that we need for interacting with red hat virtualization As well as, you know, the api module that it needs to be able to do that By the way, this is much easier if you're trying to do this from a rel host Then if you're trying to do it from something else and install these things manually as I found out yesterday I'm trying to Drink our own champagne and and I've been using sento s8 stream From my for my management host so Which works really really well with the exception of that package doesn't exist in sento s. It's a rel package So right I had to go through the manual steps Um other things in here and these are standard install things right making sure that you trust a certificate generating the ssh key pulling down the various Installation programs now where it starts to change is with these ansible playbooks So Let me switch back over to my terminal here And inside of my working directory. You'll see that I've got this playbooks So these are the playbooks that came from That I literally executed that same script and it pulled down these yaml playbook files And the important one here is this inventory file And this if you've ever deployed open shift 3 this Kind of feels familiar right because remember with open shift 3 We had an ansible file that would say I want these worker nodes and these control plane nodes and these xcd nodes and so on and so forth So in here we're defining Very similar to an install config dot yaml what we want our virtual machines to look like Hey, I want a control plane node that has 16 gigabytes of memory and four cpus right four sockets with one core each I wanted to have 120 gigabytes disc that is running on this storage domain I want to have a network adapter that is connected to this network in this profile Same thing with our worker nodes And then down here at the bottom we have which vms. We actually want right. Hey, I want a A bootstrap node that uses the control plane profile. I want us a control plane node three of them actually That use same thing right here's the name I want you to use here's the type in open shifts And here's the profile I want you to use and then down at the bottom my worker nodes So pretty straightforward, right? Hey ansible This is how I want you to provision my virtual machines And if I were to run that script right now And I may be getting ahead of myself here because I think it wants me to do an install config first Actually, let's go ahead and do that So I'll flip back over to the documentation. So we just looked at the inventory before I get ahead of myself Because the next step is Okay inventory.yaml here we go install config.yaml So effectively we need to create our install config and the recommended way of doing this is to go through like you were doing a IPI install So let's go up a directory and we want to do an open shift install Config dir equals and we'll name this assets So I'm going to connect to my overt infrastructure. So I've done this before so it is cache the credentials. That's why it's not asking me I wanted to use this storage domain. I wanted to use this network connection This vNIC profile turns out that these don't actually matter as we'll look at in the next step My base domain is work and my cluster name I don't remember if that one's available or not, but it doesn't matter And we'll look at pull that All right, so inside of this assets directory I have my install config and when we look at These commands over here. It's telling me I want to Set the compute replicas to zero. This is just like with any UPI or bare metal deployments non-integrated deployments I want to set my machine network appropriately. This applies to any deployments regardless of infrastructure type And then I want to completely Delete the platform section. So remember when I said it doesn't matter what I actually provide for those values That's because we're doing away with all of these So we're just erasing all of these values here We'll put none Like this, I believe it's the correct value Yes And then I'm not actually going to run this so you want to always set the machine network to be correct for your particular environment Um, you'll note that I used a uh, Chris I used a uh, a validly formatted but not actually valid pull secret So I don't have to worry about uh revoking it like we've had some others on the stream have to do nice So we'll save this guy now and now we can actually create our assets So we can do our Open shift install creates manifest Um Dirt equals assets I'm in the right directory. So I don't need to provide that assets So now it consumed our install config. We created our manifest. So I need to edit the manifest Cluster scheduler I'm more or less following The correct documentation here So that way we set the control plane to be non-schedulable, etc And now we can generate our ignition configs So all of this should feel pretty familiar except for that step of Generate the install config but using an ipi type config and then make some minor modifications So now I've got standards ignition files So now we diverge back and I'm going to flip back to our documentation right generate manifest files Um, so we've got all of these files that it's talking about make the control plane nodes non-schedulable Um building the ignition files. So that's basically what we've just done So now we're at this step of creating the templates and virtual machines And this is where it gets really interesting to me. This is something that I really appreciate In that I don't have to go through the manual process. I don't have to Automate this myself. Essentially. I execute a playbook Um, so if I do go into playbooks and I copy and paste my command over here. So I don't have to remember it All I'm doing is saying create templates and vm's And ansible As soon as I figure out why it just completely failed to me Cannot find metadata. Json The ansible controller We're yeah, I don't know what just happened there For something that I literally tested Yeah, you're in playbooks. Yeah, I mean Well, that's an interesting one. What did I do to myself? I don't know This is the best part of the show though. I know you get to see us troubleshoot Yeah, christian says like I said, maybe you're in the wrong directory Cannot find her access metadata. Json. Oh, um, I know what it was Oh, well, that'll do it I forgot to set an environment variable which which is in the documents I was kind of get ready to blame sc linux, but No, we love sc linux on this channel All right now that I actually did follow the documents and set my correct environment variable. There we go It is going through and you can see it's probably scrolling by very fast But we have these crate vm operations that are happening So let's flip over to our rev manager And what we see inside of here is here's all of my virtual machines Right. It was it was that easy I didn't have to go through and click new vm and set all these settings or anything like that It it takes care of it for me Now interestingly if we want to do DHCP Right, we can go in and we can look at each one of these now And check our network interfaces and I can pull my mac address and I can do a static DHCP reservation and so on and so forth So It gives me this break in what's happening so that I can take appropriate action as an administrator to go through and set things up So remember with vSphere, we're essentially requiring all of that to happen at once Right or vmware does have that vapp property slash vm property thing where it can pass in static ip's Which we don't have with rev as far as I know Yeah, that's not that's not part of that and and it is you were correct to point out, you know, it's philosophically different than than what folks have done before because we felt that You know having a more complete document that lets you do a ton of stuff, but got very very deep in the weeds very quickly Right rev is really Hey rev is easy to use. That's why people like it. That's why people use it Let's make the up experience similar to that And that's how we ended up with You know making ansible do most of the heavy lifting and and like you said if you need to go in and tweak stuff afterwards It's absolutely possible to do that, but we don't we don't make you face that You know those sharp edges right away. Yeah So I'm not actually going to turn on the VMs here Mostly because I don't have the capacity in my lab for that. But turning on the VMs is super duper easy Um, literally if I execute for example the bootstrap playbook, what it's going to do is it's going to attach the The ignition config to the virtual machine So rev as of and peter, you'll have to remind me. I think it's Whenever we first started supporting ipi so 4.3.9. I think rev 4.3.9 Not not 3.0. Yeah, no, no right product name version name. Yeah, it's Rev 4.3.10 is actually what okay, so they they modified rev to be able to attach Ignition files not just cloud init files to the virtual machines So it will attach that bootstrap Ignition file to the VM it'll turn it on so that way bootstrap does what bootstrap does same thing with the control plane Same thing with the worker nodes so on and so forth So it's all managed through ansible, right? I turn on my bootstrap I turn on my control plane. I turn on my workers When it tells me that bootstrapping is complete. I destroy the bootstrap, which is also a Ansible playbook and I've got a cluster at the end of all of that. So very very straightforward Very easy in my opinion. Again, I don't have to fiddle with you know, VM images uploading them You know, mainly provisioning or creating my own automation to do all that Remember because it's published as a part of open shift. It's supported which is great You know, it it just handles it for me So if you were quick guide there When I showed my virtual machines here, you see I've got all of these other VMs running inside of here this is a IPI cluster. This is the one that I was showing a minute ago an IPI cluster deployed into my rev cluster there And the reason I wanted to deploy this IPI cluster and have it ready for us today is csi So Peter if you don't mind, can you tell us a little bit of the the background here on the csi provisioner and Importantly the the win and the where it's supported and not sure So, uh, the csi work has actually been underway for some time right if you know Over it is our upstream There was actually a previous generation that was called flex valves or something like that But what we allowed is to provision rev managed storage right essentially a storage domain to have a provision pvcs for the cluster right and It originally like I said, we started development in open shift 4.4 and 4.5 So it technically will work under those versions, but again, it's not tested nor You know, it's it's not supported in those environments. So you'll need to be an open shift 4.6 Right, and that's where we we've got ci pipelines and and it's gone through a full regression cycle there Okay, and and the reason I did ipi here is because it is deployed automatically with ipi I didn't have to do anything to to deploy this when the cluster was done deploying it was here and it's ready for me to use Yeah, and that's again philosophically. It's slightly different than some of the other virtual platforms We have to that's an extra step, but like we just and this is something I try to You know instill in our team The default behavior should just be what you want right is because if you're good if you're going to be using ipi You pretty pretty often going to want to provision storage out of your rev cluster to the to the open shift cluster And we just do that by default Yeah, yeah, and I'll note that and I think I mentioned this a week or two ago So when there is a provisioner that is deployed as a part of the ipi or upi deployment It is actually controlled by the cluster storage operator And the cluster storage operator will make sure that that storage class always exists So if I were to come in here and I'm going to try it just because I want to see what happens and and see whether or not I'm right If I were to delete this storage class Then effectively And it might take it a little bit Or it could have went really really fast and The operator is going to recreate it and make sure that it's always there So I can't not actually it's probably not deleting because I think that there's a yeah, there is a pvc That's already using it, which is the registry So it'll it'll make sure that it's always there now I can modify it If we look at this one, I can do things like I could change the storage domain that it's using I could change From thin to thick provisioning, etc But and I can change it so that it's not the defaults, but it will always exist There's nothing that's just the function of the storage operator cluster storage and And you bring up a good point, which I don't know if it's that clear in the documentation, right, which is The default storage domain this class has created and is the default Storage domain you install the cluster in right the ipi you have to put it in a storage room You say okay great. We'll just create a class that The provisions out of that But if you actually have multiple types of storage like one storage domain is, you know Gigantic spinning hard drives and you've got another storage domain. That's all flash I would actually just create don't don't mess with this one right just leave the default there Create a new storage class or multiple storage classes And then like you said go into the go into the yaml and just change the change the storage domain it gets deployed on Yeah, exactly So real quick just because I love when things act exactly the way that you expect them to act Which I feel like is the the name of the game particularly when it comes to Many open shift things Let's create a persistent volume So i'm going to call this one, you know a test because i'm super creative like that We're going to give it a 13 gigabyte size because it's lucky number 13 And hit the create and you'll note that it's already bound um, we can see the persistent volume down here by click over I can see my 13 gigabyte persistent volume ending in 788 alpha and if I flip back over to my rev environment here and Filter these down some so that way they're easier to find I've got a disc There we go A disc right here 788 alpha and we can see it is 13 gigabytes in size so It literally does precisely what it says it's going to do right when When you create a pvc it creates a disc inside of that storage domain on the rev side If I were to mount that it would attach it as a disc to the virtual machine and I'm trying to remember peter. I had a conversation with the engineering folks Um, and I don't remember the precise number, but do you happen to recall what the maximum number of disc per virtual machine is? Oh, good question um it almost I'd have to go look that up it you almost end up being bound by What the guests can actually handle in terms of like scuzzy Units and stuff like that. Yeah, I I remember The the phrase that they 255 most likely no, so the the phrase that they told me was effectively unlimited Because it's something like 65,000 Discs can be attached to the virtual machine To your point. That's not the same thing as how many discs can coro s handle but There is just because you can just because you can doesn't mean you should yeah Yeah, so if if you have concerns about, you know, oh, can I you know, if I if I have a pod that needs, you know, 40 50 60 pvs I don't know why you would but maybe you do Um, you know, can I can it handle that? Absolutely. It can handle those types of things And it can handle multiple pods better like that if you so choose Uh, so just to round things out here. Um, we'll go back to our pvs here We'll hit the Delete no, not the persistent volume the persistent volume claim We'll delete the persistent volume claim And you see our persistent volume over here is already gone and if we come up here This might take just a second or two longer. There we go our disc inside of the storage domain is gone. So Again behaving exactly the way that you would expect it to behave um Chris we were having a conversation in in our team chat the other day about how When things magically fix themselves, it's kind of terrifying because you don't know why it broke and you don't know Why it fixed itself like when things behave the way you expect them to and want them to that that is Priceless to me. So yeah That is kind of our rev philosophy Um, and actually I do want to touch on one other thing here that that does come up as well, which is Rev is one of the only platforms that you can do a essentially a hybrid cluster right in the sense that I can have rev manage nodes in my open chip cluster, but I can also attach bare metal nodes Right now normally you can't do that But rev actually doesn't have an official cloud provider, right? We have cloud provider like capabilities So you can actually go sneak in and say hey, I've got Some bare metal nodes bare metal nodes that I've deployed You know, I don't want to virtualize anything there But I'm gonna actually attach them to my open chip cluster running on rev and that works just fine Oh, that's interesting We get we get questions about that fairly frequently Um, I see a question from Kenneth in the chats. Yeah vgpu supports Oh vgpu for Rev now this this is inception level. Well rev it does do gpu and vgpu support and that can be passed through The worker nodes that you do create In in open shift Yeah, I was gonna say I know rev itself supports gpu pass through and vgpu and multi gpu vgpu I I but I don't know off the top of my head whether or not Open shift on rev with vgpu is supported, but peter just said yes Yeah, well, let's be clear right the use case for this is more about right There's actually two use cases for a gpu right one is You know compute intensive workloads where I just want the the boost from the vector processing that I get for say aiml And the other is remote visualization Right, so the the first use case is way easier to handle because you don't have to worry about video drivers or What's my display protocol to the vm? It's just Hey, you connect to the gpu and the The process inside the vm or inside the container just talks to the gpu and handles it and there's not any other craziness So that's easy to do if you want to do something like remote visualization. That's gpu accelerated. That's slightly more complicated indeed Well, I believe that brings us to the end of our time. I can only barely see the clock on my screen Yeah, it's great fun here man, you know come on more often As long as I have something intelligent to talk about and usually that means I have to bring andrew along But yeah, I'm happy to join you you you spoil me. You flatter me how your checks in the mail There's one last question here Okay, as a point of clarity for bare metal nodes can these nodes be added after an ipi install or Only after slash during a upi install Uh, only the only place I've seen at work is during the upi install. Yeah That's what I figured. I it's not supported an ipi at all and it's yeah Mox next we don't do that. Yeah the the only ipi platform that I am aware of that you can mix physical and virtual is open stack with ironic right And I uh, do I got 30 seconds to drop take to telegraph something? Um, no this so the trick is You know There's some the nice thing about red hat. There's so many options, right? So there was the use the bare metal installer on rev totally You know totally non-integrated then there's ipi and now there's this new upi we shift Um, I think we've talked publicly about the assisted installer. Yes, yes All right, so that's something we're exploring We've got no plans to drop it in a future release, but it's absolutely something we know we want to do it's it's essentially A sort of the best of upi and ipi together Um, which works out well for rev and we just got to figure out let me get the cycle to do that It is how I stood up my cluster here at home. So yeah, I thoroughly love assisted installer Yeah, yeah, it's it's convenient when you can use it Absolutely All right, well Before we steal too much more time Um, I want to say thank you to peter. Uh, greatly appreciate you coming on Thank you to our audience We do appreciate all of the questions if you have additional or follow-up questions Please feel free to reach out. Uh, so I am At at practical andrew on twitter or andrew.sullivan at redhead.com Uh, peter you're What's the best way to contact you? Are you so uh, uh, social media twitter is pc launerback That's probably yeah, that's probably the the best way Okay, and as always you can reach out to me at chris short on twitter and c short at redhead.com And I will make sure I get your question answered All right, so the last thing before we go tomorrow at 9 a.m 10 a.m Is 10 a.m. 10 a.m. Is the Open shifts 4.7. What's new presentation? So I will be here on the stream helping answer questions helping pass questions over to the product management team that's giving that's presentation So be sure to tune into that one Again, feel free to ask questions and we'll make sure to get those answered during the presentation or or as close to afterwards as we can Yeah, no, we'll be there. I'll be there with 40 of my closest colleagues as we talk about all the stuff that's coming okay, um Islam could you email that question to us, please? And we'll find you an answer. Yes, please andrew andrew.sullivan at redhead.com C short at redhead.com too And without further ado, thank you peter. Thank you andrew and we will catch y'all tomorrow on the what's new and 4 7 briefing