 working on OpenShift and sometimes. This is so awesome. Everybody has something to think about. On top of Red Hat portfolio. Hello and welcome to everyone. It's a great day here at Red Hat. It's a wonderful Wednesday as you might expect and I am still extremely excited and very privileged to be joined by my co-host Johnny Ricard. Johnny, happy Wednesday. Hey, happy Wednesday to you Andrew and the privilege is online so thank you. Yeah, it's funny. We were both talking about how it's been a crazy week. You've switched roles. You're now fully in your new role. I've been just running all out because, I don't know, it's November and for some reason November is just a good day to be or a good time to be busy. That's right. You've got to get everything caught up right before the holidays. We've got Thanksgiving coming up. Time to get happy and full and then going into the Christmas break. It's the last mile. Yeah, I know here in the U.S. we've got two weeks until Thanksgiving. It's holiday season almost, which is kind of crazy. Already. Yeah. Well, Leeds, thank you. I can take absolutely no credit for the intro. The first one that you saw, the one that still has Chris doing the intro and all that other stuff. That was actually created by the intern team. If you all remember when Bobby was here and all of that other stuff, that group of folks created that. I think it was actually played for Red Headers. I think it was played during one of our all-hand sessions, maybe during one of the shows, something like that. Then the second one was created by our creative team. Behind the scenes, our producer Stephanie worked and got all that stuff done. I can't complement her enough on just how professional she makes us a bunch of amateurs look. It's really great. As always, I'm trying to fill the role of host here, which is still a bit unfamiliar for me. Of course, please remember to subscribe, to like and all that other stuff. We do appreciate all of those from everybody. This week is a bit different. If you noticed, if you're subscribed to the streaming calendar, we extended this one to two hours. The reason for that is because of our topic today. Our topic is, of course, disconnected installs. Specifically, we want to do a deep dive. This topic came at special request. For some of you may remember, I went to an in-person event for the first time in about two years, which was the Red Hat Summit Connect here in Raleigh. One of the folks there, who works on what we lovingly refer to as the Synergy Team, they had asked me, hey, we're doing a lot of and seeing a lot of disconnected installs. Can you do a deep dive on that? It just so happens that Johnny is an expert here. It's something that I know you've done more than a few deployments of. Oh, yeah. In all kinds of different environments in cloud and on-prem data centers. I'll never call myself an expert, but definitely have a lot of experience in the field of doing this air gap. Yeah. It's constantly changing. There's little nuances, stuff like that. I know we had reached out to the product management team. There's some stuff in the pipeline that's going to make this a lot better. We just can't show it yet. I will make some allusions to it. If I can remember what the GitHub repo is, Johnny, if you happen to remember the one that they pointed us to for the disconnected registry mirror, we can show that in a little bit. I forgot to grab the link before we start. I got it. I'll put it up in the chat. Yeah. We'll talk about all that stuff. My goal here today is to walk through the different phases of a disconnected install. For longtime viewers, long time watchers, you may remember we talked about this way back in episode 13, but that was just an hour long and it was crammed in with a bunch of other stuff. I think I got as far as kicking off the install, but never actually finished. Hopefully this time we can get through. Christian's here. Christian, I hope you can see because I wore this shirt special for you. Do you see my shirt here? It says, I'm a few records shy of a DNS zone. That one's for you because you're my favorite DNS expert. All of that being said, thank you for the compliments on the intro. I appreciate it, everybody. Let me read through the questions here. DMI 3, OpenShift 4.7, full support ends October 27th. Oh, yeah. I think the way that that technically works is there is an overlap period for those releases. Essentially, when it's the current release, it's under full support. Once it moves off from being the current release, so when 4.9 was released, theoretically, there's an overlap period of a few weeks or a month. Then 4.8 rolls into maintenance phase. Then once 4.7 would then roll off and become unsupported. Nothing has really changed in that respect rate. It's always been the policy. What does change is essentially it should be described in the life cycle page. Let me see if I can bring that up real quick. No, I don't want to play a video. I want to share my screen. We'll share this guy. If we do OpenShift Life Cycle and we check this first link here, what we should see here is this maintenance support. This breaks down the differences. I'm not going to sit here and read it or anything like that, but it should describe the differences. What those are technically, I don't honestly know. I can see here qualified, critical, and important security advisories. These two paragraphs seem to be very similar, but they are not the same length. I would have to review that in more detail to get you a better answer, but it should be here. I'll post this over into the chat as well. Do keep in mind that we did recently change the support policy for the even numbered releases. Actually, no, that's not fair. We changed the support policy for all of the releases, but we called out specifically the even numbered releases. Historically, we did it based off of a rolling release window. Support was based off a rolling release window. Current minus two was supported. Actually, that may be what the change was, DMI three. With that change, we moved to it being timeframe based. It is for 18 months, regardless of how many releases are inside of there, although it should coincide with, I think it's four releases with the yearly or maybe five releases with the yearly, because now I'm remembering Mike Barrett's blog post. I'll put a link to that in the blog post that goes out that has that summary of the support changes. The short version, if you missed it, is that starting with OpenShift version, I think it's version 4.7. With OpenShift version 4.7, we now support all releases for 18 months. If all releases are supported for 18 months, you might be thinking to yourself, well, then why are some of these called EUSs? EUS usually means that it's supported for longer than whatever that period of time is. The EUS is one in the immediate term, starting with 4.8 with the update to 4.10, you'll be able to do an accelerated update. Basically, what that means is that when you choose that option, the control plane will update from 4.8 to 4.9 to 4.10 linearly, because it has to, but we will pause the worker nodes so that they only go straight from 4.8 to 4.10. It's basically only one reboot for all of the worker nodes, which can dramatically shorten the amount of time needed for an update. Longer term, those even numbered EUS releases, if we have the opportunity to extend support beyond 18 months, it would happen with those even numbered EUS releases. For example, let's say that API stability or whatever it is that they use to measure that is where they want it to be. Hey, yeah, let's go ahead and extend support to 24 months or whatever that is. That would happen with those even releases. I'll dig up maybe on the side here. Mike Barrett's blog post on cloud.redhat.com. Man, I'll get it right eventually. Time is on your side. Here we go. Let's post that over here in the chat. This blog post from Mike Barrett goes through all of this in excruciating detail. If you have any questions about this, don't hesitate to reach out. You can always contact me here through chat if you happen to be watching. You can also contact me social media at practical Andrew on Twitter or you can send me an email, andrew.sullivan.redhat.com. Let me check through the chat here real quick. While I lead YouTube on the TV, I'm so sorry that you have to see my gigantic face on the TV. Johnny and I were poking fun at each other about the talking heads before we got started here. I also apologize because I do have a giant dome. Andrew, you are not alone in that regard. Piyush, how can I check the calendar? If you're referring to the streaming calendar, the easiest way to do that, and I don't think it's red.htslivestreaming. I'll post that over here as well. I'm a lucky guest, by the way. If you go to this web page and you scroll down, we have this calendar here. This calendar goes through and shows you all of the streams that we have coming up. If you click this Google calendar down here in the corner, it'll open your Google calendar. I'm not signed into an account, but if you are, it'll open that Google calendar and you have the option to subscribe to that if you so choose, or just bookmark the calendar so that you can see it whenever or bookmark this page. This page is a good one, right? We have links to the YouTube channel. We also have all of the other streams that are available here. Along with all of the other stuff that the new media team is working on. Lots of great stuff. I definitely recommend if you don't bookmark the calendar, subscribe to the calendar, at least bookmark this page so you can keep up with what's going on. Oh, Johnny, thank you for posting that image or that link, rather. We'll take a look at that in a few minutes. Faced a lot of issues while downloading operators for external registry using pruned index images. Yes. I sympathize with that. I was going through this yesterday as I was trying to get ready for this particular stream. Johnny and I were chatting back and forth because we were both having infrastructure issues. Me due to a misconfiguration and Johnny due to his ISP being flaky. I don't know what it is about my co-hosts and flaky internet. I got nothing. I think I've got that resolved. We'll walk through that. The good news is I went through and documented everything that I'm going to show today in a markdown document. My goal is to put that up onto GitHub after the stream and then we'll link that from a blog post. Everything that you see here and probably more will be available there. You can always reach out and ask questions about that. Okay. So by all means, please keep the questions coming in, kind of regardless of topic. I skipped over it, but this is or I skipped over my normal introduction of the office hours. So I'll go ahead and remind you that the ask an OpenShift admin office hours is exactly that. It's office hours. So if you've ever had a professor or manager or anything like that who had open office hours, that's where you can come in and you can ask questions about whatever and anything on your mind that you have for us. So Johnny and I both have a pretty deep administrator background. We've been doing this for a number of years and that's kind of where our expertise lies. So we are here to answer those questions. And usually we do that. Again, you can ask anything that you would like regardless of what topic that we're talking about for the day, but we also have a theme or a topic for the show. So that way, we're not wholly and only, you know, we're not sitting here staring at each other and waiting on you all all the time. We also have, we also have, I'm trying to look and see if I missed any questions here. If I did, Johnny, do you see any that I missed? So we also have, okay, what I call, I'm usually loathe to use the term reoccurring segment, but that's kind of what it's become, which is the top of mind topics. So essentially, these are just a handful of things that we see that we have encountered commonly or that have come up in the last couple of weeks, really since the last time that we had a stream. So just things that we think are relevant or important to you all that we like to bring up. AJ, I think we did answer that question. So we don't have an article that discusses that aside from the docs, but we'll be sure to cover that when we go through. AJ, you're welcome to ask about MTV. I can answer to the best of my knowledge, but that is admittedly shallow knowledge. So feel free to ask your question. I'm going to plow on with our topics here. So the first one that I want to talk about is, oh, the docs UI. So if you have been to the OpenShift documentation page recently, and let me refresh it here. If you've been to that page recently within the last few days, you'll notice that it looks a little bit different. So they did their best and really the goal here was to improve the overall experience with the docs to make it as friendly as possible. So you notice some little things like this side menu now scrolls independently of the rest of the page. Hey, Andrew. Yes, sir. You need to re-share your screen. Sorry. Oh, yeah. Good point. I can only paint so much of a picture with words, right? So the page we're looking at here, this is just the docs landing page for 4.9. They retroactively applied the formatting to all of the docs versions, but you'll see things like this side menu here scrolls independently of the content window. If we click on one of these, we'll pick on this one because it's relevant for today. See, our center section scrolls independently. We've got a table of contents over here that's much easier to use and find things. So this is, and I think I've mentioned before that I have or tried to have kind of regular meetings with our docs team. So this is just one step or one phase of the improvements that they're working on. That team is really great about really trying to take into account the things that you all have, your opinions, your perspectives, usability, stuff like that. So don't ever hesitate to send that feedback. Either you can send it to me. I'm happy to pass it on. You can also open BZs or GitHub issues, and they all, they look at all of those. So the second thing I wanted to talk about is a couple of features for open shifts that have been recently released. So remember that open shift releases are decoupled from many of the features that are a part of open shift. So for example, the one that we're looking at here is service mesh. So I think it was our hope nine. I don't know if you're on today, but you had sent me some messages asking about some features in service mesh, kind of all dependent on 2.1. Well, 2.1 is released. So if you're using service mesh, definitely take a look at all of the things that are inside of here. I will admit that service mesh is one of those things that I am not super familiar with. So I cannot necessarily answer a lot of questions about it, but I will certainly try my best, which brings me to an excellent point, which is if we don't know the answers, you know, Johnny and Andrew only know so much, we're more than happy to go and find those answers for you. So even if it's something like service mesh that I don't know a whole lot about, so I don't know about Johnny, I'm happy to take those and we'll figure it out. AJ, I migrated a virtual machine having a web server configured from V center to open shift, but after migration, the VM inside the, persisted the source VM IP. Yeah, so if if the virtual machine had a static IP configured, you would need to go into the operating system and basically tell it to use DHCP or something like that. I don't think it will reconfigure that automatically. I can take a note and see if I can find out the answer for that. If you'd like, please send me an email that has your question that has the issue, Andrew dot Sullivan at redhead.com and I'll follow up with the product team and see if we can get an answer for that. Thank you for posting that link, Stephanie. So the second thing that I wanted to talk about, which conveniently, thank you, AJ is a OpenShift virtualization. So last week, OpenShift virtualization released their version four dot nine. So their versions tend to match the version of OpenShift that they are associated with. And the integration team here posted a couple of things that are interesting that are coming out of this particular release. So one hot plug disks, two is MTV, there was an update to MTV, and three Sysprep. This one is particularly interesting to me because I was just doing some work with Windows nodes. And while I don't know a whole lot about Sysprep, I know that it is, it is fairly important to know a lot of Windows admins take advantage of that. So OpenShift virtualization four dot nine, they made a bunch of UX improvements as well. You can kind of see an example of that integrating this Sysprep thing in there. If you've used OpenShift virtualization previously, like you might be able to see if you squint a little bit, this SSH service, there's now you can upload your public key into a secret and then it will automatically add that to Linux virtual machines that are using cloud in its, it'll automatically create a service so that you can SSH into the virtual machine, all that other stuff. So they're really making improvements on the usability there. That's awesome. Yeah, the other thing and, you know, hot plug disks, some people might be thinking, Oh, it's, you know, what do you mean I can't hot plug a disk, you know, that's, that's been in KVM that's been in virtualization forever. So this is one of those like, on the surface, it seems like why wasn't that there but when you think about it, their mantra is Kubernetes native virtualization. And if you're trying to add a disk to a running, you know, a PV, essentially add a PVC to a running pod, that's not possible. So they work some magic. I have no idea how it actually works, where you can add a PVC to the pod. So interesting stuff that's going on there. I'm looking forward to diving into it a little bit. The OpenShift virtualization stuff is interesting, right? There's a lot of possibilities there for, you know, having VMs and containers literally on the same platform, lots of lots of cool stuff for application modernization. All right, I think that is all of the stuff that I had for today, at least for the top of mind topics. Yeah, that's all I got on my list. So disconnected. Disconnected is, it's an interesting topic. It's an interesting problem, if you will, of, you know, hey, I have this isolated network, or maybe it's not isolated. Maybe I just don't want my cluster to be randomly talking to the internet all willy-nilly for whatever reason it happens to be. So disconnected. And if you remember, if you're a OpenShift 4 customer from way back, disconnected was not in version 4.1, the first version of 4 to ship to the public. I don't think it actually shipped until 4.2. So disconnected allows us to instantiate an entire OpenShift cluster without requiring internet, without requiring access to all of the various Red Hat servers and services to provide all of those images. And when you think about this, it can be a little bit complex and a little bit intimidating, especially if we look at the documentation, right? So let's come over here to the documentation. You know, I was just showing this documentation page, and you'll notice that like, this is a, this is a long page, right? This is, this is a lot of stuff. And this is just to mirror the content. This isn't, you know, doing all of the other steps that are associated with it. So I want to walk through and, you know, look at a bunch of different aspects of this. You know, I, like I said, I went through, I stepped through this again. So it's been a little bit since I did a disconnected install. So I went through and stepped through all of these things. I want to look at some other things like the OpenShift update service, mirroring, registry catalogs, excuse me, operator catalogs, all of that other stuff to hopefully get a good perspective on what that is. So keep your questions coming. My desk setup today changed a little bit. I've got a monitor way over here, so that way I can share this entire monitor. I will warn you all it is an ultra wide. So if it's too small for you to see, just speak up in the, in the chats and basically I'll shuffle things around. So that way we can just share one window like you're seeing right now. But yeah, that'll make it easier for me to switch back and forth between browser and CLI and all that other stuff as we go through this process. And if you notice me looking way over here, that's me referencing my notes and or the chat. And I'm giving an eye on the chat to enter. So like I'll, I'll try and bring away as many questions as I can. I appreciate that. All right. So first let me stop sharing that. And then I'm going to share my entire screen here. And the first thing that I want to do that is a part of this is I think we want to consider kind of the different phases of doing a disconnected install. So when we think about deploying a cluster, you know, we think, okay, well, I need to create my install config. I need to generate my ignition files. I need to deploy my, you know, virtual physical machines. And then I go in and instantiate the cluster. And then some magic happens behind the scenes. And, you know, roughly 20 to 40 minutes later, I have a cluster that's up and running. So for the most parts, none of that changes, right? The actual install process is effectively the same. Where it gets a little complex is that, well, we need to make sure that all of those resources that it's using in the background. So remember, open shifts is Kubernetes that has OLM installed to it operator, lifecycle manager. And then a specific operator, CVO, the cluster version operator, basically works to pull in all of the container images, all of the things that are open shifts and deploy them to that Kubernetes cluster, converting that Kubernetes from, well, Kubernetes into open shift. So all of those container images that are open shift need to be made available in that disconnected network. So that's kind of step one, that's getting the deployment done and being able to just stand up a cluster. Step two is deploying useful software from the Red Hat operator catalog into that cluster. So today I'm going to use the example of the open shift update service, hopefully, as soon as I find my mouse. So if we come down here to the updating clusters, we've got this understanding the open shift update service. So this was released in the 4.8 time frame, 4.7, 4.8 time frame. And it effectively mimics the same service that we have on the internet coming from Red Hat for if you're using a connected cluster, you can go into the admin console and you can say update cluster and it has that nice channel where, oh, I'm on 4.9 stable. I want to go to whatever the next one in the 4.9 stable channel is. Hey, Andrew, real quick. Yes, go ahead. Your screen's coming across blurry. Okay. So I will, I'll just switch to sharing a window then. So my apologies to Stephanie in the background, because that just means that she'll have to be, she's got to click more buttons. Much better. It's still a little bit blurry in like the middle. So I don't know if, but like you can actually make out like the big taxes, just it's still a little blurry in the middle. That's strange. I wonder what's causing that. Yeah, I see it in our, in our view here as well. I'll refresh this webpage and see if that changes anything. No. You know, we were just complaining. We were just saying that the internet is sometimes flaky. So anyways, if it, if it persists, let me know. We'll see if we can, you know, maybe I can drop and rejoin to the restream here and see if we can fix that. Let's see. I'm just checking a couple of things here. Yeah, everything's set to full definition and all that other stuff. So as near as I can tell, it should be okay. All right. So please let me know if it's just completely not legible, you know, completely unusable, and we'll take more drastic measures, but hopefully that'll resolve itself here in a little bit. Yeah, even switching tabs. See that one looks okay. Maybe. No, that one's a little blurry too. All right. I'll forge ahead, but Johnny, just let me know if I need to drop and rejoin. I can zoom in a little bit. Let's try this. So that just bumps it up to 175. Well, eventually I'll find the right window here. No, that didn't help either. All right. Yeah, welcome back to 240p. Let's, let's hope not. Okay, let me, um, I'm going to close from the stream and I'm going to rejoin and see if that helps. So let's stop sharing that and I will turn off my camera and audio. So Johnny, if you don't mind doing a little dance, I'll be back in 10 seconds. Okay, cool. Yeah. So where Andrew was kind of, you know, trying to get to is the open shift update service and it's using the Cincinnati operator, which it, you know, when you have in a disconnected environment, you'll build out like a graph image and what that'll do is it'll allow you to see like your update path. So if you're on like a specific version of 4.7 or 4.8, it'll tell you like, you know, you can update to this path without, um, you know, without having any issues or, you know, like if you're trying to go from 4.7 to 4.9, it'll tell you like the exact path that you would have to update to in order to get to that next major release that you're wanting to get after. And Andrew's back. Let's, let's see if it helped. That looks better. Okay. All right. Well, strange browsers. I got nothing. I'm afraid to look at, uh, at my memory utilization or whatnot on, on the browser. It's probably through the roof and that might be either contributing factor because I have a lot of things open. Anyways, please let me know if that happens again. Um, we'll, we'll figure out how to address that. All right. So let me close out some things here so I can refocus. All right. So Johnny was saying that, um, so the OpenShift update service, uh, which we in OpenShift refers to as the Cincinnati operator basically mimics that over the air update experience. Um, so I'm hoping that we'll have the ability that we'll have time to go through and deploy this and maybe attempt an upgrade inside of the environment. Um, and then the last aspect here is, um, disconnected operators. So as, uh, and my apologies, I lost all the chat when I disconnected and reconnected, but as somebody was, was highlighting, right, mirroring those operators is a little bit confusing because it's not just mirror the container images. It's mirror the catalog that operator lifecycle manager uses to, you know, be aware of those things. Uh, so what I want to do is show how to prune that down, um, because if you've ever tried this before, if you've ever mirrored the entire operator catalog, it's something like 350 gigabytes. Um, and not all of the operators are supported disconnected. So we can prune that down. We can cut it down to, um, well, in my test, the smallest I got, it was 14 gigabytes. Um, but that's better than 350 for sure. And I posted a link to the disconnected operators, the ones that are supported. So, um, if you have access to a red hat, if you have access to a red hat portal, go ahead and log in and you can see it's a nice, you know, table of what isn't, isn't supported, disconnected. Yeah. Thank you. And, uh, you can, you do have access to access.redhat.com, the portal. If you have a developer account as well, uh, you don't need a full on customer account. I guess it is, I guess a developer account is a, uh, a, uh, customer account, developer account is a customer account. Yeah. All right. So let's get started. Let's look at a couple of things here. So the first thing that we want to do is mirror that data over. And if we come up here in the documentation to installing and mirroring images, effectively I'm going to be going through and following this documentation. The difference is that I've actually created a couple of bash scripts to make this helpful for me. So what I'm going to show in just a moment is if we scroll down here to the actual mirroring of the content. So what I've done here is basically take all of these commands and drop them into a single script. So that way it's just easier for me to be able to do this in a repeatable manner. And then at the end of all of this, we get down here to step three. And you'll notice a couple of things here. So one, this first sub bullet is if your mirror host does not have internet access. So basically if you're going to a disconnected installation, fully disconnected installation, this is the set of steps that you want to follow. Whereas down here, if the host that you're using, uh, so you're what I call a bastion host, if it's straddling between connected and disconnected, then you can basically use that bastion host, that admin host to pull images from one side and push them to the other side. And then from there, we go through and do all of the regular installation stuff. All right. So let me grab this terminal window and then change which one I'm sharing. Okay. So just to try and eliminate as much confusion as possible, this is the normal host that you'll see me use. So anchor, you'd see the prompt is green with a little bit of yellow or blue or excuse me, yellow, green or red, depending on the outcome of the last command. And then this one over here, which is going to be all white and named bastion. This is in my quote unquote disconnected network. So it's not actually disconnected simply because I don't have enough infrastructure to have everything replicated twice. But we are going to be doing everything exactly as though it is disconnected. And hopefully I can prove to you by showing things like the registry logs that, hey, it really is pulling from that local mirror and not trying to access the internet. So the first thing we need to do is start with our connected host. So this is our public, great, our connected network. And I need to go into this particular directory. What I've done here is what I've got in here is a couple of directories just to help me keep organized. If we go into four dot nine, I've got a number of different scripts inside of here. So let's first look at this dry run. The dry run is that first command that we run under step three. And you'll notice that it basically mimics everything that we want to do, including this to release image line. And you'll notice that that has this local registry, which is intended to be your destination registry. So your disconnected registry. So even if you can't connect to it from this host, basically what they're suggesting is use this to release image. And here with the dry run option, because it's going to output the image content source policy that we need. So let's say, for example, that you, your host names are sensitive for whatever reason, you don't want to even type those into your connected machine. It's perfectly fine. Just use dummy values inside of here. It's not going to actually try and push anything or anything like that. It just uses those at this phase for the dry run. It just uses those for those image content source policies. So all I did here, as I kind of walked through in the documentation there, here's all of my environment variables that we set. And then I execute our, our command. So this dash a, this is our credentials. So this would point to my pull secrets that I'm going to use. So just like if you were to go to console that redhead.com slash open shift, you grab your pull secrets stored in a file. That's exactly what you want to use here. From there, we just build out this quay.io URL using the parameters up here. Documentation walks through examples for each one of these. I selected 494 for my OCP release. So two, again, this is based off of those temporary things, right? These don't have to be real values. They can be fake values if you don't want to post those in, because this is just the dry run. And we'll look at the actual mirror command here in just a moment. Let's quit out of that and let's execute our dry run. What it's doing at this point is reaching out to quay and it is going to basically determine what are all of the images that make up the particular OpenShift release that we're talking about. So you'll notice that it is 141 images that make up OpenShift 4.9.4. Having downloaded this previously, it's about nine and a half gigabytes in size. So it takes it a second to digest, pull down, then digest all the metadata, do all the stuff that it's got to do. And what we're going to end up with here is it's going to scroll by a ton of information. It'll list out every single image, it'll list out how big each one is, it'll list out the shop, you know, here, all of that information about each one of these images. Ultimately, we really don't care about that. I mean, I'm sure somebody does if you want to validate it. What we really care about are these two values down here. So one is image content sources. This is going to be the stanza that we use in our install-config.yaml. So that when we install the cluster, it knows that any time I'm trying to go to quay.io, OpenShift-release-dev, blah, blah, blah, I'm actually going to get redirected over to this location. And same thing down here for this ArtDev. It'll get redirected over to this location. The object down here, and this is a Kubernetes-yaml object. If I had a cluster that is disconnected, or excuse me, connected today, and I wanted to change it so that it is going to pull all of this content from my mirror, so whether it's a connected mirror or disconnected mirror, this is what I would do. So this is particularly useful if maybe like Johnny, you've got a less than reliable internet connection. I better knock on wood. I'm going to jinx myself here. So if you have maybe a less than reliable internet connection, maybe your throughput isn't great, lots of edge sites have limited throughput, stuff like that. Maybe you just don't want to download nine gigabytes of data for every OpenShift cluster that you deploy. If you're downloading or you're deploying 2, 5, 10, 100 clusters, that's a lot of data to pull across your internet pipe. So you can mirror all of that, you can point everything over to here. This format is also good because you can use it to kind of blanket redirect. So we'll talk a little bit about that. Hopefully I'll be able to go into our node and look at what this configures, because it's basically configuring or setting the registries.comp file on the node. So if you have, for example, a caching registry, you can point everything by default to all go through that caching registry and then it will take care of that, you know, you can offload a lot of that bandwidth. Okay, so dry run, really what we care about here is these image content sources, stanzas. So what does art dev stand for? Honestly, I have no idea. I don't, artistic developments, and I would have to ask the engineering folks. It's interesting that you also noticed it's v4.0, which there never was a public v4.0. So yeah, the naming schemes that they use are a mystery to me. So with the dry run done, we want to make sure to record these values. So remember if I put it in here or not. Anyways, I can pull it later if we happen to need it because what I want to move on to is the mirror. Much like before, I followed the documentation exactly. So just like you would copy and paste this or type it in on the command line from the documentation, I did the same thing, right? Open shift release version, where I want to download these images to. So in my case, I have mounted an NFS exports to this location that NFS exports doesn't need to be super high performance or anything. Obviously, if you are on a physically air gapped network, this would be something like a thumb drive or a USB hard drive, USB SSD, whatever that happens to be. So you would want to download all of that information into some location. You'll also notice that this command, this OC Adam release mirror command is different from the dry run because we're not doing that too, right? There's no dash dash to or dash dash to repository, whatever it was on the other one. Instead, it's just to directory. And then we specify that path slash images and where to pull it from. So what I'm going to do is move my images directory to something that I know. And then we'll kick off that mirror. And I don't know if we'll let it run the whole time here because my internet is, well, I have gigabit internet, it still takes about 10 minutes or so. And I don't think anybody wants to just stared us while we stared a screen. So I'll probably interrupt it here in a little bit after it gets going. So this is going to download every one of those images. And they're going to land in the directory that we specified here. So remember, it was the directory that I'm in slash images. So we'll give it a second, just like we did with the dry run, it's doing all of its metadata thing, finding out all the information about it. And then it'll begin that pull operation. There we go. Let's make it a little bit wider so we can see it. So you can see it's uploading is a bit of a misnomer here. It's uploading it to the file system. And then this is all of our container images. You can see all the sizes out here to the side as it goes through and downloads all of those. So, oh, here's the size. It outputs it at the end, 9.412 gigabytes. So at the end of all of this, if we just let this run, it'll download all of those. We'll end up with some stuff on the file system. So let's see if I can interrupt that. I can. So remove that directory we just created because I did this yesterday. Look inside of here. You see that this is effectively a very similar layout that you would find with a registry. So if I do a tree inside of here, all the way back up to the top, we've got all of these different files, folders, right? These are links, symbolic links, right? All of the information that we have for each one of those container images. And let me do a quick view dash, sh, sh in here. It comes out to our 9.5 gigabytes in size that we would expect. So now that I have downloaded all of these images, we also want to grab a couple of other things that we're going to need for the destination side. So in my instance, I have downloaded these things into this other directory. So basically, in my hypothetical situation here, everything in this mirror directory that I'm currently in. So this mirror directory, this is all of the data that I would expect to move over to the disconnected network. So I also did this for 4.8. We're using 4.9 today. But the other one, the other directory has some kind of tools and utilities that we need access to. So one is Cincinnati. We'll talk about that in just a moment. So I'm going to do a single node open shift deployment. So this is going to be a disconnected CLI static IP single node open shift deployment. So if that's something that's interesting to you, please stick around for a few more minutes because it's going to be interesting. So inside of here, you can see I downloaded a couple of things. So one I need, so podman images. So I simply downloaded the Docker registry container image, right? Just a simple podman poll docker.io slash library slash registry. You can tell my Southern coming out there library. Version two of that particular image. And then I did a, so it's a podman save. And I think it's dash oh, where's my it is. Yep. Yep. Dash oh, so dash oh, and it's some kind of name tar. And then we just do like this version two. Oops. Yeah. And you can see it does exactly as you would expect it outputs our name.tar file. You can see that that and this match size. So we're going to need that because well, we need a registry on the disconnected network. So the good news here is or the bad news is, yeah, it's kind of a pain, right? I have to set up a registry on the disconnected network. You know, maybe I have a third party something, you know, maybe I'm using, well, Red Hat Quay is one, of course. Maybe I'm using, why is the name of our Nexus? Yeah, Nexus, JFrog. JFrog. Yeah. So there's a whole bunch of them out there. I just pulled the Docker registry because it works great. It's pretty simple and easy and straightforward to get up and running. So we'll need that on our disconnected network. Aside from that's the other things that are in here. So I've got OpenShift install and the OpenShift client. It is technically possible to extract those from the container images, but honestly, if we're already mirroring content from connected to disconnected, it's easier to just pull them. I went to the website, you know, just access or console.redhat.com, went through the normal download process to get those. So OpenShift install, OC, kubectl or kubectl, all as a part of those packages. Our registry container image. And then the last two that I've got here is our CoreOS install ISO. So remember, we do need CoreOS. We have to have something to install. So I grabbed the ISO here because I'm doing single node OpenShift. We need to use the ISO for that. If you're doing VMware install, grab the OVA. If you're doing a OpenStack or Rev install, grab the KVM, the QCOW2. Whatever your platform is, grab the appropriate image for that. Isn't 4.9 installer going to have a built-in registry? So no. So the installer won't have a built-in registry. So we were talking, Johnny and I were talking with the product management team about this. So effectively what they're working on, and Johnny posted the link earlier, is a way to, with a single command on a rel host, deploy the quay registry and then pull all of that data into it. So that way you have a very quick and very easy way to do everything that I've shown you to this point. So all of this dry run, mirror, create, stand up a registry, copy all the data in. All of that stuff will be dramatically simplified. So I don't have a precise date of when that's available other than somewhere after the 4.9 GA, which is now, and before 4.10, which is sometime in Q1. So the installer won't have an integrated registry, but we will have the ability to use a dramatically simplified way to stand up a registry and then mirror all of this data in. So hopefully that answered your question. Okay. So got my platform install ISO here. I've got my clients here. I've got my registry pod here. I've got all of my image data that has been made available. So let's take all that data. Let's mirror it across. So remember I said I'm going to cheat. So I did cheat and I have the same data that's mounted underneath Mount here on my host and it is mounted as disconnected because I'm super creative that way. So if we look inside of disconnected, you can see it is the exact same file folder structure that we had on the other one. If I go into other, for example, you can see that's this November 10th at 1148 Eastern named out TAR is the thing that we just created. So literally just pretend like I mirrored this over to a disconnected network. And that is what it is. So before we can do the next step, we need to have a registry available to us. So like I said before, I use the Docker registry and I created a very simple instance of this. You do a pod man load and that imports that image exactly as it is. So if I do pod man images, you can see I have my Docker.io slash library slash registry. You can see the image ID is going to be exactly the same. So that is our registry image. From there, what I did was I generated some certificates. I used the CA that I have running inside of my lab to create these certificates. So it's just a server certificate. It does have the sand, the server additional name. Is that right? Yeah, I think that's right. Subject alt name. Subject alt name. Thank you. So I did add sands for other host names. So you can see this one is name bastion, but it also resolves to registry. It also has an IP in there, all that other stuff. The reason for that is because, and you can create just a traditional common name certificate, you just get an error when using OC against it. So if you look in the docs, there's a go environmental parameter that you set, and it'll ignore those errors. Otherwise, OC will come back or any other go command that you run against it will come back and complain saying that, hey, you need to update your certificate to use sands. So, and I will validate. So if I do a curl dash v. And if I can do a shameless plug real quick, if you're looking for an easy way to create a certificate, or I say easy, relatively easy way to create a certificate with the sand, look at Ansible with the open SSL modules, because then you can actually, you can add the constraints that you need for like the CA equals true. And then you can add a common name with all of your subject alt names, if you want to do that as well. And it's really simple, you just add, you know, a vars file with all the things that you want, and then loop over it. So it makes it makes it pretty handy to create a certificate pretty quick. Yeah, I, I just used a GUI in my in my router that has that I cheated. So all I'm going to do here is just to kind of prove that, hey, this, this Docker registry instance is up and running with a certificate, right, HTTPS port 5000, she's doing a very simple curl. I did not include any sort of authentication on this registry. Of course, you know, depending on your registry, whatever third party one you're using, whatever your security policy is, that's probably a good idea. I just did not. And you can see that when I query that, it goes through and it finds all of the certificate stuff, there's no complaints or anything like that. And then it spits out all of the repositories that we have inside of there. So this is my just quick and easy, you know, CLI based way of saying, Hey, my registry is up, it's running, it's going to respond exactly as I would expect the alternative being. And if you were watching there, it's all that I had re tagged my registry image with that registry instance. And then I just did a push of that. And it pushed successfully. So that's kind of the one it's responding as we would expect into it functions as we would expect when we're doing things like pushes. So, okay, registry up and running in my in the doc that I'll push before the blog post goes out. I have some steps on how to do that as well. So pretty straightforward, you know, creating those, those certificates. So now I need to go now that I have all of this data here, now that I have everything that I need, I need to actually import all of those mirrored images, all the data that I downloaded in the previous step, I need to import it into this particular registry instance. Go to mounts and this connected. And I'm actually going to use this with 4.8, because I don't want it to interrupt what we're going to do later in case it does a partial upload and corrupts anything like that, because I'm going to interrupt it as well because this process while downloading it takes me about 10 to 15 minutes. Uploading it takes closer to 30 minutes because the storage that I'm using for this is not very fast. But the only difference between these two is just literally the version number that we're doing. So let's do import. So as before, we have this OCP release. I just changed this from 4.9.4 to 4.8.18. You can see here, same sort of thing. I'm just setting the path correctly for where my data was mirrored to or copied to from the connected network. I'm going to import it into our bastion registry instance. And you can see the repository that I'm going to use here. So you'll notice that inside of here, this is the only time that we're using that local registry local repository information. Again, if you use something different when doing the dry run, that's fine. It has no meaning at this point. So just make sure that here on the disconnected network, these values are set correctly. So let's go ahead and do our import of that data. And oh, one other thing that I wanted to highlight here, notice that this format is pretty specific. So we want to make sure that it points to the same location, right, this base location that it mirrored all of that data to. So over here, so remember, this is now back on our connected network. Let's do our mirror. So in our mirror, our removal path is this. So mirror slash 4.8. And then down here, because I copied this from the docs, and this is the way they did it, they appended slash images, moving back over to disconnected, we have 4.8 slash images. So make sure you use that path that will have the, it has the v2 folder inside of it. And then it will import based off of that. Okay. So let's run our import. Like before, it's going to look at that image that is on the system. It will basically determine, you know, hey, what's all the stuff that I need to import. And then it's going to go through and do exactly that. So let's expand this a little bit. Hopefully it doesn't mess up again. Oh, yeah, all the data has been imported. So it finished very quickly. So normally this would go through and it would be literally uploading all of the data into our registry. Again, it takes like 30 minutes on my system, just due to storage performance, not for any other reason, just it's using spinning this on the background. Not just any spinning this 5400 RPM spinning this. So now all of our data is accessible from a registry on the disconnected side. There is a way that I don't remember off the top of my head, you can use the CLI to search for essentially this repository, anything that's a part of this repository using curl from the Docker registry. I don't remember it off the top of my head. So I won't, I won't cover that. But if you needed validation of everything being there, that is one way to get it. Okay, so at this point, I am effectively ready to do an install, right? I copied over. So I copied over my clients and install. So OC and OpenShift install. So copy that data over just like any other time. So we can do a have those in here. Do a CP of disconnected other star.gz to here. So I'm just going to copy those locally. Do our tar XCF these to unpack those tools. So out of my directory, I now have kubectl OC and OpenShift install. So move those into the path. Use your local bin here. And then all that's left to do is remove the read me and the gzip the and I'm typing what I'm saying, and the gzip files. Now we're back to where we started. But now I have OpenShift install version 4.9.0. Oh, I did forget. I'll talk about it more in depth in just a moment. I don't have it in these notes because I referenced my other one for doing single note OpenShift from the command line. But we'll also need CoroS install. And there's one other binary that we need. What is the other binary that we need? Anyways, there is one other that we need as well. OpenShift install or CoroS install being the major one that we need to add that bootstrap or excuse me that ignition file into the installer ISO. Okay. Got my CLI tools. Got my data. Let's make a cluster. Up our previous try. We'll make your cluster. And I want to edit my install configure. All right. So there's a couple of interesting things that we're going to see with this particular install config. So first, a couple of things to note. My base domain here, lab.lan, this is my disconnected environment. You can see that the network that we're going to be using here is 10.0.101 slash 24. So this is just the IP address assigned to my bastion node, which is also my registry instance. So I can remember when we did that dry run. Remember it was like, I think it was registry.lab.lan. You can change these after the fact to whatever the mirror actually is. So again, if you don't want to or cannot expose those names on the connected network doing that dry run, no harm done in doing that. So this additional trust bundle is important. So remember I created a custom certificate on my disconnected network for my registry instance. So this is the CA bundle for that certificate. So I need to add this in so that way when it connects over to my registry, excuse me, this, when it connects over to that registry, it won't throw an SSL error. And this automatically gets added to the cluster at deployment time as well. So it'll automatically propagate out to all the hosts and everything else. I don't have to do anything else for that. So that is an important one. I think there is a way to configure insecure registries at this point. I just, I don't ever do that. So I would have to refer to the documentation. Johnny, maybe you know off the top of your head. Yeah, I've actually only done it this way. I mean, when we talk to our customers, we essentially kind of we push them to do as secure as possible. One thing to say about the cert though is that when you're doing a self sign certificate, I kind of alluded to it earlier, but you have to have the CA constraint within that certificate that says it's essentially a CA. Because if you don't have it, then the OpenShift will, the OpenShift install will essentially, it'll create it, but it won't use it. Got it. So yeah, that was actually that answers a question that I had, which was I kept encountering an issue when I was trying to do a certificate with a SAN and that was self signed. So good to know. Yeah, it's I learned it the hard way. It was painful, but yeah, it's definitely you have to have that CA constraint or the CA true constraint. Okay. So after that kind of moving down, this is a standard install config. So we've got our networking here. You can see I'm going to use OVN Kubernetes. I'll accept the defaults for service network for cluster network. It's only a single node. So I'm not overly worried about it. I do set my machine network correctly here. Down below, remember this is single node OpenShift. So we have no workers and only a single master or a control plane node. Again, single node means that we use platform none. So I was corrected by the way. So single node OpenShift today is only fully supported with physical servers. So you can, it does technically deploy and virtual machines. You're getting ready to see me do it. It is not fully supported when doing however. So that's just from what I understand it's a QA QE thing. But for now, single node OpenShift only fully supported with physical servers. Networking type is OVN is that our requirements. It is not a requirement. So you can use S or OpenShift SDN if you would like. I'm trying to, so we changed the default to OVN Kubernetes with 4.9. So I'm trying to, you know, I'm trying to be brave. I'm trying to try something new, but I get it if you want to stick with what you know and what's comfortable, familiar and trusted. So one thing to note here, my pull secret, this is obviously not a real pull secret. I don't need one, right? Remember how I said that I didn't configure authentication on my registry. So therefore, I just need to meet the format requirements, the bare minimum format requirements for OpenShift install to pass the check. I actually stole this from OKD. If you look at the OKD docs, they tell you to do the same thing. If your registry instance does have authentication, you would just provide that authentication inside of here, just like you would with your regular internet connected pull secret. So the one fancy thing to note here is that I have this little copied network parameter here. So remember I said that we're going to do a static IP install of single note OpenShift. And the last time when we talked about single note OpenShift, I kind of hand waved over this a little bit and said, oh yeah, it's possible to do a static IP install, but I didn't go into any detail. So it's a two step process, or we'll call it 1.1 steps. So the first step is, and the point one is to add this parameter here to installation disk. And the magic that happens here is effectively in the logic of what's going to happen at the bootstrap level, it will append this string to the end of the CoroS install command. So in our instance, we want it to copy the network, the static IP config, into the installed CoroS instance. So that's the point one. The full step is just like you would with any other CoroS install where you're setting the static IP. I want to interrupt that boot prompt. I want to hit the E button and then append the IP equals blah, blah, blah, blah, for my note. So we'll look at all of that in just a moment when I go to actually boot our virtual machine here. So that's our install config. So pretty straightforward. I'll scroll back up here just to review. Additional trust bundle is important. That is what gives or allows it to trust the registry instance that we're using, our image content sources, which again, if we come back over here, this guy, and hopefully if I scroll up, nope. Anyways, if I do that dry run again, that is the stanza that gets output at the end of that. I just copied that over, pasted it in here, maybe copy paste that, save that as a text file in the content that gets mirrored across. I modified it to be actual reality on my destination network. So again, I set the host name, the actual destination registry correctly and we're ready to go from here. Let's exit out of there into our working directory, copy the install config. So now I'm just inside of a sub directory. Remember, OpenShift install consumes the install config. So I want to preserve it for future mistakes. So let's move forward. So one thing to note, let me open this. So at this point, I'm more or less going to be following the guide or the set of steps that I created back when we did the single note OpenShift stream. So right now, I just completed what is effectively step two in that link that I just posted into chat, the GitHub gist. I'm getting ready to do step three, which is the OpenShift install create. So we'll do OpenShift install create single node ignition config. And I don't need to specify a directory because I wanted to use the one that I'm in. It does its thing. And at the end of all of that, we have our ignition file. And in the auth directory, the expected Kube admin password and kube config file. So the next step, if I'm looking at that link that I just referenced, in my apologies, I would have both of these up and I'd switch back and forth between them. But when we tried to do that whole full screen monitoring, full screen sharing thing, it borked. So I'm going to try and rearrange my desks that way next time we do one of these, I can share the whole screen and it's just easier. Anyways, so the next step is to take this bootstrap file and embedded into the ISO we're going to boot our single note OpenShift from. So remember, I copied, I moved that core OS ISO with the other data that was mirrored. So oops. Inside of this other directory, I have my core OS ISO. So remember that the core OS install files, you see this is 4.9.0, they don't have to match the version of core OS of OpenShift that I'm installing. It will automatically adjust, it will automatically change or update core OS if it needs to be. So they only re-spin these ISOs periodically, basically if there's been a major change or something inside of there. So like if you look back at 4.8, I think there was like 4.8.2 and then 4.8.14 are the two core OS versions you can download. That doesn't mean that there wasn't any changes to core OS between there. It just means that we didn't see or there was no reason for them to recreate the installer. So let's copy that into our working directory here. That'll take just a second to copy the gigabyte or so of data across. Is there a dock link for that install on disk? Unfortunately, no. So as it stands right now, the documentation only shows single node OpenShift with assisted installer, unless this has changed in the last few days. I don't think it has, but it could have. So the problem with that is assisted installer itself is a tech preview feature. So using assisted installer, great. It works in my experience. It works well, but just be aware tech preview means that AI itself, assisted installer itself, is not supported. The resulting cluster is, we don't really care how you got the cluster installed. If it's there, it's supported. But assisted installer also is not available disconnected yet. So you can absolutely, it's very easy if you've seen us do it before. You can go to assisted installer on console.redhead.com. You click the button that says install single node OpenShift, generate ISO, blah, blah, blah, blah, blah. Unfortunately, that is for disconnected. That's not an option. So all of this stuff that I'm doing comes in large part from Ben Schmouse. So he worked with the engineering folks. So at the top of that GitHub gist that I plugged into the chat there is a link to Ben's blog that I used for all of this. So, and he also documents how to do a disconnected install. I think I modified his blog post to be connected. So, okay, our copy command finished here. If I look inside of here, we've got this CoroS ISO. So now I want to do a CoroOS installer. So remember I said that there was another binary that we would need CoroS installer. So this one was downloaded just like all of the others from the OpenShift mirrors. So there's also a way that you can extract it from the release in the documentation. Johnny, I don't know if you happen to have that link at hand. But it's available through the normal mechanisms. I didn't do anything fancy to get that. mirror.openshift.com. I'll find it. So I'm going to use CoroS installer. And what I want to do is do the ISO ignition embed. And I want to embed our bootstrap ignition file. And I want to embed it into our CoroS ISO. So as Ben points out in his blog post and something, he reminded me of a command that I haven't used in about a decade, which is, you know, there's no output from this. I have no idea whether or not it succeeded or failed. So I can do an echo dollar question and it will output the status of the previous command. So the output's result was zero, which means that it exited successfully. So our bootstrap or, excuse me, our CoroS ISO here now has our ignition embedded into it. So notice that the file didn't change sizes. I actually asked them about this. That's expected. They do some magic inside of there that puts in like a dummy file or something like that. I don't know exactly what it was, but basically they said, yeah, that's the way it should be. So at this point, I have my installation ISO. I have the ignition file for my single node open shifts in that ignition or excuse me, in that CoroS install. Now I need to take that ISO. I need to attach it to the host that I'm going to install to. So as I said before, we would expect that fully supported. You're doing this with bare metal. So burn it on a physical CD. I don't, is that still a thing? Put it on a USB drive, you know, boot through IPMI or Redfish or whatever it happens to be. Use the DRAC or whatever your server management software is to mount that ISO and boot it. In my case, I'm going to use a virtual machine. I'm going to cheat a little bit since I'm working on a remote host. I can't exactly, you know, bring up the VMware GUI and upload the file. I've actually mounted an NFS data store also on this local host that I'm going to use for that. So that is mounted to Mount NFS blue. And what we want to do is move our CoroS install ISO over to this directory. Platt AOA disconnected ISO. So that'll take a couple of seconds for that to copy over. Oh, PC ladder box. Peter, thank you for posting Ben's blog. And it is a great blog post. Ben, which I think you all may have, may remember him from the single note open shifts or the assisted installer stream that we did. Ben's the guy that I go to with all the crazy esoteric questions. Like, I wonder if anybody has ever tried this before and Ben almost always has. So while that's finishing copying, I'm going to change the window that I am sharing here because now I need a browser window. So we'll go back to what I hope is this one. Yep. All right. So if you're a quick guide before you saw my vSphere host. So this is just a standalone ESXi host. I'm not doing anything fancy here. You can of course use vCenter. I just don't have vCenter running at the moment because it uses precious, precious resources that I just don't have to spare. Is this replacing the part that you insert the vApp properties in the isotemplate? Yes and no. So normally when you do a VMware install with the OVA, it will use a VM property. So it used to be a vApp property. We changed it to a VM property due to size. So the vApp property was limited to, I think it was 80 kilobytes or something like that. And the boot strap ignition specifically is way more than that. Whereas the VM property has a much larger size limit. So it does replace that, how we get the ignition into the boot process or into the CoreOS because with the ISO install, basically not the OVA, there's no tools. There's nothing that allows it to reach up into that vSphere layer and look at the VM properties to ingest that information. So we have to get it somehow. This is one option. Another option would be to just like with any other install, you could do the kernel parameter. So I'll highlight that in just a moment. So let's look at our virtual machines here. I left it running. That's convenient. So we'll turn this guy off. Let me go down in here just to make sure I don't accidentally power off my host, which would not be the first time I have done that either on stream or off stream. So I created a virtual machine here. So one thing to note, if you use assisted installer to do single note open shift, it's going to require you to use four or eight CPUs and 32 gigabytes of memory. If you do it from the CLI like this, it won't enforce those checks. So you can go, see I'm using six and 32. I've done it with four and 24 before. I generally don't recommend folks go too much below that because particularly for CPU, it starts extending the timeline for install. Other than that, it's pretty straightforward install, 240 gigabyte disk. Let me see if I can, I'm going to blow this up just a little bit to make it easier for folks to read. It's connected to the same network that 10.0.101 network. And I'm going to attach our CD device. So let's come in here to our CD drive. I want to connect to a data store ISO file. I want to choose the one I want. So our NFS blue data store, and we want our AOA disconnected ISO. So this other one in here, that was from earlier when I was testing. So we can just ignore it for now. We want to make sure that our connected power on and our connector selected here. We'll hit save and we can power on. So for this, I capture my, so I set a boot delay inside of my virtual machine. So I'm going to zoom this back out since it's a little bit big. So I set a boot delay in here so that I can click in and hit escape. This is so that I can reuse the virtual machines. Normally, if it's a brand new VM, right, never been loaded or anything, it'll automatically boot to the CD drive first. So that's why you're seeing this because I don't constantly destroy and recreate the VMs. I reuse them. So I'm going to have the boot to our CD ROM drive, which remember has the ISO for the core OS install that we added the bootstrap in place ignition file to. And here is where we want to capture it. So before that screen times out, you want to hit E. And it's going to drop you into this prompt. And if you've ever done, you know, a traditional open shift install before, you're familiar with the kernel parameters that we can configure here to do things like set static IP or tell it where the bootstrap file is. So that's exactly what we're going to do here. So I'm going to go down to the make note of the Linux line here, not the in it, our D line, the Linux line, we're going to go all the way out to the end and we're going to set our IP equals one. And because I can never remember the format, I got to cheat off my notes here. So IP equals the first field is going to be the IP address of the node double colon. And then we want the IP address of the gateway. Single colon, we want the net mask. Single colon, the host name. And single colon, the interface to use. So this is, it's a VM, obviously. So NS 192 is the first nick. If you're using a physical host, you know, you probably have, you know, four, eight, 12 interfaces. One thing I found is you can boot to the live ISO. So not the one that we have here with the bootstrap added or the ignition file added, but just the regular live ISO. If you just boot to that, it'll drop you at a command prompt and you can do like a nmc li con show or IP show IP, a IP space. Yep. Yep. IPA. And it will show you those interface names and you can kind of map back from there, right? It's, it's a regular Linux environment at that point. That I found is kind of the quickest and easiest way, particularly because if you want to do everything in one step, you can boot into that live environment. You can configure the network and then you can do the core OS install dash dash copy network. If you want to do everything in an automated fashion like, Hey, I'm setting up for pixie or I pixie. You just look at those and then set it correctly in your, your I pixie file or your pixie file. And then so last field here colon none space name server equals use my name server, my DNS server that has all of the things that I need. And now just as the directions at the bottom say we hit control X to start the boot process. So at this point, we, we basically sit back and relax and we let this thing go through and do its thing. So effectively what's going to happen is just like with a regular open shift install, and you notice here that it utilize the IP address we configured 101.60 along with our host name, SNO dash node. So it's going to go through doing normal bootstrap. So I'm going to switch back over to our CLI and because I want to show you that I can SSH over to it. And I connect as core just use it, expect I'm going to do my journal control here. And it's going to go through and do its thing right there starting cluster bootstrap waiting up to 20 minutes for the Kubernetes API to get started. If we were to go over to, I'll open up a new tab. So if we were to go back to our bastion host here, do a logs on our registry, what we should see is as it goes through, it's going to be pulling in all of those container images. So if we sit here and watch this, we'll see all those container images get pulled as it does its thing. The other thing we can look at is so I can go into my cluster. So this is where I was running the CoreOS install where I did open shift install, create single node, blah, blah, blah. I can do my open shift install, wait for bootstrap completes. It's like always. And you see it comes back exactly as expected. I'll also take a moment here to point out. And again, this is in the blog posts, or excuse me, the GitHub gist that we linked earlier. You do need those four DNS entries. So this is in step one of the gist. So you need the node. So sno-node as we saw. Where did that go? So sno-node to be there. We need the api.clustername, api-int.clustername and star.apps.clustername. So those three DNS entries need to be there, or four DNS entries need to be there. In the case of single node open shift, they all need to point to the same place. Because it's one node. So there is no virtual IP addresses. There is no load balancer. It's a single node. You can't load balance with a single node. So they're bootcube.service succeeded. See, that was pretty quick and painless. Here in a minute, we'll see that complete. And what I'm going to expect here is this will probably drop my connection. So before it does that, we'll exit out. And then if I were to switch back over, I would see that my virtual machine will eventually go through a reboot process. So while we're waiting on this single node open shift cluster to finish provisioning or finish installing itself, I wanted to talk about mirroring operators. So let's come back over to this guy. Where am I at? And just real quick, Andrew, like the thing that you're just doing, whether you're following the journal CTL along the bootcube service, that's a good way to determine very quickly if your registry and comms between registry and your cluster are good to go. Because what you'll see on a disconnected side is when you go and you run in that bootcube, if it can't talk to the registry, if it can't pull, then you'll see where it's kind of misleading where it says, hey, pull in from quay.io, the image name, and then the hash or the shaw. It'll say like, try in this mirror, which is going to be your registry. And then it's going to say, I failed. And then it'll try and kick out to like actual quay. So just something to keep in mind when you're troubleshooting or debugging or what, you know, like just, you need that warm fuzzy that things are going well. Go and check out that service and you'll know right away. Yep. Yeah. And you'll notice as I was scrolling up, you probably saw a bunch of red lines go through and there's some, as we look through here, there's some failed, like your failed to create, blah, blah, blah. All of that is normal. What you're seeing here is the sausage being made with regard to the bootstrap process and the install process. So don't be alarmed with the stuff. It's one of those, it's normal because like particularly with up here, these failed to create, basically we're waiting for the, the Kubernetes API to understand what are these things? What are these objects? Like you see, you know, it's saying the namespace open shift machine API doesn't exist yet. So we're waiting for machine API to be deployed, you know, the operator to be deployed to create its namespace to define the CRD so that it can then add all of these various objects inside of there. And after a few seconds, you notice that they eventually do succeed and it moves forward. So generally speaking, these things are, if they don't last for long, they're not to be alarmed about. If they last for a while, like to the point where your install fails, obviously, that's when you need to look at it. So you can see here the connection to the server API and was refused. That's because it's not up yet, like literally the pods hadn't started in order to accept those connections. And then once it did, it moved on, it succeeded. Okay, so while that's doing its thing, just like any other open shift deployment, it's going to take a few minutes. So interestingly, if you have done connected installs before, usually the vast majority of the time, unless you have just a really, really nice internet connection is just waiting for it to download those container images to each node, right? Every, every control plane host has to download a couple of gigabytes, every worker node has to download a couple of gigabytes and so on and so forth. And it can take a while just to download all of those. If you do this disconnected, then it's running at the speed of whatever your local network is, whatever your local storage is. So this is one reason why I will often recommend to folks like, Hey, if you're doing a lot of installs, a lot of deployments, doing it disconnected for the install, and then to reconnect a cluster, all you have to do is delete the image content source policy. So after deployment, hey, it deployed it instead of taking 30 minutes or 40 minutes or whatever, it only took 20 minutes, go in, delete that ICSP, and it's a regular cluster again. So as well as saving all of that bandwidth. I know, particularly for some home labbers, you know, some ISPs limit your download bandwidth. So something to be cognizant of. So all right, that's doing its thing. We're going to ignore it for a little bit because I want to talk about mirroring operators. So I'm going to switch back to the browser briefly just to look at the documentation that's associated with this and share that and bring up this guy. All right, so that's doing its thing. What we want to do now is move from here to our, if we scroll down here, there is, where is it? I can never find these things when I want to. Excuse me. If I just search for disconnected OLM, it should come up using OLM on restricted networks. There underneath operators. I was looking for operator lifecycle manager. So this page walks through all of the steps that we're getting ready to do on the command line, starting with pruning this SQLite based index image and going through and then looking at, you know, which ones that we want to do. I will give you a hint and this is something that I do all of the time right now. I don't have a cluster deployed and honestly doing this whole pod man run, blah, blah, blah, going out and getting gRPC curl or gRPC URL. That's just, it's a lot of work. Andrew's lazy. So if you go to learn.openshift.com and if we scroll down here to open shift playgrounds and I select 4.7, you notice I haven't logged in anything by the way. You know, it's all of like 15 seconds and I'm getting ready to get dropped into a running OpenShift 4.7 cluster. So what I'm really looking for here is the name that is used internally to OLM for our operators. So let's go to, if I go to the console here and wait for the console to come up. Come on. There you go. Developer. No, thank you. I don't need a tour. Oops. I want the admin view. And well, I don't have cluster admin privileges, but if I did, right, I would be presented with the list of operators. I'm going to assume that we've all seen it. So each one of those is represented as, and it is, it is a package manifest inside of OpenShift. So if I do an OC get package manifest all namespaces, you see I have all of these things that come up inside of here. So for example, if I wanted to mirror, if I wanted to only mirror the MTV, the migration toolkit for virtualization operator, then I just need to know this name, right? If I wanted to do Kita, I have no idea what Kita is, right? If whatever these happen to be, you basically have to map those over. If you have a connected cluster, it's, oh, I don't know which one I want. So it took me a minute, for example, the other day, to figure out that the OpenShift update service is the Cincinnati operator. So if you can't figure out what it is, if you have it on a connected cluster, if you have it deployed, you can just go in and you can do an OC get subscription. And that will print out what each one of those that you have deployed is. And that's an easy way to find the background name here. So I want this Cincinnati operator. I want to prune down my index image to only be the Cincinnati operator. So that way I'm not having to copy all of the red hat marketplace or all of the red hats operators, certified operators. So let's dispose of our temporary cluster here. So that's what this process is describing. So the other way to get that is what they describe here, which is, you know, pull the operator index, use GRPC curl to get the list of them, and then extract the one that you want. And then you use that name or names with this opm index prune command. So let's come to, so this is my connected node again. I'm going to let's go into other working directory for us here. But what I need to do now is basically follow exactly with that. And I realized that I am, oops, I keep clicking the wrong button. So I realized I'm now we're sharing the wrong window. So let's switch over to our CLI here. All I've done is switch back to my connected host. I'm now in just a working directory. So remember, this is the set of data that will get mirrored across, excuse me. And I'm going to do exactly what that command that we just saw in the docs was, right? So opm index prune. So I copied and pasted from my cheat notes over here. So opm index prune, the source is our red hat operator index for version 4.9. I only want the Cincinnati operator. And I'm going to deposit this. So this modified operator image, I'm going to tag it with this particular flag. So I'm actually going to do a 4.9.5 tag here because I already have the 4.9. So this is actually going to fail. This is an error that Andrew hasn't checked out yet or hasn't figured out yet. It's going to fail because I'm in an NFS directory. So if I just go to my home directory and repeat the same exact command, it will go through and it will succeed. I'm assuming that's something to do with how Podman builds layers and how NFS doesn't agree with it and something like that. I haven't dug into it. I just know that I have to use an actual local file system, block file system for this. All right. So now if I do a podman images, and if I look somewhere up and through here, what did I call that? Cincinnati version 4.9.5. So this is the one that we just created. So the reason I specifically tagged this with quay is because this image does need to be pushed to an actual registry for the next step. I don't have one deployed on my connected network. So I just use quay and I push it up there and use quay to do that. So it is a public repository. So if you happen to browse to quay.io. Slash A and solo, you'll see a Cincinnati repository there with two tags of 4.9 and a 4.8. So let's see what happens when we do this with this 4.9 tag. So basically they're effectively the same. I just added the .5 for illustrative purposes here. So imagine that I have pushed this up to quay itself. And then the next step is going to be to actually mirror all of that content. So let's go back to our old directory here. So this is our, again, this is in our removable media, our working directory for all of this mirror content. And I highlighted something. Thank you so helpfully, Mac OS for copying that for me. Come on. There we go. So again, OCADM catalog mirror. This time I need to provide my pull secret. So this is the same pull secret that I used before, right? Just as if I were to go to console.redhat.com, pull down all my pull secret, it needs to be able to authenticate to the Red Hat Registries to pull down content for OpenShift. This is the source operator index file that we're going to use or image that we're going to use. And then last but not least is the destination that we're going to put all of the data at. So you notice that this is going to put it into the local file system that I'm currently on. And it will go from there. So if we hit return here, it'll take a moment to do its thing. So you can see it pulls that image. Again, I don't know why it can't use or doesn't use the cached one that I have on my host. It wants it to be in a registry for the reasons that it wants it to be in a registry. So it'll use that. It'll build up the index of images that it's getting ready to download. After about 30 seconds here, it will spit out literally a couple of thousands, a thousand lines of text. Like too much. It'll scroll off of the backlog here in my browser window. So we'll let that go through. And anyways, at the end result of all of this is it's going to pull down all of those images. I'll let it get started before I cancel it. Because just like before, I've already done this. Do note that this is, it mirrors a lot of data here. So the OpenShift install itself for 4.9 is about nine and a half gigabytes. Just doing this with the Cincinnati operator is 14 gigabytes. If I do OpenShift logging, it's like 18 gigabytes. So it pulls down a lot of data. And there's multiple reasons for that. The most non-obvious one in my opinion is it includes previous versions of the operator. So that if you wanted to, maybe the current version of the operator is six, maybe six doesn't work for you or whatever, you can deploy version 5 or version 4. And it has all of the relevant images for those as well. There is a way, and I think it's in the documentation, there is a way where you can tell it to specifically only pull one version. I don't remember how to do that. I didn't test that. So I'm not going to show that. But I think you can reduce this content pretty significantly if you try. They're planning took a minute and a half, but it's doing exactly as it was before, right? It's pulling down all those images. It's storing them into my local file system. And if I hit control C to exit out of this, hit control C to exit out of this, there we go. You can see we now have this v2 directory inside of here. And that v2 directory is what gets mirrored over. We don't need this manifest. It is helpful if we look inside of there. It has a, oh, it didn't finish. So we'll look at the other one. If I look inside of that, you can see the same name, just a different iteration. I have this mapping.txt. This is a fairly helpful file because it maps which source image and where it landed on disk. And you can use this with the ocmir command. You can use this with the scopio command. I'm probably not pronouncing that right. But so it is a convenient file to have to use, you know, for all of those things if you happen to need it. So I've mirrored all my content or I pulled down all of those images. I have now mirrored them over. So let's move over to our disconnected bastion node. I come into 4.9 and I go into operators. Yes, I know this isn't the other directory from before. I'm just using a different iteration. And we still have our Cincinnati inside of here. So you notice a couple of things about this. So this is all of our data. So remember I said it uses up some room. So if I let this run for a minute, it will eventually come back with 14 gigabytes. So now that we're on the disconnected side, I want to now import all of this data into the registry. And that works basically exactly like the previous command, right? When we did it for the open shift installation content. I'm not going to let that run. So let's paste this in here. So we have the same ocadm catalog mirror commands. This time we're telling it the source. So an interesting thing to note here. So remember on when we created this local slash index, it appended this A and solo of Cincinnati v 4.9. This is the name of the repository. Remember over here, I created my image and solo of Cincinnati. So I with a tag of v 4.9. That's where it's getting this information from. So where it mirrored the content to and then the tag associated with our catalog index and then where I wanted to send that data to. So if we let it do its thing, it will do exactly what you would expect. It's going to go through. It's going to take all of those images off of disk and shove them into our catalog or excuse me, our registry instance. And that's really all there is to it. At least as far as importing the content and the data. So I'm actually going to halt this because I've already done it again. That takes that one takes closer to 45 minutes on my infrastructure. Important thing here is and see if I created it here or not. It did not. So let's see if I can find means that it did do it. So we have this manifest index folder. So if I were to let that previous command run through completion, so it took all of the data that was in this V2 directory off disk and shoved it up into a registry. At the end of that, it's going to create this folder structure. And inside of here is another mapping file. So this time I sourced it from disk and pushed it up into the registry. But it also provides us a sample catalog source which we'll need in order to point OLM at our new or our mirrored operator index. Words are hard at this point. We've been we've been here for an hour and a half, right? So our mirrored operator index, you can see the spec points to exactly where it created it at. And the other thing that it's going to create is an ICSP. And this image content source policy is, as you would expect, it tells the cluster exactly how to find all of the images that it might need in order to deploy that particular operator inside of the cluster. So at this point, if our cluster is up, and I should probably check on that, it's hopefully done by now. If our cluster is up, then we would be able to add that catalog source, add that ICSP, and then we would have a new operator. Let's see. So I want to go to our cluster to export kube config. All I'm doing here is exporting the kube config location, so that way I can use an OC it's on our, oh, did I specify the right host report? Well, I thought I did. Did my install fail? It didn't go right. Let's see what happens if I SSH failed units. That's never a good sign. So something went wrong with my deployment here, so that's not good. With only 10 minutes left, I don't think we have time to go through and troubleshoot it. But that is a little bit disappointing. I'm going to do the by the standard Windows fix. That's right. When all else fills. So we'll let that reboot. I have slight hope that it might come back up. In reality, what would normally happen is, if I weren't talking about other things at the same time, you would do an open shift install wait for install completes, and you'd be able to get that output and see what all's going on inside of there. Since I can connect to it, I know that it was at least mostly successful with the deployment. So we'll see what happened here. If I can figure out what happened, I'll put it in the blog post just to point out what it is. So this is also the first time that I think that we've shown or in any way documented how to do the static IP stuff, the single note open shift. So I'll be sure to highlight that in the blog post as well, which reminds me, I don't think I published last week's blog post yet. So my apologies to anybody who's been waiting on that. I'll have to do a double deal. So I know I've been seeing chat out of the corner of my eye. Is there anything that we should address, Johnny? No, I think it's more around all the questions have really been about capability and stuff like that. There's been some questions about documentation, but I think we've got most of it under control. And we've just got a new question in about whenever we get the disconnected OLM. Something always goes wrong. Yeah, 100%. At least this, well, we haven't even gotten the disconnected OLM yet. It was just the disconnected single note open shift. I may have been a little over ambitious with the, instead of doing a regular old cluster install with something like vSphere IPI. So I'm relatively confident that the OLM side will work. Like I said, I've got a markdown document that I'll publish as a gist that has all of the steps that we've gone through here. So that way, you know, anybody who wants can go in, take a look at that, you know, would love comments, suggestions, et cetera on that as well. So we can maybe if it's popular, we can turn that into a blog post for cloud.redhat.com. Yeah. And something else to mention, too, that ICSP, when you're looking at that, you know, you see like the source and destination. If you go look in your, if you were to go to your open shift nodes, you could actually see that in your Etsy registries, registries.com, or Etsy containers, register, whatever the path is, I can't remember, but that mapping is actually there. So that way you can kind of see like, okay, I'm going to try and hit this, you know, if I go to quay.io slash whatever, actually hit my mirror registry first. And then if not, then back up and go to the upstream. So it kind of helps give you like a visual of what the cluster's actually doing. I don't know what happened there. I don't know if you can still see my screen, but I rebooted it and it came back with a different SSH key. That's strange and slightly alarming. So where is that file at? It's under... I think it's Etsy containers. Etsy containers. And then it'll be registries.com. Yeah. So if we do a cat on registries.com, very much to your point there, here's our registry mirror, right, going over to the appropriate locations. So in this applies regardless of whether you're using connected or disconnected. When you add those image content source policies, ICSPs, this is what it does. It goes into every node in the cluster. It does not require a reboot. It is not machine config. So it doesn't do a node reboot, but... And it adds these files in. So this is how it redirects all of those things at kind of the lowest level or the level closest to where the pods get or the images get pulled at. So I'm a little curious now of what happens if I am... Well, I am thoroughly confused as what has happened here. I'll have to do some troubleshooting here to find out what went on. Strange. So we've only got just a couple of minutes left, seven by my clock. So if anybody has any questions, if there's anything that we haven't addressed that we happen to miss, if there's anything you want to bring up, please, by all means, go ahead and do that. As always, you're welcome to reach out. I think Stephanie's been flashing our Twitter handles on there a few times. You're welcome to reach out on Twitter, Practical Andrew for me. Or you can reach out via email, Andrew.Sullivan at redhat.com. I don't mind that at all. I'll also note, so next week we have a stream. I haven't quite nailed down the topic yet. So we'll hopefully by the end of today we'll have that down. Stephanie's grimacing right now because she's dependent on me to get a lot of the organization stuff done. So I'm sorry. So next week we will be streaming two weeks from today. We will not. That is the Thanksgiving holiday here in the US. So yeah, so we won't be here that week. We will be streaming three weeks in December, the 1st, the 8th, and the 15th. And then we will go on holiday hiatus. So kind of the last two weeks of the year will be off the air. I don't know about the bigger Red Hat live streaming or OpenShip TV, but this stream will be off the air. We do have some interesting sessions or episodes planned for the rest of this year. So sandbox containers. I'm going to get together with the VMware guys again. Hopefully we can get an update from them on all the stuff that's going on. So tons of stuff that's going on all before the end of the year. I know some people, one of the guys on my team is taking off. He goes on PTO Thanksgiving and doesn't come back until the new year. And he's US based. So I know for some countries it's normal to get several weeks like that. But which I'm envious, by the way. So hopefully don't go on holiday mode just yet. We would love to have you join us for those streams. If not, they're of course available through whatever streaming platform, Twitch, YouTube, et cetera, as well as we'll have the blog posts. If anybody has any requests, again, this session itself was a request. So if you have any requests, suggestions, ideas, corrections, I love corrections, by the way. Andrew is, as my shirt says, a few records shy of a DNS zone. By all means, reach out to us. We love to get those. And making sure that we're able to interact with you all and answer your questions is really good. Our hope nine. So this is the two hour length. While I kind of like it, it gives us the ability to really dig in. Right now, this is just a one off. So we haven't broached the topic of going longer than that consistently yet. But it does feel good to me. So we'll talk on the back end. We'll see if we can find some other topics that make sense for the longer stream length, where we can go more in depth with those things. Sometimes we do have some, not to disparage product managers or things like that. But if we're not diving in deep and getting our hands dirty, sometimes it doesn't make sense to have the super long sessions. Loki would be a cool topic. Yes. So there's actually several things related to logging that I'd like to cover at some point. Not the least of which is the move towards Fluent D. And so Fluent D, Loki, all of that other stuff that's happening with the OpenShift logging service. So the observability stuff is it's becoming more and more important. And as a result, we're getting additional PMs associated with it. So hopefully that means that we can get some more in depth information, both from product management as well as from engineering. So Johnny, I know you've been super busy in chat. So thank you. Is there any questions that you want to highlight anything that you think is important to bring up? No, I just, you know, just key in on some of like the stuff that you're talking about, you know, like if you're, you know, I think everything that you went through today is really good for any type of cluster install, right? Like mirroring the images, if you're, you know, checking the bootstrap logs, checking, you know, using curl to, to hitch a registry to make sure that your repos are there. Like that stuff is all like, it seems basic and simple, but that is literally like, I'd say a killer and like 80% of the installs that we do where it's just something silly. But just having those tools in your tool bag where, you know, to like, you run your curl to go check the authentication, can I log into my registry if I have it set up? Yeah, you know, use an ht password or using podman login to output the login credentials. So that way when you do have to use like a pool secret in your install config, you actually get the format and everything that you need. Just all you have to do is really remove the spaces. So just having all those tools available and knowing like, okay, where I'm at in this process, how do I check this? And yeah, I think it's, it's awesome. I think you hit on a bunch of like awesome big key points that we need to get out more. Yeah. And this has been a fun one for me too, because it's stuff that honestly, I rarely do like the catalog, the operator index mirroring. I semi regularly do a disconnected install, which started back when not to not to sound disparaging, but you remember there was a short period of time there where the red hat infrastructure was a little shaky. You remember Kway was going down and stuff like that. And it's one of those like, I gotta have a cluster. I need to deploy this. I'm getting ready to do this. So I started mirroring content. And I would do those do the install offline and then just remove the ICSP and reconnect it. And, you know, everything went went well from there. So I always find that that's an excellent way of also speeding up your installs. You know, especially if you're doing a lot of I hope nine OSSM Federation. Yes, I would love to learn more about that. OSSM has been on my backlog of things to learn more about basically since it was released. So I would love to get some experts on to talk about that. I'll see if I can find that find, find the right folks. So I won't dilly dally. I won't hold folks any longer. Thank you so much for joining us today. Really enjoyed doing the two hour stream with you, Johnny. Thank you so much for helping and chat and all of that. You know, I know that there was a bunch of stuff through there. I will work on those blog posts. My apologies for being a week behind essentially. So we'll get that blog post out from last week. We'll get the one out from this week that has a link to all of the resources we used here. If you have any questions, if there's anything that comes up, anything like that, please don't hesitate to reach out. We're happy to help as much as we can. And like I said at the beginning, if I don't know the answer or we don't know the answer, because Johnny knows more than I do, if we don't know the answer, we'll find the right people and get you connected with that. Yeah, 100%. Thank you so much, everybody. Have a great rest of your week. Have a great, if anybody's going on holiday anytime soon, have a great holiday and stay safe out there.