 Good morning. Good evening. Good afternoon. Wherever you're healing from welcome to another episode of the open shift Administrator office hours. I am Chris short executive producer of this thing. They call open shift TV I am joined today by two of my fellow Red Hatters the just incredibly brilliant Andrew Block and the Cuddly Kermudgen Andrew Sullivan. So we have two Andrews on the call So this will be fun and interesting and today we are talking about what mr. Sullivan. Yeah. So today I asked mr. Block to join us to talk about registries Specifically, what are the registry options inside of open shift? So as always, this is the open shift administrator office hours, which means that it is very much and ask me anything style of interaction So please don't hesitate to ask us questions at any point in time. We'll keep an eye on the chat We'll answer those as they come in But that being said registries are a topic for today So, mr. Block, if you don't mind, please introduce yourself. Who are you and what do you do as who is your daddy and what do you do? Well, I was afraid I wasn't gonna be able to be let in because normally I join on the developer side This is one of my first ones on the administrator side I do a lot of get-up stuff and we did some sessions with the developer of console customization This is one that's really focused on the administrator side of the house So thanks again Andrew for inviting me on Chris once again for Wonderful I know many talents. Yes, my name is Andy block. I'm in the same was architect of red hats I work customers primarily. So if you're a customer of red hats, you may have seen me Especially when it comes to the open-gift world. So I'm here to answer questions. Please lay them on it. Yeah, so a couple of as we kind of always do with this show a couple of follow-ups from previously so really the big one I had one of our Watchers one of our streamers reach out to me saying that they were having some issues with the hostname stuff So I know that there are a couple of open support cases. There are a couple of open bz's on how to resolve that So I did respond to that person But if you have any questions about that, please feel free to reach out. So Andy I saw you cock your head there. So last week we talked a little bit about and it was a result of a couple of Threads that we were having internally around How does the hostname for an IPI node get set? And the crux of this is if DHCP is handing out hostnames those override the hostname that OpenShift wants to set for the node So unfortunately the best option that we have right now is to just not let DHCP set right push hostnames Give the host a hostname And instead disable that feature of DHCP You might be able to work around that By SSHing to the node and doing something like a hostname control set right to set the hostname or even populating Etsy hostname But that of course is not scalable and not sustainable, you know long-term. It kind of defeats a lot of the purpose of IPI So the other thing that I wanted to follow up on is somebody last week also asked about the disconnected Side and they followed up with me via email. So yes, we did sort of abruptly we got to the point of doing the Disconnected OLM mirroring for Red Hat operators and then I cut that and jumped over to the update manager because I wanted to talk about the update service So I will follow up with a blog post on how specifically to do that So that content that we had planned of let me prune the Index image and then only mirror a subset of those images. So I'll follow up with a blog post I'll get that into the queue. I don't know when it'll actually get published But we'll get that that content out there for you. Thank you 4.6 made that a lot easier It used to be if you're on older versions, you might be having some challenges get up to open just 4.6 Easier. Yeah, the index images make it just way way better. Yeah Cool. All right, so registries What what what is a registry mr. Block? It's a location where you can store images because everything an open chip runs as a container. They got to come from somewhere I mean, you know, it is it is it would be very Kubernetes like for you to create something other than nothing But yes, they do have to you know come from somewhere because technically technically you don't need a registry Because you couldn't build an image on a node Tech the pod to that node and it just runs right I don't recommend that quite scalable and a number of other challenges But so you don't need a registry You probably will use a public registry like quay that IO Docker hub you name it or you might have like for most of my customers. They will have their own enterprise registry There's a lot will go dive into that in a minute But that is where I see a lot of questions regarding registries and where do my images come from Right. Yeah, so I think that's a good one, right of whether or not, you know, you're using a registry, you know as an administrator, it's not something we often think about because You know, we typically leave it to the developers right the application teams of hey, I'm producing code I'm producing application components. I need to build my container image I need to store it somewhere so that when I deploy the application, right? I can pull that image in and be able to use it But reality is we we do right every time you deploy open shifts. You're using a registry It pulls everything in open shift all of the open shift services components, etc. Are deployed as containers themselves So you you're absolutely taking advantage of that so Let's let's talk about the open shift registry itself so Open shift has a registry right and That is not the same as quay, right, so let's let's differentiate here There's three things really there's quay.io the public's hosted version of quay There's red hat quay, which you can go ahead and deploy on-premise inside your environment or there's as Andrew said the internal registry So what's the difference between those latter two in particular the two on-premises options? so the Open shift version is a very lightweight image registry But it does have a number of key functions that are really built for open shift one It has our back support by default already integrated to open chip that is crucial to this is concept and open chip called an image string Which is basically a virtual representation of where images are sourced from it can be sourced from the internal registry They can be sourced from external registry They're basically like I'm very used to image registries like a nexus where you can use a virtual group To represent a number of registries you just point at it and somehow it magically figures out where the image is So I'm gonna I'm gonna tangent on image streams for a moment. Yeah, so Where or who is it that's responsible for defining in image streams? So a lot of times in the mainstream if you're using your application. It's just a developer the developer is just gonna go We lost your lost audio for a second about that. Yeah, no worries Created creating the mainstream and when you complete in when you push to the registry that integration between the internal registry And the image stream is automatic. It will automatically create a tag on that image stream Okay, so image streams used by application teams right as I'm gonna say as a Constantly updated source for a particular image. Is that accurate? Basically, I say I want to use this image stream as my base image as my underlying image and Any time whatever the source of that image stream is gets updated the image stream is also updated And if I'm using builds as the application team It'll automatically trigger a new build and if I have the deployments or you know, the rollout scheduled and everything there What is very important here is the image stream will only get updated automatically if you are using the internal registry So if you have a mistream that points to an external registry, it will not be updated You can schedule that action or you can manually trigger that but that's a big thing that took a lot of my customers up A lot of users of OpenShift is I go ahead push an image to quit at IO Why doesn't it get updated on my F in my cluster? Got it. That's that's important information and I I didn't know that you can see I don't do a lot with image streams. Yeah So, okay to me as well. Yeah, so the internal registry So what we call the OpenShift registry and if you were to look at like if you were to do an OC get CEO right the one that shows up as registry is Is what we're talking about here, right? Yep So how does and and I'm kind of starting at the base here because I feel like this is one that's particularly if we're talking on-prem it gets overlooked a lot because Most of the time it's not deployed by default, right on-premises Actually a lot of times we do we do deploy it because we don't have to involve any external teams That's one of the benefits of OpenShift is that self-service capability You can just deploy an image build it and you name it if you have to integrate with an external registry You have to maybe get credential setup. You need to do passwords accounts There's a lot of additional setup that you need to do that slows things down There's a lot of organizations I do work with you still have to work with another team unfortunately DevOps hasn't fully Embraced every customer I've worked with so you have to kind of throw it over the wall wait for the service now Take it to come back and then you then get access to it But it as a it's a day to operation if I'm unless I'm doing IPI correct Yes, if you're doing IPI will then automatically stand that up if you're using UPI You have to hook up the registry after the fact and there are a number of different storage options So let's I'm gonna take advantage of of that segue. So let's talk about storage for the registry So if you look in the documentation, let me share my screen real quick here Where's the button there's the button and For those of you watching at home, you you notice that I'm in a different office today. Everything's slightly rearranged I'm also slightly discombobulated Yeah Well, that's fair. I guess, you know, it's my cameras over here instead of here Great different monitors for the for the camera to be at so it throws me off a little bit No, I get it having moved offices recently. I went through the same thing and it was kind of weird Yeah So I'm on the documentation page here very simply, you know docs.openshift.com I went to 4.6. You can see that we have a top-level item here for registry so if you look through the Through the documentation here and I should have found the page beforehand. So I'm not left scrolling here on stream There are a number of different storage options available for you So the big three and I'll distill it down into really the three protocols, right? There's file There's block and there's object. So I'll Toss you what I hope is a softball Andy, right? What's the difference and what and why should I choose one or the other? So a couple different reasons one. I'll say it so many times. Don't do file. Don't do file Don't do file you will run into so many challenges so many challenges so many challenges if you're using quay Enterprise file is not supported. You can't hook up a persistent volume and assume that it works only object storage is Supported one of the benefits of object storage is that you can have more you can hook multiple Instances to it. The problem with block is you can only typically have one attachment at a time So you cannot have it highly available that is so basically if you're doing any head for registry storage grab a Object storage one of the best best options out there is open different inner storage which has an object end point You can hook up very easily Yeah, and I'll also offer so Usually the way I describe it the short version is Block is fine so long as you only need one registry instance, right? You can't scale it you only get one because you can only have one instance access that that PVC at a time file is File you have to be careful with right as you said, there's a lot of sharp edges that you can accidentally encounter so knowing that there is a lot of background there if you look in the documentation and I'll dig it up and post the link But if you look in the documentation we specifically say that we strongly discourage the use of the rel NFS server for the for file storage However, many of our storage partners They will say that it's perfectly fine to use file storage for, you know, multiple access for highly available Right multiple registry instance instances pointing at that same piece of storage So just work with your storage partner right net up is one that they they say absolutely use NFS all day long It'll be great no matter how many instances of the registry you have deployed against it Just be aware that from red hats and specifically the rel NFS server. We don't suggest doing that And then the last one is object as you said object is the one that's really recommended, especially at scale one of the things that I thought was interesting and that I didn't learn until recently is that the the behavior actually changes between file and block and object and So what happens is with file and block the registry instance has to proxy access to all of those Registry layers or image layers. So when you say, you know Podman pole or you know when Kubernetes has to do a poll of that image The registry instance is proxying off of that file or block storage and sending it to whatever that client happens to be With object storage, it gives it a pointer to the object in the object store so the issue here or the the the thing is right if I've got a cluster of a hundred nodes and I deploy a pod across all 100 nodes All 100 nodes are simultaneously going to say give me that image and if I have one registry instance out there I'm going to now be limited to the throughput of that one registry instance for all 100 nodes Whereas with an object store, I'm going to be limited by the object store Which is probably going to you know for talking like Amazon S3 or you know OCS or whatever that happens to be right It's gonna be much more distributed. It's gonna have a much better much more robust architecture for that type of pulled operation You're also brought up a really good point there with AWS because you know many many of the default images from OpenJip come from Queda IO and Those are being stored by AWS S3 in the background So you need to be cognizant about firewall rules that you have in your corporate environment because even though it's going to Queda IO It's actually getting sent to S3 directly. Yeah Yeah, and I think we have there's a KCS out there That has all of the all of the things that you need to whitelist in order to do that So okay, all right. I've beaten the storage thing To death here. I will say because I was a storage admin before Sometimes they get asked about sizing storage for the registry. I don't know if you have any thoughts there Typically it comes down to how many development teams you're going to be working with some of my customers They're not building that many images themselves. They are consuming Commercial off-the-shelf images that come from partners or other images that are not in any of the registry So they don't worry about it But it comes down to look at how many images you expect to produce being able to produce in a given month How many teams and then just do some math and give yourself some fudge back there of course because you might have some Some sprints were like, oh my god, I need to build everything and others that you don't and most importantly Put a pruning policy in place So you can go ahead and recapture some of that storage because I know I have images that I built five by five years ago. So sitting up on coita.io Am I using them? No Sorry, coi team. Sorry. Yeah, no and then So again coming from a storage background sometimes we'll get asked. Well, what's the read write mixture, right? How many apps do I need right that type of stuff and My answer to that is very much in line with with what you just said it entirely depends on your application team You know, if you're if you've got a CI CD pipeline and you're doing, you know, a thousand builds a day That's going to be a lot of pushes that are happening and a lot of polls that are happening If you don't or if your pipeline is once a week or whatever that happens to be that it's going to be pretty dramatically different So it it's really really variable I will say that it's easy to skew towards reads the more nodes you have in the cluster Right, because every time that image has to get pulled. That's a read operation Whereas it'll only be one right one push operation for each one of those images Um, let's see. What was I going to ask next? Uh, I don't think I don't think there's any networking considerations anything like that that we really need to take into account I will say and I had it up here just a moment to go So accessing the registry. This is really about if you're looking the docs Our back right and ensuring that we have the right permissions in place Exposing the registry is an interesting one. So this is For the internal registry, how do I enable external access to that which effectively just creates a route So I do want to take a moment to go through some of the, you know, how do we configure? How do we set up? How do we manage the registry real quick? Because as I said a few minutes ago If I'm doing upi Then it's not going to deploy the registry by default Essentially, you have to tell it what storage to use and tell it to deploy that registry and then the operator takes over from there So let me let me switch over here So I do get Config I think That might be oops Here we go. So this cluster is probably a bad example because it's actually in azure And by default it's an azure ip i install so we can see things like the management state here is set to managed So i'm going to switch back to the docs And we'll look at this configuring registry for bare metal So essentially you're editing this config object for the registry instance and Primarily, there are two things that we are concerned about One is the management state, which is basically telling the operator Do you want me to manage this or do you not want me to manage this? There's actually three states. There is completely removed the registry. There is Completely managed the registry and then there is let it be present, but don't Don't mess with it right the the unmanaged state So typically it'll either be removed or managed however Beyond that right what we're looking at is And so they changed the documentation. We no longer have the nice pretty listing inside of here But beyond that is configuring the storage that you want to use for that particular registry So if we go to github And we look at the registry So we have this image registry So the image registry this is the actual image that's associated with it and if you were to Look at it and I mean they they updated this again as well since the last time I looked at it to be fair It's been like two months. This is effectively the same as docker registry You can see there's even some references inside of here to the way that these things work and this is how right, so the configuration options are going to be the same as for the docker registry So when I went to configure and I know I'm scrolling and talking at the same time Here, this is a better example So when I want to configure for example s3 storage against a local s3 store, you know, maybe I've got men io Maybe I've got ocs. Maybe I've got sef, you know, whatever that happens to be Um, I go in and I configure that using, you know, this particular instance So for my cluster, right, let's pretend like this is a non-prem cluster So I can do an edit here And I would go in and I would do something like Add the storage option here. I would say it's an s3. I want my region to be local because it's using a local store Region end point is going to be, you know, whatever I've got configured here that I need to be able to use So effectively by configuring this right if I were to now write this out At that point the operator would take action to go through and reconfigure the settings of the image Container that's actually deployed inside of there. So I'm not going to do that because I would very obviously break My my azure deployed registry here And if we do an oc get project and grep for registry You see I have the open shift image registry I can dig inside of here If I look at the pods We have this image registry to image registry pods So by default with ipi and the cloud providers, it'll deploy a in a highly available right two instances in this in this case Of the registry and if I look inside of one of these so if you So if I jump inside of here there is The same configuration files that I'm used to so I got to look in the right place though So I have this config.yaml. So if you've ever managed if you've ever, you know, used the docker registry This is the configuration file for that and you can see we've extended it some with this open shift stuff and all that But as you come down through here, you can see all of the different configuration values that are available inside of here So you don't modify this file, right? You do all of that configuration through the operator. So through that oc edit config slash cluster So just to be clear that's that's how we actually do all of that stuff So up here is our storage you can see All the storage configuration that we have inside of here what to use for it's how to connect to it's All of that would be put in place by the operator not by us going in and managing it manually Andy Chris anything to to add there No, yeah, everything making sure not to I actually had a colleague of mine yesterday Manually modify some configurations for the registry and actually was also for the router and they were like, well, why is this? They actually managed manually managed the um deployment associated to it. They were trying to set a node selector. I'm like, no, no, no, no Let's take a step back and talk about how open shift is managed everything through operators There's an operator that manages the registry and the router You need to go ahead and set one of the one of the configurations on that operator to specify what node selector They wanted to have their resources to be deployed to Yeah, that's a good point. Um, so the registry is an infrastructure workload as well so it is eligible to be deployed to infra nodes and Take advantage of all of the licensing benefits, etc there So one one thing I have for you, Andrew. Do you typically set tolerations or node selectors? Oh good question So for an infer node, I usually do Both I'll do I'll put a taint on the infer nodes and then also use So taint on the infer nodes along with the labels and then on the workload set a toleration and a node selector Yep, that's just simply what I do as well Definitely a toleration because we try for two reasons when we do not want to have our infrastructure resources be constrained but also It's a licensing concern because like infrastructure nodes are no included subscription So that keeps all it keeps everyone status quo Yeah, the the rule of or the way I remember it is Taints repel workloads So right by it unless you specifically tolerate it it basically pushes it away. So that's what we want with infer nodes That's what you want with special resource Considerations so GPUs stuff like that. You don't want your You know, you don't want to waste CPU and memory resources on a non-GPU workload on a GPU node You want to dedicate those? um, okay, so OpenShift registry the the internal the default registry that's deployed alongside open shift That's managed as a part of one of the default cluster operators. That's what we've been looking at so far um to your points, uh, you probably see a lot more of these I would say that the The number of customers who ultimately deploy this registry is very very high Right, it may not be deployed on day one right right after open shift install finishes deploying Especially if you're doing a upi or a bare metal install um, but it's almost universal used Right for for a number of different reasons One of the biggest use cases that I use is actually being a cache Because you can have have it run as a pull cache So you're not dependent on an external registry So if you are using docker.io As your image registry You use the image pull cache and you can get around the pain when it comes to some of the rate limiting that they're They're implementing. Oh, that's that's interesting. I I know that that is possible, but I've never configured that before It is configured by default. So if you set an image stream Up by default and have it reference an external registry It will automatically pull it in to the internal registry and then serve out of there Very cool Because when you when you deploy an image or a deployment that references an image stream, it always points to the internal registry Okay, good to know um Thank you So the the last thing that i'll say about the internal registry or the default registry before we move on um, how do we configure or configuring for insecure registries? and and other type of scenarios um, which is Non-obvious, but so it's tangential non-obvious, but very much related of how do I get access to my registries or to my container images? Um, that's maybe local or or maybe image streamed even Um, and how do I tell OpenShift to allow that? So again, I should go ahead So, uh, let me just talk about the internal registry because that we can just get rid of that because For that it's all our back through OpenShift. So that makes life a lot easier Also uses the certificate that OpenShift generates by default. So the all the nodes trust that image and also very interestingly The and I've been playing around with this The dns operator has an explicit configuration So if you do like an oc debug node and actually look and actually get into the node itself and look at the sd host file it actually writes out a Host entry for the internal registry nice I did not know that me either So I just posted a link to the registry docs. Um, I also brought up a second page here. So this is the uh I'm wait, where did you paste a link into youtube? Oh, but did not come through Okay, that's not a good sign. No, let's see what's going on there Well, I look I know I now posted it into both. Okay. Thank you separately. Yeah, so, uh, I have youtube up and uh, You Posted from twitch into youtube, but not from youtube into twitch. So just So the second page that I brought up here is the image configuration resources page I'll post this one into into twitch since twitch seems to be going to youtube, but not the other way around I'm trying to make a habit of posting into youtube because youtube has a 200 character limit Whereas twitch doesn't so when we when we Chat in twitch sometimes not all of the message makes it into youtube So this image configuration resources is important for doing things like adding additional registries that you want to act as sources and importantly Configuring things like insecure registries and adding ca's for those other registries So a lot of times we talk about these things together even though they're you know, they are separate because people say I I need to You know configure my image stream and image stream uses the internal registry But really I'm trying to pull it from an external registry and it needs to have You know the same set of accesses grants right etc for permissions for accepting certificates that type of stuff So just be aware of this page. I'm not going to go in depth on this particular page here You can see if we scroll down far enough apologies for scrolling on stream If I want to block a registry, this is an important one You know, I We don't want to use docker Dockers registry because you know, we're we're always throttled right they throttle by ip sometimes You know, if all of your nana traffic goes through one proxy, it's always throttled therefore You know stuff like that isn't isn't necessarily going to work Uh, and then insecure registries is the other big one, you know Hey, I have an existing registry. Maybe it's the defaults docker registry that I just pulled from docker hub I'm in deployed and I don't have a certificate configured for it something like that Um, this is how you would add that insecure registry in there Note, however, if you update that it will trigger the nodes to reboot Maybe because we we still have to update that config on the nodes to allow the nodes to So if when you update this it will trigger machine config to reboot your nodes according to whatever the machine config pools policy is Okay So 32 minutes in let's talk about quay and quay.io Um, so andy, I you are far more of an expert on this than I am Um, I am cognizant of quay. I have some awareness of it, but Let's just say not a lot beyond that for the on-premises, you know quay deployments Can you can you tell us about that? It's the same code base, which is good There really is no difference, but the benefit is you become the SME You now have control of quay.io in your environment. So all the best parts about quay You now have inside your environment. You can go ahead and set up, you know authentication RBAC policies organizational Structures very much the same thing that you have with quay.io. You now can take advantage of yourself and all the the replication capabilities and geo replication and Scalability options that quay.io provides you now have inside your environment and that's a A big game changer because a lot of other solutions don't have those those replication capabilities so that so replication scale great are there other distinctions advantages features of quay that And specifically quay on-prem that the default open shift registry doesn't have Advanced RBAC policies organizational components that are tuned into basically It makes it so you aren't aligned to the open jiff world because the internal The internal registry is aligned to open jiff. You have to have you have to have a user account or a service account To be able to access it. There is no user interface for the open jiff registry aside from the open jiff console So if you want to have that was removed with four right right there was kind of a Very lightweight console that was provided in three that was as andrew you just mentioned was removed in four but You have to have access to open shift to get access to any type of visualization of the images inside the open chip registry Not everyone should have access to open chips so quita io provides that nice visibility for your organization because I can go and send a link to my cio and say hey, this image is out there Here here it is here's when it was created. I know you might have had some concerns about what's in that You can now see it because quay also has claire scanning So you can she's awesome claire scanning is awesome as well as there's a operator called the container security operator where you can actually see Vulnerability inside your images that are deployed here a cluster available within the within the console Okay, and and I think that really answered or answers my real question, which is Why how do I know I want quay versus using the default registry? if you want to actually use a enterprise Scalable router or registry that should be they can provide the benefits for your organization or just you Go to quay That's that's a big thing all right, so I think so so one getting quay Is pretty simple inside of open shifts. I think it will show up inside of my cluster here Right. There is an operator for for quay So i'm not going to deploy it into azure. I think you have an environment I do and I and I say that and and the reason you know Some people might be thinking. Whoa. Why is andy on the show to talk about uh to talk about registries Andy was actually the creator the originator of the quay operator that we see inside of here So if anybody is an expert on this We we have we have that guy for a couple more days The new version of quay operator has just been redesigned and redeveloped So I will be I am passing the torch as they say Yes As we heard earlier on the channel this week, uh, there's a new version of quay coming out this week and It'll help Number one It should help a lot of people installing quay just on rel 8 kind of a standalone instance and then yes It is updating its operator There's also a bunch of other fun features, but I won't spoil the surprises So so big shout out to the quay engineering team who I've had the privilege to work with over the years Not only developing this operator, but there's also a second operator. They are called the quay bridge operator Do you know what the bridge operator is? I don't I don't either you asked me that earlier this week and I Nope The bridge operator does a little bit of magic because it allows you to Emulate a lot of the constructs that Openjet has within quay So I can go in and create a new project in open shift And a new organization gets created in quay I create a new image stream in open shift a new new rep repository. It's created in quay I create a new I create a new build And the build hooks automatically get rerouted to quay Oh, that's really nice. That's so we use so we use the mutating web hook configuration to rewrite the build spec So no, so we look basically point it to quay and it also automatically sets up Robotic counts which are your service accounts with in quay A little bit of magic there. Oh, that's fancy All right, so do you mind if I if I uh, toss you under the bus so to speak? Yeah, watch out for the for the we throw tomatoes not rocks So I will avoid should of course. Yeah. Well, yeah, you're gonna get dirty, buddy I didn't say anything before but I it looks like you're in an office like he is in an office. Yes So, yeah, that's uh, that's fancy. You're now the second person this week. I've talked to that's been in an office So, um, so do you mind do you have can we can we see the quay operator? What deploying that looks like? So I did a little bit of cooking show magic just because you know, I don't trust things But I do want to show a couple of the patterns that you also described as well So I'm happy to show some things off as well today. I learned Andy has trust issues with technology Uh, no, it comes down to let me know when you can see my screen. Yeah, we can see it Uh, the big one was Andrew you mentioned earlier when you make some modifications. It does roll the cluster Right and I didn't want to have any downtime. So I might as well until I was having to, you know Wait a few minutes for paint drying. Yeah paint drying Or we can go ahead and spin this up So we have a nice open jet 4.6 cluster if we go over here to operators and we go look at the quay enterprise project we have the redact quay operator And if we go look at the quay ecosystem, which will be renamed quay registry next later in the week We have the configuration that defines our quay instance and we'll go past all the wonderful managed fields and we can basically specify All the different access I had a custom SSL certificate as well Because I wanted to by defining that I can then go into open shift and have open shift itself Trust it because quay has its own certificate So we need to go in and configure open shift to trust that certificate So any images that were pushed or pulled would be able to trust the quay registry And I'll show you how I did that as well Uh, everything else is fairly vanilla everything the register the operator itself goes in and configures a lot of these fields automatically but it's basically go in Set a couple fields and you have quay running And coming up with a new version you can also get a small instance of open shift container storage with it too So if you have a quay enterprise Quay, um, subscription you then get a very small instance of ocs to allow you to hook up object storage That's wow, which is really cool A lot of one of the biggest challenges or for my customers is I only have nfs Yeah, and I love quay. I want quay Do I have to go in and buy this massive new Object storage in my environment? No, look, you know, we now have that in the available option and It's only the tip of the iceberg when it comes to ocs, right That's nice. Yeah, cool. So how did I ever set up open shift to trust the quay instance? So inside the quay enterprise project. I have this secret called quay SSL quay custom SSL. It's your basic TLS certificates you know typical Self-sign because I was doing some testing if you have one is corporately signed by your organization Or you have one from a trusted certificate authority. Go daddy. You name it. You can then hook it up very easily And then we need to go ahead and tell open shift how to trust it and to do so You need to configure a config map inside the open shift dash config project big Here and I have this user ca bundle and it's basically an identical copy of what we had inside that other secret And then the final step is just to interrupt you there that Open shift this config map in open shift config applies not just to the registry. This is kind of like the global Certificates to a trust for anything open shift is doing. So if you have Uh your internal ca for all of your internal sites and all that other stuff that would be where you configure that as well And exactly and you then tell open shift You create that config map, then you tell open shift. Hey use that as my global Um reference and you do that inside this proxy object We just set this trusted ca field to be whatever the name of the config map that you create inside the open shift config project And it then goes allows us to go ahead and pull and push from play Now the next thing and many of my organizations are We only want to deploy images from certain registries Whether they be quitta io. Let's say they're coming, you know, we want to get the images for open shift We also then want to be able to get images maybe from just quay No, nothing else Because I have many of my developers go in and say, okay, how do I deploy on kubernetes kubernetes 101? Okay, let's go to the kubernetes documentation or the open shift documentation. Okay. I have this wonderful engine next pod, right? Yeah I'm gonna go ahead and just take the example. This is basically with the 101 from kubernetes. Yep We'll go in and we will just create a brand new project So we'll go over here to the developer council because I love the developer council develop the tools team does an amazing job Yes, they do So we'll just do my app Create that and we'll just go ahead and import from yaml Go ahead and create this It's basically want to create three replicas of engine x Click on create watch the pods come up And give it a second. Uh, oh I am feeling to pull. Why am I feeling to fall? Hmm How do we investigate that? Well, let's go ahead and look at the events And you'll see here basically We have based we have what do you see right here? I certainly can thank you It basically says rejected by policy because we only have certain registries enabled by default docker.io is rejected And that goes back to what andrew mentioned earlier the image configuration options that we set We'll pop back over to the administrator page or perspective. Pardon me go over to the explore tab Click on config And then we go find it down here image And this is for the registry yeah, it's for the registry not the actual configuration options Too many too many overlapping names Yeah, everything's config everything is a config your configs need configs Let's go with this one I'm she I'm going to go ahead and cheat and just say this is what it looks like. Oh, it's image Basically, it says right here. These are the these are the registries that I trust Yeah Yeah, so that was that documentation page The the second documentation page that I linked up there the image configuration So obviously quit iio registry dot red hat dot iio Registry dot connect dot red hat dot com which is for our partner connect set of images image registry dot assets that red hat dot com which is our older Image registry with some which we don't recommend but still some there are some references to it So if we want to keep that in there Very important is do you want to trust the internal registry of open shift? Which is this internal address and then this last one is basically my image registry that we saw a minute ago Wait up wait enterprise So I can go back so which basically refers to this deployment of white that we have running inside our cluster We showed the operator a minute ago What else do I have out there? I have We did that we did that trusted image trusted locations. So this is important because let's say I have an organization Let's say I have a a Linux team and I have a middleware team And they want to make their images available to all users of open shifts They have their images sourced in quay and they want to go ahead and deploy them in an open shift But they still want to protect them How do I go and configure an open shift for open shift to trust those images? So not have to deal with whole secrets. Just globally allow all images Anybody know No, there is a config map apartment. There's a secret Inside the open shift config project called pull secret That specifies all the trusted sources for images for image content That's so when you go in and set up open shift you set up your pull secret You grab your pull so you put that into the install config that gets popped into this secret And then you can go ahead and update it and then open shift will trust all those sources and to be clear there's There's sort of a global pull secret which is the one that is provided when you install But each project you can also specify Specific pull secrets so like I could provide a different set of credentials for my project to use by default when it's doing its thing Or chris code or you could right right? So change over to the open shift config project So a question for you, Andy. Do you often see people replace that default global pull secret for any reason? Replace it. Uh, so you can't replace it. You do not want to replace it. You will break your cluster horribly. You know why? Because you needed to pull images for everything updates and I feel like almost everything is dependent upon it. Yeah all your releases From registry.io all this stuff call it at openjip.com That needs to be here. You cannot And you just don't break. I mean, I actually ended up overwriting this last night by default by accident And I went ahead and started seeing failures to pull. I'm like, what did I do? What did I do? Oh, that's what I did So another question that I see sometimes can you So if if One of the three of us goes to cloud.redhead.com and gets the pull secret It's quote-unquote our credentials Is it possible to have sort of a A group account if you will or or like, you know, is is it possible for me to delegate that? So that if I have a team of administrators that I can utilize that manner And or is it always tied to an individual? And then the follow-on to that is What happens if You know, Andrew used his pull secret to to deploy the cluster and now Andrew less the company yeah So you you can go in and replace it There are a few steps that you need to take to replace all of the references This does this these credentials do get replicated in a couple different projects The big one that I I know of offhand is the open shift Project because that contains all the samples to all the templates and image streams get put into that project So if we look into the open shifts itself project open ship proper project, you'll see there should be one for samples Let's see And I know i'm Distracting you I sent you off on a tangent here. Yeah, it's okay. That's what we're here for uh open shifts samples operator apple operator So You do need to modify a couple different locations, but it is possible to change the default set of credentials that your account uses And there is a way also. Um, I have to dig up the link from way in the bowels of my email If you and and how did I discover this chris because well, some of us have accidentally shared our our pull secrets on the Yes, I've noticed You can go in and reset those credentials if you you know accidentally compromise them I was going to go walk through the um, you know how to how to set your authentication credentials And i'm like wait a second. I don't want to show the whole world all the credentials So I went ahead and just modified it here locally just because I didn't trust myself because just that exact situation So yes, if that is something interesting to you just let us know and I'll I'll dig up that link and I'll share it I'll also put it in the uh show notes blog post that we have so if you need to go in and You know, I accidentally uh shared my credentials once and and now I need to reset those that's There's a way to do that We kind of and just circling back to the security side of things, you know, we mentioned the machine config operator goes in and Makes changes to the cluster. How does it make changes? We just saw a minute ago that the Uh image policy did not allow images from certain registries like doc.io. What actually happened? That's what you know, I get is okay open chips magic. We know that but there's got to be some We're gonna go ahead and uncover some of the secrets from open shifts. So let's show How that occurs. So let's I'm on my cluster right now. I can do oc get nodes I can go in and See what changes got Made so I'm going to pick some random nodes pick some one of my welfare nodes And if you ever wanted to get on the node itself, you can use the ocd oc debug node command And that gives you access to the node itself Give it a second. It basically spins up a pod Start the session inside that pod and it uses a host mount which gives you access to it So I'm going to be I'm going to be lazy and just follow the steps You know change your route to the host path and now we have basically access to the host itself And if we do cat etsy containers Policy.json And that's very hard to read business. Yeah, is jq in this pod It's not I don't think actually it is. Yeah edit it finally. I forgot about that. Yeah, so I can just do jq R and get a nice fancy fancy fancy. There you go. All right That's what I'm talking about. So if we look at the top the default policy is reject everything And then go ahead and yes always step one reject everything So especially if you're doing image signing Same exact situation reject the inch that you actually go in. This is the file you actually modify for that too reject that reject everything and then go in and only allow certain fields and one of the benefits of Open to the operators is that I didn't have to worry about having to figure out this whole configuration file I just specify I want these image registries to be allowed It does the rest And it says basically use the atomic or the docker transport mechanism Make transport and then basically say I don't care what type of content it is. I'm going to allow everything And then it's it's this is for the image the internal registry Quite a IO the quay enterprise that we ended up deploying and the other registries So that's some of the magic that happened on the cover Yeah, it's it's funny to me the more familiar I get with open shift the more I realize you know, we've created this whole system and open shift of Things that basically are an administrator. So before it's very much your point It's I got to go to each host I have to update this file and then I either have to reboot the host or restart these services and do all these other things Or using an operator. It's I may I update the configuration once and then the operator does all of those other steps for me In some ways really convenience Right in some ways it can be frustrating if you're used to the old way You know, I want to be hands-on. I want to be able to go in and touch and manipulate and change all of these things and I we have At least on on my side. I don't I don't know about you know for you out in the field Andy You know, we have conversations fairly regularly about, you know, well The advantages of the operator paradigm particularly at scale, you know, and having this known configuration and It being done, you know, it done if you will in the way that red hat has has created from a support perspective Right, so we know how to help you when things go wrong and all that So it it can be intimidating when you're first learning open shift. I will fully admit that So I've been working with open drift for since the beginning basically at least definitely in the beginning since the kubernetes world and also back into the version one and version two and It's a little bit of a shift when it comes to three versus four a lot of my customers Move from three to four three to four They're like we have a lot of ansible work We love the ansible for depth because all this had to be done to be ansible previously Does that mean that we Can no longer use the ansible anymore and I felt no no no no no wait a second here ansible Is complimentary Ansible I use as especially if you're using upi you're still going to be using ansible Because there's a lot of manuals Manualish, but you're obviously using ansible to automate that but what I use ansible is is you're basically your quick start Use ansible to help Any of the install configs setting up vpcs if you're using ipi setting up your environment and then basically just What letting the flame to the fire which is basically going in and starting the cluster and preparing it so Chris, you know this better than I do. I love get ops. I love argocd I use ansible to help just stand up the base components of argocd and then let argocd do the rest Yeah, and the rest is all of the day two stuff right because that was that's one big change that I think a lot of people don't realize is with openshift 4 The installer is only responsible for getting the cluster up and running not for all the stuff inside the cluster So it's it's this bare minimum. What do I need to consider the cluster running and then Day two is where all of those other things come in whereas openshift 3 We had that massive how many it was over a thousand different variables in the the the yaml file for ansible And so all of that stuff has been pulled out and it's just that bare minimum in the install Is it a problem where I know the actual names of the ansible plays that were in openshift 3 because I've seen them so many times All right, so, uh, we've only got a couple minutes left. Um, so any anything you want to close with No, uh, I guess the last thing I'll leave everyone with is that registries are awesome Yes, they suck because they are dependency. They are in many cases an external third party dependency You're either leveraging or waiting on quit. I oh Docker hub you name it Have that in mind when you prepare and plan for it assume that it will never be available So especially since I have a lot of customers that are in a disk net in a moment One thing we did not discuss is the whole disk net in space because that itself is a Entirely different episode and I think you've had some other that's a discussion that we need to have over and over again Right like because as things change that gets better I definitely love to have a follow-up I think a good follow-up would be everything we need to do to manage Images in a disconnected space. How do we get images into the disconnected space? How do we configure openshift to use those images instead of going in and talking to the public registry? And then how do we make changes and maintain them over time? So here's a question for you Andy Even if I'm in a connected environment Because I'm relying on the internet because I'm relying on like you said external third parties right registry red hats quay.io, etc Do you see do you recommend? Basically pulling those images locally and then deploying from that local cache To speed up deployments and also to remove some of those external dependencies if I'm deploying, you know To five ten a hundred clusters inside of my environment. It's give and take If you go down that pack, you then have to maintain over time We push images and updates all the time, which means you then have to continuously mirror those images locally Yeah, that itself is a killer for me A lot of times what I recommend is you set up an image of cache So like a pull-through cache So so you can cast them locally openshift basically references that image that image repository industry. Pardon me and then You then we somewhat remove the decrepit dependency on the external source, but you're still Using the external source and not having to maintain it yourself But that's yet another conversation that we can certainly have in the future So we do have a little bit of time for overflow and one question just came in In a disconnected environment instead of downloading images locally Can we point the installer to an artifactory instance? Which is just what that's what I just mentioned. Yep Yeah, so it has now now nexus and certain versions of artifactory you have to be at a certain version This is very important because it uses image reference by hash and not by tag Which makes a big difference. Yes Which is good because remember if you reference an image by tag it can change if you reference it by a hash It is immutable right Yeah, awesome man in the middle that's uh manipulating those All right, so we are at the top of the hour Um, I do want to thank you very especially, uh, mr. Block. Thank you for joining today Appreciate all of your knowledge and all of the effort that you put in here. It's always great having you on buddy Yes, I love it. I learned something new from you and by something. I mean about three dozen things every time we talk Yes, so Um, thank you very much for coming on the show really do appreciate it So for our audience, please don't hesitate to reach out Even after the show ends with with questions you can reach me andrew.sullivan at redhat.com or practical andrew on twitter You're more than welcome to reach out with questions right happy to answer those if I can or uh bring in mr. Block as needed Chris if you're Yeah, no we I am at chris short on twitter see short at redhat.com and I just drop a link to to our discord channel So there's plenty of ways to engage with us if you have questions. Thank you very much and andy. I will give you the last word Thanks a lot. As always for the opportunity to come on If you want a new you need to reach out to me a block at redhat.com on twitter at saver s a d re 1041 sadly my hockey team is on Covid hiatus until the eighth which stinks So I have to go find myself and fill myself full of super bowl stuff for the weekend All right. Well, thank you very much everybody. Have a uh, have a great rest of your day Thanks everyone. Yeah, stay safe out there y'all. Thanks