 Let's get into the big topic, open source, something that we actually have in our heads. This is so awesome. We are an open culture that is actually under some sort of process that a developer or, let's say, as the Kubernetes ecosystem really brings. Welcome to this week's Ask in an OpenShift admin office hour. I am very privileged and very happy to be joined by two of my favorite people here at RedHats. My reoccurring guest host is your Johnny Richard. How are you, sir? I'm good. How are you? You know, it's a great day at RedHats. Here in North Carolina, it's gray and rainy, but it's a great RedHat day. We had a recharge day last week, which was very, very nice. Good to get a little bit of time to, I don't know about you, but I did a whole lot of nothing as much as I possibly could. I did as much nothing as possible as well. And the cool part about it is I found out on the 17th that 18th was a recharge day. So I was like, this is going to be a great day. So yeah, it was awesome. Yeah, it's, I can't say I did nothing. I, you know, for US folks, you know, it's tax season. So I had the great privilege of assembling all of our tax documents. You know, lots of fun. Yeah. I'll wait till the last minute on that one. Oh, no, we, I always try and get it done as early as possible just to get the pain out of the way. I should, I know I should, but I just, I hate it. Yeah. Yeah. So we are also joined today by Daniel Messer. And I apologize, Daniel, if I just butchered your last name. I think you've joined us on the stream before, but it was in early days. It was a while ago. It was definitely a while ago. Yeah. I think we've talked a lot about one of my products. So I'm a product manager here at Red Hat in the OpenShift group. And one of my products is Quay, Red Hat Quay to be precise. And also the SAS version of that Quay.io. So, and I think we talked a little bit about how you would set up Quay if you're running your data center and your own. Yeah. In your own cloud region. We demoed setting up Quay in an HA fashion using traditional virtual machines and me in itermin free parallel windows, receiving broadcast input to do configurations across free machines. So a very rudimentary demo of how you would install it for sure. Yeah. So I know we were talking just before we started of you're the guy who gets all the complaints anytime Quay.io has any blips and availability or anything like that. So. Yeah. I tend to. And, you know, there's nothing wrong with that, of course. Right. So we'll always, you know, work towards making your service better and more available, increasing the uptime. And yeah, sometimes we stumble, right. But we are continuously improving the service. We've just actually conducted a large fairly large migration of the code base. So if you're familiar with, you know, Python Quays written in Python and Python. The community essentially discontinued Python version two. Right. And so we had to migrate whole code base to Python V3. So you can imagine, you know, how many hundreds, thousands of lines of code of a large SAS small undertaking. Is a little bit entertaining. Right. So so yeah, we're going hard to get up until last Sunday where we actually did the migration and finished it. And yeah, so pretty excited about what Quay will have to offer this year. Yeah. Quay is, it's almost like a sleeper product, right? Like everybody knows it's there, but I don't think a lot of people really know what it does or really what its capabilities are. So I'm looking forward to, looking forward to learning more today. Absolutely. It's part of the invisible plumbing that makes all the high level cool looking stuff work, but it's pretty important. Yeah. Yeah, definitely. The product behind the product. So I feel like that's, that's definitely a product manager's point of view right. So before we move, move on to today's topic, I do want to cover a few things. So first and foremost, this is one of the office hours series of live streams here on red hat live streaming. And what that means is that just like if you've ever had a manager or a teacher or anybody like that who had an open door office hours type of policy, we're here for you. Whatever questions you have about OpenShift, whatever's on the top of your mind, please don't hesitate to ask that in chat on whatever platform you happen to be watching us on. The, the, the service that we use here, restream it, it rebroadcasts, resends all of those chat messages across all of them. So it doesn't matter where you're at. We'll see your messages and we'll do our best to answer those questions. And as I like to highlight and point out, you know, Johnny and I and our guests Daniel today, right, we only know so much. So we will do our best to answer your questions here on the stream. If we can't, then we're happy to take this back. We'll go and talk with whoever we need to inside of red hat, whether it's the product management team, the engineering team. I have had to reach out to the product marketing team even a few times surprisingly. We'll get those answers and then we'll follow up with those whether it's in a following stream in the blog post that comes out, or you can always reach out to us directly. So on social media, you can find me at practical Andrew on Twitter. And I've also recently started paying attention to Reddit as well, the OpenShift subreddit. So if anybody happens to be on there, it's another way to contact us. And you can also of course reach out to me via email if you'd like. And my email address is there in the corner, Andrew dot Sullivan at redhead.com. So with that being said, again, don't hesitate to ask any questions that you have at any point in time. We'll keep up with those in chat. I'd like to talk about some of our top of mind topics. So top of mind topics are things that we talk about, you know, kind of every week or every stream and it's things that we feel are important or I've come up recently that might be interesting to you all to our audience. So the first one of these that I have on the list here is actually a follow up from last week when we had Frank bought in on to talk about the performance out on operator. And I got an email afterwards asking about some configuration things that have to do with OpenShift deployed on to Red Hat virtualization and hyperthreading configuration. And I'll admit it's been, it's been a few minutes since I had to think about straight hyperthreading and all of the stuff that's inside of there. And this was a really fascinating question that chased down and work with our, you know, KVM performance team to find out what's going on here. So the short version is effectively in Red Hat virtualization, you have the option of specifying not just the number of cores per CPU or per socket, but the number of threads per core per socket. And so effectively the question was, you know, should I set the number of threads to be, you know, two, because, you know, if hyperthreading is turned on then, you know, in theory each core has two threads that are available to it. And the response that I got back was one that it makes sense, although it didn't occur to me at the time when the question was first asked and that is essentially under normal circumstances. So let's ignore OpenShift for a moment. Under normal circumstances, you would want to set that value to be equal to the number of cores that show up or the number of threads that show up from the output of like an LS CPU. So on your rev hypervisor nodes, you would go in, you would look at that, you would set it to that value. And this is important when you're doing things like CPU pinning, because as we talked about on the stream, things like NUMA awareness right all that other stuff that can provide some tangible performance benefits. With OpenShift on the other hand, because there's now an additional scheduler and additional layer of abstraction inside of there, the response I got back from the engineering team was more or less. We don't know, but we don't think that there's going to be substantial benefit that's available there. So they recommended keeping it set to the default of one thread per core per socket. So long-winded way of saying, you know, if you want to choose it, you're not going to invalidate support, you're not going to do anything like that. But definitely be sure to test both before and after so that you can verify that it is a benefit, right? It is positive effect on your applications that are running inside of there. Because, you know, best practices or even recommendations are not always universal. They're not always global. So I always recommend testing there. And they did also point out to me, I'm going to share my screen here. So you don't want a window and I want this guy. So they did point me to, like literally as we were asking this question, you can see last Thursday, which is the day after the stream was Kuvert Summit. So Kuvert is the upstream project behind OpenShift virtualization. And it utilizes KVM, it utilizes all of the same things that effectively read virtualization and rel, et cetera do as well. And basically what they pointed out to me was there's a session that is exactly around what this particular question was asking. So it should be posted to the YouTube channel. And I don't have, I thought I had a link. So it should be posted to the Kuvert YouTube channel in the coming days. I think they usually have a lag somewhere between one and two weeks. But if that's something that's interesting to you, you know, definitely go in and look for that. I know that they do a great job of getting that up onto their YouTube channel. And hopefully that'll help us to dig in. Is there any questions, Johnny? Anything that we should? Yeah, there's a few coming in about Kuvert specifically, but I figure we'll let Daniel answer those since they are right in his wheelhouse. There is one question from Alvaro. And sorry if I just butchered your name. But he's asking like, hey, what do I need to keep in mind when I'm migrating from Rancher to OpenShift? Yeah, so in theory, it should work exactly the same. So the reason why I say that is because Rancher is a quote unquote vanilla Kubernetes right when you're deploying like an RKE or something like that. And OpenShift is Kubernetes with some things added on top. So usually the things that most people are concerned about is creating ingresses, an ingress object, versus a route object. And OpenShift will automatically create a route when you create an ingress. Some things do change. So for example, with an ingress, you can specify a secret to use for the certificates. Whereas with a route, you add that certificate information in directly to the route object. I believe that it will automatically accommodate that as well. Basically, it'll automatically create the route with all of that information inside of it. So in theory, it'll work exactly the way that you would expect. Of course, I always recommend, you know, that you test and validate. And you also have the option of using the migration toolkit. So if we look at, and I can never remember the product URL, but if we look at the conveyor project. You can turn my screen. Yep. So here. So the conveyor project, which is the upstream project for the migration toolkit for applications. You can essentially deploy this into your OpenShift cluster and then point it at, you know, the existing cluster, point it at the new cluster and say, hey, I want to, you know, migrate this application and all of the Kubernetes objects associated with it into my, into my, you know, the destination cluster, and it'll handle doing all of that for you. So, including migrating data. So I think there is two strategies for that, right, where you can, thank you, I don't need a subscription. So where you can either have it literally copy the data. So, you know, mount the PVC in the source cluster, mount a new PVC in the destination cluster and have it copy the data from one to the other. Or you can have it, remember the terminology they move, they use, it's either move or re-home or something like that. We're assuming it's an on an external storage system that is accessible by both clusters. It'll create a new PVC or a new PV in the destination cluster and a PVC to map to it so that it can connect to both. We should really, I need to take down a note that we should really have the migration team on. Yeah, definitely, that'd be awesome. So one thing to add to what you're just saying Andrew is like, another consideration to make is when you're coming from K3s or native Kubernetes, right. And you're going to OpenShift, all of your pods in OpenShift are restricted out of the box, right. So that's going to be an issue if you're coming from Kubernetes and you haven't built your application for security, right. Or I hate to say it like that, but like if you're not using SEC or pod security policies or anything like that, then if you're running everything as root or trying to use in a UID, that's going to be an issue when you come over to OpenShift. So it's just something to be aware of. Yeah, that's a great point. The security policies in OpenShift are much more strict than, you know, most other Kuberneteses, Kuberneteses. So things like, you know, no root in the containers and all of that. So yeah, that may, that is probably going to be the hardest part if you haven't already accommodated that in the application. Let's see. Next one I've got on the list here. Oh, we updated the subscription guide. So this is one, it's a little bit self-serving because I'm a part of the update process. So we updated here. You can see it happened February 8th. I might have mentioned this last week. I don't know. I described to yesterday, Johnny and I were chatting and I described my day yesterday as basically like imagine a ball of cats that you just like set in the middle of the floor and let go. And they all scatter in 100 different directions. That's been my week the last week. So my apologies if I already mentioned this, but the two big updates that we had in this particular release are one, and I think I scrolled to the right place. One, we specifically highlighted that with OpenShift virtualization, guest entitlements for REL virtual machines are included. So if you are using OpenShift virtualization, you deploy REL VMs, those entitlements are included as a part of your OpenShift subscription. And then the other one which is related, you know, tangential to today is there has been a number of updates to include the various OpenShift platform plus components. You know, Quay is of course a part of that. We'll talk more about that in just a few minutes, but they added in the OpenShift data foundation essentials stuff in here. So if you happen to be interested in that, we talked briefly about data foundation essentials recently. It's included with those platform plus OpenShift platform plus entitlements up to I think 256 terabytes of capacity. So and I'm not paying attention. I think Stephanie, yeah, Stephanie's, she's on it. Stephanie in the background here is posting links before I even remember that I need to post links here. So thank you, Stephanie. All right, just a, I've got a couple of other ones here. I just want to hit them quickly. So we've had a number of questions. It's one of those, you know, when it, when it rains at pores thing, things. So I've been asked a number of times recently of specifically to OpenShift virtualization for some reason, but can we deploy REL CoreOS as a standalone operating system. It's sort of like we used to be able to do with atomic or, you know, CoreOS container Linux. So before CoreOS became a part of Red Hat. The answer to that is no. REL CoreOS is a, it is not meant to be used standalone. It can only be used as an OpenShift node and only supported when used in conjunction with OpenShift. You can't, we sort of alluded to that or it's sort of the same as when we talked about the various CVEs and security bulletins and all that. They don't call out, you know, Red Hat Enterprise Linux CoreOS as being an affected platform. It's OpenShift is the affected platform because you can't get CoreOS without OpenShift. Next one. This is one that I, I didn't know about this. And it's more because I like the docs are just huge and massive, but somebody had asked, is there a way that I can limit the bandwidth that APOD is consuming? And it turns out, yes, it's actually a built-in functionality. So you can see it's very simple annotation to go in and say, I want to limit the ingress and egress bandwidth. So this came up in the context of, you know, I want to figure out or I want to find out what happens if I limit my application to a very low bandwidth, right? Is it going to cause a ripple effect of other things to my application? And I was watching the email thread about it and was interested and turned out it was a very easy thing to do. And I was expecting it to be some arcane magic and like the, you know, using OVN commands or something like that at the CLI, but it works as you would expect. And then last but not least, just a real quick reminder for upcoming streams. So John and I have been planning out the next few streams that are coming up next week. We are probably going to look at and not just next week, but maybe the week after as well. We'll be looking at some OpenShift 4.10 stuff. So last week, we didn't have this stream because of the what's new presentation from the product management team, Daniel included in that. But we want to dig into some of the features, some of the things that are coming out. So rather than having that kind of high level, here's what's happening. It's let's dig in and let's see what's really changed and how it works. So we've got some ideas for things. There's some stuff that John and I are both interested in. If there's anything that you all the audience are interested in, if anything you'd like to see us dig into. Please send us a message. Please reach out. We'd be happy to look at that and see if we can find some SMEs if needed in order to bring on and really to talk about and dig down into those. And then also coming up in March, we have finally our hope nine. I see you're in the chat there. We have finally identified somebody who is willing to come on and talk about service mesh. I feel like we've been trying to find somebody for the last year or so. But one of the folks on my team, Ortwin is going to be coming on to the stream to talk about service mesh and all the really cool stuff that we can do there. So I'm looking forward to finally getting that on the stream here. So be sure to subscribe to the channel, whatever platform you happen to be on. Subscribe to the channel. You can also go to red.ht slash live stream. You'll land at the live streaming landing page. And on there is a calendar you can subscribe to that calendar if you'd like. And then you can get updates on all of our various streams. Okay. I'm done with all of that. Let's let's talk about quay. And, you know, when we first started talking about this, this topic, and, you know, kind of the title as you will even references directly the open shift mirror registry. The open shift mirror registry is one of the more. It's not one of the more visible things, but I think it's one of the things that has caught a lot of folks attention, because it solves a very tangible problem. Right. And the result has been some increased awareness of quay itself, which is kind of how we arrived at today's topic. So, so Daniel, as the product manager, can you tell us about the mirror registry kind of what what it is as well as the use cases and limitations surrounding it. And then I do want to I've got a quick little CLI thing I'll show of, you know, hey, what does it look like to actually deploy this thing. Sure. Maybe just to level set. I just want to assume that everybody kind of knows what a container registry is like a very fundamental level. So I'm going to skip that part. If there's interest in that, you do that. Just let us know the check. But yeah, so it turns out that open shift, but we need to work needs registry, right? Open shift is a bit more special in that sense, in the sense that all of its parts also containerized, right? So if you look at what runs on our costs, these are containers, right? So the API server, the schedule, all of these fundamental things run as containers as well. And if you know a little bit about containers, you know that they have to come from somewhere. The place where they're running is not the place where they are stored, right? You pull them down from these registries. Open shift has a day zero requirement for registry when you want to install it. Now, normally you don't really see or know about this because when you are using the open shift installer, you're connected to the internet and that's where container images are downloaded for Red Hat registry. It's actually going to be coi.io, but also registry.redhat.io, which is a proxy for coi.io. So for customers who are not connected to the internet, you have a little problem, right? Because you're behind your firewall and you have a green field, right? Nothing is running there. You have a bunch of servers that run open shift on them. So how do you do that if you don't have registry, right? You can't. So you need to think about that first. You kind of need a registry to store all the open shift images that are used at installation time and then also made on the clusters for me, right? And what people have done so far about this is either they had already some registry running behind the firewall or other storage reasons, so people use coi, others use other registries. Again, others kind of resort back to a upstream solution where you have, you know, an upstream CNCF distribution registry running in Parkman, you know, in a single node and single system using open disk storage. And that's fine, right? Technically, this will work. But it creates a couple of problems, right? So the first problem is, like, you know, if you're paying for, which is a subscription, you're enjoying all that great support we give you. But then one tiny piece in front of it that makes it all work is kind of unsupporting. So that's mainly not an awkward strategy, right? And I'm saying this because you will want to update this registry all the time just for a mere purpose of getting rid of CBEs. And what you also want is getting rid of content that you don't need anymore, right? And maybe you want to also manage and actually access that content, right? This is essentially licensed software or software that you pay subscription money for that's in there, so you don't want to open it up to the world. So you want to have a real registry there, right? With some, you know, wide problem management features. So it's open to mirror registry, right? Sorry to interrupt you. I just want to, I want to highlight that very much to your point here. You know, when Johnny and I did the disconnected deep dive four months ago, you know, we weren't ready to show the mirror registry at that point. So I think, and I can dig up the gist. Like we use the upstream Docker registry and all of the stuff associated with it. And, you know, one of the challenges of that is, you know, Koi has this really nice, really robust management interface where I can go in and, you know, exactly as you just said, manage that content and all that other stuff. So whereas with the upstream registry, that isn't there. Like, yes, you can technically go and improve images using the API and stuff like that, but it's kind of rough. So for me who was going through and it's, you know, now it's, well, I still have a registry, you know, in my, in my lab environment that I want to continue to use. But how do I prune off the tens of gigabytes of images that I need to. And so yeah, I'm very excited. I'm happy to roll over everything that I have in my lab over to the mirror registry. And to me, and you, you kind of touched on it, but didn't outright call it out, which is for me having a mirror of the registry for an install on premises is really helpful. Because I mean, yeah, it's my home lab, right? So I'm using residential internet service and all that other stuff. And even though it's a gigabit, you know, fiber link, it can still take a fair bit of time to pull down all of the images every time I'm doing a cluster deploy. And, you know, I've talked with partners before who as a part of their CI system, you know, they're, you know, one of our storage partners, you know, they're, they're building their CSI provider and then a part of the testing process is to deploy an OpenShift cluster. And they want one that's running, you know, the latest 4.6, 4.7, 4.8, 4.9, because those are all the currently supported versions. And, you know, it takes a non inconsequential amount of time to pull that stuff down. And what they told me is they cut that time in half the time to deploy a cluster in half by having it all local. So sorry to interrupt. And just to key in some more stuff there, Daniel, is you guys are doing, you guys, you're checking all the boxes that a lot of our enterprises are having problems with right now with the upstream registry, right? Where they're trying to use like a Nexus or a, an artifact or something like that. So that way they have authentication built in and TLS and stuff like that. Because I mean, with, with Docker, you can do TLS, but to get an authenticated registry, you actually have to kind of work some magic to kind of, you know, put something in front of it. So that way you can hit the, you know, HTTP access or something like that first, where this is all out of the box at native. And it's just, I'm super excited. I'm with Andrew. I'm in the, I'm in the fanboy boat on this big time. So sorry, I didn't mean to cut you off. I just want to add that. That's the whole premise, right? We want to make the disconnected installation easier. We've received a lot of feedback over the last couple of years, since OpenJet 4 was brought into the world, that this part of it is where you have to do more pre-work, right? And it's evolving out of different tools. And in case of registry, third-party upstream tools. So that's really what you wanted to kind of simplify. So you don't have to worry about where this comes from, how this is maintained, how you update it. You just download a thing from Red Hat that gives you that. And that thing is called the mirror registry. And it has this installer, right? It has literally one command, which is install, and it will install a streamlined reduced query deployment on a system. So you just need World 8 and Pogman. And that's pretty much it. And then you get going to the next step. This is what we wanted people to do, right? We don't want people to be stuck with, oh, how do I set up a registry? How do I make sure people have access? How do I ensure it's, you know, probably secure with credentials? So all of that is out of the box there with the installer. So it's just a check box that you can check. And then you proceed to the next step, which is actually what takes more time, which is mirroring all these images down, right? So there was a question in the chat. It's quite the only thing where I can mirror stuff into, right? And the answer is no, you can use any OCI compatible registry. We don't test all of them, to be honest, because some of them have their own implementation of the Dr. B2 protocol that comes with a couple of God shares, right? So we give our domination a couple of hints of what you can use. Quay, amongst other things, like Artifactory or Nexus. So all of these are supported options for a commercial enterprise customer. But if you don't have anything, now you have the mirror registry. And it's part of OpenShift. It's part of the subscription. There's no additional cost to it. It's part of the OpenShift entitlement. And then you kind of are able to proceed to the next step. So we really want to cut down the time if it really takes to get the disconnected environment up and running by really cutting down the amount of prerequisites that you need to do. So mirroring the street was one part of that for sure. So I think I'm going to actually skip doing the demo. I just wanted to kick off the install process, which is really straightforward. And actually, instead of showing that, I'm going to paste the video that you created, Daniel, here into chat. So that way, if folks do want to see what it looks like to deploy it, they can go and see that. But what I wanted to do instead of that, because we do still want to see Quay itself, which you have up and running, is let's spend a couple of minutes on the questions that have come in so far and see if we can answer a number of those for folks. So, Johnny, I'm scrolling back through the chat here just to just define what we haven't answered yet. Do you have a list? Yep. So I put it in our internal chat. It's basically, I think Daniel started keying in on some of them, but the one was about like, if I have, is it possible with Quay high availability to set up different retention policies for images? For example, I have one node keeping images for a different period of time over the others. So in the registry, you don't think about nodes. The registry is like a service that consists of multiple nodes for high availability and scale out reasons. So in the registry, you kind of think about it as a big bucket of container images, and you manage these buckets and the contents inside at the registry level. So we actually segregate the registry into what we call organizations. Others call these namespaces, what is the second namespace in Kubernetes, right? And there at this level, you have users, you have content that's managed there. And at that level, you have repositories, which is really what holds the images, right? And at that level, you can do retention. What Quay does today is allows you to set exploration of an image at the tag level. So you can actually say this tag that is pushed is going to be good for the next two weeks, and then you need to pull down and you want to write. So that's what we do in retention today. We are working on some stuff for the future where we are sort of listening to our users who say, I have a lot of tenants in Quay. I need to kind of keep the storage group in check because they are eating up all my precious expensive enterprise storage, right? So I need to kind of put some benefits up. So one thing we are going to do in the next bit of Quay in three, seven is we're going to have a cool time management being implemented. So you can actually put a limit up per tenant or organization how many gigabytes of images they can store inside. So that would be enforced at some point with a hard limit. So when their limit is reached, let's say 10 gigs in the war, you're gone pushing more images into it. In the future version of Quay, we're going to be a little bit more touching about that will allow people to define retention policies that tell you or that tell Quay what should happen if that limit is reached, right? Maybe you want to delete some of the older tags, right? Maybe you want to sort by semantic versioning because you're releasing software that way and it's probably fine for a developer cluster so that the version of like 0.5.7 from four weeks ago isn't there anymore, right? So that's how you can keep growth in check. But for tag explorations what we have today allows us to set a time and after the time the tag will be orphaned and will not be put able anymore. And does that, when that expires does it also remove the image layers in the cluster or in the registry itself? That's a good question. So the image layers in Quay are shared on a global basis which is very storage efficient, right? So you can imagine many people upload images to Quay.io and many of that are using UBI. The UBI base layer will only be stored once in the whole Quay.io department. The Red Hat Quay, the downstream product which is using the same code base in Quay.io that's the same when you kind of delete or orphan an image what happens is that the tag will be removed but the layers may not be removed if they are referenced by other images because you have layer sharing going on, right? But from a user perspective the image tag will sort of be gone. I think you might have just skipped over something pretty important there which is you said that Quay.io and Quay are the same code base. So for folks who are wondering like scalability and performance and all that other stuff we have a pretty good idea of what that looks like. Yeah, it's pretty impressive so if you look at the numbers pretty confident this is one of the largest public registry services out there. So we have like over 2.4 billion container images pulled through Quay every month, right? So there's a sizeable traffic that comes from public registry services but there's also a very large audience of community projects up there that are using Quay.io to distribute their containers, right? Some of that are even public source projects. So the numbers in like storage consumption and tag traffic that generates out through the CDN is just mind-boggling but that's what Quay is designed to do. It's a scale-out registry and that's important because if you think about kind of where the future is going, right? Kubernetes and all which is the song ubiquitous, right? Then you get a cluster in like 10 minutes and when you have things like hypershifting you may get them in like 2 minutes. So getting a cluster is very cheap and now you're able to say yeah, every development team gets a cluster even for a temporary thing because it becomes cheap, right? So with that in abundance you need sort of a central approach to software management which is what Quay gives you, right? It's containerized software and it's the central source of truth for distributing that. So when you do that or you ask someone about this you better make sure that the cluster is a pretty big and also very safe, right? Because if you use that central registry even for just 5 minutes trust me, you will know that one cluster is very, very soon that this is the case, right? So registries need to have a very high uptime and Quay is you know, geared toward that in the first design principle because it's scale out, right? The whole design of the registry is scale out. It's the presentation there like the image streaming is essentially stateless. There are stateful parts in the back end like the blob storage for instance that actually holds image layers, right? So database that, you know, takes as well as the XR or what you're using in this, what your credentials are but the serving there is the most important thing, that's scale out, right? So you can, like, run hundreds of Quay instances in ownership in parts and therefore have a very high degree of the registry that will give you a short size. It's funny because, you know I kind of poked at it earlier when you said that it's the platform behind the platform and stuff like that and the registry should absolutely be considered a core service, right? In my opinion, it's in line with things like DHCP and DNS as, you know, providing that you know, core if it's down, things aren't going to work type of functionality. You know, last week I was, I participated in a demo and one of the things that we showed was like using an open shift pipeline to build an application to push to a, you know distributed Quay instance and then using Ansible to then push those images you know, out to RHEL at the edge and it's one of those like it's not, you know, we're not talking about just impacting open shifts hosted application instances. We're talking about many, many things now are running on, you know, RHEL nodes are running on individual nodes you know, all across the data center and the edge and everywhere in between you know, they're running applications in containers. So it's the registry is a core and central part of that. So I did want to touch on I'm reading through chat here. There's a couple of questions here. What is an image stream? It's quite something like satellites, colleagues so that one, that's an interesting one I hadn't thought about it that way. They are slightly similar in that they are a content repository but satellite offers some other functionality as well for things like registering and entitling systems and stuff like that whereas Quay is a container image registry. So it's reaching out or providing that storage for those container images. Is it the same as open shift serverless? No, so serverless or I usually, I don't think they're the same this is Andrew showing his ignorance when it comes to these things. I don't think serverless and functions as a service are sort of similar to me. So serverless would be I am able to define and provide a piece of code directly to the cluster and the cluster takes that piece of code and any triggers around it and implements that. So what do I mean by that? Let's say that I have a function that just multiplies two numbers and then I have something that it subscribes to or some sort of trigger point that it hits and when it says it gets that input it fires up that function, usually in a pod and then executes that service or that function and then terminates whereas Quay would be just a regular enterprise application that's running inside of there. I may have misunderstood that question too I thought he was asking if Knative was the same as open shift serverless and if that's not the case and that's my fault. So Knative is a component of open shift serverless. Yeah. So yeah, good call. I missed the context amongst all of these other things. So the latest question is does Quay have a verified publisher program so kind of like Docker Hub where you have like this this is my stuff authorized or verified. Does Quay have a similar future? Yeah, so the question pertains to Quay.io so I think you understand kind of the friendship between Quay or Project Quay, the upstream technology and Red Hat Quay, the downstream product that you can install and find yourself from data center and then Quay.io which is our hosted version of Quay same code base and not all features are needed for all these things but the answer is Quay.io today does not have a verified publisher program so that's something where we're looking at the market to tell us if there's actually interest for that. And that's where you guys kind of need to make some noise and say we want something like this so Quay.io and Red Hat Quay as open source project are very open. So we have a Jira in which you can actually follow the whole development screen by all the releases all the features that are coming and you can also raise RPs you can also raise GitHub issues if it's pertaining to Quay technology itself but I think the Red Hat Quay Jira will be a good place to raise those RPs or you need this Red Hat account and we're always looking to get back from you as users and interested people to know about this right now it does not have that but it's probably an interesting thing to think about in the future we have seen a bit of an uptake in Quay since our registry offering has changed policies around how much the free tier you can still use so I think something like this could really work well when there's interest also from the projects to use Quay.io as the primary distribution mechanism to give it a little bit more of an official experience and a first-class experience but as of today it does not have that awesome and then we kind of talked about this in chat on the side when we were going through the initial build up but if you have one Quay in several OpenShift clusters can you use the bridging operator on say four clusters and use the one Quay to rule them all that's the case yeah that's true so what the bridge operator does for people who don't know it's a thing you install in OpenShift and the bridge operator will give you the same type of integration that OpenShift used to have or actually still has with the integrated internal registry right so if you think about image streams, S2I builds built automation inside OpenShift using these primitives that is all coming off the internal registry where you transparently push the image to the internal registry and then it's triggering really problems of your software now that the registry with Quay is outside of the cluster that integration was not available and this gap was filled by the Quay bridge operator the bridge operator gives you back all this automation so that when you create a namespace in OpenShift you have an organization in Quay you have when you have an image stream in that namespace in OpenShift you have a repository with the same name in that Quay organization you also have the pull secrets and access credentials be automatically pulled down into OpenShift with a so called robot account that Quay has in order to give you access without giving you the password right so all of these things are all straight in the background by the Quay bridge operator which makes dealing with S2I you know image streams and builds in OpenShift very very natural as if it were the internal registry but it's the external one the one thing you need to be aware of is that when you are using names in one cluster to name things in the central Quay instance you need to ensure that there's no collision QBO makes sure Quay bridge operator makes sure that doesn't happen because it prefixes all the namespaces and organizations and repositories in Quay that it creates as part of that automation with the cluster ID that the bridge operator is running on so you only need to make sure that you have various clusters by default have different cluster IDs and that's what disambiguates potentially similarly or equally namespaces in OpenShift so yes the answer is yes to have one central Quay and you have n OpenShift clusters all around the bridge operator and it would happily run on outside each other no problem awesome and then one last question I'm sorry Andrew and this is because this is coming up quite a bit about essentially the purpose of the Quay the Quay mirror registry versus getting content into the registry there's two different paths here you have Quay the mirror registry which is really your mechanism to get images to the cluster and then there's mirroring the content it's a totally separate process and totally separate pathway but there's a tool that's out tech preview in 410 called OC mirror and one of the questions that we got is OC mirror and mirror registry are they ever going to be I don't think so to be honest because we are following Junix philosophy here the tool that does one thing and does it very well so the mirror registry's sole purpose is to get your registry in like literally under 5 minutes as simple as that and then there's going to be another tool which is centralizing all our different mirroring approaches into one tool so that's going to be a separate tool which has only that mission if you stick with whatever registry you have if you have already Quay properly installed or you have another registry you can also use OC mirror with that but OC mirror is the next step to simplify all this prerequisite process of getting a disconnected cluster because today you need to follow different documentation steps using different tools to mirror different parts of objects to differentiate between the core images the core cluster operators and the optional layout item provides in form of the old image operators so these operator catalogs are managed and mirrored in a different way so it's also different steps and different customization mechanisms OC mirror is going to unify all of that in one central tool that does all of this in one command in one go of a central configuration file these are the OC mirror these are all the catalogs I want to mirror these are the operators inside those catalogs I want to mirror, down to the version or the channel and then off it goes and it can do that with any registry you have it could be a mirror registry it could be a production query you name it that's why it's a separate tool but it both comes as part of OC and it's both supported so it shouldn't be a problem that was another thing that when we did the disconnected deep dive we were just a little too early to show instead we ended up walking through the process of doing the OCA to ADM mirror and all of that other stuff and walking through all those steps which has now been dramatically simplified so I'm going to use this to change direction slightly so there's a question here from Khalid about the best way to deploy Quay one, I think we can use that as an opportunity to show the Quay instance that we have running but two, talking about ACM and Quay together going back to what we were talking about before of Quay entitlements are included as a part of OpenShift platform plus and so is ACS and so is ACM and so is OpenShift container platform OCP entitlements you can absolutely deploy Quay via ACM whether or not it's the best way in my opinion is a very subjective thing like everything Linux and Kubernetes in OpenShift there's at least half a dozen different ways to do things one thing I will highlight and Daniel please correct me if I'm wrong is that ACM, ACS and Quay all qualifies infrastructure workloads so you can deploy so you can deploy a quote-unquote hub cluster that is effectively control plane and all infer nodes and requires no entitlements and then you can scale those services as you need to for all of the other managed clusters that happen to be out there that you're using and again because it's all infrastructure workload there's no entitlements required with that so I thought that was a really cool thing when we announced OpenShift Platform Plus and all of those being infer workloads yeah it's actually true also for regular OpenShift so if you run regular OpenShift non-platform plus you have infrastructure nodes you can run ACM, ACS and Quay there you obviously need a subscription for each of those components but you don't need the subscription for the underlying OpenShift nodes that run DNS workloads and parts right OpenShift Platform Plus is a step further because all of these products are part of that bundle right so very simplified way of buying all into this and Quay is actually part of that bundle as well and not only that it's an all-you-can-eat Quay offering so if you have OpenShift Platform Plus you can deploy as many Quay instances and registries independent registries as you like and as you need talk to many customers who actually have new instances always because they have a staging instance and then a production instance we also have customers who are doing this with the geo-education feature of Quay where you can actually deploy Quay that's across your data center in the US and your data center in the EU and you can make it look like it's one big federated Quay with no difference to the client right so that's all available and you don't need to worry about any of that because it's a production-wise idea and when you talk about deploying Quay do you want to create a cluster running Quay operator because that would kind of get some insight into how it works and while you're sharing your screen real quick one other thing I wanted to highlight with OpenShift Platform Plus this is something I recently learned so Quay is the only one that you can once you deploy it it doesn't matter what nodes or what clusters are accessing it and what nodes outside of OpenShift so with OpenShift Platform Plus with the ACM and ACS deployments you either have to use other, it can only manage other Platform Plus entitled nodes or you have to have add-on licenses effectively but with Quay and the Quay deployments you can deploy that Quay instance and then other Kubernetes other non-Kubernetes things that you need to access and store images can all access that Quay instance so I thought that was really interesting of just a way to gain access to Quay and centralize that registry instance but so yeah Daniel if you don't mind please do share we've got about 10 minutes ish left I do have a hard stop today so I want to keep an eye on the clock for myself but as always for anybody who's watching you can follow up, you can send us emails you can send us messages on social media whether it's Twitter whether it's Reddit now whether it's LinkedIn etc so you're welcome to send us any messages and we'll be sure to follow up on any questions alright then let me share my screen in a second as you probably can imagine many tabs open so I need to ensure I have the right one open I don't know what you're talking about as I'm looking at my screen with you have like one browser instance and just one tab open but you should see OpenShift at this point if the screen share is coming through the screen as well there we go so I'm in an OpenShift cluster here it's 4.9 the latest version what I've done is I've actually installed the redhead Quay operator in this cluster and that's really how you deploy Quay with the Quay operator the Quay operator is taking care of everything that's needed to lifecycle Quay deployment and that makes it immediately subject to any kind of GitOps mentality because all that's needed is you need to lay down an object called Quay registry so this is the one API that this operator has a custom resource definition and once you have it you basically have control over a Quay deployment so I have prepared one here you see the kind here it's called Quay registry and you see this operator is already reporting quite some information at this overview stage it essentially reports general availability of the Quay instance as well as all the components that make up a Quay deployment so Quay deployment really is many components working together so let's take a look at the components here so what you need for Quay is a database we use Postgres for this you also need an object storage in that object storage Quay will store the actual layout blocks so the individual parts that make a container image will be stored in an object storage and we support various object storage providers and vendors but that's what's needed there's also Redis Redis isn't necessarily needed for every Quay deployment but when you use a feature of Quay that's called Builders which is Quay building your container images right inside the registry Redis would be required to have these build blocks stored for you to review how your container file slash Docker file build went we also have auto scalers here that make the Quay deployment go scale out and scale down in case that's needed from a load perspective there's also route management with the operators the operator will also create all the necessary ingress configuration for your registry to be available to the external world it sets up monitoring of course that's integrated in the open shift monitoring dashboard actually haven't looked at this in this instance here let's actually just try that real quick because it integrates into open shift monitoring so let's see if it's there there's Quay so I use open shift monitoring to essentially monitor the Quay deployment that's managed by the operator and we have an integrated dashboard here so you see I've been fiddling with this over the course of the day in preparing this demo now it's been quite down but here you can see what the usage of the registry is we pause CPU percentage memory and so on so it's pretty neat and integrates right inside open shifts monitoring dashboard so that's what this monitoring component is doing and then there's TLS this is the way of the operator allowing you to provide your own TLS certificates in case you have one in this configuration it is actually managed by the operator and that means it's just going to delegate this TLS certificate management to open shift and then last but not least there's Clare and Clare is the image scanner component of Quay Quay is not only storing and protecting access to your container images it also scans them for security vulnerabilities and that means you can actually look at your images before they hit the cluster and look at which packages are installed there, DBN packages Alpine packages, Red Hat RPM packages and reason about whether or not these packages are actually containing known CVs known security vulnerabilities right so you can take a look at that up front before you even hits the cluster I skipped over one component here that's called mirroring, mirroring is another functionality of Quay normally the way you use registry is you push content inside either regularly manually with your part man slash Docker client or as part of a build pipeline but Quay can also ingest other registries using the mirroring feature which means you can have Quay autonomously mirror every night at midnight certain parts of Docker Hub or your own registry for availability purposes improving pull performance and so on so that is Quay able to act as a proxy registry? It will be so we are working on that feature in Free7 so proxy means Quay transparently fetches something from an upstream registry like Docker Hub or GCR and serves it as it would be stored inside Quay itself and it does this transparently and it caches this so it does this once the first time the client pulls the image and then it downloads it into its internal cache so that is the download speed is the same as if you would pull the image directly from the upstream registry but once you pull it a second time for instance think of another node and go open shift running the same workload and you are scaling out that pull will be served directly from Quay it will be very fast and Quay maintains cache coherence by itself so this is going to be coming in Free7 it is a pretty neat feature because Docker Hub put up these rate limits over a year ago which are pretty nasty if you have a CI pipeline so you can use Quay as a cache in Free7 this quarter to avoid that Benjamin asks what I think is a tangential question which is will allow mirror by tags be supported within image content source policy that is something that we are working on at the open shift side right now it is not supported we only mirror if you reference the image by digest I know that these are very tightly in-between feature requests caching in Quay and mirroring by tag and open shift because what it would allow you to do is to say at the runtime level in the open shift all pull attempts to let's say everything in Docker Hub library project pretty popular project is cached by this Quay instance here right and that would be totally transparent to clients because of this mirroring by tag feature we are not there yet so last I heard is we are looking at that for 412 but we are working on it at apartment level you can just do this with registry.com in open shift you sort of have the stance that you only do this for images referred to by tag by digest so you can be absolutely sure that what you wanted to get is actually what you get right yeah and Johnny do you know the context for Khalid's question there scan and scan the image before? yeah I think he's asking about Claire when you pull an image in is Claire going to scan the image before pulls it into Quay or maybe it might be before the point like when does Claire do the scan essentially? yeah that's an interesting thing so Claire scans everything that's inside inside Quay the scanning part is actually interesting what it will do it will actually look at the image content first images are immutable so the image content as in all the rpms dbm packages, alpine packages, apk that are found inside stays the same over the lifetime of the image that's the immutable part so Claire actually starts to index what's inside all those images when all the layers it understands what an rpm database is it can read this and figure out which rpms are in that image and on a continuous basis every 60 minutes by default it will compare that to a list of known vulnerabilities per rpm from vendors like Red Hat because we provide this data as part of the subscription value to our customers so while the image is actually exploded on this only once by Claire when it's being loaded into Quay it'll be continuously updated because it's matching against cve databases so it's not like we need to re-scan the image every time which means you need to download the image, you need to unpack the table and you need to figure out is this an rpms database, is this an alpine image and what not that only happens once that only has to have once because images are immutable and then we only continuously update every 60 minutes which cve's are known for these packages identified in this image and that happens autonomously in the background you don't need to do this, it's continuously updated by Claire and that is kind of how it scans images, so the images available before the security vulnerability results are available you can pull the image with the scanning still being in progress no problem but when the image is arriving in Quay it will actually be dispatched to Claire for scanning and matching and to be clear Claire only operates at the package level it's not doing like code analysis or anything like that so for example let's say that your container build process is going in and it's doing like a pip3 install requirements.txt it's not going to look at each one of the python modules that's being pulled down it's actually going to do that no it is actually going to do that so what Claire does is package analysis could be operating system packages but it could also be programming language package manager dependencies so pip is one example and we actually support python so Claire will report python vulnerabilities in your image we've actually started to add more programming languages as well so java is next we had this pretty public log4j cve in the news a couple of weeks back so that's perfect timing right now because we are extending this package coverage so that Claire understands what a java archive is and it can also start to index content there so that's coming what Claire generally does not do it's not doing source code analysis so it's not going to tell you oh this malloc call is probably very dangerous so you have a little bit of tea here that exposes you to some sort of bubble overflow scenario or you're not parsing that user input correctly so somebody could trigger an sql injection so that's not what Claire does Claire's no access to the source code but it has access to the binaries and because there are databases that we maintain that others maintain that tell you which binary as in which package has certain cve's attached to it we can create those reports and that extends in the future way into programming languages so java is the next first step I think of npm right everything that's very popular among cloud native workloads well I I've learned multiple new things today that was a that was a big one I didn't know that yeah so if you want to return real quick to the deployment so this is a running deployment so you can see parts running here right you can also deployments all of that is managed by the operator and if you then log on to quay this is what it looks like right so I'm a user here and Andrew let me know if you can see the repository screen but I think it looks good yeah so this is kind of the user view of quay so I'm a user in quay so I see sort of my organizations that I have access to I have my home organization that is named like me my user demessor and I've also access to another organization that I created here in mirror so organizations are like projects and open shift or namespaces and Kubernetes it's a place where collaboration happens and you can dive into these right and see the content in there so I have like two repositories in here postpress 10 for instance and I see the various tags so I pushed the tag here a couple of hours ago that's the RHSCL official postpress 10 image and that as you can see has two vulnerabilities right so we can actually see that the Python package scanner has detected that there is a Python CDE in that package in the URL lib3 Python import so that's not an RPM that's a Python import that's probably used in some capacity by this postpress version in there and Claire sees this and sees the version and also tells you what the attack factor is it's putting this data from the national vulnerability database and it will tell you that this is actually an attack vector coming over the network and it's very low to exploit this attack factor so you should probably better update that image here right sooner or later so I'm going to use the fact that you pointed out that that's an RHSCL image official image that has some vulnerabilities highlighted in it so Adeel asked about how scanning red hat images with things such as FluidD or sorry with things such as JFrog etc reveals vulnerabilities inside of there and is that something that basically what does that mean does that mean that there are red hat as an updating images does that mean that the customer needs to do things to update those images yeah we don't care about any of that like who needs updated images jokes aside that's a big problem in the industry right so like my engineering manager used to make this joke saying you know you ask the man of a thousand watches what time it is and he will not know right this is the problem that you have and you're using third party vulnerability scanners with images coming from red hat because these scanners don't understand how we backport security fixes right they also don't understand how the vulnerability that may come from a library being very abstract actually relates to an exploitable path in our code I give you an example right so in the quay image even as of today quay itself is also a containerized piece of software as you can imagine there is a certain XML library that we use to parse some configuration data and that library has a known vulnerability attached to it and this vulnerability can only be exploited if you feed this parser some sort of very bogus XML and that's when it creates a attack vector right the quay image will never be reported as being affected by that cvd because the red hat project team went in and looked at it and saw there is no way you can feed an XML file into quay that it would be processed by that library and the only place in quay where we deal with an XML file is a static XML file that's part of the image that's not user manipulatable right or that's not maintainable by the user and that's why it's not affected so you need to differentiate between is there in cvd or are we actually affected by that cvd and you will get lots of false positives from other scanners if you just feed them ubi because these scanners don't understand that we back put security fixes into rpm versions that we are not bumping the rpm version at the patch level we are appending a build string or another build id to it and these scanners don't see that right they see oh it's version 1.0 it must have this vulnerability and then ubi suddenly has like 600 vulnerabilities and then they are also not taking into account that sometimes these vulnerabilities aren't applicable because there is no way you can put in a user manipulated XML file so it's not a vulnerability that's exploitable and that's why it's important to not only have a scanner but also have good scanner databases and we retards maintain these databases for retards products all of the retards products and these are the databases that clear in quays using and that's why there's a much higher accuracy about what's actually vulnerable right so if I look here I pulled every ubi release that is currently on the red head registry so all the way back to 8.3 and you see this is a source image here and you see some of them have vulnerabilities right take your time to look but you see that updating your images is also pretty important and ubi in particular because you need to update to the newest minor version because no CVEs are backported to older minor versions in ubi it's basically obsolete as soon as the next minor version comes out and that's why in older ubi images like 8.3 here you see lots of vulnerabilities that are being found new ubi versions you can see this that's why it's important to get images from red head because we spin our image builds very regularly right so that they are running off the newest ubi most of the vulnerabilities actually come from the base image or they come from dependencies that your package manager pulls in like play python or npm right so that's why it's important to always pull the latest software from redhead.com and scan the software with a red head approved scanner we actually launched a scanner certification program last year already that certain versions of jfrog xray for instance are part of so you need to also look at are you using the newest version of jfrog xray and then this problem of many false positives hundreds of CVEs in one image that's just regold will go away but not everyone participates in that because it entails to using our CVE feeds which not everybody is doing but claire is doing that and that's why claire has very high accuracy that content that it's scanning and it gives you a much more precise statement about how vulnerable are you how bad is this what you're currently facing right yeah so in the interest of time and because there's supposed to be somebody ringing my doorbell at any minute I'm just going to ask one more question which comes from user bin cat our hope nine I see your question if you don't mind following up via email android.solvinatredhead.com we'll get that answered anybody else who has questions please don't hesitate to reach out if we missed your question if like earlier I completely missed the context and answered your question wrong don't hesitate to reach out we'll get those taken care of and we'll also address those during next week's stream too so cat asks is the public quay service ready for cosine yes it is so cosine is the utility to sine content images it's actually a effort that we as redhead support together with google in the six store project very important very interesting project go check it out at sixdoor.dev cosine is a utility that essentially science contain images and stores the signature as a OCI artifact in the registry itself so it's not the image that is kind of mutated there's an additional kind of payload that comes with the image that looks to the standard registry as a OCI artifact and quay since version three six and quay I also I also want to add a little bit more first-class experience around the UI and UX of that in the quay user interface by you know visually highlighting these right now they'll just look at like normal image tags so you can use cosine to sign your images and store them in quay.io or quay three six or newer and you will see the signatures appearing there that's what makes these signatures and the images verifiable and we're also looking to add a little bit normal image tags but they're actually the signatures and in the future we'll make that look a little bit nicer. Very cool all right well as I said and as Stephanie posted into the chat there please don't hesitate to reach out with any questions email is of course fine you can also reach out on social media at practical Andrew on Twitter or on Reddit Johnny I will throw him under the bus as well jonny at redhead.com and jrock tx1 on Twitter so just a quick reminder for everybody be sure to subscribe to whatever platform you happen to be on we've got a lot of interesting shows coming up. Daniel thank you so much for coming on the show today I feel like we might have put this one together slightly last minute which was more my fault than anybody else's but this has been phenomenal it's been you know really really good and I've learned a lot I was aware of quay and many of the things but I wasn't like knowledgeable on it and now I feel much better about it so thank you very much for joining us today thanks for having me and Johnny as always good to see you again I'll leave you with the last word all right thank you yeah good seeing you too Daniel seriously thank you this was very enlightening I hope that everybody that watched got a lot out of it and feel free to hit us up if you have any questions Stephanie thank you for being awesome and seeing together for us because you are the reason why we're successful so thank you my pleasure