 Good morning. Good afternoon. Good evening. Hello everybody. I am Chris short principal technical marketing manager on the OpenShift team at red hats I am joined today by the one and only Daniel Messer. He is a product manager here at red hats I'll let him introduce himself. Please Daniel go ahead, but Before we get started this show is about how to get started with quay container registry So Daniel, please introduce yourself and let everybody know what we're gonna talk about Sure. Thanks Chris, and you know, thanks for having me excited to be here finally awesome. It's very timely Yeah, it is so hi everyone, my name is Daniel Messer. I'm a product manager in red head and I look among other things after a product called quay, which is an open source project like every red hat product and really a enterprise-able container registry and I'm gonna walk you today through the process of actually setting it up in various different fashions and Along with that show some of its features its architecture and you know kind of get you started with you know, how you do in a Sanctuary registry, which is gonna be really important in the in the in the near-term future, right as we are talking about having Multiple clusters these days running right and they all need kind of a central place to pull content from so quays the perfect Project and technology is actually make that happen Awesome. Yes, and quay is such a critical component in my daily workflow, right? Like I cannot tell you how often I am pulling images from quay on a regular basis It is a fantastic container registry and has quickly become my favorite container registry and it's not just because I work at red hat I would probably still use quay if I wasn't at red hat. So that's that's probably the best endorsement I can give of it I Mean it's it's great to hear and it really hits home with a with you know that notion that you know Registries are usually in the cloud native discussion somewhat in the background, right? I'm talking about Kubernetes and the stuff that runs on top of Kubernetes and different personality to technology has but The registry said to be a fairly Vital component, right? So if if you are Running your stuff of a central registry and that registry is down. You'll notice pretty quickly, right? So yeah, it's sweet only, you know, you cannot deploy anything. You cannot update anything. You know, no rollouts are going to happen And if your registry is right only you're gonna have a lot of angry developers on the back of your heels as well because they can't push the artifacts to a central store so Tends to be a little bit of an underrated topic, I would say But it becomes super important as we are moving into this future where you know with open shift It became so easy to run a cluster now, you know in your own machine on your top provider account in your, you know, Virtual infrastructure, you know The only thing that's gonna lead to is more classes are gonna pop up, right? And I want to have a central view on you know, where's the software that I'm gonna run store, right? because yeah, and You also want to make sure that that stuff isn't full of security vulnerabilities either, right? Right, and you want to make sure that only the right people have actually access to this stuff too sometimes so That's where Quay becomes sort of in the in the picture and becomes really critical to that a whole Open shift multi cluster management future that we are embarking right now. Yep It's it's one of those things where if if you don't have it's like storage almost, right? Like if you don't have this registry You're not going to have a critical component that you need inside your infrastructure, right? Like if you're always reaching out to the internet for something or if you're always pulling, you know Think you're gonna have something break and you're not gonna have access to it at some point So having this stuff internally having it running on your own infrastructure is vitally important in my opinion Yeah, and like even if you have one registry and it's not of good quality and availability, right? You'll notice pretty pretty quick real quick real quick. Yeah, like oh, I can't pull wait a second Everybody's, you know ripping the hair about that error and parts crash looping or errors Putting images, you know, and it's not something you can do something about usually because it's not your registry. So Yeah, so let's let's let's let's show people real quick, right? Like how to dive in to like quay.io The internet side, right? Like the public-facing side and then if you want, you know, we'll continue on diving deep into what you're doing Let me share my screen real quick. I just want to show people like it's if you haven't heard Docker hub is changing its policies around image retention. So come November 1st if believe the Guidance has been if your image hasn't been either pulled or pushed or anything touched essentially in a certain amount of time They're getting off their platform essentially. So, you know, if you've got this project that you only release every six months guess what? You better be very careful with your your Docker hub You know usage because you might very well end up without access to that image So, you know, you come to this page here on quay.io You know super clean looking page It's not very obvious like these two buttons are like try for free on premises, right? Like that gets you started on what Daniel's gonna talk about try for free in the cloud same thermos But just different infrastructure, right? But if you want to create your own account real quick all you do hit the sign-in button up here I know that's counter-intuitive, but then hit create account and You know, I'll just create a simple You know dummy account real quick my email address. No wait username Let's just do barf email address is me at crash short net and then we'll just make up a password real quick with my fancy handy-dandy password manager Thank you very much. Nope. I don't want to actually sign in I want to create See now this is where it gets interesting, right like now my No, go back create. Yeah So, yeah, it's really that simple folks, right like Just dive in you know, these passwords aren't gonna match, but you know, let's just do it real quick and I can tell you That it will be very very simple process once you don't get interfered with by your password manager Wow, that was super weird behavior. But anyways, yeah, I'll save that for sure Thank you registering. We have sent you an email to activate done, right? Like I got the email We're gonna open that up Get the link to confirm Copy pasta done Hi, you know and now it asks you some information about yourself just so you know very like hey blah blah blah This is my person so forth But once you get past that it's all yours. You just log in to quay.io from you know Docker you do a Docker login from pod man You do a pod man login quay.io and you're off and running with your new credentials and everything and it's that easy So, you know Hopefully this will help a lot of people right now that are in that dire situation where it's like, oh my gosh What do I do with my images, right? Like this is a quick way to just get you going now Daniel will get you going the right way Yeah So what you've just seen is actually Quay.io, which is a hosted quay instance and it's based on the very same technology that you know I'm gonna showcase today exactly how to use on your own premises your own server your infrastructure. So It is battle tested at that scale, right? So we literally have you know thousands of users and You know hundreds of thousands of images petabytes of storage in the back end. That's all managed With quay.io and the same code base is what makes up the quay product like so Let's just do a quick intro on What quay is and then then let's start right into the you know hands-on stuff hands-on part of this of this dream. So Who am I already kind of told you right so I'm a product manager. I can also do other stuff at Red Hat So next to do others Yeah Basically responsible for that operator how I oh thing. Yeah, that thing that's kind of important Yeah, it tends to be right. So it's our upstream operator catalog right and the operator framework is like all the components that make operators In the workload space work nicely an open shift. So I'll do that stuff in my you know time left of the quay basically you know doing this now since almost two years now and Been with red head for for quite some time as well my newest hobby project actually is my son So he's a good project to have nice much nine months old. It is a fairly involved project was super happy though and Yeah, because you've been in the input space for a long time as well. So go check that out, you know get up the calm quays or Open source upstream present. It's fully open source project And then there's also the other part of my work, which is the operator framework And that's actually going to be some overlap today because I'm going to show you how using quay operator to deploy quay on your open shift cluster So, um, what are we going to do? Let me just quickly run down, you know What quay is what quay can do and then how quays working from an architectural perspective and then we're going to install it on a single machine Right, that's going to be really straightforward And that's like your typical, you know at home, you know poc laptop demo install And then we're going to go a bit more serious, right? We want to make quay ha So highly available. Um, look balance probably with no single point of failure Um, and um, that's going to be how you know customers run this in production and don't run into this situation where You know, you have image pool errors all over your cluster because your registry is kind of down or you know, You know, uh Coms spread out across, you know, the the the fruited planes of the globe. Yeah Exactly Right of the globe, right? We're going to take that deployment in the third part of the demo and um, I'm going to replicate it across to the us So I'm going to start in e.u. Um, which is where I'm based. I'm based off Germany. So I'm going to host this in a AWS location in frankford um And uh, but it could really be an example of, you know, your data center and then we'll have another data center in the US At which we will also install quay and we will start to replicate a quantum across the pond, right and see how that works Nice If we have time left and you know, we can be flexible and we can you know, depending on the interests of people on the stream comment section Skip some parts. Um, but if you have time left, I also want to show you how you Going to go up and get up and running with quay super straightforward with uh, the quay operator Which is something we are currently rewriting for scratch actually So with that said, what is quay, right? So You heard it. It's a registry. Um, I like to actually call it a registry platform because You know, a registry can push and pull images. I mean, that's not really That complicated anymore But as soon as this thing starts to become central to multiple clusters and multiple teams and multiple departments um And surfs production workloads, you need to think about a couple of additional things, right? So you need to think about multi-tenancy security. Uh, would be really nice if you build automation on that as well Um, and it would be really cool if you could store not just contain images But basically all cloud data artifacts out there, right? So, um, that's what quay does, right? And you know, the usual disclaimer, um, there is redhead quay, which is the, you know, commercial product You know chris and i are basically in selling and then there's project quay, which is the open source equivalent, right? Same thing open source That's what you can download, you know, play around with as well um Cool, so you can totally run your own look registry like just locally on your local host project quay if you want Exactly. Exactly. Just to kick the tires on at poc at whatever you want to do. It's right there for you. Yeah Yeah, absolutely It's going to be a little bit more resource intensive than your usual, um, you know local docker distribution registry That's basically because quay has been, you know It's part of the stock for scalability. So the initial footprint is a little bit higher, right? Um, so you want to have at least like, you know, four gigs of memory And and when I, you know, one spare cpu court, you know Not unusual anymore these days, but I just want to throw it out there that it's not like a very tiny thing It's actually going to go up a little bit of capacity on your system But that gives you a lot of in return, right? And some of this is what you see here, right? So, you know, it's the same thing that powers quay.io, which is our which is actually the first You know private registry out there So before docker even introduced, um, you know private Repositories quay had that quay provided that So that's how quay started and you know quay.io is the online service with the same functionality So chris just showed you how you sign up when you sign up and you don't eat any Private repositories the thing is free. Um, so you can store your content there You can build your container registry as well. You can make them get scanned against known vulnerabilities. That I think is the best feature I mean by far hands down bar none in my opinion, right? Like Having a vulnerable container register container image out there Is a big problem Having your registry point that out to you is a huge one Absolutely, right. So that's what the security piece is going to do for you and we'll show how you set that up and how that works today right, so, um I think without, you know further feature setting here. I'm basically going to show you, um, you know, how you get started, right? So, um This is what you know quay, uh, looks like from an architectural perspective, right? So quay is basically, um, a python layer, um, which is, um a enabling you to store content according to the docker v1 and v2 to protocol and It has several additional parts to it to make that actually a fully flash registry. So, um because it's like Containerized from the start. So you actually start quay in a container, right? There's nothing to install on your host. You just launch a container Um, it runs on every infrastructure. Um, you know, you can run this with, you know, docker podman on rel. Um, and you can run this with kubernetes and open shift In a container registration platform, right? Um, and then quay needs a couple of things and actually to do its job, right? So the first is, um, you need to have a place where to store content. Um, that's usually an object storage um And then there's also a database which handles some of the metadata that is involved in the process of actually serving container images, you know And also models, you know different teams and organizations your automation settings and all the integrations that that quay has And there's also an in-memory cache involved. That's usually involved when you do builds Um and build is a feature of quay where you kind of give it your docker file, right? Or you hook up your repository and then once you push to it, uh, or you push the docker file, um, You know quay builds a container for you. So these quay builders are making use of redis Um, then there's quay itself, obviously. Um, that's the registry compliant oci registry endpoint And quay has a feature called repository mirroring where you can actually tell it to mirror a repository of your choice For instance to pull content closer to where it's needed or put it behind a firewall, right? So that's what these mirroring workers are going to do. I'm gonna, you know, look at how that works as well And then there's claire claire is actually a system project of quay and it's a security vulnerability scanning solution for container images So, um when deployed with quay claire will look inside the images that quay stores and will tell you If there is a known security vulnerability in that image Either in that operating system base part, right, you know Or now actually as a recently also in your application level dependencies So we started with python there because that's kind of our home turf, right quay python claire can actually now look into your python um dependencies and python applications and you know shipping in the container and can tell you if those dependencies Also have known vulnerabilities, right? So you're also kind of looking at the application now now Yeah, I was I was adding a bunch of uh, hugo containers very versions that I use Based off the website and like having having built those like months ago, potentially Those docker files being written, you know months and months and months maybe a year ago having you know, like pulling those docker files in and like making sure they're good and then Publishing that image and then having it scanned by quay gave me just this highest degree of confidence that These are good. I'm going to be able to use these for whenever and however long I need until the security scan fails and have to rebuild basically Right, yeah Yeah, so claire is going to continue doing that, right? So claire is going to be constantly receiving updates to databases where you know has all the vulnerabilities that are known stored and Once those get updated you'll also get a notification if there was previously unknown CVE for instance Discover that actually measures the signature in your image, right? So it's not just this one-off thing It's actually constantly looking at your content and and yeah, claire also needs the database and To store some some metadata during this processing but other than that like this entire layer is completely stateless, right? Which is makes it super easy to put it behind a little balancer and just scale it out by adding additional copies of it, right? ideally another host and then You know the way this usually gets used is behind an htps endpoint So encryption as well. So there's content ingress coming in, you know from auto registries from, you know Your system with parking and push Or we have users that are actually going to use the ui or the api Which is completely According to what the ui does So everything in quay is actually api driven and that console you'll see in a minute is essentially just, you know Python front end calling and using ui You know, and that's that's how you have a sample registry, you know Other clients can then use it or Kubernetes clusters can use it Open shift obviously an open shift you have an additional integration, which I'm not going to show today We may do another stream on actually how to use quay with open shift But there are two operators on an open shift that make it make for some real interesting use cases One is going to cover security scanning on vulnerability right in your cluster and the other is going to automate You know quay with with an open shift, but yeah more to be known on that and maybe another session Yeah, for sure Um, I just wanted to ask real quick. Can you share your slides with me? Just add them so I can toss them out on our slide share real quick. Oh, yeah For sure. Um, let me actually just do that right now. Um, yeah, thank you people people already said like that slide right there is gold And they want that so I'm doing I'm doing the audience justice here Alrighty sweet, thank you Yeah Anyway, that's nothing confidential in there. Um, yeah, I can just check them out um Cool, so Let's go all on one. Let's do it. Let's do it, right? So I have a um a system here That has nothing else installed and sent us right, you know sent us eight one Um, so nothing out of the ordinary and we are going to basically build this right We're going to install a postgres database quay can use postgres among other databases And we're going to install redis. Um, which is also a requirement for quay And then um, we're going to configure quay to use the local x4 file system Now that is actually something that is only meant for pocs like this, right? So, um, that's really just your local demo You don't want to make your quay registry dependent on your local file system because quay has been built For object storage, uh in mind and it'll it'll not be very happy with with the limitations of the local file system in the No, no, no, so, you know, just do this with that one off demo and they're enough Cool, um That's quay, right and uh, we'll also have um, you know in ssl endpoints So then I have a sort of bit installed and then I've already prepared the dns record at points to my machine All right, so Let's jump into the console This is readable if it doesn't uh go up on audience, please let me know if there's issues there. So, let me just resize this a little bit. Cool. All right. Dan, I prepared, like I did with TV Cook already, and Instance. Multiple questions here in chat already. But go ahead, keep going. I think this will actually answer some of them. Yeah, so I prepared Instance to Instance. Nothing fancy, too coarse. Eight gigs of memory. As I said, you want to do six gigs at minimum. Especially in those daughter farms, you don't have small space, right? So if you run out of money, you'll be killed. And also already have a public IP here, right? So that's what I prepared. And I also prepared a DNS record for that. So it's going to be quay-standalone-dmeser.io, right? And that's going to resolve to exactly that 165 address that's in this instance, right? Cool. Okay. So let's get going. Going to look into that thing and walk you through what you need to do and install quay. So the first thing we're going to install is some dependencies, right? So I'm going to run quay as a container, right? That's the official way to do it. So I need to have something that will run containers. And what's better to use here than our good old friend is a wrapper. And then I'm also going to do the Postgres client. I'm going to run the Postgres database actually in the container too. But I'm going to install the client as well to connect to the database in case that's needed. All right. And then Wget, because that comes in handy every now and then as well. Some people curl, some people Wget. There's no right way to say it. I think what's funny is Curls there. And Wget isn't. If the signs of the times aren't changing to the Linux land, right? Netstat to SS. Wget to Curl, right? All these things are changing, right? So we're at this weird transition point where there's still this group of folks that like, oh crap, I haven't learned Curl yet. It's still Wget. Oh crap, I haven't learned Curl yet. It's actually a lot easier than you realize. It's almost as easy as Wget now. So yeah, a lot of advancements has been made there. So don't fear the Curl. LibCurl is a good thing. All right, cool. So everything installed. All right. So first thing I'm going to install is the database that Qua is going to use. That's going to be a Postgres database. So let's create some directory where the data is going to be stored. It's going to be usually more on PGSQL data. I'm going to mount that into the container in a second that I'm going to run. So one thing that I will do here is, and I'll apologize in advance, like seriously, is I'm going to run that container as root. Oh, it's simply because there is there are some components here that do need privileged ports like ports under 1024. And those need to run as root even with potline, which is actually really nice and good at running stuff as non-root. Also do copy and paste here at some point. So adjust permissions to that directory a little bit so that Qua container has access. Then we're going to set a couple of environment variables to make the Qua command not so long. So the database container name is going to be Postgres. The database name is going to be Qua. The username is going to be QuaUser. It's going to have by default all permissions on that database called Qua. That's going to be the password. I'm not super worried about sharing this with you because that port is just internal. You can't reach it from the outside. That's the same password here. So yeah, we're going to run a database now. And I'm saving a little bit of typing here because I don't want to bore you with my, you know, abysmal typing skills and that is the container run command, right? So, you know, proper run, give it a name. We detach the container because we want to run it in the background. We inject some known environment variables into a container that is used during the initial part of the database setup that only runs once that container starts. And it's going to set the user, the password, create a default empty database on which the user will have all permissions. That's the database that Qua is going to use. Then there's also a super user here. And then we're going to publish that port and then we're going to mount that directory that I created before, which is the PostgresData directory. Take notice of that little Z switch here. That's going to make this thing fly with CVNUX enabled, which is enabled on this platform. And then we're going to push the official Postgres 10 image from the redhead software collection in version 10 based on the rel7 base image. That's just, you know, me being lazy, I could have used, you know, the version based on rel8 or like a different Postgres version. It doesn't really matter. Qua is pretty happy with everything starting with Postgres 9.6 onwards. Right? So, let's run that. And that's going to pull the container and launch the database. There's nothing else in the database, right? Qua will actually set all of the stuff that it needs in there up for you. So, that container started. Let's take a look. There it is, running happily. Let's check the loss very quick if I did any errors. Looks healthy to me, starting a server, listening on port 5, 4, 3, 2. That's the one on Postgres port. Okay. One thing I want to know is the IP of this container, and that's a portman thing, because I'm not going to be able to reach this port inside that Postgres container from another container that's also running root, at least not with the host system. So, I'm going to inspect that container. I'm going to look at the IP address that portman assigned to that container in its internal network, which is this one here, in 80802. That's the IP I'm going to use to connect Qua to this database when Qua is also running in container. That's stage one. Let's actually try to log in real quick to see if that also works and if I didn't type any passwords. I'm going to use that IP. Actually, I'm not sure if I can do this from the local host. Let me actually use the local IP here to just save. Yeah. But from container to container communication perspective, it's definitely going to be part of an internal IP. From the host perspective, I need to use the host IP because that's where the port is. Yeah. I will do connect to the database called Qua as the Qua user. The password was QuaDemo Twitch. There I am. Cool. So that thing is up and running. Database is already there. I can log out of that. Cool. Step one completed. Now I'm going to basically deploy Redis. Very similar. I'm going to create a directory in which Redis container is going to store its data. I'm going to give that the appropriate permissions and I'm going to run Redis which is also available in the Red Hat software connection. Nothing out of the ordinary here. Redis is super straightforward to set up. Run a container, make it detach as a demon. Restart it always. Once it's coming down, there's just one port that Redis needs which is 6379. That's where Redis will be serving its functionality. Give it a name. And then we also mount that directory. Redis is also really small. Really easy to install as well. We won't use it because we're not going to run builds on this install. But effectively it's a requirement that the installer checks for. Okay. Let's see if that thing is up and running. Yep. There it is. And then let's check for any errors in the port. Looks good. Nice ASCII logo. Server is now ready to accept connections on port 6379. Let's actually find out what that IP is. That's the IP. Just to refresh our memory, the Postgres IP is going to be that. So it's 8080.03 and 8080.02. All right. So we're just one step away from running our Quay instance. One thing I want to do here is I want to give that Quay a valid SSL certificate. And one awesome super straightforward way to do this is using Let's Encrypt. So let's actually get a real trusted certificate from Let's Encrypt in like one minute. For that, I'm going to quickly install a web server on this system. If you have ever used Let's Encrypt, it's actually serving certificates as a service and it lets you respond to a couple of challenges they will give you in order to verify that it's actually you behind that IP, behind that host name, making that request. So the term that it's going to be using here is something needs to be created behind an HTTP server. So I just installed an HTTP server on my system. I'm going to create a dummy web page for that thing. And I'm going to create a virtual host. So I'll make that web server listen on port 80 and make it own under the server name Quay standard on the web server. This is standard Apache stuff. It's the same thing since 20 years probably. So now, I'm going to basically start the web service now. And now I'm going to download the Let's Encrypt bot. There's a bot called CertBot which will automate the retrieval of the SSL certificate for you. So it's from the Electronic Frontier Foundation and it's downloaded. It's awesome. I love Let's Encrypt. If you have not heard of Let's Encrypt please, please, please. It is an electronic frontier foundation project. They are literally encrypting half the internet now I think. Or like something like 40, some like high 40% I think last I heard. Janessa Peterson, if you are out there please ping me with stats if you have them. But the idea is that you get a cert based off, you know, a number of verification methods. One of them is DNS one of them is spinning up your own local HTTP host. The DNS method is the preferred method because that's obviously hard to mangle and hack kind of deal. So you can get your own like valid publicly signed like in, you know, all the browser registries cert for free. And you just have to cycle it 90 days. Every 90 days you got to life cycle it. That's all. And they make it super easy to do that too. So like, I mean it's a lot to like wrap your mind around, you know, public key encryption but public key infrastructure I should say, but like it is worth the time and investment and investing and, you know, learning Let's Encrypt and how to use it and all your needs. Yeah, it's a great, I mean there's tons of companies that use it in proud right now. You know, so it's not like it's something brand new. I mean, it's been around for a few years as well established. It's very stable. Yeah. Very awesome project. Right. So that's the command I'm going to use to just request the cert, support auto dash dash and patchy and that will Rick of MassMed patchy install in the system. It'll request a certificate from Let's Encrypt using the ACME protocol and that comes with a challenge and the challenge would say create a web page called XYZ behind your web server and make it has ABC as content and that bot will respond to that challenge and create the website for me. So the only thing that I need to enter here is kind of my e-mail address, which is something I mean, for this case you don't necessarily have to because you're tossing the box later but yeah, I actually already registered that so I'm going to I'm going to cancel the, you know, do you want to be getting messages from us? So I'm already getting messages from them. Exactly. And they ask me, which is my domain name is quay dash then on the message.io. So I'm going to say one and it says not obtaining the certificate performing a challenge. It's an HTTP challenge, which is, you know, create a website towards you and that's what the thing is doing automatically. So what it's actually doing is creating a dot well known directory in your root, you know, path of your web server and it's creating this special file that only it knows about during the transaction and it goes and checks for the availability of that file and that's how it verifies that the server is known good place for this certificate. Right. And it already works. So it's already done and then I put my fully trusted certificate now in this location and the private key is in this location, which is an EDC, that's incorrect, right? So that's it. I have a valid fully trusted public SSL certificate now that I'm going to use for my quay instance. So I'm going to shut down that web server again because it's port is going to collide with quay. Port is also going to be port 80. And actually I'm going to copy those certificates out of that directory to my home directory. I need to do Sudo for that because you know, obviously it has been closed for a long time. Yeah, it's locked down to root 066 like any good key should be or I'm sorry 0600 as permissions for Shamad folks out there. There's a chat thread about ChangeMod all up in the chat right now. Let me just save the certificate as SSL.SERC. The private key is SSL.Key in my home directory I'm going to change the ownership of that one to central so I can download it in a second to my to my computer. So let me look out real quick on this mission and I'm going to SCP this to my download directory and I'll tell you in a second why I'm doing that. So that's the cert coming down and that's the key coming down. Because the way we are going to configure Quay is with UI and that UI is based on the web page and to this UI I'm going to upload those certificates and this key which is going to create a config file which is Quay's main config file and to make it not so boring to create by hand we have a UI for that. So let's go ahead and actually log in to Quay.io because I'm going to install the official Quay product it's behind the login as you can imagine only paying customers will use that and they will get a login. I have a login as well which I'm going to not share with you but what I'm going to do is I'm just going to paste it here to the custom portal that's where the login is stored as a paying user you already have your own login right, paste alright that didn't seem to work did I paste wrong? did you? I don't know that's where the red hat plus Quay is definitely the login that is weird you copied that out of how are you doing? I'm an idiot I'm trying to do this on my own machine obviously so I need to log back into the system oh, duh I was trying to log in as part man so yeah I don't know if I'm on Mac so I'm not sure if something happens if I enter my sudo password wrong three times so let's not do that again and then a user which is available to paying customers to get the real Quay bit if I am typing correctly I'm going to log into Quay IO with that user and that looks much better so what we are going to do is basically run the config app of Quay so I get that nice config UI let me do that sudo part man run give it a name Quay config actually it doesn't need to run as privileged because the port is high enough the port is going to be 8443 8443 I'm going to t-tatch for it and it's going to come down of the Quay image which is on Quay.io slash red hat we are going to run 3.3.0 that's the main Quay image that's where everything is I'm going to say the entry point is config that's going to launch the config app I'm going to basically set a password here because that will have a password protection and that's downloading so that continues because it has the complete Quay product in it so everything is inside that container and the way you run the different components of Quay is just launch that container multiple times of this in config directory with a different entry point the entry point doesn't need any data because it's a stateless app basically but all the others need a pointer to where the config yaml is and we are going to create that config yaml right now he says slides are uploading right now folks yeah actually Chris I'm not even sure if I deleted all those standard template slides in the back oh shit it's still in draft status so oh it's 132 slides that's why it's taking so long exactly you may want to remove those yeah let me toss those real quick whoops I used the standard redhead template and it comes with like 200 example slides yeah it does and most of us are just like oh yeah I might need one of those and we just start adding to the top of the slide back and we just leave the rest behind and forget that I delete them all the time you know what it'd be interesting to see how much Google's dry space we waste on just leaving standard templates around okay so that container has started and I'm going to switch to my browser here and I'm going to go to quay-landalone-dmeserio you can't see it right now because the address bar is hidden but I'm basically going to type this is unsafe which is something I learned from a colleague so instead of hitting that advance button and say accept the risk you can literally type this is unsafe you know one word that from on my case Brave will forward you alright so really do that again yeah we'll do it again okay okay okay like I need to see this because cockpit and chrome are not getting along right now for me so like yeah I need to get that fixed cool so I'm in the setup app now so and I'm going to basically start from scratch here so the first thing that quay needs is an empty database and you can do Postgres on my SQL I'm going to do Postgres now I quickly need to remember that IP address right so that's from the container it's 88.02 I'm going to paste that in here the quay username was quay-user the password was quay-demo-twitch and the database that I'm going to use is just be called quay now I'm going to validate and that will basically test if that container can reach the database and can access that as a user and it will fail because I need to enable an extension in Postgres that prgm extension first for quay in order to use the database so that's fairly easy because you just literally copy that command and I'm going to do one more login on my on my database whoops I think I mistyped where we are there we are I'm already in the database I'm going to paste that SQL statement create the extension pg-underscore-trgm beautiful I'm actually I need to be super user that makes sense put on your cape I'm going to be doing that as Postgres that's the super user same password I'm already in the quay database one more attempt and there we go all right switch back to config app and let's try this one more time and there it works and now it's creating a schema bunch of tables and now the second step is going to be the super user super user is like loot in quay can do everything so I'm literally going to call this guy super user it's going to be my email address not that that really matters but I'm going to basically use a web supposedly no and I'm going to create that super user all right that's the config app now the last stage is actually configuring my system here and we will basically only do the very very basic setup right so the one thing that I'll need to add here is the server name under which that instance is known that's quay-standalone the messer.io I'm going to basically say I have redhead quay handling TLS so the SSL endpoint and now you know why I downloaded these let's encrypt certificates from the server to my Mac here because I need to upload them in the config app so I'm going to do that real quick there's the SSL cert there's the SSL key and the config app will actually validate that all right the only other thing that I need to add here is redis in the redis hostname it's literally that IP so I have a back memory so I need to go go back and look at that again it's 8803 in that other redis container that we just started a minute ago and then paste it here and that's basically it right we'll configure more options in a bit in that other HA setup but effectively that's all you need in that default configuration quay will make use of local storage again this is not supported for any kind of production usage and please use object storage at all times when you want to get your data back but for this demo it's going to be fine and yeah that's basically it right you can set a bunch of other stuff yeah we'll work through this in more detail oh yeah you can set up github authentication google authenticate all the providers and everything else yeah there's all kinds of fun stuff you can do oh yeah use the default now say safe configuration changes you see it actually belated that it can connect to redis as well that it can touch the storage that the SSL certificate kind of makes sense and that's it it's going to give me a toggle on which I'm going to download and that is what we call the configuration bundle so I'm going to put that in here and that's pretty much it I'm going to close this now and stop that container because I don't need it anymore no container needed and now I'm actually ready to install quay itself right so I need to copy that config bundle there so I'm going to quickly do that quay config target oh did it save it like that apparently it did that's mech for you and I'm going to save that to my machine which is going to be simtas at quay-standalone dmeser.io and then I'm going to put it in that user's home directory if I can type quay correctly as a product I should probably be able to do that there we go uploaded the config bundle from my Mac back to the server looking here and now we are going to basically run quay so like the others it needs a couple of directories first so I'm going to create a directory where quay is going to store its configuration and then also the directory where it's actually storing the content that I'm going to push so I'm going to cp that config bundle to mount quay config go there very quick and extract it there's only three things inside it's literally the config it's literally the config YAML I'm not going to show that to you because it will contain some sensitive data but effectively it contains connection spring the host name, the SSL search the configuration and then this is the SSL search all right I'm going to remove that looking top all and I'm going to make sure that the container user which is of ID 1001 as a proper unauthenticated container should be that config directory it only needs read permissions and I'm also going to do a read write execute permissions on the storage directory because obviously it needs to store its container which is there so I could have also changed the ownership of that I used file system ACLs for that to add additional users access to an existing directory something you learn when you are doing real certifications all right now we are kind of ready let's start to run Quay it's going to be another sudo portman run command and I'm going to walk you through it step by step sudo portman run container name is going to be Quay obviously you want to restart in case it fails I'm going to publish the port 443 and port 80 that's why it needs to be a privileged container I'm going to set a kernel setting here to increase the maximum connection settings on the TCPI stack in the kernel that's coming from all official docs for something like this it would actually not need that but Quay can have a lot of connections so your system should be configured correctly it's going to be a privileged container because of the port and I'm going to mount the config directory into the container on the con stack and again notice the z for zlinux and then another volume which is going to be that storage directory and I mount it on the data storage that I'm going to detach from that and yeah run Quay no entry point means it's going to run the default entry point which is going to be our lovely Quay server so that was much faster because obviously portman we used the already cached image there it is you can also watch it start starting with Quay can take a little bit and I will complain about a couple of missing things on the configuration but none of that is really a problem but after a minute or so that Quay instance is usable nice so this in theory like how does this relate to a disconnected install what you're doing right now it's very similar but you got to bring in some images I'm assuming right literally you only need to bring in the Quay image and the Clare image if you want to run Clare as well but that's about it the thing with Clare today is that it actually needs internet connectivity to download the the signature updates and everything so that's a limitation of Clare v2 we are actually releasing a new version of Clare which is going to be Clare v4 kind of skip version 3 which was never really released Clare v4 is completely rewrite Clare v4 is much more versatile than Clare v2 and one thing that's going to bring us in October is basically disconnected connectivity so in the log I see now Quay is kind of up and running and running for its normal loops here I'm going to control C out of that and I'm going to go to my browser and open Quay standalone dmeso.io and that should look very familiar right so that's almost exactly like exactly man, don't create a count right there sign in if you already got one the whole nine yards, yeah I'm going to log in as a super user quick because I'm not going to spend much time here but effectively that's a super user I created in the config stage and password that I set there I'm going to sign into Quay and there I am I am a 10 user Quay I have Quay running, I can now actually start push and pulling images container image registry and scanning all in one there you go so that's pretty much it you have a working container registry now you can create repositories and say this is a test with this public and they have a repository there's nothing in here so I cannot pull anything you can maybe could you use scopio to like search something over real quick I don't have maybe that's fine but you know if you give me the command I can probably just do it you know no problem but don't worry about it so there's a question I would love to see how Daniel might map org and team to a repo and that's a very good question but I think in that regard you would like use github authentication and do your organizational management through github if that makes sense yeah I mean you can you can do multiple things right so we have integration with OIDC providers like github for instance or google you can do and all kind of stuff in general this just authenticates users right so it's not going to map you know your github organizations all your github organizations magically to redhead organizations in the coins you're going to do that one by one so it's actually a setting that you do on an org setting so or on an org basis so here I'm in my own organization with just one repository and basically say external logins and I can authorize stuff like github or OIDC providers and that would then map one let's say github org to one of my coi orgs and then within an organization in coi you also have teams to further subdivide access and basically limit people from accessing stuff right so it's it's not like automated where you know it's going to be captain synced with whatever you have access to in github for instance but so there's a question way back in chat I'm scrolling up to find it real quick yeah it's past here where to go where to go so I know we should ask this earlier nope where did it go I thought it was before the t-shirt discussion yeah there's a whole thing going on here um just download an example image and push that to my little quay here yeah go ahead okay here's a question does quay allow robot accounts to sign and check in images that meet certain criteria to make it more obvious and to ensure it's not changed after it's been evaluated right like I guess this would be part of a c-i pipeline right like where like I know that my static analysis my dynamic you know scanning all my stuff all my stuff has been done to it and I'm just going to let this robot account through and see if it can just upload this image for me does it allow for that I'm sure it does yeah so basically robot accounts can do anything a normal user can do so it's basically an extension of your personal login to quay and what it's what's going to happen is that you are basically going to give it permissions like you give user permissions and now it alleviates the need for you to actually share your password so in a robot account it's really just a programmatic access to quay exactly for you because it's like c-i pipelines so whatever you can do a robot account can do for you as well it just needs appropriate permissions robot account works at the organization level so robot account can access multiple organizations at once but yeah I don't know if that answers the question in the meantime I'm going to just push my example in the chair to quay and uh that did fail because I'm actually not logged into that so that'll do it yeah it's important when you change over to quay that so there's a etsy what is it containers.conf file that lists the order of registries that it's going to pull from right and making sure you're logged into quay is a very important process in switching over to quay right like quay requires a login yes you can pull from public quay image registries all day long but if you want to upload you got to log in just like you would for docker hub so put quay first put docker hub after, log into quay and off you go and I just pushed my image successfully so that works against store data um and I'll also see that image now here 171 megabytes and I can actually look at the layers of what's inside that image as well so apparently my quay is working and I think I want to leave it at that real quick demo of standing on quay because that's going to be what you do on your laptop that's not what you're going to do on a data center basis boom yeah so let's get into the big deal exactly so what are we going to do next we just built that run a bunch of containers then we had a quay after we inserted the certificate and created the config so what we're going to do next is deploy this in an H8 version which is going to be very very similar just do the last part multiple times and I'm going to simulate like a real setup like customers will have with data centers that have fair your domains and have existing database services so I'm doing this in the example of AWS but effectively you can do it all of that in any cloud environment and data center environment doesn't really matter I'm doing this for convenience here because I just ran out of data centers and I basically want to have the pre existing the real problem I ran out of data centers yeah that's such a great problem cloud real quick what we're going to do here is we're going to use AWS S3 storage an RDS back Postgres instance and then lastly cache manage Redis instance so I don't set any of that up myself and I'm not going to use quay to store stuff in disk but rather in S3 I'm going to deploy free machines in free availability zones and I'm not going to install quay on it very much like I did before then I'm going to put that behind a low balancer so that's what's going to happen as usual with the cloud stuff there's a bit of overhead so you need to create some security groups here define some subnets make sure that stuff is able to connect to each other and then put the low balancer from the certificate and all this kind of stuff so I prepared some of that already so we're not using top 2 thank you this is probably going to start right the database running already has security groups attached we have elastic cache running with the right security groups as well and we have noise free thing we have no easy 2 instances so let's get that up and running and I think the 2 pieces you set up already are the 2 that take the longest so thank you yeah definitely and you know it's really easy to do there's no value in showing this here yeah I mean it's just walking through the AWS commands so I'm going to power off this machine because I have some money alright okay so I am basically in the EU central one this is a data center in Frankfurt which is not too far from where I live and I'm going to show you that I already have an existing network here VPC that's the 10 0000 network has that VPC ID and that's what we're going to use so the first thing we want to do is create some security groups so that clients can actually reach our Quay instances so that's going to be very much similar to what you do in the data center with configuring firewalls so I'm going to give that a description here allow Quay traffic and Quay really needs only 2 ports right so I'm going to call this group D messer Quay Quay SG I'm going to use that VPC ID I saw earlier and I'm also going to give it some tags which you don't need to worry about right now because I just want to make it really easy later to delete stuff okay that's it that I'm going to it's going to create a security group with this ID and then I'm going to allow ports so Quay basically needs port 80 also rise security group ingress you can see I have already done this once as you can probably imagine so I'm going to open port AD protocol TCP and the port is 80 and it's going to allow that traffic from the entire internet this is only hard the first time right that worked it's always hard the first time I'm also going to allow HTTPS 443 and I'm going to allow 8443 which is going to be that port of the config which is only run for a little while alright that's Quay let's do Clare basically needs two ports as well going to rename this to Clare Security Group allow Clare traffic so this is this particular security group ID and I need to allow port 6060 TCP port 6060 but only internally so Clare is not going to be exposed to the internet the Postgres database and Redis that's posted to the internet leader so only Quay will communicate with Clare directly and so it's fine for the security group to restrict traffic coming from the VPCs IP address right and I'm going to put this rule against the correct security group which is that one of Clare it's port 6060 for the Clare API and it's port 6061 for the Clare health checks I realize my screen sharing session is not quite aligned with my console windows so now you see everything it's kind of spilling out yeah it's spilling out a little bit but that's about it right okay cool now the next thing I'm going to create is an S3 bucket that's where Clare is going to store its data so I'm going to create a bucket and the bucket's name is going to be the necessarily Quay demo EU it's going to be in this region so normally buckets don't have a region right but with some new stuff and AWS buckets are now localized to certain right way depending on your storage here and everything else how you want to classify it and everything yeah you can lock it down to where that data is just replicated in that region so what I'm going to do now is basically I lock it down to the Frankfurt location which is EU central one so I have good latency to that location and Clare is going to be there as well so we're going to have good latency too and we'll create that so there it is now I need to give Clare Quay access to that S3 bucket and I'm going to do this creating an AWS user account that only has permissions to that particular bucket I'm going to call it Dmeser, Quay, Demo S3 access and I'm basically going to dump that user's access key I'm going to create one and dump this in a file I'm not going to show you this one because that's kind of secret all right, there it is so I have just created an access key then I need to basically attach a policy to this user that makes the user able to access that bucket and I've already prepared that it's Jason thing, not very complicated basically say can do list buckets and put operations and bucket location on that particular bucket and everything that's inside so I'm going to assign this policy to the user first I'm going to create it I'm going to give it a name Dmeser, Quay, Demo S3 bucket and the policy document is in a file that I just showed you which is literally called IAM bucket policy.json all right, now I have that policy and the last step is to attach that policy to that user that Quay is going to use to write stuff in the S3 bucket user name is the one that I created Dmeser, Quay, Demo S3 access and the policy is actually the same name but it's requiring deep anyway it has resource identifier ARN so I'm going to just quickly copy and paste it here and now I have a user which can actually write to an S3 bucket all right, so that's what I just did we created security groups we created a bucket and we created a policy so the next thing we're going to do is create those instances and for the sake of time I'm going to actually do this with the UI because it's a little bit elaborate on the CLI yeah I have already created an instance template so I'm not going to need to put a lot of data in so here's an instance template of what a typical Quay deployment would look like it's instance type t3 a large that's two CPUs and eight gigs of memory a decent EBS and network bandwidth it's going to have one root volume 50 gigs, nothing too special doesn't need to be high performance there it's going to have it's going to get a network interface and it's going to have the ability to talk to the internet in public IP which is by default and it's going to be attached to an IAM instance profile which is a neat little trick that I usually apply that policy will allow the virtual machine to update its own DNS record nice nice nice nice DNS resolution so let's do this real quick I'm not going to create a large template I wanted to create an instance all right so it's going to be center space that instance type my key I'm going to be in that VPC that I selected, going to have that storage and now I'm going to add some resource tags here so I already prepared all of those and the resource tags are going to be used by a little script inside the machine that updates its own DNS record so it's going to be a 1A QuayDemoDemoser.io and the machine is going to be called Dmeser QuayDemo1A for that particular subnet so and the way I put machines into a certain subnet is with selecting a subnet of a certain region a certain ability zone so I'm going to say 1A and that's going to be this subnet here Dmeser QuayDemoEU1A I already created this obviously and then I'm going to make it have access to the right security groups so it's going to be it's going to be let's say Dmeser wow that's a very weird menu Dmeser QuayDemoSG so it's going to use the Quay Security Group it's going to use the Clare Security Group as well because we're going to run Clare on the same machine which is something you don't have to do but it's actually just a very easy way to get things up and running and I'm going to allow SSH traffic with an existing security group for that as well awesome and that's pretty much it the thing we'll get in public IP address by default so let's start it alright I'm going to two more times in order to get two more instances so bear with me just one second so now I'm going to change the host name this one is 1B which is the ID of the availability zone and it's going to be called Dmeser QuayDemo1B I'm going to make it have an IP address by default in the public space and I'm going to put it in that existing Dmeser QuayDemoEU1B subnet and I'm going to put it in the in the proper security groups as well so that Quay traffic is allowed clear traffic is allowed and SSH traffic is allowed as well that's it second one is under go and then one more time third time is a charm this time it's going to be 1C so that particular AWS region has three availability zones this is just a nice number you don't get into Chrome problems with this right so it's going to be Dmeser QuayDemo EU1C sorry this is a bit messy it's just the UI that's just in your name cool let's do clear and clear security groups as well as SSH traffic and then we should be called all right cool so I should have three new instances here there we go one pending yeah they should be they should be ready in a second I'm going to log into those real quick, deploy my little script that updates the three records then log out and reboot the thing so the script will run on the boot so log into the first one send those that IP address I trust this host and then I'm copying then I'm going to copy and paste a bunch of stuff so what I'm going to do here is I'm going to install unzip download the AWS CLI from the official location because it's somehow not package was centralized in the most recent version so I'm going to download a zip file from Amazon unzip that run the installer, remove the zip file and then move the script into place which I should have done before so the script is here in my local directory and it's called update route 53 that's the IP so it's going to be scp update route 53 with that IP there you go log back in let me actually do this right now for the free other machines as well while I'm here just so we don't lose too much time yep put it there and then there's a third machine and there's third availability zone and put it there as well cool all right so log back into the first one complete the instructions there which is really just moving that script into place make it executable and then reboot the whole damn machine all right I'm going to repeat the thing on the second one on the third one too this time since the script is already there it will just work in one go already prepare this third IP address in my clipboard install 808 CLI move the script in place and reboot that system too and now the third system as well and that's pretty much it so if I look at my route 52 records I think the first machine already rebooted I should have now a 1a record for that system if I refresh here whoops that is 1a query demo the meso.io mapping to the IP address that the system got from Amazon without using an elastic IP address which is kind of nice so I can give you that script afterwards if you're interested but it's just a real nice way to actually don't need to memorize those IP addresses all right cool now I'm going to install query again so I'm going to log into the first system again with this DNS name that it self assigned and I'm going to basically install portman again my favorite editor, Postgres and my curl alternative and it's going to be really useful not to what we did before on the system packages yeah actually I can speed that up a little bit because I haven't have a couple of other console windows as well there you go that's 1b here 2 and then as age into well and if that works that also means that the IP addresses have registered correctly and the machines have rebooted and are up and running which is awesome I'll install that stuff here as well okay let's continue focusing on that machine I need to log in here to query.io as well with the secret password succeeded and that actually lets me download this machine here too and I'm going to do the same thing I'm going to run the config app this config is going to do most of the work for you yeah exactly so we'll basically configure a couple of additional things compared to our initial attempt we're going to use the existing Postgres and since we're going to use the existing Redis instance we are going to enable repo mirroring which is a feature of Quay and we're also going to have security scanning so we can actually scan some of the features too so the way I will install Quay and Clare on these three machines is I'm going to create a config bundle once I'm going to download it to my machine I'm going to SCP to all those three hosts and then I'm basically going to run that same Parkland command on both of these hosts as well so that's basically all it takes to run Quay and Clare prepare directory I'm going to run the config bundle in it and run the container with that directory mounted that's in the natural how Quay works everything is inside that container inside that part and it will work nice I mean this doesn't this seems like a very thoughtful amount of work went into the install process like getting this up and running seems like the team has put a lot of work and effort into as simple as possible given the complexities of running a container or registry at scale exactly very encouraging I appreciate that I really do now we're going to show you that trick again I'm going to launch the config app now I already figured out the trick I googled it just so folks understand Chrome by the fault is doing the thing where it's like I can let you into a secure an insecure site just forget it this certificate is not good so for me locally here on my instance where I just spun up a server last night I want to log into cockpit and that is a self-signed cert it's not going to let me work so essentially what I have to do is this I log in with 9090 as I normally would and then I add pound sign or hashtag and then I can safely treat insecure origin as secure and that will allow you to do that so yeah you can now before I was using a developer edition of firefox I was the only thing that wasn't allowing me through because I didn't realize this was a freaking option because I'm not a browser genius just to get people not lose people and all those commands and you know that's where we are right now we have the storage, we have the database we have the cache, we have the virtual machines we have the correct security groups in the correct subnets and now we're going to install quay and I'm basically running that setup routine again with that one quay config part on the first machine set it post-pres again only that this time I'm going to paste the connection endpoint of that AWS RDS instance and that already has a user that already has a password as well and that will get a database too so I'm going to run on the same issue that I ran before that we need to enable that extension that's why I'm going to circle back real quick to my system here to log in to that guy and enable that extension so let me just log in here that's the connection endpoint the user is post-dressed and the database is called quay and the password is one that I remember and I'm going to copy and paste that post-dressed command in there that created the extension look out again and now I'm able to proceed here I'm going to set up a super user next out of HABIT it's going to be called super user it's going to have my email address it's going to have my password and we are back in the config game cool so now I need to pay a little bit of attention because the server host name is not going to be 1a because we want to do a AHA setup so remember we want to basically go to this picture here where there is one open answer across all of those instances cool so I already have the name quaydemo.dimesser.io and that's in route 53 domain that's actually where the host name is now going to as well and I also have a certificate by AHA as a certificate management and that's going to be attached to the load balancer so quay itself actually doesn't need to handle any certificates in this case and I'm going to select my own load balancer handle stls which is not recommended and I'll tell you why it's not recommended it's because you put it behind the load balancer like that the traffic between the load balancer and quay will be unencrypted that's one thing so I'm just lazy here the second thing is if you are using an hdbs load balancer quay in its logs will always have the IP of those load balancers and as the client IP who just pushed the container so in our production setup you wouldn't do this, you would actually make a network load balancer and you would do SSL pass through but for the sake of simplicity we are not going to do this here a couple of interesting settings that exist here as well data consistency I can make it so that repository pulls are allowed even if audit logging fails so if audits are not available by default quay will not work which is good for enterprise environments maybe not so good for you so that's why this is configurable Redis is also running already it's provided by AWS I have this endpoint here that's a free now Redis cluster it's probably oversized for this but it's going to be okay again Redis is behind it's not on the internet as well it's just purely internal reachable from those subnets and this is done in port now we're going to implement and enable repository mirroring and I'm going to show you that instead quay instance is running and repository mirroring will make quay mirror from any kind of other container OCI docker v22 compatible registry as long as it's OCI combined you're good exactly and we want to make sure that the mirroring process uses SSL and verify certificates in the process now it becomes interesting now I'm going to basically enable s-free storage here instead of local amount of directory I'm going to say Amazon s-free and I'm going to give it the credentials from that AWS s-free user that I created earlier where I extracted that jason right so I'm basically saying the bucket name is dmesher quay demo EU that is what I used before when I created that bucket I'm going to paste my access key here is this hidden or not that's not hidden but next one is hidden so okay thank god I didn't want you to paste your key live in case it was something like important right like it's my worst fear that we somehow capture somebody like using their real good key and I'm like nope you got a life cycle that now and that's why we created the user specifically for this right so special quay user now on AWS that I created before just for the purpose of quay accessing s-free accessing that particular bucket right so if you switch on the console again um I'm gonna quickly look at the policy so that policy only allows that user that I just pasted credentials in the UI to access this particular bucket so you know blast radius is fairly limited already but again the config bundle contains sensitive data right so don't hear that and also pay attention to the s-free host so the default is s-free amazon AWS com nowadays all these s-free buckets have location constraints and that means you need to actually put the region in the URL as well otherwise this won't if I don't do this config I will complain because it thankfully validates all of my config file oh good cool so action lock storage that's where all the auditing data goes right so that's going to by default be in its own postgres database we can also use an external elastic search instance for that um we can enable lock rotation and we are going to also enable security scanning because we want to show clear right mm-hmm so that's going to create a key which clear is going to use to authenticate the quay it's not a problem that I show this key here because again that clear instance isn't on the internet you know you don't have any access to it anyway I'm going to just save that here externally in an editor also the private key because I need to upload that later on as well it will ask me to download that pen file and I will do that um all right um cool close that I'm not going to enable application registry we can talk about that in a minute when we have some time when something is you know processing and we are waiting I can basically also make it validate emails I'm not going to do that here because that's just a test environment authentication you can you can use elder keystone any external JWT or ADC compatible application um I'm going to use a local database and that's where you integrate with github so you can make quay integrate with your github enterprise github enterprise account or google for authentication purposes and then there's a bunch of additional stuff here I can allow external applications allow this allow anonymous access we definitely want to allow that because we're going to use that later and we can also restrict v1 push support so docker v1 is still supported with quay you can still use that with old clients and it'll basically still work right but you can restrict that support just for a couple of namespace which is enabled here I'm going to leave that as a default so I'm missing one particular entry here which is the security scan endpoint that's why my Claire will live and I'm already pre-filling this here with the domain name of the Claire load balancer that's going to be an internal load balancer it's going to be behind Claire demo at the Messer IO on the port 6060 that's the port that you put in a couple of minutes ago in the security now I'm ready everything checks out everything is regional that's why it's important to run the quay configure where quay is running right because it will test connectivity from there I'm going to do next I'm going to download the bundle and it's going to call it quay config app yeah what yeah tartar cheesy yeah interesting so I need to pay attention that I'm not mixing up the names here okay because I already have a couple of files right that are named like this naming things are hard yep naming things is super hard okay cool so that is almost ready to go now can close this go back to the CLI and log into the system again the quay config container and what I'm going to do now is quickly log in to the same Postgres database and create a database for Claire so Claire also uses Postgres and normally Claire would also use its own Postgres instance but again I am lazy so I'm sharing an instance this is not why not 30s but it's going to at least have its own database right so exactly there will be some separation yeah oh there it is already so a previous run crop that just to be on the same side created again so now that's all you need you just need to create the clear like is the user you're just going to reuse and everything yeah I'll show you that in a second so Claire is also a config file right so oh that's right Claire is config file and we will basically now look at that config file we have a example on the web page already and it's something I will download that you can download I already prepared that config file the only thing you need to do is actually update the domain names of where Claire and quay are going to be vitally important yeah I'm accessible so I'm going to paste this here and we can run through it really quick so you basically configure the database right ports and stuff is already correct you basically say where is my quay because Claire needs to talk to quay to report security vulnerability and scanning results then you also need to Claire where it's running behind which domain which is going to be Claire demo D measured 6060 and it also needs to authenticate with quay in some form of fashion and it's doing a key exchange in this aspect and that's the key I downloaded earlier right that pen file that came down from the config app that's the key which will be used by Claire to authenticate with quay so that is already there and that's already working so let me just check one more thing which is going to be the endpoint of the Postgres database to make sure that hasn't changed because it's always a long name and there it is it's still the same name again I don't mind sharing the password because database is offline okay alright so that's it basically that's all the configuration you need to do now you just need to upload that so I'm going to unpack the existing configuration here that's that quay config file that's just one file because we didn't generate security keys or that's handled by law balancer right so they're going to scp that to my host cool so I got some questions in chat if you want to tackle them real quick how often is quay released like what's release cycle look like I guess is the first question yeah we try to release every every three months we haven't been doing that right now because we were actually in a big rewrite process of quay so quay is written in python and we just are transitioning from python v2 to python v3 and as you can imagine we're such a big code base that's a non-trivial undertaking so basically that's essentially what has kept us busy so far and this is essentially what we're going to release in quay v3 4 which is going to be in October and then hopefully we can get back to that in about three months nice and then the next question was does the quay container itself still need root or is it rootless now it's rootless now again this is just me being very very lazy overly efficient awesome place right so it's the same thing as before I uploaded them with scp and there's no clear config yaml I'm actually missing my so I already moved that config yaml out of there so there's a clear config yaml and there's the security scanner one thing I just forgot is that I actually need to update well do I yes I do there's one thing I need to update in that clear config which is the key id so that is something that associates that pan file with that key so that is oops I should probably do that correctly do this on all three notes now with with my term so I should probably do this I on all three notes so like once everything is set up you're just updating the image that's a question in chat unless there's some change that would be documented that you need to make in your existing configuration it is literally just updating the image exactly so it's updating the image but also database schema yeah that sometimes needs to change too in an AJ environment like the one we are going to deploy I'm just going to paste the correct key here that I was given by the config app in an HA deploy you would scale it down to one and you would update the image and with that there is a little migration script running on the start of the new image that migrates the database schema once that's complete and Quay starts up you can basically restore all your other instances again with the new image version but the database schema migration needs to run in a serial fashion and that's why you shouldn't actually you should scale it down before updating it okay so now I updated the config YAML for clear with the correct key ID this is how Quay knows an authenticated instance can talk to it and pull images because it will pull all images right and I moved those config files in the right location so for clear we have another config mount which is mount clear config that's where the config YAML is that's where the security key is and like before we have a config mount for Quay that's why it's config bundle is and that's pretty much it I can now run clear I want to do that first and I need to make sure I'm locked in on all nodes so let me just repeat that process here one more time for love hotman login dash U dash U which is going to be that user red hat Quay and then that's Quay.io Poppin is not installed on this node you see that's why just the one though yeah it's just a one screenshot we did that on everything yeah me too maybe we missed that on that it was one box you know I'm wondering so it's 10 to 59 so it should be the correct subnet and it's that one B so yeah it's the same system the other one is called 177 and 39 in the end yeah and 177 39 so careful with those broadcast commands right yeah right just mentioned in chat Ansible for the win you can totally system configuration process and install cloud services and everything right like that's kind of the purpose of Ansible the container or not container orchestrating tool but the configuration management and setup capabilities of Ansible all right there so yeah you wouldn't learn as much you would learn all about like Ansible modules and how to interface with them you wouldn't learn the guts of setting this all up okay now I'm typing well I'm not typing on this machine though it's just really really weird it's are we going to use an iterm or yeah I'm using iterm my broadcast see it's only that it should actually broadcast okay I'm not seen okay so now poor man's automation solution super portman login we're going to authenticate on all nodes so that we are actually able to pull down that image and that image is going to be clear first and it's okay and now we're going to launch clear clear is a different image and really easy to start basically say publish port 6060 that's the main api that's the health check and then you launch it from that image and you also mount the config right and while that is working I'm going to basically create a lot of load balancer for that because these are free independent instances clear is also going to be stateless and I'm going to create a load balancer for clear so it's going to be an hdbs load balancer it's going to be internal it's going to be called the measure way more clear it's going to basically proxy hdbs and it is going to listen on port 6060 that's the clear port that we use to talk to clear so the load balancer is going to be available or free of my subnets one A one B and one C and then it's going to need a certificate and I already created a certificate as you can imagine which is really nice it's a service in AWS that lets you use something very similar to what we did with let's encrypt before only usable within other AWS services but I already created an SSL certificate that is publicly validated so that load balancer will have an SSL certificate that's valid it will also have a security group which is going to be the one of clear right traffic and then I'm going to point that to that load balancer to my free clear instances so it's going to be the measure way clear targets and I'm going to point it to instances in a second the load balancer will talk to clear with HTTP so it's not encrypted and it's going to be port 6060 as well and there's going to be a health check which is going to happen via HTTP as well on the endpoint health but it's going to use a different port it's going to use port 6061 so there's a different endpoint in clear so the health checks and that's the clear load balancer I'm going to add these three instances and if clear is running happily I should see three valid targets in that target group in a second so at one point this will basically turn green and we can peek at the logs of clear so it's going to be quite quick to see if it's happy with what it's found maybe it did a typo in the config YAML or something but that actually looks quite okay everything started no error messages when clear has a problem it actually fails pretty pretty soon so that's going to be very obvious so that's basically what it takes to run clear and we're going to go and run quay and we already know how that works so I'm going to run quay we have podman as well same settings as before and it's very fast in the first node the two other nodes actually need to download that image again and in the meantime I'm going to create another load balancer which is the run for quay so create load balancer HTTP demaster quay demo quay lb it's going to be an internet-facing load balancer this time because obviously we want to have clients be able to push and pull that's going to listen on HTTPS the rest is pretty much the same as before so it's going to be available for instances in all my free subnets I'm going to use the quay demo certificate which I created already again this is also a pre-existing certificate that I created before and as a security group I'm basically selecting the quay security group that we created before which is going to allow traffic on HTTPS another target group just for quay quay will basically serve the HTTPS of course the health check is going to be via HTTP because quay itself is not handling as itself our load balancer and the endpoint by which the load balancer determines if it should actually forward traffic to that quay instance it's going to be called forward slash health forward slash instance and the rest is fine so the traffic port is going to be port 80 but it's actually also the traffic port so let's register those free virtual machines that we created hit create and that created the load balancer so let's see if my clear targets are happy they are success so Claire is happy up and running and the load balancer is forwarding traffic now there's going to be another group here that doesn't have targets registered yet so that is apparently still configuring its status normally it's pretty fast I'm not quite sure what's happening here but maybe the load balancer load balancer is still provisioning so this will populate in a second and when that is all green we have basically a highly available Claire and quay instance so the only thing that I actually need to do now is create the nest records for those elastic load balancers so we create one for Claire so it's going to say create record and I'm going to define a simple record which is the Claire demo DMSI forwarding traffic to our load balancer in our Frankfurt region for our Claire load balancer which is this one and that's it I'm going to do the same for my quay D mode domain define a record the quay instance is going to be behind quay D mode DMSI.io and it's basically going to forward traffic to my load balancer in Frankfurt which is DMSI quay demo quay and and that's pretty much it so what we did so far is basically we created the instance with security groups and subnets we deployed Claire on made sure it's healthy we deployed a quay load balancer and probably at health checks and we deployed quay and a public load balancer for quay with health checks as well beautiful so let's see how our favorite product is doing ordinary logs as a quay and it looks pretty happy to me yeah you can't help but I can tell if it would complain executed successfully is usually a good location this is obviously going to be the target provide so if that thing is going to be healthy it's not so is that love and is it still good isn't it no it's active so let's see didn't reduce the target well maybe I need to do it again you're you know it's all api calls behind there you never really know what's happening I might just not get out of my place I guess yeah maybe but once that's coming up back green I will essentially go to quay-demo at the meso.io and see if that is working so that's green it's happy it's still a big gateway so I think that the glow penance is just a little bit behind good old AWS everybody's demoing on a friday too makes it slower could also be cashed oh yeah it's very fresh not yet still healthy yeah but actually you know I think um 480 yeah it should be 480 targets monitoring I think I did that wrong so this is not correct basically okay so I need to do register basically I don't think I can edit that anymore um yeah I did a typo somewhere so let me just do this guy one more time hey if you're not breaking stuff building stuff you're doing something wrong right you gotta crack a couple legs to make an omelet yeah interesting it's forwarding traffic um to 440 that's not correct okay then why don't we delete you um goodbye we create you a quick so this time it's going to be um correct so going forward it's going to be an hdps listener it's going to be 440 it's going to go in these three subnets I wonder at which stage I did the mistake um but I'll probably never find out but uh you can watch it in the video if you want thank you so I'm going to use the right certificate I'm going to use the right um security group probably the routing so yeah well okay so it's team lesser kway demo kway targets it's 480 and the health check is also on 480 and the health check goes to the instance and the health instance endpoint so we just saw those targets I probably forgot to click this blue button here and there it's now saying 480 so there we go okay um I need to update that um route 53 record of course because that's now updated um yeah and we'll do that while the low balancer gets its act together um and we'll hopefully start serving traffic but um from an architectural perspective this is probably not too difficult right so you put up three instances to share the same configuration the instances themselves outside of the configuration are stateless um they access the same database and um you can add as many as you want um behind the low balancer um and um so there's no well yeah it's not a cd it's red as or not red as postgres yeah so yeah it's a bit of a bottleneck at postgres level right um so um that's usually where our problems are appearing so oh and that's a good forwarding rules hmm still hmm weird well we have had shows that have ended incomplete failure before we will probably end incomplete as well but I think um now it has poured 80 here so that this makes me more hopeful yeah target registration and progress I think once that happens you might be good yeah exactly I mean the next step would be to make this um go with georepo we are out of time but I'm gonna quickly while this is registering show you what this would look like so um that is the end state right of locally of a local region right with multiple failure demands so that could be a data center um it would probably love balance so um georeplicated quay is gonna work like this um you're gonna have um quay instances that are very far apart and they are going to have access to the same database the same red as but to different storage back ends and quay is gonna be responsible for replicating content between those storage back ends right um so um the way you deploy this is with something like this and that's what we have what we would have done here in this uh call um we would basically have the exact same setup in the US these two so really far away right yeah except for RDS and Redis which would obviously still reside in the EU but the latency to Postgres and RDS isn't that much of a problem was actually causing slowdown as bad latency to the storage and that's why we have S3 in the US these two region and we're gonna have um pods with quayer and quay running there as well they're gonna be configured with access credentials for both S3 buckets basically um so and there will be workers starting inside those pods that are starting to replicate um storage from left to right and from right to left depending on the origin and then these instances are configured to prefer their local storage so whenever we are um answering requests um it will basically um look does the object already so it's gonna be a char or char block basically does the block exist already in that bucket if not I fetch it from the remote bucket if it does exist I'm gonna um fetch it from that bucket and serve it locally the way clear a quay source content is basically um handing out pre-signed URLs that are going to end in my case on quay demode but it's gonna actually redirect to the exact HPP URL to that bucket with that block and it's gonna be in this case for clients in the US gonna be that US these two bucket for clients in northern Europe it's gonna be the EU central bucket right so you as a client don't see the difference between you know quay right here and quay right there it's all one stretch instance but you're always gonna get storage at superb speed speed cost they're coming from your from your local S3 region okay so that thing registered now um so shall we try again? yes let's give it a stab well how about that there you go there you go so um that's our highly available um quay installation with quay with S3 backing with RDS fully load balance I mean this is cool man I'm glad you did the stream today yeah should we maybe um end on one interesting thing which is gonna be the repro-marrying with security scanning should we maybe do that okay go ahead let's create a quick repository um because we enabled repro-marrying security scanning so let's actually prove that this works make it public and I'm going to configure this repository to be a mirror of a different repository and I'm gonna do that in the settings I'm gonna say repository state is not normal and also not read only it's gonna be mirror that means you cannot push to this repository it'll always be filled by quay with content from a remote repository cool I will basically configure that to be um and that's gonna be a little bit of an inception right I'm gonna basically do I'm gonna mirror quay itself to quay so I'm gonna mirror the upstream image which is behind quay.io project quay slash quay yeah I can keep the latest local image of the project and just update it as you need it because it's already there exactly so there's an image behind that that's the upstream up coping source version of quay I'm gonna just mirror one tag which is quay on so you can see how the engineering team is kind of into star wars um but you can um you know set sync intervals for mirrors um a user is gonna be um a robot account doing that mirror for you so you're not gonna share your own um credentials I'm just gonna create a robot account here uh that's gonna be used for that um that repository is public anyway so there's no so it's gonna be fine gonna enable mirror I'm gonna start the mirror now and you can see in the logs that this is actually going to start to work and um it would work if I had done one thing which is actually starting the mirroring workers so uh yeah well let's get easy because you know it's it's it's all just one more container right so we can start mirroring workers as well by saying podman run and again the quay image by the different end point which is gonna be mirror and it's gonna fetch from the same config so everything else is gonna be the same so that will start mirroring workers and the task of these workers is to look at a prosperous database look at outstanding mirroring tasks and start to mirror content from the mode registry to us wonderful wonderful I'm gonna start that uh one more time um now that the mirror workers are actually up think in progress yeah so I'm gonna cancel it though so um it's gonna start fresh and um now I basically see that um mirroring has been scheduled right um and in a couple of seconds we will see here additional log entries to tell us about that content is starting to get mirrored that content will end up being in this repository um so you can on a per repository basis mirror content from anything that's out there and the nice thing is that because we deployed Claire it will also scan that content right um so you could mirror from anywhere and it's gonna get scanned no matter what exactly this is this is exactly how customers are essentially getting untrusted content into their you know trusted data center environments um with uh um uh with uh with quay right they're going to basically um uh uh to get it in and then they're going to scan it uh with with Claire and um you see mirroring finished successfully so we now have a new tag which is that quite one tag and security scanning has been huge so um in in one minute or so um we don't need to do this but in one minute or so that will out I mean if you have the seconds to wait for it we can yeah unless you gotta jump um I'm just gonna switch back here to Claire real quick um um and see if Claire's still happy now that it's actually going to do some work for us um but uh yeah it looks exactly seems like it's fun some stuff yeah it's definitely uh communicating and um yeah it's 577 I didn't see how big it was I mean it's not going to take long but what Claire's going to do is it's going to um see that it didn't scan certain images yet so um it will actually um download that image um unpack it on disk and run security scans on it right and you know it has a vast amount of um security a list of security vulnerabilities that it's going to identify and um that is um why um you should not underestimate how many resources Claire will be using in a very busy environment so that's why Claire's a separate part a separate container and you can totally run it on a separate host and scale it independently of of quay um as well so um yeah uh yeah so you could have 20 Claire scanners and just three Claire nodes or Quay nodes yeah yeah yeah you would totally make that something that auto scales right um so that you don't you know waste um uh resource yeah obviously but yeah if you're if you're group of Claire scanners are all pegged out at 80% you might want to spin up another one right you see here Claire's actually starting to peck to see view of the system right so it's um running on all nodes here here and the second nodes actually oh yeah wow so it can get fairly resource intensive but it's also doing important work right it's extracting all your content so it recognizes are these rpms installed in the base image or dvm packages or alpine stuff and it's going to scan against all of that um so that's basically what Claire is doing all day long and it does that whenever a new image arrives right um it starts it yeah and and it'll index the content of that and it'll basically um look at that index content um over time as security vulnerabilities are published and um unfold right so even if this image has been scanned already um Claire will update you when there's a new cv that came out so that's why Claire is a two-way street right um quay tells Claire when there's an image and Claire tells quay when it receives new data about an existing already scanned image right so um there it is security scan past um beautiful that is actually nice and it's also important because you know we shouldn't have security vulnerabilities in our stuff oh there are two well that's interesting huh just but there it is oh there we go okay it's fixable yeah it's fixable so fixable basically means you just need to update that base image so in this case um the vulnerability seems to be the debuff package that comes from that rail um base image so I guess the engineering team has to do some work here to actually update uh that image to the newest version so we can um you know we can do this with all of our abilities but um I mean but that's a simple clean thing right like that's just you know update the base image to the latest version exactly that's it cool um this is great this is awesome Daniel thank you so much and again you can create your own quay.io account right now just hit up quay.io create an account and off you go and you'll have quay and clear scanning right there for free you know essentially exactly cool so I hope um this was somehow something one could follow so if I would just repeat that um we had basically you know existing databases existing memory caches we created buckets and we created the user that has access to the buckets we created free machines and free azs um we deployed clear um we deployed um a load balancer internally for clear so that um you know you can scale this up and down depending on the load as you've seen could be considerable load and then we deployed quay in the same way we deployed it on the scene machine only that we copied the config over to the two remaining nodes and put all of that behind the load balancer as well and the result is what you have just seen that's a fully functional fully scalable full tolerant enterprise great registry that scans your content happily as it comes in and um that sim job that I just started um that's going to run every 12 hours so it will be up to date um with the stuff as it is released in remote registries as well beautiful um thanks for watching thanks for having me thank no thank you Daniel really appreciate it we look to have you on in the future sometime right like that'd be really cool so I'll go back through the chat and see we can come up with a couple other extra topics for you maybe uh but just as the stream has finished the local uh baby turkey flock has arrived so it it must be time to go because the turkeys are here if you follow my twitter stream you'll understand um they're a very interesting bird looking uh and their youth rather ragged anyways um so thank you all for joining us today appreciate it you can always catch this on twitch for the next 60 days and then after that it will be arch it well right now it is archived on our youtube channel um and when in doubt just the the dog just saw the turkeys if you heard that my bad um if if you are looking for content that was on twitch previously uh just head up our youtube channel and then be sure to stick around for later today so we can talk on open shift commons about uh devops anti-patterns with kevin bearer and author of the phoenix project and if you have never read the phoenix project please go get yourself a copy of it um and with that thank you very much daniel and we will see y'all soon in less than an hour thank you very much thanks chris thanks everyone have a nice day