 Good morning. Good afternoon. Good evening and welcome to another edition of the open shift. Oh my gosh It's open shift container storage office hours. It's Thursday. It has been a long week for me. I'm sorry today. We're talking about a Auto scaling that multi-cloud gateway. So if you remember from a couple weeks ago We were working on my cluster here at home trying to get a quay installed We're just gonna skip over that part and assume quay is installed And kind of push forward from there in the experiment. I did get quay installed I had to remove the operator clean up some stuff Reinstall the quay operator and everything worked fine but Today we're going to be using a cluster that we can really kind of just beat up and not have any issues with and Michelle, please. I'm sorry to sorry to kind of run through your whole spiel But Michelle, please introduce yourself. Please like tell us what exactly we're doing today Okay, okay. So Michelle Dupama here again I'm from the cloud storage and data services BU, etc, etc We're going to pick up where we left off last week Except not on Chris's cluster. We're going to use my dear teammate Chris Chris Blum's lab out in the middle of Actually, I don't even know where it is But it's VMware lab and it works and it's it's not quite as under resourced As your lab is so yes right now everything is set up everything is actually already set up So I was I was wondering if we wanted to start fresh except that It takes too long It takes too long to serve fresh and as we go through the setup I'll show you where things take a long time and why I wanted to start with a working product So yeah, and to to that point right you remember the last show it literally took us the entire show just to get to You know running the Koi registry and you know, that's because of stuff on my cluster not being fast, so We're not going to worry about that anymore. We're not going to worry about that So so okay, so the remember we were what I wanted to do, but I thought was going to be simple to do Was yeah exactly to to watch the nuba end point scale With you know the horizontal pod scaling so I wanted to see it. I wanted to create an image storm Using we have the Koi registry and the idea is the registry part of quite not postgres or whatever I think it's postgres or red is whatever it's doing. Anyway, not that part the registry Yeah, the the registry part actually talks to the nuba endpoint and uses obc It uses dynamic object market claims so that's kind of That part works and it's can and I wanted to hit that endpoint and then just watch it scale up so That we can look at that now We can start at the end after I like let's go through the setup and you could like just show Everybody how it's kind of set up and then we can do it and then you I need help actually Bringing this making it have impact like the utilization. It's just like I can't get it to move So so I would like some help with that. So Do you want to share your screen and I'll go through so of course you want me to share first a terminal Console, what do you want? No, let's do console So I can just show because I want to talk about some of the gotchas the things that I learned doing this process Find that tab and all the wonderful screen sharedness I just want to throw out there. OCS was the easy part. That was the easy part of this I'm sure a registry person would say the opposite but for me that was like, oh, I can't believe I have to mess with this registry No problem. Yeah. Um, all right, let's see which I don't want to share desktop Do you have like a ton of tabs I have so many tabs Do you want me to send it to you again? No, no, no, no. I've got it. I just need to find it All right, so while you're looking Wait standard setup right like this advanced. No, I want a nope. That's not files. Nope. That's All right, so I don't want to share this little desktop. I just want to share a I'll share the right Browser, that's the problem. I think this is okay Wait, let me actually switch tabs to the console. Hopefully that would help. Okay Are you let me know when you're sharing this is screen sharing song folks There it is All right, all right, let's see Okay, perfect. Cool You see what I got? I'm in the open shift image registry. I will go to open shift storage That's not my console. Is it? I don't think that's my console Your domain. Let me check. Hey, hey, hey No, no, no, no, no. This is mine. Sorry my bad Uh, I believe nope, that's recent Did I share the wrong thing I did I shared the wrong screen my bad No problem. Oh good. Don't worry These bits here. Why is it so slow open? Why are you waiting? Well, did you you're on the global VPN you tunneled in? Okay, just checking Just checking I've had my coffee. There you go. Okay, so it has not had its coffee yet, but yeah Okay, so many clusters a little time. All right, so I know so latest and greatest, uh Open shift for 617. Yeah Yeah So if you just just to show everyone if you click on the persistent storage tab and the object service tab underneath overview Just everything set up I regularly forget to click the when you install ocs and you have to click the monitoring button I regularly forget to do that. You really have to Yeah, because none of this works if you don't do it right because you don't monitor it yet It is a mess. So ocs is up and happy object services there. Um You can if you want to click on the A multi cloud gateway. You don't have to but I do I always go because you know, uh kind of an mcg person I'm sharing the whole whole browser window. Yeah, go ahead and log in and with same creds Let's see what it should ask you do permission and stuff. Come on. You can do it. I believe So I will say a friend to everyone there. There There is a little bit that's in limbo between mcg and getting things back into the dashboard in Open shift, so you will see some that yeah, just be aware. This is it's new But everything functions, but sometimes you don't see the updates the way you should so When you say that You don't see the updates and what exactly? Okay, so I'll show you I should so here's a in fact I have to raise an issue for this so back to the web console Back to the open shift console. Sorry. Yeah, okay, so this is up and if you go to operators installed operators I would just want to show you the registry Okay Hang on All right click in there when So a few few gotchas when you yep keep going you can drill down into it It's things that I just learned that I didn't know maybe someone else. No, this is good. I like it Okay, so when I went to install this the normal way through the operator hub um, I had to ignore the dock that came the deployment to So I'm I'll raise the issue for that. No worries. So, um I turned off if I had to do it in the open shift storage namespace I couldn't do you know when you install an operator and says all namespaces, right? It for clay that's the default So you I think it yeah, I said I and I did that and it didn't work I didn't troubleshoot because I that wasn't my goal I was like, let me just install it in the open shift storage namespace and voila. It worked in addition When you go to install it when you when you bring up the registry you have a bunch of choices about what components you want. So I turned off mirroring It's only one cluster. Yeah, it's and and I also thought um, I could have turned off claire I could have turned out so I could have done more. I didn't I know I like claire But I did turn off mirroring. I I just said, okay. Let's let's just bring it down a little bit I don't need all these extra things. So I turn it off. Yeah, you don't need to mirror to some other node or I'll stream quite a diode type thing So, um, the other thing I wanted to mention and this is something particular to my labs Like when I went into this lab the first thing I have to do before I install is yes is I have to scale up Right, I have to add more more Nodes that can house ocs does not compact. I didn't do a compact install. I did a real install of ocs But I had to add one more for this registry. So that was Yeah, I got stuck in pending mode. It's like I don't have enough CPUs on memory. Everything like that. I'm like, okay, fine So I scaled up. So that's something to be aware of in the beginning Right, if you're scaling up for ocs add another one for this registry, right? Okay, so if you click on example registry the registry endpoint right there, that's you see it on Yeah Hang on. You can find it. There he is So I I did the usual I created an account and I uh, I pushed an image This the next gotcha and feel free to create an account if you want or I think did I send you my um You're not have a query account No, well not for this is local. This is yeah, not for this cluster. So Log in as mdapama hang on. I'll send it to you if I haven't already. Can I send it to you? I should have shame on me Uh, try this Okay there Just so you can see I put in a I think that's it I can copy and paste it right if I can remember anything from yes. Oh, there we go. Okay. Cool. You got a repo here I got a repo. I put in if I put in um I put in the awscli image because I want to do something with bursting for next time. Um, and then if you go to tags I think I have the Yeah, I have the ubi ubi image and then I wanted something beefier. So I was like, okay. What's a big fat image Oh, I know so latest Yeah, like I like I'm like let's put some weight in this so so anyway So the registry setup Here's your first gotcha on the registry side. Um, this is uh, I didn't use let's encrypt for this cluster I don't even know I didn't get into it like I so fine. Yeah, all right except that this registry is insecure Yes, I did notice that So so then the next gotcha is I have to modify um The image config for the cluster to accept it in secure registry Okay Did you know I I forgot like I I knew that but I forgot so so if you go to the command line go to a terminal window and Share the terminal window. Yes, please I want you to see This is what I had to do I'll I'll send this to you have to go in so all these little it was like, oh my god all these little gotchas All right, if you do if you Hang on Hang on I gotta find it first the joys of screen sharing All right, there it is. Sorry. I'm not really sure that all right. Yes. Okay. So oc edit and you'll see um Yes, it's hang on right oc edit image.config There's more dot open shift that I owe Slash cluster that's it. Yep Do you need a dash? Do we have to tell it yamla or anything? I don't think so. Let's see what it says There we go. Okay. There we go. Okay. So notice at the bottom when when on a fresh cluster under spec There's nothing Should be clean. So it should be clean. So I have to tell it Allow my my registry to be insecure But the minute you tell it that it starts like You know allow it to be insecure and and then you see underneath allowed registries it blocks everything else So I had to go Yeah, so if you add it as insecure it blocks everything else. So it's like insecure or nothing So I put insecure and then allowed registries So and I just had my little you know example registry blah, blah, blah, blah, and then I I exit it out And then this in turn has to go and reboot all of the nodes like one or two at a time because it's got a restart Yeah, yeah, so this is why I didn't want to do this live. I'm like, it's gonna take it did take a long time So when you actually edit so then I'm I'm sitting there. I'm watching pods fail and I'm like what's going on Why are things failing? I just did this little this is like a not a big deal tiny little thing. Yeah Yeah So in small prints in in one of the deployment docs it says, you know Oh, by the way, when you add an insecure registry and you have allowed registries You have to put in The red hat registry. Otherwise it by default will block everything Interesting Yes, so I will I can find that link for you if you want to send it out to people But it's I was like this needs to be like in big bold letters, right? Exactly Yeah, that like so our docs are great, right our docs team is there but some of the Some of the styling of the pages. Yeah, it could be Like we could break it out into like a gray box with the huge like red stop sign kind of symbol Remember and to be fair to be fair when I read it. I'm just skimming it because I'm like, I know this I'm not really reading it. The curse of knowledge. Yeah, exactly. So I'm like, they did put it there It does have a little box around it. Like I'm just not actually reading it So just so here's I just wanted to point it out as a gotcha, right so far We've had to the registry has got to be installed in the namespace the open shift storage namespace When you if it's insecure, which if you're lazy like me it will be you have to go in and modify this Cluster image config Be aware that when you start when you say allowed registries It means you really have to specify all of them including the other ones that you need for your very base core images So that's all I just wanted to make sure everybody was aware of that. So these are some of the gotchas So I don't exit out Don't change anything because then everything starts rebooting. I'm not I'm not going to but I will share how to set up registries And or how to set up not registries how to come on copy link. There we go Uh, this is a blog post for getting what's on your open shift cluster. So you go through this blog post And not have to do this Exactly that would be that would be the way to go You just add your one allowed registry and you know, you're done. It's life goes on and running. Yeah often writing Okay, so once this was all done, sorry Yes, and go back to the console And I pushed those images that you saw then I said, okay, let's look at the um, the horizontal pod bottle scaling thing So oc gets hba You can actually you're in the sorry. I'm my bad if you could go back to the terminal You can see it two ways, but I like to see it on the command line. I did go back to your terminal Sorry, yeah, no, it's okay I like how it moves things around based off how you used it. That's at least useful All right, here we go. Okay, so oc get hba. You're in the open shift storage You see it over there on the far right? Oh, yeah. Yeah. Okay. Cool. Yep. Just do that You should see two things the registry and the nuba endpoint So normally so if you just look at the nuba endpoint, that's that 3% that's normally 80 I took it down Okay, so um Yeah, so because I'm trying to do some so if you want to edit it we can uh, if you did um oc edit hba slash nuba endpoint, but we can also just try and start stuff So, you know, remember the so I sent you the image storm Yeah, yeah, do you want to show that just so people can see this is really simple like I'm not trying to do It's super simple and I think this is where I can help you as far as like Getting which getting to the goal of the show, right? So you are spending up here your deployment of a thousand replicas of a sleep Command right which essentially uses no cpu by the way Okay, so but wait so so my thought was this I want to I want this the nuba endpoint to use the cpu So it's the pulling it should be like the I've contacted the registry the registry has got to go figure it out So that's the in terms of the pipeline. It's the it's between Uh quay and the nuba endpoint that I'm trying to get some activity. So tell me is this so that's I thought A larger Okay, so maybe my thing I'm also like, okay, so I thought let me get a nice fat image I'll go out and get like sense or something like I'll get something beefier that requires Time to bowl. I don't know. I'm I was just we could do something more intensive like uh a curl or um I don't know. There's a lot of okay, but wait we could run So if I make but that's in the pod, but that's in the pod We're trying to make I want a pre pod nuba, right? Okay, right. So that's why I thought 5 000 Okay So and maybe yeah, five thousand was probably too much and that's kind of the goal, right? I mean I could set it to all nine's who cares. You know, it's I'm going to blow away in a second Okay But this is essentially what we're doing folks, uh The image is coming from the repot. I'm pointing at the screen like people know what the hell I'm pointing at Sorry, uh, the image is coming from the registry in the example repo You know M. De Palma my repo column sent to us great There's a pull secret there and everything that you'll need And I'm always pulling Is always pulling always always pulling. Yeah, so I the only thing for this example for this example Right, don't do that normally. Um, so the only thing I mean if you want we could show where in the registry configured says that it's using object store That that you can you can see that if on the uh console if you go down into the the operator for Quay you can actually look at the It just it's one of the components like it's it's nicely done. Okay Proof that it's in use And by the way, I didn't do anything like other than when you go to install it You say yes use that out of the new object store. There's nothing There's no more configuration on my side as long as you install it in open shift storage Yeah, what's up, let me stop sharing it just won't change for something probably been sharing screen for too long or something Who knows? Okay. All right. Let's stop and start it and here we go share Ta-da Oh wrong desktop Why did it do that? I hate you Oh, and I don't see chat so I'm not I'm not actually watching anything. I'm watching chat. There's not going on there. Okay. Cool. Uh, let's see. Where am I? What is this? That's right. Yes. So, uh operators installed operators quay Okay, so wait, wait, uh Yes. Yes. Yeah. Yeah. You're right. You're right. Okay, so Remember how you clicked on registry before the registry endpoint see the one below config editor endpoint? If you go there It does get you have to get the creds. I think If it prompts you for using password to to get into the config editor, but they're back. They're right back in the previous screen Oh, are they Yeah, oh it didn't even prompt you. What is this? I've got a little sign in pop up. That's probably not visible Oh, you do. Okay. It's not visible. So if you go back and you can grab it's like the username is uh quay config But the password is generally mm-hmm so, uh Yes, and then say reveal and it'll do it another tab just I have it just in case Uh reveal values. Oh my goodness. Okay. Oh wait. No, no, that's not it. Hang on Unless it's in there. Uh, so go back to the previous screen just to make your yeah, that's the Scroll up a little bit I'll see the config editor credential secret. Oh that one. Okay. Got it. Sorry. I was and then do reveal and then you can just cut and paste it Yeah Yeah Nope back back up one and Going to the config editor. Yeah There must be an easy way. I'm sure I'm sure there is but you know, okay. All right We're in Okay, so if you scroll all this stuff is you can actually a lot of it you can't customize but at this point But if you scroll down You can even search for nuba. I think uh davis consistency keep going somewhere in here. It talks about the object store It's got like the key. There it is true storage. There you go. Yep And it says there red hat openshift container storage nuba s3 You can't can you select that? I don't think you can even change now You can't change it at this point, but just just proof that it's in it's in the works. It's there It's being used that that's all I wanted to show from that actually. Okay When it's been up 5,000 pots Let me uh change screens here for my turn Here we go or not. Okay zoom Share screens today. It's fine. Stop. We can be special today zoom. That's cool. You know Oh Nope, I did the whole desktop again, even though I selected the thing my turn window. Oh, right Can you tell zoom doesn't like me? I did it again. What in the world? All right, so here's what we're going to do folks. We're going to minimize everything on this one Like ignore pop up here. Yeah, exactly. Don't uh turn on my do not disturb I'm sorry folks. This is all this is stuff. I should have done beforehand, but I didn't. Sorry Um That's This all right now we will share. Oh, no, that's fine. I can leave that open and I'm sure that there's nothing sensitive hanging out on my desktop Uh And now we'll just share the whole dang screen because why not Move this box over here. Just so it's not weird looking. Okay So, uh, here's 5,000 you make the uh font bigger. Oh, yeah, sorry for those of us who are blind Yeah There you go All right So if you could run this we're at 5,000 pods We deployment that's going to spend a 5,000 pods and then after it It gives you the prompt back. Could we watch the endpoint again? Watch hba do do a series of that? Okay All right, that took that took time It's it. Okay. So it so can that oc get hba. We'll just watch it and see not that HPA Yeah, and completion. No oc get hba. That's it. Yep. That's it because we're in this names. Yep. Oh, that's right Yeah, nothing nothing like I want to see That's because they spun up Well, you see and you see the registry's got some action. Oh, no, it was already there before so yeah, nothing So the first number is supposed to be what you're currently using your current utilization And the second number is your threshold So i'm just going to get the pods here. This is going to take a second for it to run Because okay, this should just be 5,000 of them Right and they're creating pending I'm sure running Yeah, like It's creating it should very active. Yeah, right. So Okay, so what am I doing wrong? Let's see Not even like a touch not even not yeah, that even sneeze at it, right? So let's let's delete it and And We'll go nuts. Let's go absolute Ape poop on this. Okay. Okay Trust me, I know what I'm doing So it created that very quickly that time this should be interesting Okay, so so all right So here were some of my thoughts when I did one I was doing like in the hundreds, right? But uh, my worry was I thought Image pull policy always really means that so this should be no docker caching happening, correct? Like I I don't want to hit the cash Exactly. It happened a little quick. So that was my thought. I was like, I don't understand what's happening at that level like Oh now it's taking some time now still zero percent Let's see get pods Let's see get hpa. Yeah, I can grip for storm Oh, no, sorry Let me get pods Yeah Piping to the max not watching chat, but thank you for following us And continuing along with us here 7,036 pods That's not a million pods No Are we hitting the compute limit come potentially here? I'm wondering so So I was I was gonna ask you like what are your thoughts about what? What the pipeline looks like where the bottlenecks are it is but I'm like, are we hitting a different bottleneck that I'm just not thinking of like I don't know much about the lab. I don't yeah By the way, you can see hba in the console as well. Yeah those who want to just let's go look at it Where where would I find it? Um It's under is it compute compute wait a minute No No, it's like is it there scroll down. I said I saw I just had it my bad. Hang on. Oh, you're fine. Where was it? Where was it? I literally just saw it Monitoring would help right now. Let's pop that open real quick. See what's going on Where was it? Isn't that crazy? You know crazy you can do it Sorry workloads. Yep. Very last one in workloads Okay, notice the disk sync duration RPC rates like everything's going up and Everything's going up, but nothing's actually Yes, if you yeah So have I structured this incorrectly like I'm what am I? That's why I thought maybe a fatter image more to pull the larger Things I don't know. Yeah, I'm like, hmm. Something's not I mean it's still going like You see it's still going but now it's starting to plateau. So that's what I'm wondering right like Right and in your setup We would hit other uh bottlenecks first, right? Because oh, yeah Yeah, like so that's but in this one I'm like, it's a reasonable Okay, I'm I'm gonna be honest. I don't actually know much about the lab like the details of what's under Like so um underlying infrastructure. Well, let's go. Yeah, that that's it's it's plumb stuff So I I'm just happy he set it up and that I didn't have to do anything A ton of nodes. Jesus. Um, wow Well, and that's me scaling too. That's my I I ended up. Oh, it's auto scaling Scaling up that's what that's what it's doing Look, this node just popped up. It's not ready Or we just crashed it one or the other I don't know Yeah, so this is where I was kind of stuck I'm like I'm I think I'm set up properly everything looks right and yet I can't get it to move Why is this working so well? Right. We don't want it to work well. I don't want it to work well. We want it to be a disaster. Yeah, exactly. Um What do you think And and also then I so then given that I don't really know much about my cluster like I was like, okay So let's step back one of the things I can tweak I can I can increase the the number of the Pods being run. I can increase the size of the image being pulled. Um I don't know what else should I have it kill it like I was I don't have a lot of knobs. I feel it's a kind of turn so I was Yeah, I got an idea. Okay Bloom might hate me for this, but I got an idea Well, no, if I screw with this stuff will just start breaking tragically, right? So Hang on. I'm not going to decrease the node count And I don't want you to decrease it for OCS right because three of those one was for the registry the other three Yes, I need to keep us just functional, but We had a million pods yet is the question Oh Oh, there we go Sorry wait wait wait for it Bump this up in the screen here I mean, so it's active. It's very See what's going on here, okay Okay, I'll just serve the stuff Muba just having a look Yeah, poke around see what's going on Is it that cd? I want Okay, 37,000 Okay But it's not like I got cpu use of just going up, but it's not by a lot right like right Dropping the bucket as far as cpu goes. Yeah, no scaling happening yet because it's it's not needed yet. So right um, well, there's no auto scalers anyway, so That's a good sign. I guess Do you well, you probably don't know which worker it's on Yeah, because we could check Shoot, this is no fun Ah I know so Last week we had everything going wrong this week. We have everything going right. This is terrible So it's a terrible balance we have here. Um, I'm going to run this command again. Just because it took so long Last time let's see Trying to see network see bandwidth So where's the where's the new dashboard? Sorry, I forget It is okay. So if you go to Up to the dashboards home click on home and then yep Overview Yep, and then down in there and I think you clicked on it before yet, but if you click on object service, you'll see it Okay, but you see the pods moving and stuff. I'm all right. Yeah So there's a little bit here. Um yeah One thing I wanted to note was uh The object bucking bucket claim is still in pending but It's actually functional So so when I was talking about like sometimes updates don't seem to get back to a dashboard that that was one thing I saw And I'll go ask someone about making sure I don't I'm pretty sure I Pulled the latest and greatest stuff, but I maybe I did a pre-release or something like that. So I also pushed. Um, it's not moving So I have an elastic search image, which I think is a little bit bigger I mean, it's fantastic It should be nice. Yeah, like it should be nice in fact like I Um, we're up to 52,000 pods, but it's nowhere near a million. This is gonna take forever Yeah, yeah, no, this is really Okay, so this isn't working how we want it to So, I mean we could try your cluster But I think that's a lot it's gonna be a lot of work to use my cluster It is it's and and just just the registry setup is is intense. So, um So, yeah, so this is where I kind of got stuck. I was like, oh, what else do we what else do we do here? All right, we've got like half now low less Yeah, we're good my cluster could find out I think that going through the registry install is is useful to people like they should see right Yeah, the things you have to think about and So Do you have you said you already have uh a registry setup? Uh, no, are you good? We need to go set that up real quick. Okay, and I have a lot to do either Which means we'd have to read so we got it. We have to reboot everything Or we can go through the process of putting lots and creep on here Oh Up to you. I've never I haven't done it. So if that's um, I'm game for anything I mean, what's is it easy In theory it is if everything works, right? I've done it many times before so let's see Someone is messaging me. I wonder if it's important Um, I'm waiting for bloom to be like, what are you doing to my cluster? I'm gonna have a lot to explain No, it's fine. We just deleted and everything goes away. Um I'm speaking of that. I'll watch it while you're doing this. Okay, so Just can I just operators installed operator Koi, okay Like succeeded. It's an open shift operators, by the way. Okay. I have main registry here Which I have no notice. So this is what I saw when I install it in all namespaces that registry endpoint never pops up Maybe there's something additional that we could do to fix that like it's But if I install the operator into the open shift storage namespace that works in particular what I noted Yeah, so when I when I was at that stage when I went into the config editor for the registry I saw that it was trying to use nuba and that it had credentials But it was complaining the pod was complaining that it had bad credentials So I'm like That sounds weird. I don't know what that's about. I don't have time to debug that Let's just install it in our namespace and run from there. Right. Okay So let's get this thing up and running Okay Okay, so I'm just gonna save this over here. Actually, I'm gonna DM it to you just to have it somewhere. Okay Since we have a running notes thing here So do you can you Can you can you delete this and uninstall the operator and then really you want me to go back through all that Because I couldn't get around the namespace thing if you know how to like what the fix for that is we could do it but So what was the problem? When the operator installed in the open shift operator namespace, you never got an endpoint I never got an endpoint even though nuba credentials were present And I saw agic bucket claims So I'm like I'm being blocked in some other storage. Yeah Yeah, this piece doesn't actually work. There's nothing there. Yeah. Oh, yeah, like it's like, yeah You can't edit it Weird. Yeah All right. Well, let's go back and put it in the over shift storage repo or project Okay, he's enough But when we I'm sorry when we go into so go ahead and uninstall it and then when we go in to do it Let's just take out parts like we don't have to do the mirror We can leave Claire if you want but we could take it out like we don't have to install all of those other pieces. Yeah Going away. Okay. Okay. Do you To the open shift one. Oh, actually, let's just go operator hub So That's good. Yes. What does it say that you should install it to Right. I'm sure I never actually read them any either Okay, so specific Civic Yeah, should show up. You see it. Yeah. I know why are we always Whatever It's weird that you had to in storage like put it in the store. There's got to be a reason for that Yeah Right like something we can change to make it not do that And what's also weird is in mine. I got I did get credentials. They just apparently weren't working Like so when you looked at the log of the pod it was something it was trying to do I was like So I will investigate. It's probably me Don't worry about it. So advance configuration And then I think there's a component thing. Yep. Yep. Okay. Okay, so Postgres true How big storage true red is true managed Yes You see mirror in there somewhere very last one mirror Remove it. I just removed it. Oh just remove it. Okay. Yeah, you just don't need it at all. So, okay So you got your horizontal Yeah, yep. Yep. Yep. Yep. All right Create Yeah, sample registry Okay And now if you do oc get pods in the open-shift storage namespace, you'll see it You'll you can watch these pods just for example and you'll see them spin up among our I got a log. Oh, yeah Yeah, yeah other cluster. Okay Which I totally forget. Oh, I was supposed to be watching this cluster. Sorry. Yeah, not bad. Oh, you're fine Yeah, nothing. Yes. Nothing qctx. Where am I? It's oc status. I forget I think it's oc status. Yeah, you're right. Yeah Always forget or just you know, I do oc get nodes and then I that's yeah, that works. Yep. Okay. All right, right thing. All right so oc project Storage, yes and then oc get pods You don't have a million pods yet. So we should okay. You see all those example Mm-hmm. Yep. Just you guys set away for them to finish, but it's nice. They're here. They're working. It's all good Happy happy. I'll just set a watch on this And there's nothing the other clusters seems to be fine. Like happy as a clan. I can yeah Awesome Right, I'll graph out. Let's see how many okay now. Wait a minute. I think yeah on this one. I'm still on Your cluster and I'm getting utilization At least I think I am Unless it switched contacts in both terminal labs, which it probably did I had to log in again Ooh Change our cluster. No, it did not change. Okay, so You you might have some progress here. Um, can't reach the console anymore. Can't log in anymore Ah Okay Yeah, it's back on my cluster. I think no, no, no. Let's still sign into yours here No, it's mine. Yeah, sorry. I can't I can't I can no longer reach the the cluster in the lab It's done. Oh fun. Okay. Let me see if I can touch it real quick Which is the lab cluster right over here Yes Yeah, can't get to the dash Oh It's just a bunch of api. Let's make it happen Hmm. Well, I didn't want to bring the whole cluster to its needs. I just wanted to bring this end point. Well, that's what we did. Yes I bet if I could see it it's scaled Uh Hmm Yeah, let me know it's not happening from my side. Okay. Well, then let me Ctx this thing I can walk back into it. Maybe hopefully who knows You see status anything? Bueller? Bueller? Bueller Oh boy That's the lab cluster you're on. Yeah, I think we broke it. We broke it. Nope. It's definitely unhappy. Yeah. Yeah Okay, well, good morning. I feel good. We broke something. Yeah, we broke it. Hooray. Um Uh, okay, so push the project storage it even lets me I hope it lets me switch projects that'd be awful if it didn't to be really screwed then Nope, okay. Yeah, it's done. See delete Yeah Uh Storm we'll see if we can even make that go away But you gotta be historic graph somewhere for the for nuba, right? Yeah, and interestingly enough Uh, hang on. Hang on. Shave time out. Dagnabbit. Yeah Yeah, we've really destroyed this thing We did Yeah, so my the noodle console is still responding Interesting. Yes. Do you want to share your screen? Yeah, and you have it actually you I think you brought it up It's on one of your tabs if you look close to where you are No, I think I closed it on accident by the way. Okay. All right, so here link this was it but hang on Okay, I know I can also select too. It's like, okay. Oh, yeah, send me the Yeah, there you go That is unless it's cash, but I don't think so. I'm doing that What does it command are? Yeah, I get a response from nuba, but I don't get Well, it's nuba busy because that was the goal, right? Yeah, um, okay, so hang on This thing hates me right now, by the way This cluster is very unhappy. It's very unhappy Okay, so some parties. All right, so that so that's weird because, um No, the new burger thing It's not responding to mine just cash that that's probably what it is. So, okay So all right, so from an open shift perspective I can't even list the projects. It's very dead But at least you got that far mine just gives me an error right there Okay Maybe needs to catch up time out. Yep. Just notice toast Yeah Very much needs a kick in the butt All right, so that note's destroyed. Tada. I'm gonna go back to mine Yeah, yeah, let's go back to yours Okay, so and where are we? Did it finish or did you uninstall and you succeeded? Let's see where How far away we got? Okay Okay, so check your Yeah, that's weird. So let's check the the pods um in In your cluster in the open shift storage namespace just to see If they're done So, you know clear takes a while to update because it's got a download It's a little busy. Yeah. Yep. Yep. Yep failed But you can start right over Okay, the monitoring one Anything pending do you see anything pending or? Yeah, I do see some pendings clear clear Yeah So this one is Putting everything in the database. This one is the database So, yeah, that's kind of important Yeah first download Nick if I should try it without claire just for demo purposes, right? Like You don't need to allow it claire is good like claire. Oh, what's the error up top claire erred? Okay That's fun. That's probably because this stuff isn't running yet. Right, right Okay, we got 10 minutes I don't know if it's going to be done in 10 minutes, but I don't think so Crash loop back off Do you remember how long it took? Oh Did you change or it's running? I didn't do anything. Here we go. Okay, something And that crash loop back off Okay So it's still trying to work I haven't think yeah, we probably should have skipped claire, but oh well. Yeah, so I'm going to go through that let's encrypt Tutorial like setting up your cluster with let's encrypt so that I can start Doing that and just skip all of the Because I don't want to do the insecure registry thing again. That was really And I when I did it it was after everything was installed So then of course everything gets a little upset while the pods are restarting and other that I was like, uh next time I would I would scale up Not just ocs, but also quay initially And since I already in this lab, I know my uh, I know what the registry is going to be called I know the name the full domain I would go ahead after I scaled up I would go ahead and modify that image config cluster And let it reboot like I would get everything set up all the rebooting that needs to happen All the cluster wide configuration changes done first And then I would install ocs and then I would install quay and then I would do that I would instead of what I did is I did it while I was going as I was discovering I needed to do it. So That would be the deal There's a lot of failures here This might be mine my cluster just being a but so Hmm I wonder why I wonder why that's a no SD failure Yeah, that shouldn't happen. No, like I said my cluster is probably under provision Do you run in compact mode? OCS Yeah So you're being mean to it CPU request commitment 115 percent Well, there you go It's a problem, but that's okay Okay, so I have a question because we have like eight minutes. So do we want to try To do this again like and really like should we should I work on it? Should I What would you like like it's and then So I would go through the whole process again But we could start at the right if we work if I work on it and try and get it to go We could start with Everything set up perfectly watch its scale like but or we can just drop it and say like, okay What do you want to do? You mean for next show? Kind of yeah, yeah, because you know, I always have other stuff like I'm I'm gained to show you some other cool stuff Like I hate I hate to leave this in like a failure state though, right? Like right We did not achieve our goal. So that's a problem But um, but no one's really complaining either, right? Like they're seeing how we're looking at stuff and you know troubleshooting accordingly and open to anyone who might be listening like If you've got some ideas by all means pipe up and let us know like they owe your images to stall or whatever So Yeah, this I don't think this is going to go well to be honest with you. Let's see. Okay. Okay. All right. So let's have a think about it My my I have a burst use case for mcg. I'd like to show you um, I was I definitely want to talk to some teammates about um Some edge stuff like we could do yeah, I don't know what but but be patient some of these things they just come back like You are over way over committed on this Cluster, so I know All right, so I will work on it. Okay. Yeah, like talk to my team and see like, okay. This is my setup. What did I do wrong? Like let's think about how we I'm sure I'll hear from chris about it I want to clean up this cluster somehow because I don't know how the chris is going to debug this otherwise So I can do that. I can go into um, I'll do that from my side. I'll just kill it You just kill it in the yeah. Yeah, totally. Let me start it. I can do that You want me to do it now? I can do it now before you do it now because I don't want to leave chris's cluster in a state of debilitation that it is Uh No idea who who else uses this or something. Yeah, right like I I don't I don't want to destroy everything for everybody I want to make sure that cleaned up. Okay redeploy done Done The whole cluster so he set up a lovely he yeah, no He did a lovely page like you go right to your page and you can like you want the he just did it Right. So I'm okay. Literally click a button and the whole thing redeploy. So it's very nice. I like, okay All right Okay, so I'm going to work on this and see if I can make it so that maybe in the first 10 minutes of the next show We can really show it right. I'll have it up and running and then we'll try and slam it And then we can see it scale and then we can put the issue to rest and Move on to something else more move on to something else. Yep. Yep. Yep. You got it. Perfect. Okay. All right. I actually make that happen. Yes All right All right, cool. And how's your cluster? Are you okay? Yeah, my closer just and I should show right like it's not Like the fact that Everything else is failing right like that just tells me that I've under provisioned this horribly or Or not under provision. It's being over provisioned right like right right So maybe some time it'll come back. Hopefully all right. Yeah, we'll see You never have any you never know any questions. Do you see anything in chat? Let me find the chat window No, just a couple people followed us if you have questions, I feel free to ask because now it's the last time Five minutes left here with the lawyer and everything Let's see stop sharing my screen, right? Okay. Yes. Yeah. All right Well, I don't feel too bad about it I was kind of hoping we'd get it done today, but we did manage to bring the whole cluster down and that's something Okay, so I do I have a screen here Yeah, that shows the nuba console Okay, let me here This might just be cached But yeah, because I I definitely restarted it Oh, well then never mind Yeah, no, I took it down because I was like getting scared there. Yeah, the image is the cluster's not up yet It's gonna take another 20 minutes for it to come back. Yeah. All right. Never mind All right, never mind. It's all good. Okay. Okay. So let's see Yeah, see you in two weeks says uh john shubek from over on youtube. Thank you very much Yes, thank you so Two weeks. We'll wrap this up and then we will go about our business on some other fun adventures Yes, so we'll be sure to update the calendar as we see fit for that And then coming up on the channel today. Wow, even my calendar is frozen Dev nation is coming up next at noon eastern 1600 utc the show with seby their back should be streaming to the channel today And then get ups happy hour three o'clock eastern eight o'clock utc 8 p.m utc Stick around for the get ups happy hour and it'll be awesome. Yeah, seby's watching. So yeah, seby says we are back So awesome great great to have you back seby really appreciate you and your show and all the Dev nation crew out there and folks, uh, you know, we have If you want to kick the tires on your own on a real cluster head over the dev sandbox That cluster will exist for two weeks for you right now And you can install various operators. I don't think you can install a storage operator yet but you can install things like already workspaces or serverless or service mesh And you can tinker with those operators that are like, you know, not requiring more more nodes in the cluster So, yeah, kick the tires on that. Tell us what you think give us feedback We really appreciate all that all the time and uh, yeah We will see y'all on this show in two weeks and I will see y'all This afternoon. So thank you, Michelle for taking us to this awesome adventure of crashing clusters. It was fun Hey, it's whatever whenever you need to crash a cluster just spend up a million sleep containers and you'll be off and running Thank you so much. Be well. Be well to everyone. Yes. Stay safe out there everybody and we will Make sure that we stay safe as well. So thank you very much