 Good morning. Good afternoon. Good evening and welcome to another edition of the get ops. Happy hour with myself Chris short Executive producer of this thing we call open shift TV and the one and only Christian Hernandez get ops extraordinary Christian we're talking about managing machine sets today. Yeah. Yeah, so Also, I'm gonna say good morning. So we we we spent across across the globe here. Yeah so Yeah, so we're talking about machine sets and auto scaling actually You know in a get ops world, right? So I I tried to prepare I was telling Chris You know right before we started I try to prepare some slides, but You know work You know my day-to-day day-to-day gets in gets involved actually a lot of people don't know that like this isn't considered Like the streams we're doing isn't considered your day-to-day, right? So So it's like I can't I probably should work on that. Yeah. Yeah. So it's so I got Got a little sidetracked. So I'm gonna be doing an Andrew Sullivan So if you guys haven't never checked out Andrew show Open shift for administrators office hours, I believe it's called Where he when he talks about a topic he usually just goes through the open shift documentation Which makes total sense sometimes which makes total sense. Yeah, so I think I'm gonna do that and you know A lot of times we catch things. I know Eric Eric Jacobs when he goes through the docs he finds bugs and he just files them as he finds them Yep, so maybe we'll find some of those to write. So You know first and foremost, I guess You know Talking about every putting everything in a get repo right get ups world where everything's in a get repo if it's not in a Get repo then you're not doing get ups sort of, you know drawing that hard line, right? So and I talked a little bit about this when we talked about operators using operators and get up And I should actually yeah So Just said that I need to have a get-ups extraordinary on business cards. Actually you can actually pretty much put anything Yeah, like red hat allows us to put whatever we want on our business. Yeah, so I think I think I'll put that put that as Yeah I actually just got my RHCA. So I actually do new business cards, right? So by the way, if you guys don't know that's you have to take five tests on top of the RHCA. So it's So in the end you're taking like seven So thank you. Yeah, it's it's a it's a it's a bear of a gauntlet but but now I need new business cards and I'll have a Something else to put on on the business card. So I would feel Disrespected if you didn't put get-ups So now I have to do it right right So yeah, so so, you know back to the That's what I was talking about about the operators, right? You're wearing an operator shirt. I am see I'm only wearing the open-shift shirt. I don't think you can see it. Um, they're all one big happy family, man Yes, it's all so I actually joked that Chris short's closet is just nothing but swag t-shirts that he's gotten through So the events so funny story and we do have a hard stop it for today. I have to interview a potential intern. Thank God But so like I bought what was it cube news dot IO like way back in the day thinking I thought your Kubernetes newsletter at some point, you know, it would be like a spin-off of devops my current newsletter but I never Ended up doing anything with it. Yeah, and I kind of don't like Dot IO domains anymore because I learned the history behind them. Yeah. Yeah, so I was like, hey CNCF do you want the cube news that IO? They're like, yeah, sure Here's a three hundred dollar voucher for the gift store. Oh, and it's like, oh Okay, let me get every single project that open-shift uses Stickers galore. Yeah, so you're gonna have stickers I bought very few stickers, but I got all the shirts. That's how I got my Argo CD shirt It's how I got my Etsy D shirt Thanos all those shirts came in that order. Nice and like I've got a nice beanie and everything with the Kubernetes logo on it. So yeah, so then yeah So then now you have the the operator shirt, but you probably got that. No, this came from that store as well Oh, okay, there you go. Yeah, because remember CNCF adopted in the incubating side operator operator And if you're in an incubating status, you get all the swag you get the shirt Okay, well now I need an operator shirt from the swag store. So Because I did yeah, so yeah, so if you remember in the previous episode we talked about operators, right and how operators is the like the automation core of open-shift for right and how do you kind of get ups the the The the operator right like so if like, you know automation everywhere and oh what Shari's on Yeah, he was on earlier. No nice. Nice. Yeah, welcome back If you could drop the also Chris if you could drop the the bang link of the the playlist, right? So if you've missed past episodes of The happy hour. Yeah, we'll drop those in the link I talked about operators and this is kind of the The same idea with machine sets and machine auto scalers, right? So what's a machine set? So let's let's do a Andrew Sullivan here Let me share my screen. Well, I mean and layman's terms while you're sharing your screen Yeah, a machine set is a set of machines that you want. Thank you. That is API to manage, right? So you create a machine set to help you with scaling and You know management of those machines, right? So that's the layman's terms now Let's go to the docs and get through now. Let's yeah So that's you know, basically a machine set is a set of machines as Chris said and I went to the official docs, right? So docs.openship.com and I kind of you just kind of navigate to machine management and creating machine set. I Use AWS a lot. I mean, it's the same idea no matter what which one you go to GCP Azure Same premise, right? Like same premise, right? So Kubernetes API is talking to other APIs to do. Yeah. Yeah API is talking to other APIs. So and so So let's talk about a little bit of the machine API over here, right? So the machine API is like the the guy that's doing all of this stuff here The the the controller as it were, right? So the custom resource definition. So we have something called Let me make this a little bigger. Oh Yeah, please. Thank you for those of you watching at home. I forgot to check. Yeah. Yeah and so So we have two primary resources, right? So we have something that's called a machine, right? So a machine is a Definition that describes a node, right or a host or VM, right? Whatever it is. Yeah, whatever it is, right? Actually, Andrew did a did a stream about machine sets and machine API. He goes really really in depth into it I recommend you you guys watch that as well But a machine is a a description, right? of What an instance is, right? So we're talking about things like a machine has certain characteristics, right? Like what like CPUs like for CPUs, right? Ram this many disks, right? So Basically a description of of what a machine is right in AWS land, right? Those are these The node types, right? So like you're talking about like M4 Extra large or whatever extra large right for 2xl, right? Like that's how it is in in Azure It's different right in Azure. I forget how they define theirs in GCP, but the concepts pretty much the same is a Definition of a machine, right? So a machine set kind of like the same paradigm from like replication controller and A pod same idea, right machine set is kind of like a replication controller, but for machines, right? So it describes a group of machines, right? It controls The scale of it, right and and it controls The the overall health of it, right? So kind of like here it actually says in the documentation machine sets are two machines as replica sets are two pods, right? So if you need more machines, you just scale the machine set, right? Or kind of like a replication controller where Where you have You say I want four pods you don't you don't modify the pod definition modify the replication controller, right? You know that was back in the day Actually, you modify the deployment which then deploys a new replica set and you know yada yada, but conceptually you can see that You know you you change the number of replicas, right? And if a machine goes away the idea is the machine set will then re redeploy that that machine, right? So it's kind of it controls the scale so So Once the machine set controls the machine, right it will instantiate the machine and then the machine will then Bootstrap a node, right? And then it becomes a node, right? So you go from machine set to machines to node So that's kind of like the the difference between a lot of people get understand machine sets And they don't understand like there is between machine and a node. I'm like, well, aren't they the same thing? I was like, no a machine. Yeah machine is the definition of a node like yeah a hardware, right? Node is actually has Red Hat CoreOS installed on it with the kubelet running and it's joined to the cluster So they're like your machine is basically a blank VM and a node is that VM that's running Part of the cluster, right? So it's kind of like the the differences there. So And so there's a lively check going on about the Every time I mentioned that IO domain it always starts alive. Yeah Well, I've been kind of following you know, I guess it's like to say it's a good lively chat though. So And then we have the concept of a machine autoscaler, right? So the machine autoscaler Identifies Actually, sorry is connected to a machine, right? So you're saying that You know for us Take for example AWS US West one, right or like one US A right or whatever, right? Like you're you're you're identifying that specific Zone right within the region and I say I want you to keep this at x scale and you have to do that for each Availability zone that you have I think you use the cluster autoscaler in conjunction with the machine autoscaler, right? So you have a machine autoscaler defined saying You know when you reach x amount of you Utilization then spin up another machine the cluster autoscaler use that in conjunction with it is saying okay, like my cluster as a whole I Want I want to make sure you know no more than 80% utilized, right? So it takes kind of like an aggregate of all your machines and uses the machine autoscaler to scale that right so And so the machine help check is exactly what what you Think it is right It detects when the machine is unhealthy and deletes it right and then creates a new one, right? So this is the idea and this is where kind of people get tripped up With your with your journey like with just in general from your journey from To cloud native right because that's like a journey in itself and then from cloud native to get offs It's like a whole another you know journey. It is like a lot of people treat their Machines as pets, right? Yeah, instead of cattle. So this is the idea of just treating your your servers as As cattle essentially right like you you wanted to be a commodity you wanted you know The important thing is the application not not the machine So you kind of have to get that that paradigm shift of like well, you're deleting machines is like, yeah, yeah Like it shouldn't matter. Yeah, you know, that's that's your journey to to cloud native is right you're deleting These things as cattle and you set points out it's kind of like the horizontal pod autoscaler, right? This is just doing it with actual like Nodes in your cluster, which is cool. Yeah. Yeah, exactly exactly And so yeah, you're pushing the You know you keep pushing the What do you call it the pets? Sorry the cattle the cattle you keep pushing the cattle further further down and the pets further further up, right? Where as now the pet to be that bespoke container your developers working on not the cluster Yeah, so I come from an operations background. So like I Always but I'm like one of those like rogue Operator guys. It's like yeah, same like you shouldn't you shouldn't care about your VMs and like, you know, it's like, what do you mean, right? I had had across that that same that same barrier, right? Like you shouldn't have to care about your VMs like you shouldn't you should I should be able to delete it So I had Mike, you know, I always do dollar sign last job in my last job My ex boss he would say I I should be able to pull this machine Out of the wall socket throw it out the window and you shouldn't care, right? Yeah, so it's Same idea, right? Like I should I should be able to go into AWS a console delete that instance and you shouldn't care that I did that. That's the idea, right? So And that's the idea about this this kind of automation So it goes through this example here, I'll go through action an actual example that I'll use later, but Kind of just a complete My thought here is that That's this is a point of automation, right and get ops is a point of automation as well, right? You know, like where where's where's Where do you control these things, right? So Kind of similar to the What do you call it the operator, right? So the operator stream that we did is like especially this is basically is like Where's your point of demarcation? Do I put my machine sets? Do I put my machines to put my auto scaler and get like where where should I release that control of automation, right? So I think I came up with I've been playing around with it again kind of like the operators. There's no magic bullet You know, it all depends what you want to do and what works for you, but I think you can leverage both the automation from the operator framework in OpenShift Alongside with get ops practices, right? You know, it doesn't have to be an or it could be an end So I do have so usually so I was actually telling Chris. Yeah, usually I have like things teed up, right? Today I don't so you guys are gonna watch a lot of a lot of things called paint drying, but I think I think it'd be Event it invetages to actually see what I go through in order to set something up, right because You know, it's a lot of the times it just seems like magic to people and it's like no I actually beat my head, you know against the wall for a few hours before showing you this demo So you're gonna see some of that Some of that head beating, right? So I was up to my beat your head beat my head here to my CLI here, so Oh, by the way, I've switched Speaking of a derailing right I switched from using power line to pure line. I don't know if you guys seen pure So it is this guy called Chris Marsh He kind of maintains it You're still in using bash So it's so you switch back to bash. So it's using bash. So it's basically because you know power line is basically Your your shell wrapped in Python, right? Or pearl I forget which which one of the two, right? But this does it completely in bash, right? So it's actually pretty easy You don't have the over overhead of running a daemon is this basically but anyways, it looks the same, right? And it's customizable. So anyways, nice kind of little geek thing that he's I like it So I do it OC OC status was What am I looking for? Oh, yeah, so I am I am on the right cluster here. Okay, so So that's good always a good always a good thing to check Well, you know, like, you know, we use so many clusters, you know I know trust me might get my cube config directory is just like yeah, it's great. I Need to go through that and like version it or something at some point. Yeah Machine sets so machine sets. So let's look at the machine sets actually instead of doing dash a Let's do the right thing Right, it's in the machine config API. So you see here that I have a definition of Of a machine one in each availability zone, right? So US West to a to be to see right So this machine set controls a scale as you see here, there's one desired one current one ready and then for this cluster, there's one in There's a machine set created. So there's an odd one automatically created for each availability zone available to you Yeah, but you know, it'll set the scale as zero, right? So cool. Yeah, you know, if you only need three clusters Yeah, if you only need three, it's it's fine to be a three So if I do OC, but it can utilize all four regions if region goes down somehow or Yeah, or like scale that up. Yeah Machines right so if I do OC get machine Opuship API. So this here to a corresponds with this to a here, right? So Right and it gives you the status, right? It gives you the type right here again. Remember we're saying in five large That's like that type. It's it's running on and running in this region this zone, right? It's been up for four hours, right? So I installed it this morning Because that's what you do. Yeah. Yeah, right. So like if you do OC get OC scale machine set Right, let's take D here And then you do replicas you want one right like I want to because it goes to that zero right now So let's just say I want one Right, so then now if I do get machine sets You'll see that I scaled right if I do OC get machines You see that there's a new one Nice, you know in the status of provisioning, right? So it's kind of and if you don't see get nodes eventually this will become a node, right? And then I'll have four workers, right? So and then so someone It's over. Yeah. Yeah, Hillary said half the time. I wish I had a OC where am I? That would be great. That would be pretty good. Yeah Oh There's a who am I now even where am I where am I which cluster, right? Which cluster what namespace, you know, am I in a pod right now? Where am I right, so it's Yeah, I do OC Yeah, OC status will show you the Yeah, so I'm like, oh, okay. I'm using this cluster. So yeah Um, so And then there's OC something else. I Always find stuff and then I oh, yeah cluster info, right? Yeah, and then there's cluster info And then you could do like a dump or something cluster info dump. Yeah, and that just you know anyways, so a little Because that's what you need. Yeah, like if you I guess you could jq it, right? Well, that's like good. Yeah, yeah Right so Yeah So I have OC get knows maybe maybe the nodes are Notice up they go Cooper net ease. Oops. No, it's node roll Yeah So yeah, so So one way to kind of get ops this here is that I have You have to Kind of do what Kind of do if you guys I've looked up get ops You're probably see a lot of we've works coming up kind of do what what we works guys recommend It's just basically export everything and import everything, right? So if you do a OC get Machine sets Open shift machine API, okay You basically take these and export them Yeah, and put them in get so I kind of did this with my old cluster. So let's look at my machine sets here Let me look at my overlay. What's your mind? Okay So I kind of did this with my other ones, but essentially I just scraped the definition out. Oh Okay, that way you can yeah, okay So that way it'll apply because it Argo CD does doesn't apply, right? It doesn't do a great. It doesn't apply So So it'll apply it, but if there's no change then you know, it goes. Oh, okay, then it's healthy And I did this for all three of Please increase your font size sir, please zoom. I'm at 120. All right. Let's go here 133. Is that good? That's better 150. It's the it's the the fix with font. That's hard to read. So yeah, got you got you I Think that's good. By the way, I love this new night mode. They have or dark mode that get Yeah, you don't like it. I liked it like as soon as I turned it on I was like I I'm in love with this like There's already a like darker Mode extension because like the dark mode they used isn't quite it like accessibility ready, right? Yeah. Yeah, it's like Interesting well, I like I can actually see the colors better like when I'm looking at yaml or like shell or something. Oh, yeah No, the color coding is a must but yeah So so yeah, so essentially what I did was I just exported all of these right so And I control the machine sets there. So that's What would be the easiest way thing to do here I can do this I Could probably do it through vi right? So that's Let me As you can see I'm working on like 300 things at once here Covid note I have SSH enabled. What the hell is that? Yeah, so So I leave my laptop on so I can log into it. Oh Makes sense, but I usually don't have that. I usually just stable SSH. Yeah, so Yeah, so this reminds me that when covid's over. Hopefully soon sometime maybe I Can I can continue blocking SSH from my laptop so very yeah by the way You shouldn't have SSH turned on on your laptop unless you're me I guess and wants to log into it remotely. Yeah, I I only turn it on when I'm in the house if I know I'm gonna be like Working on that machine. Yeah from someplace else in the house Or outside of the house potentially so like all of my machines in the house have SSH enabled and on But there's an SSH gateway that I use and it's very locked down and if you start hammering it It will ban your ass for a life kind of thing and I'll get you It'll ban your ass So if I do machine machine sets oops machine sets and then I am Overlays cluster to right so I have This here so let's do this one at a time right so I have So let me move So which one this one is a right so to East one a To this guy YAML Right, so they do that guy right so then what I'm looking for is to change this D To this cluster name, right, so I'm gonna look for this cluster name here So what I can do is oh if you want to see some vi foo right if you if I want to search if I want to search for cluster 2 dash ng df 7 Replace it with cluster 3 do that globally right Not found oh percent s yeah, yeah, you're right search for this guy replace it with this guy here and Then do that globally right and then that's all changed right so that if I know that look for that guy He's there and then the other thing is that I'm not in US East anymore US East One right. I'm not in US East one Here you go US West to yeah West side. All right, which is what is that Seattle or Oregon? That's Oregon, I Believe it's Oregon. I think you're right So then do you're right percent s search for us East one And replace it with West to globally sweet I still use that in VS code by the way Because it's like with the Vim extension installed it's still the fastest way to find and replace stuff Yeah, even though the S code has like all these find and folder find in whatever functions It's yeah, just easier for me to be like easier just do the vi done and So so that's you know one the cluster name rich is usually your cluster name dash some hash right and then wherever you are is you You know in in the region right and the rest is pretty much the same right one thing I do have to change is I do have to make sure If I do get machine sets if I get a 2a Great dash. Oh, yeah, I'm all there's other things I need to look out for like This in the instance type right so the instance type I'm using I'm five large here, so I have to make sure that matches, right? instance type, yeah, so I'm using that and then One last thing is the the AMI right the AMI ID is different so So where's the AMI why am I not there we go so Let's see if this is different or the same or it is different. Oh because I'm in a different region. Yeah, yeah I shouldn't it be the same. No, it's not so and the rest is pretty much the same, right? If you guys are interested Let's look at the documentation, right? There is I basically use this right the And I'll drop this in the yeah in in the chat here I'm using just basically this right as as my template right because it's the same no matter You know if you have you know one or ten, right? So So that's thing here. Oh, let me make sure the the IOPS things are correct here. So let's do Where am I? Okay. Where am I? I say, oh, see where am I? There we go, we should have used it. So yeah, so that's fine. I Okay, cool So that's one right so I Have to do this two more times, right because I have two more Oh You should just use said I was just thinking about that. I should just use said yeah on the See here how much time do we go that because I'm being conscious of time because I know you have the at the top Yeah, you need to drop so All right, so cool. So that's That's a let's do be now So as I'm doing this if there's any questions Feel free to ask related to machine sets or anything get up Right, like yeah, oh, I get to oh I get to use the oh command. Oh That's right. You're brand new command. You love it There we go And I stole the definition of office hours from Cornell University. So try me on that one Here I'm doing percent s of cluster 2 That's and then percent s So you're really seeing all the sausage just made right here, so if you guys guys Know a better way. Let me know. I'm always US West to the problem is Well, no, it is consistent throughout right except for the am I like you're changing from this cluster name to that cluster name You totally could have you said on that directory Yeah, I probably could have just done a star a set and like just every file in there You see here this graph am I using this am I I am I Hope I do mess this up because you know, it's always good to see error sometimes right We see tight. Oh my gosh, we're streambot. You're killing me this week Large and in charge and then that's about it. All right, and then I'm trying to think here the last one So closer to To to see here we go To see or not to see this one is percent s search for this guy because I don't want this guy Place that with this guy here There we go Globally I could type there we go I want us east one gone place it with I am in US West to globally And then it's do you remember it's M something, right? I am in five large. Yeah large and in charge And then the am I Just different from what I understand Where is it there it is Cool All right, so that should be good And then oh, yeah, I need to update my customized file customization Right because I don't want to load these guys I want to load these guys. Oops. All right All right, cool And then the customization file basically also reads base right and base all it does is just make sure The API is there right the Openship namespaces there. You don't have to do this Because Openship machine API is like a protected namespace However comma for the sake of being complete I always put it in there It doesn't it doesn't hurt because if it's already there doesn't do anything though, right? But if somebody stumbled across the file and it's like, oh, what does this apply to? Yeah. Yeah, exactly. Yeah, right so it's So get add to the Langdon thing get commit That's my thing too. It's an everybody thing actually Master I had an interesting error earlier with Argo. Oh, it wasn't an error. It's me user error I'm so used to typing master and new repos are now main. Yeah, I'm like, why isn't this working? Why isn't this working? I'm like and then I started like spelunking through the logs and it's like master doesn't exist I was like, what? Oh, yeah That happened to me this weekend actually. Well, I had it like a what moment because it's like what do you mean? It doesn't I oh wait. Yeah, you're right So, let's go back to the repo here. I reload The page the file you file you want is not there because I changed the name, right? Let's go to cluster. There we go so now I have the Machine says it right so essentially As you saw the the paint dry Essentially, it's the only difference is a cluster name, right where you're at, right us, you know, I mean us to region right and then the instance type and like the AMI so so You know the basic things that would be different, right? It's you know, no, no, this is a surprise You know, you can use my customized patching, right if you want to have like a generic machine set And it's used customized patching for it to do Cluster three files in a folder called cluster two. They're hanging up now So, um, yeah, so there's a reason for this. I actually can't say it on air. I'll tell you later, Chris So why I'm doing it? Yeah, it's exactly it's one of those things So, um So so yeah, so it's it's actually I can you know, I can what I can do Sheen sets I can copy it, right? I can just do right you could totally just be like Done. Yeah, CD or that and let's do this live. We'll do it live CP dash a cluster two to cluster three Ta-da done done. I am done But now cluster two is messed up Yeah, oh well See oh For open pixel, there we go. There you go. Yep. There it is. All right, I'll happy Yeah, I could do a get revert. Yeah, sure you could and so now if I reload this page I get closer three right so now closer three has closer three. All right, sweet All is well in the world, right and which is what's cool is that the customized works, right? I use the same base for all three of them. So I don't even have to change the base. So again if you If you've missed any past episodes, I do actually do go over customize earlier Yeah, in one of the episodes. So check out that playlist on YouTube Yeah, let me drop the yeah, or do I have a I wonder if I have that I set up one for you the play I think if you do bang get ups. Oh, was that it? Or happy hour, I don't know. No, it was get ups There we go. That's the playlist. I actually go through customize and how you can and we're gonna get spammed by that now. So yeah, yeah And so So there we go So now I'm gonna apply this to this cluster, right? So if I do See here new app Right. So application name. What do we call this? Let's call it something It's called a cluster three app Right, so let's call it whatever this machine is here actually to see if our new a new Yeah, your new cluster. There we go. We got four new Easy as that right so we'll call it cluster C app Again, we're putting a default namespace in Argo CD. Remember, they're different Argo CD handles its own projects differently than Get an open. Yeah, then open an open shipped as well So I'm gonna do a manual sync right just because so we can see what's going on Repository URL. We'll get get ops examples We'll use Instead of cluster to we'll do cluster three so that all matches Destination clusters this cluster What's the name? What's the namespace again? It's Open ship machine API Yeah So I like how like this The old expect us to do all this typing in the OCC li and they give us like the longest names, right? It's like, yes, I need to know this is a worker maybe the region Yeah, I know the unique ID thing has to happen to so it's like oh Well, and I every time you do tab completion like it literally hits the API so it depends on a faster cluster So you're like right bashing the tab So this namespace is the namespace within the cluster, right? So I'm gonna target that so let's do to create here I'm gonna just pick invalid and able to get app Machine says overlay. Oh, I got it. Really? What? What doesn't exist Get status so machine. Oh, I Know I know what I did wrong. It's out machine sets. There we go There it is so So it's so then it actually creates this here and if I do This this this run a watch here. So let's do a watch of that Right, and let's sync it. Let's see what happens, right? Synchronize Synchronize happens quick because it's already there, right? It doesn't have to change anything nothing changed here my machines didn't go away and I think it deleted because It was it's like for like right. It's the same essentially I exported the config and imported it and make now I can handle this with with in a get ops fashion, right? So So actually so now that I have this in in git the idea being that If a user right wants to come in and do a PR, right? It's a say I want to scale this guy for one replicas to Right, I can actually control the scale with git now, right? so now when I When I sync this guy Just notice how now I have this machine set now has modified two machines, right? And then the status here change from desired to the current to two, right? and Then that will actually if I do go see get machines Now I'll have a new one provisioning right so now in a get ops fashion now I can control the scale of my cluster if I delete a machine It will reconcile that for me I Have also The way a developer now say like a developer wants to Scale the cluster now they can submit a PR, right? So now they can submit a PR we have it all tracked and get right I can connect You know you can have hooks into whatever service now whatever you want, but the idea is that now you have an audible track of like Developer saying okay, well then You know, why do you want this scaled? Why do you need more machines, right? Like you can have that conversation openly, right? So the idea is that You know anyone get like you know if if you know the If the marketing manager is gonna run a Campaign and he knows that we need to scale the cluster he can go and do a PR anyone can do PR, right? Yeah, that's the idea And so hey, we're doing a Super Bowl commercial Yeah, I'm the scale scale right like I want to scale this right. I want to be preemptive about it So actually I'm actually Hillary I'm Braided silver is actually part of our OSD team right? Openship dedicated. I'm actually curious to If she wants to chat it out how we manage this in OSD, right? Because it'd be interesting How they manage in the sheen set so if if I'd be I'd be curious, right? So if she wants to answer she can drop that in but that'd be awesome Yeah, it's too long of a story. We can bring you on the channel. You can bring on the channel later. Yeah So did you see open pixels question about can you explain machine sets and how they relate to machine configs? Yeah, so I'm gonna go through it very briefly because That is a topic. It's a deep subject. Yes. It's one is a deep subject and two. It's a topic for That I have Inline so I am going to do a stream about machine sets. Sorry machine configs. So When so this here you see this is is Starting provisioning, right? So that provisioning step it actually do if I do OC get a machine config configs Right, it actually takes This machine config and passes it through ignition, right? So what is What is ignition, right? Ignition is a way to set up your server Or set up red hat core os before pit one runs, right? And it runs it in ramfs and I'll I'll I'll explain all that in In another stream, but the the idea is it's like, how do I run cloud in it? But only once right? That's a big difference between cloud in it versus ignition is ignition runs once cloud in it runs Every time the machine's booted But the idea is it's like it has things like um Things like ssh configuration Kublet configuration container runtime configuration, right and all that gets glommed together Into what we call a rendered worker, right or rendered master, right? So if I do OC get machine configs Rendered worker Right oops dash oh yaml, right it's essentially a dump of Configuration things that you would want to see right like There's a c system d units here. There's like username passwords, right if I do If I get the the ssh one is Here dash oh yaml It's basically you're setting up authorized keys, right? So I'm going to Do I'm going to go over this more in detail, but essentially when you bring up a machine a machine config is how Is how the machine gets configured hence the name machine config and it has all kinds of stuff and it's based on ignition, right? So You know, I can do a whole hour of just this which is why I'm gonna deserve it the rest of this for for another episode because You know, we have a few 10 minutes left here. So Um, one last thing I want to talk about is it so this is one way to managing it, right, right? another way of managing this is actually Setting up machine autoscalers, right? So if I do um, so let's go to my repo here is I have A machine autoscaler. So let's go over here overlays cluster two. I will edit this to cluster three, right? Um So I have an autoscaler setup, right? So like this is this is an example of autoscaler, right? So I want to say I'm going to target this machine set And I want minimum of one replicas and a maximum of three replicas, right? So what is that? So that's kind of like the lower end and higher end, right and this will Um, it won't go lower than one, but it won't go more than three, right? So that's and you know, I kind of have the similar configuration for each one of the machine sets But what you do need to Um You need to map a machine of an autoscaler to this, right? So because it won't work without an autoscaler definition, right? um So I want to say I want a max Total of nine nodes, right? And that's what my autoscaler does Um, and these these are kind of some of the things that you can set up, right? Like Delayed after add rich means like how long to wait After you add it for it to trigger another one, right? Delayed after delay how long to wait, you know after you delete it to trigger another one, right? So you can kind of and you can find these in the In the dock, right? So let me put the dock here autoscaling. Yeah, there you go Nice. Thank you. So yeah, so it's just info about the The machine autoscaler, right so you can target things like Not even just total nodes. You can do I want minimum number of cores, right? So you can you know play around with this As as you need to write so um So I basically target the autoscaler The autoscaler sets up like okay what I want to autoscale and then um in the machine autoscaler actually schedules the machine set, right? So let's Let's go to Um Where am I here? There we go. Oh see where am I? Machine autoscaler, right? So let's go to the base first Um Open machine api, that's still the same Um Cluster autoscaler, that's the same doesn't matter. This is generic enough to where it doesn't matter where where you have it here So but my overlays, right? So let's do the the pixel The pixel is it open pixel thing here Or I have cluster three and this is basically um the same thing East 1a to uh, which is this called here uh oc get machine sets Let's call this uh us west 2a And then we're gonna target Whatever you're gonna target we're gonna target this guy here man All right, and then um Uh for the sake of time, I'm not going to do the other ones. So let's uh remove these guys Uh, but I do need to update The customize file because There we go All right We're ready to rock and roll here. So Get add Get commit dash am updated Write something something useful Um get push Uh, what do they do? Up these? Oh, did you not? Do a get pull and then uh, oh, you're right. Did you yeah Get pull uh Only just fast forward to me. Yep. Okay. Get push Go go go Oh, no, it said fatal not possible to fast forward aborting. Uh, oh Get rebase. Are you rebasing now? Are you pulling? No, yeah, let's just do that. There we go. That's good. Okay. Wow all that for one line. Awesome. Thanks This goes back to a conversation I was having with somebody in the kubernetes community today It's like I know we do everything to get to make it trackable, but oh my god Yeah, the amount of stuff we do and get everything we do everything I get through in the kubernetes community Yeah, you want to tweet from the kubernetes account? Get So here, uh, so here I have this um auto scaler up. So here I can add this so if I go, um Let's do oc get machine autos scalar Right, it's not there. Um Yeah, so let's add so application name is autos scalar app project defaults Normal normal normal things here get ops examples to auto scalar overlays cluster three back guy and then uh, I forget Name space this guy here Right, uh, I got a warning. What is that? Oh, yeah, that's fine And so, uh, there we go. So it creates the the autoscaler, right? Um, I go here Yeah, so it creates the autoscaler for me Um And so you can control it that way as well, right? So you can control it either through um All through uh just Exporting the machine sets directly or um Doing it with uh The autoscaler, right? So it depends where how you you know, where do you want the control, right? The autoscaler is kind of nice because um You can set lower and higher limits, right? And so it'll it'll scale I mean you can tweak those right so it'll scale based on those limits, right? So it'll scale up when it needs to and I'll scale down when you know when you don't need it So you kind of set lower and higher limits. So that's kind of nice If you want a bit a bit more control you Export the uh the machine set itself, right? And then you control the scale via pull request or whatever. So, um By the way, if you're curious what this warning is it's like hey, you're managing A namespace in two separate apps That's a good one. Yeah. Okay. Um, so by the way, you can't do that. You manage the namespace in two separate apps, right? So it's um, yeah They are The way to handle that is something called the app of apps design, right? I'm gonna go through that. Yeah, I'm gonna go through that actually in Yeah, so I'll put that You guys you guys can read up on this. I'm actually gonna go do a stream about this. Um, actually um, you you know this christie made us Uh give like five months of Show ideas in advance. So like I know what's up. Well, we're trying to we're trying to up level your show Yeah, yeah, we're trying to level up Bring the cache. Yeah, we're trying to make we're trying to be legit here. Um No, but to be honest with you, uh So we're working with our illustrious team at The corporate level now to kind of get open shift tv more polished and prim and proper and to do that You kind of like to market future shows. You kind of have to project out your schedule per se. So yeah It's a yeah, we went from doing this ad hoc To oh wait, well first this week we're doing the day before. What should we do this week? Yeah, like the day before Now we actually have like planning meetings and stuff. So Yeah, we've evolved so much I went from what was it? Uh, I went from Pikachu to Raichu, right? There you go So, yeah, so, you know, it's kind of like pick your poison. I well, it's what's really cool About get ops is that you know, you can kind of tailor it to What your environment is right as long as you are Um, you know following the principles, right everything and get everything through pull requests You can kind of tweak it to where um where it fits for your environment, right? You know, I show you both examples because none of these are right, right? None of these are wrong. It's kind of like it's just you can do it want to do it. Yeah, you can do it all works for you Yeah, exactly. Yeah, so you can do auto-scaler. You can do it, you know, export it have a little bit more control So, um, so yeah, so cool. Um, awesome Yeah, I think this is the first time we ended on time Yeah, ever. Yeah, I think ever so, you know, we we are we're growing here. And is this is this The second episode without issue we've had Yeah, it's not officially over, but I think this is this is not come with but um, it's the second episode without issue. So awesome So yeah, so we are making progress. We are making progress. So yeah, so feel free to reach out to me. I'm on uh, I'm on the twitters, right a christian h 814 So you guys can find me on twitter feel free to uh Don't be a tweet there or i'm linked in if you find that. Yeah I'm chris short. I'm always answering uh tweets and dms. So feel free Yeah, uh chris's dms like you need you need a uh a pa for just for just the uh, so like I need somebody to take like the external messaging stuff And bring it internally right like that's that's what I need right like manage the the linked in messages and dms and emails from external sources And and give it to me in like a single place somehow. Yeah, you need like a single Plane of glass that you can write exactly exactly that would that would make my job so much easier Yeah, if you don't know chris is like in the kubernetes world in the cncf world a devops ish He's an ansible world. He's I don't know how many of you there are i'm sure they're more There's a bunch of chris shorts running around actually doing this And i'm only getting the openshift tv one. Yeah, maybe Maybe Uh, but yeah, great show today, uh tomorrow. We only have one stream and but we'll be talking about Silo's and other windmills Kevin bears joining us. So yeah, it should be a good show There you go. Don't miss it. Yeah. All right. Awesome. Thanks y'all. All right. Thank you everyone. See you later. See you