 All right, I see it going. Hey, everyone. Hi, Christian. Hey, Eric. How are you? Good. So we've also got, as you can see, we've got James Lebaki in Word, and hopefully soon indeed. With us today, we're going to be playing with packet.net. But while he sorts through various livestreaming webcam type issues, we will just kind of have a chat about random things. Yeah, we were just talking about compiling kernels. Because he brought up the camera and was like, well, maybe he has to recompile the kernel to see his camera. Yeah, right? Yes, we're talking about recompile. And so Chris Short will be able to chime in on this. So the Fedora RPM Fusion version of OBS, but also the Flatpak version, neither of them for some reason, ship with the browser plug-in, which now is part of OBS, which is the streaming tool we use, has a version 25. And so I spent the afternoon the other day like trying to build my own RPM, which actually works great until you try and touch a media source, and then it crashes. And it just cracks nice. So I need to spend some more time doing that. But if anybody out there in Twitchiverse wants to RPM package a Fedora-esque version of OBS, we will gladly contribute to said project. Oh, yeah, definitely. Yeah, definitely. But I'm sure it's some good. Well, especially the Flatpaks, I've been using a lot more Flatpaks as of late trying to, just because it's just easier. I don't have to worry about RPMs and dependencies and whatnot. So I kicked one of James's James out because he was in twice. And now it's saying it won't let him. It's saying it won't let him join. So am I sharing the actual part of the Zoom window? Put the chat. Oh, yeah, I am. That's all right. I'm trying to figure out how to allow him to come back in now that I've accidentally kicked him out. Yeah, you accidentally kicked him out. So hopefully we'll be able to let him back in. Otherwise, this is going to be real interesting, slash exciting. We may have to close this meeting and then restart it. But that's OK. But yeah. Oh, so the Flatpak thing, I looked into that too. And they were saying that the Flatpak people were saying that it's like something weird about. So the browser for OBS uses CEF. It says he can't get back in. All right. So I'm going to kill this Zoom meeting and then restart it. So Christian, I'll see you back here in a second. All right. Yeah, we'll be right back then. We'll be back momentarily, folks. So let's see. We're going to do, we'll switch to the break scene. Yeah, break scene. We'll take a break. Hey, we got three heads now. This is great. James here. Say hi, James. Hello. Hi, James. Say hi, Christian. Hello, hello. All right, so now that we're all back-ish, sort of kind of what we're going to do today is we're going to play with some bare metal open. Oh, I probably should get rid of my work grave here. Mode suspended. We're going to do some bare metal open shift. And we're going to do it using bare metal servers from packet.net. And so the OpenShift 4 installer does a lot of fully automated things. And we're working on sort of a fully automated bare metal provisioning solution as well. That's coming down the pipe. But we have this sort of manual installation process that you can do as well. And generally speaking, for bare metal, that's what you would do. You would follow this manual installation process. Even though packet.net is technically a cloud, and it's technically a bare metal cloud, where you have an API that you can get infrastructure resources from, we're not tying into that in any way. So the OpenShift 4 installation today doesn't know anything about packet. They're not on our short-term roadmap. But James has actually, so spoiler alert, this does work. And James has done it before. Now, this is live. So of course, we're going to completely hose it in real time and recover, I think, maybe. But the goal essentially is to go through the steps that James had done before and maybe actually try and automate some of the things that he's doing. Do some live Ansible or something like that. I don't know. We'll get crazy. It's Friday. I don't have a beverage. I mean, I have a regular standard water. Standard water, yeah. Well, technically, it's not even standard water. It's got collagen in it because, you know. Oh, yeah. Take care of my joints. The joints are important. It's important. Yeah, it's important, yeah. It's like, you know, similar analogy would be, you know, if there's no hoof, there's no horse. So if there's no joints, there's no person. But anyway, OK. So enough of me yacking away, James, I guess. So Jell-O-Water. No, it only turns into Jell-O-Water if you use cold water. If you use room temperature water, like out of the tap, it's fine. It doesn't turn into Jell-O. And it really doesn't taste like anything. Like, you kind of don't really notice that there's stuff in there. But anyway, we're not here to talk about collagen supplements. We're here to talk about OpenShift. And so, James, I guess, if you want to share your screen. Yeah, that's the first test of the day, huh? First test of the day. First task of the day is to share the screen and see what happens. Looks good so far. I see your screen being shared. I'm going to make our heads a little smaller here. Try to. There we go. There we go. Split in the screen. I like it. OK. So for sure, you're going to have to make your text way bigger. OK. How's that on the, or should I just do one screen and flip back and forth? I would probably just do one screen at a time, but still make it bigger, even bigger still. Yeah, that's fair. Nice. Because our heads are somewhere. I will try. There we go. Is that good? OK. That's probably good. Let's let our viewers let us know. Yeah, yeah. You can't see it. Raise your hand. Yeah, apologies in advance. At some point, my laptop will decide to do something awful. And so we will lose several frames, but I will do my best to not do anything else while that's going on. So anyway, take us away, Mr. James. All right. Yeah, thanks, Eric. And so I'm glad I have your help so you can help me as I make a bunch of mistakes. So. Pair programming at its finest. Yeah, or at its worst, I guess, I don't know. CIS admins would never do that. Maybe they should. Maybe CIS admins should pair admin. Oh, just got to follow. I don't know who it was. All right, so real quick. So what I wanted to do was walk through deploying OpenShift. I think I'm going to stick with OpenShift 4.3. I know 4.4 is out, but I did this with 4.3. I'm assuming it would work with 4.4, but I'm not going to test it live. Oh, no, no. That's too much testing for a live Friday stream. Yeah, yeah. So I'm going to do it with 4.4. We've already got one person saying we should. All right, you know what, James? We're going to do it with 4.4. So here's what we're going to do. So I'm going to use AWS Route 53 for DNS. I know that Payne's Christian, from what I understand, he's a big DNS. Christian loves DNS. I'm a DNS purist. For those of you anywhere on the internet, if you have a DNS problem, you should send it to Christian. Yes, exactly. He loves DNS. All right, and you could recite all the things you're not supposed to do in domain names while we go through this. By the way, we don't have two people clamoring for us to try OpenShift 4.4. So we may have to do some runtime or audible, playtime audible. Yeah, we do an audible. Yeah, call an audible. Do both of them work at our competitors? Or is this stuff? It's fine, man. This is all about trial and error. All right, so here's what you should know. I'll be using Route 53 for DNS. Do you have a slide? No, I don't. No, it doesn't. Can you make a slide on the fly? Yeah, it can be ugly. Yeah, make a slide. All right, so then what we'll do is then we'll provision a bunch of servers on packet.net. I have Route 53 here. I have a hosted zone. Let me make this bigger. Yes, please. There we go. At least 200%. Yeah, if you think you can't see it from the back of your own room, it's probably too small. All right, so this is on my DNS setup here on Route 53. So I'm being lazy. I'm not going to use them for, I'm not going to set up my own bind server. But I am going to set up everything else, essentially. So what are these records for? These records are going to be for everything in my OpenShift cluster. OK, well, really, no way. Yeah, sure. Well, what a novel idea. Well, hey, you ask a stupid question. You get a stupid answer. Well, I asked a non-specific question. So OK, I'll accept your answer. So you have M0 and M1 are going to be the IP addresses for the master nodes and M0 and M1 and M2. And then we have two workers, worker one, worker two. And then you have C names for at CD. So we're to find things. All your apps, you have a wildcard here entry for any of your apps basically to get forward to what's going to be the HA proxies. So that's for application ingress. Exactly. Gotcha. And then, yeah, and then you just have the regular old when someone goes to hit the API of my cluster, it's going to know where to find those as well. So OK, and then that's going to be, I guess, like a load balancer that's going to sit in front of the masters. Yeah, exactly. I'm going to have HA proxy running on like a helper node. So I'll make like a helper node that's going to export NFS, if I remember correctly, do HA proxy and handle a couple of other things. I'm trying to think what else was on there. OK, and so when you say HA proxy on the helper node, which you're talking about is the HA proxy binary like RPM that we ship with rel. Oh, there it goes. Hang on. I know. I think it's Eric. Yeah, Eric is. It is. No, I told you it was going to happen right before I. All right, we're back. So there we go. There we go. When you say HA proxy, you're talking about the RPM that ships with rel. Yeah, yeah, I'm just going to install that straight away on there and configure it. So yeah, and then. So I know I note that you have for your wildcard DNS, you have two entries, right? So I guess you're round robin-ing between the two workers. Yeah, exactly. Gotcha, gotcha, cool. And so you might ask yourself, well, how do you know what the IP addresses of these are ahead of time? And the answer is I don't. I'm going to have to go change them after I kick off the actual provisioning and I get IP. If only we could automate this with. So you could. And in fact, there is actually a Terraform. There's some Terraform and Ansible that the OpenShift engineering team at Red Hat has written to do this. So it's like a both Terraform and Ansible, I think, leveraged together to deploy all the servers on packet and get this up and running. But then you wouldn't have the fun of watching us do it. True, yeah, you'd have the. Well, but you know, we were thinking potentially about us doing it. So, but anyway, let's let's move along and then maybe Christian, you can, we'll switch to you and you can try and re redo some of the steps here. Nice. I do have a packet account. I can send you some access info in the background, Christian. Yeah, we'll just, we'll just do it live. Just send, just put it on the screen. Everyone can have it. No, no. All right. So let me, let me start off by, so we'll deploy one small instance in packet and we'll configure HAProxy, NFS, and Apache on it to start. And then we'll put the OpenShift binaries on there too. And we'll create like the manifests, the ignition files, all that stuff you need to start the OpenShift installation process. Then we'll go deploy a bootstrap node, which is something that you need to deploy OpenShift and then we'll deploy the masters and the workers. We'll probably use like small instances, I think, for those, if I remember correctly. And then we'll wait for everything to deploy. So we'll cross our fingers and maybe that's a good chance to get a coffee or water. And then we'll log in and let me, you know, make sure the cluster's running. Sweet. Sounds beautiful. Sounds like an agenda. Sounds promising. All right. Yes. I hope you guys don't have plans tonight. So we're doing this on R, right? On R, yeah, let's do it. Okay. 4.4 would be scary enough. All right, so we'll let's... Spoiler alert, it won't work on ARM. Yeah, I know, I know. So let me... Oh, wait, they added some new instances here. I was going to see... I just wanted... I was going to do this in Newark, but is Newark gone now? Well, Newark. Click North America. Yeah, but... Or he clicked on all, yeah. I think Newark would be considered under all. It might be because of what I have selected over here. So... Oh, who knows? Let me see. I wanted to just do a custom iPixie. So... Oh, Jesus. Let's see. Wow. Oh, no, no, sorry, sorry, no. I want to do... Sorry, you're right. I want to do a REL7 here. It's like, wow, that's ambitious. It's ambitious, but it's defaulted to 8. I guess it's got to be Dallas. I guess Newark... I just like it because of the airport, but we'll... What? Someone likes the Newark airport. I think you're fired from this stream right now. Yeah, yeah. The airport is the worst airport. It's our greatest chance at herd immunity. I guess it's better than LaGuardia. I'll tell you that. Oh, so Parsiphani is Newark says somebody keeps elite. It says it's Parsiphani is Newark. Oh, you know, that's a... You're right. EWR, there you go. That's a pretty good branding trick there. You know, it's funny, they do what we do. They use airport codes, but it's the city. Thank you for that, by the way. Yeah, it's fine, okay, so... Keybiz, elite. Thank you. Keybiz, elite. So let's do this. I'll hit deploy now and I don't think I forgot to do anything, so let's see. This is a very... Your new devices are queued for creation. This is very stressful. Why? Okay, so basically, you could view the progress of this. Are you concerned about making a mistake? Yeah, absolutely. That's the process, like the journey is not about the destination. Yeah, it's about the journey, not the destination. All right, so my configuration, you'll see here, you could watch the process here. There's also a way to actually, if you wanna SSH and watch this thing boot, you could see that there. Interesting. And I had previously set up my SSH keys in here, obviously, so you have to do that if you didn't, but it's a booty. And what are we gonna do while we kill time? We're gonna kill time while we kill time. You could pull up the Ansible for a packet and we could look at the docs. Oh yeah, what do you think will come up here? Pack it in a guide. Well, there's a guide, well, that's good. No, I know there's a module. You could export here if you wanna turn it off stream. Hit install it. And then try and do some of this stuff. Looks like there's a lot of things you could do, of which I will do none, and I will let Christian do them if he wants to. Yeah, all right, so let's see. Let me get my packet key, get in the background here and see about sending it to Christian. We can have a race, right? I can do it on my end. Well, no, I'm saying I have it in town. Well, I'm just saying, let's see if I can do the Ansible playbook before. If you could write it faster than James can do it while he's waiting for service. Yeah, I do it manually. I don't think it'd be any different. All right, so how do I go? I need to API, API keys, there we go. No API keys. Well, I guess I'll need one. Description, Christian, it's the name of my API key. Oh, how am I gonna send this to you? You can use Keybase. No, I just, I don't wanna open more applications. Oh, more applications. I'm very concerned with my laptop here. So I have a question for James, actually. So I've actually never really used PacketNet. So what's going on now is that it's just going out and just looking for an available hardware. Yeah, I think so. It basically pulls a piece of bare metal out for you and does all the provisioning for it. And you know, I mentioned that there's a very small creature right now running around in a data center. Yeah, that's like a creature. So actually, I mentioned how there was a, you know, the challenge of having to get the DNS, I'm sorry, having to get your IP addresses first from Packet and then plugging those into my DNS. They actually started this pretty cool project called Tinkerbell out there, but it's meant to be able to kind of do all of this and automate it a little bit. I like the name. Yeah, Tink. Yeah, yeah, so they're working on that, but... If you clap, does the server come faster? I don't know if anyone got that reference. I have no idea. Yeah, and anyone on chat, let me know if you got the reference. We've got a question in the chat about, is this first machine that you're provisioning the helper node? Yeah, it's basically gonna be the helper node and then it's not the bootstrap node. So when I was first doing this, I didn't realize that you actually need a whole second. I'm gonna pause you for a second. Can we add any more servers while we're waiting so that we don't have to wait sequentially? No, we can't actually, because everything else we have to add, we have to custom Ipixie, and we're gonna host those Ipixie files. Oh, okay, so we really do need this version. Yeah, yeah, I don't know why, I should have just provisioned it before we started. That's okay. In hindsight, but... Hey, Magic! It's over networking. I mean, it was only three minutes. I know. It just felt longer, yeah. Yeah. Three minutes is a long time when you've got several thousand people watching you live. That's true. And let me see. So this will basically get configured, and then we'll start it up and get in and start configuring NFS, and you could watch me add firewall commands. It's gonna be exciting. Packinit Ansible. So now I'm searching Packinit Ansible, and I'm gonna see here. Do I have to do a pip install or is it? Is Chris Short on the chat? Does he know if I can Ansible doc? Let's see if I have it installed. So the module's called what? What is the module called? Oh, Chris Hackit-Python. Hackit. So I sent Christian an API key in the background for Giggles, so he's gonna... Well, how do I list all the Ansible doc? Ansible doc-... Hey, it looks like... List. I think the server is up. All right, I'm in on my rel7 server over here. Awesome. Okay. Nice, look at that. In Newark. It's in Newark. Okay, so a couple of things. I'm actually gonna just provision some storage and attach it to it. There's some storage on it, but if you wanna, like, obviously not, this is, A, this is completely unsupported, deploying OpenShift on Packet, I think, as far as I know, like Eric said, this is more like just for fun. It's supported in the sense that these are bare metal systems running somewhere. So installing it and running it here is just fine. You will be supported running it on Packet in the sense that these are rel7 or CoreOS systems running on bare metal. So we don't have any supported native integration. That's the thing. There's no Red Hat provided automation today. So if Packet had some weird object storage that you could use, like that's, we don't tie into anything that Packet provides in the context of being a cloud provider. Yeah, the fact, you say the fact that you're doing a bare metal UPI install, that means it is supported. Because that's supported. Yeah, this process we're doing is supported anywhere that rel7 runs. Yeah, let me do a deploy a volume. So I just deployed a 100 gigabyte volume and then I could go manage it. I always forget how to attach it. Storage actions, not clone, delete, snapshot schedules. I think you actually have to go back to the server and add it. So this is just storage space that you're going to attach to the existing server. I guess for NFS, right? Yeah, exactly. I was just going to mount this eventually and then who knows if we'll get that far but eventually just mount this up. We got nothing but time brother. We've only been going for 26 minutes. Don't wish it away yet. Yeah, I was going to ask. I didn't actually see when this ended. Is there an end? There's never an end. It just kind of goes until it fills us out. It goes until it reaches its natural conclusion. Nice. All right, so now I've got that storage attached to here, right? When the fire starts to smolder, that's when we... Yeah, when we see smoke coming out of your brain, we know that maybe we should end. I know. All right, so we'll go here. So let's just, let's get started here. So I'm on my rel7 machine. So now what I want to do is I want to add a couple of repos. I think like, because this comes up much like, whoops. Oh, it comes really bare. Wait a second. Yum repo list, maybe? Yeah, this is 7-7. You might have to install yamkin to the manager. I don't think that comes in the... What provides? What provides? What provides you? Is that the name of the command? Yeah, what provides yourself? Do you like quote bash bin slash, or is it smart enough? Or is it, or is it Yum search? It's, it's now smart enough. No, Yum provides is fine, but I don't, I don't know how much expansion it does on file names with provides. Okay. I don't know. I was an RHC in three. So in the last three days. And rel3. You guys are gonna be making fun of me the whole time. This is pre, this is pre-SNL. Oh man, I got all my certs on five, I think. Got mine on six, so. I'm kidding. I think it was on six. I'm a little newer. All right, let's wait for this to finish here. But what Eric was saying, yeah, it's Yum provides quote slash star slash, yeah. What bin slash, no, no, sorry. Quote splat bin slash thing. Single quote. Yeah. Because if, because then it does like expansion, it does glob expansion of both bin and S bin. And so it may or may not find what you just asked it for. Or they call it the cheerleader, right? Back. No, dude, I don't know what you just wrote. There should be a way to cash this. Yeah, I think it's called Yum fast cash, but you need to go through it once at least. Yeah, I'm just saying like on the, oh, I guess it's not an image, it's laying down. It must be doing an install from packages when you boot. I don't know. Something just dinged at me, which makes me believe that I have, I do have something else open still. I need to close that, here we go. Free up some precious resources here for my. Well, you can do a renice, right? Renice dash 20, negative 20. Yeah, Zoom is the most important thing. Zoom is most important in anything. My gosh. There's a lot of stuff in real seven these days and it's all. I have no idea. Well, it's because it hasn't ever pulled any of the stuff yet. So it wants to validate its cash and blah, blah, blah. So. Yum. So there's three ways to do the same thing. Pseudo DNF, what provides whatever. Yep, that's another option. So if you're on the packet team and you're watching, if you could figure out how to run this after the install. Do it. Well, but then the install would just take longer. Like, I mean. Yeah, it's true. That's true. Well, yeah, it's like, don't make me wait. Right. Yeah. Hooray, there you go. Oh, yummutiles. Why did none of us know that? I bet a Google search would have been faster. Google search would have been quicker, yeah. But I still had to go through that, right? Correct, yeah, you'd have to get the cash. Probably. All right. RPMDB altered outside of yumm, sounds like the packet does something as well. Okay, now you should have yummconfig manager. All right, so let's add a, let's add rel7 server RPMs, make sure it's enabled. Cool. And extras. I've got some. Pretty sure it's all enabled. You never know. What's that? That didn't work. So, the other thing, I'm, yeah. Yeah, it didn't work. Rel7 servers, isn't it server? Oh yeah. There we go. Oh, it's an entitlement thing. Should I subscribe? I'm guessing I have to subscribe, right? You might need to. You might need to. Do you have your employee subscription? I do. Somebody says it was enabled already, so maybe just try and do your thing. Okay, yeah, maybe you're right. WSRI, which I'm sure is WSRI, but I also can't, I'm not sure, so. One time I might, if you notice my RHN ID has engineering in it. I once fooled somebody well enough that I was in engineering, it lasted a day. A day? A day, nice. All right, so question. Can we kind of just create like a script real quick to run all these firewall commands and cheat? They're all commented out. Unless you want to see me type them all out by hand. Well, you commented them out. Oh, I know. I don't know why. Oh, you could have done a visual block there. I know. See, these are the things that I knew were going to happen. If in case you didn't know, bind is number one, DNS is number one for me, and VIM is number two, so. Have you been doing the VS Code VIM? VS Code, yes, I've been, yeah, I've been forcing myself to do that. Dude, the VS Code VIM life, is it? Let me ask you guys, what are all these ports? Let's see here. Well, I recognize a couple of them. So if you do, we go line by line, right? I'm just kidding, I'm just kidding. We don't have to do that. So HTTP, HTTPS, NFS, you got the UDP. HA proxy. I'm guessing time protocol. Automap? Wait, no, no, no. What's RPC bind? Yeah, yeah. It's been a while. Okay, so let me just. You can just do bash it. Come on, engineering, let's. All right, all right, all right. So adding a bunch of firewall rules, I probably missed some because I took bad notes the last time I did this. And then we'll, should we, let's not configure NFS yet. Let's just install, install HA proxy. Yeah, right. Keeb says that we should be adding service instead of doing port by port. Oh yeah. Better, yeah. Same behavior, better experience though, yeah. Well, better documentation. You're probably right. And all right, so then what we're gonna need, and oh man, sorry, try to copy and paste. Yeah, Chris, Chris short says that assumes we know what we're doing. No, I said that. Oh, is that, oh, is that you out there? Yeah, no, Chris is logged in. By himself. Okay, so what do we need to edit next is the HA proxy config, right? HA proxy config, SC Linux, all that good stuff. Do we need SC Linux? Well, if you're binding to different, I forget what it is. I think it's, SC will collect any, connect any or something like that. Is that four zero? Yeah. Set four zero would do it as well. Oh, right out of the, right out of the training manual, man. Copy the default config file to a rich. Okay, so here's the new HA proxy config. That's very well. That's right. It's actually, it's not because I want to do it the right way. It's because I'm scared I'm gonna do it the wrong way. There you go. So now we have to put the new, we will have to put the IPs in once we have them, right? Yeah, exactly. So this is basically just creating a simple HA proxy that's gonna, you know, what kind of load balancing, you know, you want to do and all that sort of stuff. And then you'll see it's basically saying for answer for OpenShift API server, the backend for OpenShift API server, these three masters, so on and so forth. And I should say a lot of this stuff I got from various people's GitHub repos at Red Hat. So it was very helpful. I didn't actually have to figure it all out myself. Well, but that means that if you had to go troll and get repos, that means that our docs aren't good enough, which means we have yet again found docs bugs to find. No, well, I mean, I guess setting up HA proxy is not necessarily, you know. Oh, that's true. Okay. Well, but we should be clear. I would hope that at least the docs are clear about which ports need to go where. Yeah, they are there. Okay, great. But knowing how you take that and transcribe it into HA proxy, gotcha. Yeah, see, that's the, also I see you're missing the bootstrap on the API server backend and also the machine config server. No, he had machine config, it was further down. No, I mean, he needs the bootstrap entry for the machine config server. Bootstrap entry. All right, because doesn't the bootstrap first host the machine config service and also the API? Yeah, but when you boot the, when you pixie boot the CoreOS host, you give it that. It's not hitting the load balancer for that, it's going directly. Because the bootstrap server is the machine config server. And so. Interesting. Okay, yeah, yes. Okay, I see what you're saying here. So James will confirm or deny. Well, we're gonna find out. Well, we'll see it. Let's do it. Yes. Because then, because this will simplify my, this will simplify my configuration. If this is true, because I always have to, I remember, but maybe this is my, my information is from port.o, which was alpha long time ago. So everything has changed. I love that we have a partner helping a customer right now in the Twitch chat with regards. There you go. Ansible and configuring HAProxy. This is awesome. All right, so I will, I'm not gonna start HAProxy now because. RDO agrees with me. So. Okay. Well, we'll see. Maybe James found a better way. So, I don't think it was me. I think I took it from someone in consulting to be honest with you. And it, and it worked. So. I trust people in consulting. So I'm not gonna start. I did explode, so I'm gonna keep doing it. Yeah, yeah. Okay, so I, I'm not gonna start HAProxy because we have to change the values in there anyway. So it's kind of a move point. And then all, let's just get. It's a move point? Let's. It's like a council opinion? What is it? It's a move point. I don't know, a move. Anybody get that reference? Move. Yeah. It's for movie, right? I'm installing a patchy. And then we'll go, we'll go edit our http.com and we're gonna change this to listen on 8080, right? So that way it doesn't conflict with HAProxy. Okay. Cool. What's a patchy gonna provide for us? It's gonna, we're gonna host all of our pics, ignition files, pixie, all that stuff. Yeah. I really am worried I'm gonna butcher this up and it's not gonna. Service. Wow, look at you. Talk about real, real three days. Yeah. Oh, sorry. Yeah. I don't know who decided that system CTL, HTTP dot service start is better. You don't need the dot service. You, it assumes service. If you don't tell it what type of unit you want. Sorry. Old habits die hard. I was actually gonna write a corn shell script for those part of- Corn shell. Okay. Now we're talking. There you go. Oh man. I used to do that in my solaris days. So let's see. Actually, let's just go back here because we have. So let me go back to this screen. If you were looking at your monitor very closely, I apologize for switching from like a black terminal to a white screen. And I'm messing up your eyes pretty bad. You can see that my face got brighter and the video's changed. So it's not working. So is this a permissions issue? No, you went to port 80. Oh, 88. So yeah, you're right. Oh, sorry. All right. All right. Good. Hello world. All right. Okay, so. So just for those watching, please don't DDoS the freaking servers that work. Yeah. That could be nice. While true, do curl. And, curl and. And yeah, exactly. A patchy benchmark. For fuck me. All right. So, all right. So now what do we need to do? We need to get over to. Oh, I just got a clue. We need to go to cloud.redhat.com slash open shift slash install. So maybe I should actually. The college is getting to you, man. It's not the college. It's the. Is it in cluster manager? What do you want to do? We're trying to do. Oh, go, go back. You're in the right place. Yeah, I was just trying to show how you actually get there. Oh, how do you get there? I always get there by try.openship.com and follow the links. Follow the endless links. It's funny, you know, somewhere there's someone in UXD that could be like, like this is how you get there. And then there's like the 800 ways everybody gets there without ever. Yeah. Without ever using it. Yeah. Like I go to this Reddit feed where someone mentioned it once. And then I, and then I. I've definitely done that before for finding. Weird things. It's like I do a search for the thing and it's not the thing I'm looking for, but it's the first search result or whatever. And that takes me to. Yeah. I search for Redis post about OpenShift and then from there. So we want to, so we want to run bare metal. So I'll hit run bare metal. And then you have your downloads. So you have. Did you already do this? Or are you going to show us your key as well? You could copy. Oh, that's true. You can copy it. Yeah, I'm going to copy it and I'll put it on our page. But so I think what I need to download is the installer. Cause I'm just going to drop it on that system anyway. And then this sounds like it's important. Yeah. Sounds like it's important. It might just be. Yeah. I think I need the, let me see. I need the installer init ramfs install, which one? Which one? Let me go. Sorry. You probably can't see that. It's too tight. Making a quick note, because I ran into this when 4.4 came out, the links on this cloud. redhat.com slash openshift slash install aren't always right. I had to file a bug. So, and I had to file a bug to get the sim links updated. So always just make sure you're downloading the right version of Red Hat Core OS. For the version of the installer. So, which you are. This is 4.4. How do I get 4? How do I get 4? How do I get 4.3? Click on parent directory. Looks like I am installing 4.4. Yeah. We are installing 4.4. But if you really wanted 4.3, you can go to the parent directory and then pick. Yeah. If you click on parent directory and. Should I just go through and install 4.4 then? Yeah. Do it. Do it. All right. Where it's like peer pressure. At least that way I'll find the differences. By the way, Andrew suggests that you simply copy the URL and W get the files from the server. Oh, man. I'll tell you, people are smart. I need to listen to more people. All right. That's right. Let's just make a directory called install in here. For phone. W get it? Or curl dash O. However you guys want to. Curl dash O in the redirect. From the person who made fun of me using the service command comes curl dash O. Yeah. All right. Although I'm the person that like when I'm writing code, which really is more just like hacking around. I include the URL from Stack Overflow where I copied the code as a comment. Just so that I can find it later to figure out why I'm doing this really weird thing. I'm doing this because this and just put the link. No, really. I mean, I definitely many Stack Overflow links in my code. Yeah. Let me get another window over here. I just want to see where, whoa. Sorry. This is off camera here. At least I hope it is. I do not see it. You're doing something that you're not looking at us. Is that a Peloton bike in the background, by the way? It is. Indeed. There you go. It's my. Did you gift it to your wife? No. My wife actually gave it to me like two years ago. Oh. There you go. There you go. That was a great commercial. So hold on. What was I getting? Where's my download installer? What was that thing called? Open shift dash installer. Where did that thing go? Because let me just show in folder. Well, if you go, you could just copy the link, right? Yeah. But I just couldn't find it in my downloads directory here. I'm a little slow. Maybe slash temp. You should probably do this earlier in the morning. OK, so. Dev shim, you know. Just put it in your memory. OK, so let me just move. I'm just doing something off camera here. Sorry. We're off screen. It's OK. I'm very. Apologize for being like. This is all casual. Yeah, this is all. So while we're waiting for James, I'll do the obligatory. Where's everybody from? And they can respond and chat. And that could be whatever type of answer you want. Well, as long as it's nice. As long as it's a nice place, I guess. No, I meant like PG-13 answer. Oh, OK, OK. Although I don't think we have a definitive answer about swearing yet. I know that Mr. Dobies has a mouth like a sail. Dobies dropped the F-bombs, so it's. I don't know. I'm like, I'm still being normal because I'm trying to be nice, but. Yeah. Oh, we got somebody from Norway. It's like, if I was in Norway, I would be asleep right now. It is definitely almost bedtime for me in Norway time. Well, Norway's, well, they're starting to get to the point where the day is really, really, really long to get really long sunlight. So they're up north. Belarus, Arabia, Libya. Oh, previous. Although I don't know. I only know that one word in Russian, so. New York City, all right. Uh-oh. My Chrome just froze. Let me see. Oh, baby. There it goes. Wait, it's starting to do something. Oh, for some reason, I can't click on that one installer page. I have to do this, OK. OK, so I downloaded the command line tools. I downloaded a bunch of the stuff over here. Thank you. I got this red. I've got an MFA. Everything. Yeah, I think I need everybody. I have everything I need. Gretzi. Christian, is our word. Oh, my installer is still downloading, actually, or uploading. That's word nerd. OK, so. All right, cool. OK, so I need those files. This R-A-C-O-S to be in my web server, right? So basically on var www.html, let's create something called install. Does that work? Well, it could. Let's call it deploy, because I have a directory called install my home directory. And that will mess with my head. Oh, yeah, I guess. Let me just move. Just move all the files there. I'll move. Yeah, I know. So I'll just move everything into there. I'm going to leave my installer script here, because when you run the installer, you have to paste your pull secret into there, right? Yeah, but it pastes with stars when you paste it into the interactive installer. Yeah, I'm not going to do an interactive installer. I think I'm going to use a, wait, I don't know. I'll show you how I'm doing it. But my point was that if I put that in var www.html, then my manifest is going to have the pull secret on a public website. Oh, yeah, yeah, yeah, fun. So although it would make for interesting results potentially of this up today. I'm not really worried about the pull secret thing. Then it just looks like you're really excited about OpenShift. By the way, there's no reason to, even if you were, for people watching, there's no reason for you to steal the pull secret, because you can apply, you can sign up for a free account and get your own pull secret. It's free anyway. Yeah, pull secret's free. So by the way, so it's developer.redhat.com, sign up, you get a pull secret. Nice. I like it. Okay, so let me see if my secure copy of, okay. So the OpenShift tools are still here, right? And they finished uploading. They're, yeah, 81 meg looks good. So let me just do one of these. Just a hash, make sure it's... Yeah. Okay, so now what I need to do is... Client tools, probably you need OCNs, Qtl, CubeCuttle, Qbop, whatever you want to call it. Yeah, I will need those, won't I? All right, let's figure that out in a little bit. Actually, I could use most of those from my laptop, right? Yes. They should be backwards compatible. Once the cluster's running. So you say that with so much confidence. I like it. All right, OpenShift install, create manifests, dash dash directory equals, and we're going to create them in... Oops, I guess I need a directory first, huh? What should we call it? Twitch. Manifests. How about that? Am I the first one to come up with that idea? No. You think? I created a Twitch something the other day. I don't remember what it was. Oh, when we did our metrics one, we had a Twitch metric that was a random number generator for how many viewers we had. This isn't right. Wait, I'm doing something wrong here. There was supposed to be a way... You need an install-config. So you can do create install-config, or you can drop an install-config.yaml directly in that directory. Oh yeah, I know what I need to do. So I make the directory Twitch, and then I create this file. Correct in there. Yeah. And there's a copy button, I think, on the top right. The top right corner. Yeah. Look at that. Okay. Did you hit Control-C? To install-config.yaml. So basically, this is what it's going to look like. You guys can tell me how poorly I don't understand how to do all these things, or how bad I am at all this stuff. But basically, let me look at this. I just want to see. I have a little cheat sheet here on my side where I was doing this kind of stuff before. Okay. So base domain. This is going to be my domain name that was in AWS in Route 53. So this will be demonstrate.net. Did you really buy that? That's fun. Yeah. I like it. I like it. I couldn't think of anything fast enough. So this metadata name, this is going to be my sub-domain. So if I go back to- I believe we call it the cluster ID, right? We call it the cluster ID that- yeah, well, you have tests, right, in this- Yeah. I don't know what the right terminology is for that. I think it's cluster ID. I think we call the cluster ID. What did the doc say? Oh, yeah. Let's go look. Here on the page, metadata six. The cluster name that you specified in your DNS records. We were both right. We were both in the right. All right. We'll let you know when that happens. All right. Shortling your SCAT. All right. And then poll secret. Obviously, we're going to replace that with my poll secret. And then SSH key. This is where you should place your SSH key you want to use to be able to access bootstrap master and workers. So I'm guessing what I should probably do is just drag this off screen, copy that in, save it, and then we'll be good. Yeah. Yep. Just give me 30 minutes. Yeah. So let's, let's 30 minutes to make that update. I don't do well with everything everybody watching, you know, it's really scary. Although we're hitting, we're hitting 45 users. So we're getting close to mining. I told my, I told my kids to join. So that's four. Oh, there you go. Doing my part. Yeah. That's right. I'm trying to think. Oh, they thought, they thought you were going to play Minecraft or something though. Yeah. Something exciting. So someone just says more than zero worker replicas replicas. No. No. So no. So what, so the reason it sets a zero in case you didn't know is that you're telling the installer how many machine sets or machine configs you're going to build. And since we're doing this manually, the installer isn't going to build the machines for you. We're going to build the machines. So if you, if you take a look at the dock, what numbers James, if you can just a little bit, it's number, number four. So if you scroll down and see number four, it tells you to set it to zero. So you must have the value of replicas parameters to zero or workers. So there you go. If you were doing an IPI install, let's say on a cloud provider with cloud integration, then you definitely need to set it to like two or three something other than zero. All right. Sorry. I told you it was going to take this long. So someone said I came here for modern warfare ended up with open ship. So it's, it might be hopefully it's the wrong stream. Yeah. You can, you can, you can see some sim racing stuff with on my drive with Eric stream someday soon. All right. Is that what the, the seat is next to you Eric? Yeah. It's my hero. You're racing. Yeah. Oh, there you go. See this racing rig. Nice. Let me see. Let me just do a couple of things. Okay. So I have my install config in that directory with the poll secret. Oh wait, what happened to it? Oh, no, it's there. I just, I'm just scared. I'm going to screw it up and cost us a bunch of time. So I'm being careful. All right. So let me drag this back over here. Okay. And you do a tree. Right. Tree. Find. Fine. Fine. Yeah. Free. I'm not found. I think give you the bare minimum, man. Yeah. Okay. So. So now we've got. That done. What's next? So now we have to create manifests or do we. Ignition configs. Wait, I'm trying to think through this. Oh, we do an install create manifests directory. We do. We had to create manifests because by default. The installer will try to make the, the master schedule of all, which we, we don't want. What do the doc say? Yeah. Anytime you ask what we should do, I'm going to ask you with the doc say. Reading documentation is not the way to learning. Thanks for the follow up, Mr. Harry's. Yeah. Smashing your head against the wall is how you know. Exactly. Open shift install create. I think it's create manifests. Yeah. It's create manifests. Yeah. And then it's. Does this mean James will need to write his own machine sets or none at all because UPI? So the answer is none at all because UPI machine sets only work in a cloud provider integrated environment where we have a machine API controller for that particular cloud. And since we're not treating packet like the cloud, there are no machine sets. So we will be using bear node definitions. And then we will. So we won't have machine sets. We won't have machines. We'll just have bear node definitions. And then when we do the bootstrap process for new nodes, our workers, when they come in, they will simply report in and then we will accept them. So, so just to clarify, because someone's saying there's no machine sets with UPI, that's not technically true because if you do an AWS UPI, you will still have machine sets. This is specific to bear metal UPI, which I think it's wrong. I think we kind of made a mistake by calling it bear metal UPI, because we should maybe call it like the manual installer or. Yeah, I mean the key differentiator is whether or not there's a cloud provider configured in the cluster. So like you said, UPI is possible on all of the clouds. Well, I shouldn't say all. There are a number of clouds where UPI installations are supported. And even though it's UPI where you're pulling all the levers yourself and it's not fully automated, we do have that cloud provider integration. So when you do that on AWS UPI, all you're doing is pulling more levers on your own, but those machine sets and that cloud provider integration still is available and can be used. So you can do machine sets, auto scaling, all that stuff, even if you do UPI in a cloud. How about VMware UPI? So VMware UPI is actually the only one we have right now. We're working diligently together with VMware engineering to get a fully automated installer. Part of the thing is if you think about what an IPI installation requires, it requires load balancers and it requires other dynamic infrastructure to be created. And we had to work with VMware to figure out what API call we're going to make to get a load balancer as an example. So that's hopefully four or five-ish. We're still trying to figure that out, whether that's going to be four, five, four, six, but it's definitely on the roadmap. Where are you at, Mr. James? Yeah, just a couple of things. So I ran that create manifests. It failed because my JSON file was not formatted correctly. So I fixed that. I ran it and now I have, actually, if I look inside of that Twitch directory, it now created a whole bunch of manifests for me as you can see that. Can you list that folder out real quick as well? Yeah, sure. Find dots. There you go. Lots of YAML. So I have a whole bunch of cool stuff that happened. And then what I do need to do is, if in that manifests, there is this cluster scheduler O2 config. And I think I have to change this to false, according to the documentation. And I think it was also according to the message it spit out at you. Yeah. Well, it's a warning, right? It says control schedule. Yeah. But it doesn't tell you how to do it. Couldn't they just have told me, just go. You may wish to blah, blah, blah. Yeah. You may want to. So let's see. Okay. So we've got. Elaine, thank you. Keeps. Thank you. Open shift install create manifests. Directory Twitch. Okay. So now we want to create the ignition files, right? Yep. So we're going to run an open shift install, create the ignition dash configs. That look right? Somebody looks good to me. All right. Let's run it. Your fingers. Colonel panic now. There we go. Okay. Success. So four dot four is working so far. This is great. Okay. So now what we want to do is we're going to go to our, so now we have everything we need to successfully deploy as far as ignition config files and manifest, right? So now we're going to go over to. Our. Web server over here and we're just going to make a. Oops. Sorry. I'm just going to make a file called bootstrap dot boot, which will be our I pixie file. Okay. We're going to need three. We're going to, we're going to have to have three I pixie files, one for the bootstrap node, one for the worker nodes and one for the master nodes. Makes sense. Because the bootstrap host is also running core OS, right? Exactly. Okay. So what we're doing is we're creating an I pixie config specific for the bootstrap host and the bootstrap host is the one that actually sort of configures the cluster. Like if I remember correctly, the bootstrap host starts its own cluster and quotes, it's just one node and then it stands up the masters, the control plane and sort of like then puts them in charge. Right? Yep. Yeah. Yeah. I, it sounds plausible what you just said. We're going to go with it for now. Hopefully there's an engineer watching who will correct me if I'm wrong. Which was super fun when we did the monitoring one. Apparently they're actually not that happy. The monitoring engineering team, but that's okay. Engineering in general, right? I don't think we said anything. They're not there. So here you're changing the IP address. These IP address is specific to the helper that you have. Correct? Yeah, exactly. So I just pulled that from the packet.net server. I probably should just have created DNS entry for it and make it a lot easier. If I were going to do this ever again in my life. Well, or especially if you were going to be bootstrapping lots of clusters. Yeah. You would probably make this a little bit easier. Yeah. It sounds like I've done this before. It sounds like I've done which one. Yeah. Also though, the reason we're doing IPXE is because we can't really, on packet.net we can attach an ISO and just do the install from that, right? Exactly. Yeah. They have a, they have an IPXE, sorry, they have a custom IPXE option for when you're deploying a server, but they don't have like, yeah, just drop an ISO onto this. I think it has some to do with the magic foo that they have going on in the, their deployment stuff. Magic foo. I'm assuming. Magic foo. Little, little trolls, right? The little hamsters running around. I was going to actually ask you guys, do you think you could, not obviously not with packet, but do you think you could leverage satellite to do this? I don't know. I don't know. Satellite supports IPXE. Well, not IPXE, but satellite does do pixie booting. Yeah. Yeah. We need to ask rich Dorito. Is this Dorito? Dorito. Oh, Dorito. I was like, what a cool last name. Do you know Rich? No. Oh, okay. That's a surprise. Sorry. Yeah. We need to not do that. That was the thing that Chris short posted the other day. Let me rephrase that. Wow. I really need to introduce you to Rich. There we go. Okay. There you go. Yeah. He's a, he and I were actually both going for him, me and Scott McCarty were all trying to make the transition from sales engineering. Oh, all at the same time. And so we were all in Raleigh. Which we discovered we were all there because we were all interviewing for the satellite TMM role. But it didn't dawn on us until we're sitting in the lounge at the Marriott. You're sitting in the waiting room. No, no. We're sitting in the lounge at the Marriott. Like with Chris Wells basically, who was running that group at the time. Oh, funny. Marketing group. And we were like, Oh. Oh. Like nobody said it, but we all sort of looked at each other at the same time. Oh yeah. So, which got the nod for that one. And then I ended up coming to work under Dan Young's. At the time, as the first open shift TMM. And then Scott ultimately ended up moving over to portfolio marketing and then into PM. So yeah, I forgot to replace while you guys were talking. I forgot I got to replace all these image names with four, four, three now. That was another reason not to do for. So what? Another reason to not pre-prepare. Oh, I could probably do like substitute four, four, three. I was going to say you can run a set script. Yeah. I understand. But by the time I do that correctly. Yeah. What, didn't we say like, why, what did you post Eric the other day on Slack is like, why, why do something once? Oh, here. Wait, no. I was just looking through my, my tweet stream because one of our viewers asked about me, I think, annoying engineers. And so the quote is from a guy. It says never spend six minutes doing something by hand when you can spend six hours failing to automate it. Yeah. Exactly. That's, that's what, that's when CF engine was the predominant. Wow. It's not nice. Yeah. I'm just kidding. I'm kidding. So someone, someone posted the actual actual said command. Yeah. It's actually pretty cool. Well, it's the, the syntax of the actual said command. It wasn't the actual command to run. Like you couldn't just, Oh no, somebody did post the set command. Yeah. You were talking about, I was looking at the VI one, not the said one. Go ahead. I'm looking at this. This didn't, this didn't copy and paste correctly. Did it? Don't ask me. This is, I've never used I pixie. So this is, this is your, you can do a set list. So if you do a colon set list, it'll show you where the line breaks are. Yeah. You guys are really making me nervous here. So this kernel line is supposed to be all one line. That's good. Okay. And then, then the, and then the, then the, then the. That's, that's our main. I've got to, I just got to, I mean, we find out pretty quick when you pixie boot it, nothing works. This should be. Yeah. We're not, we're not being good pear admin. Programmers. Yeah, someone, someone just says there's a delay. There's a delay in chat. Yeah, that's, that's true. We're probably a good 30 seconds delayed. Blame the internet. In the end to, it's a series of tubes from what I understand. So many. So many tubes. tubes. All right. So I know people are all getting upset with me right now for editing this all by hand. But nobody's getting upset with you. They're just laughing at you. Small shell script. Yeah. Okay. So did I miss anything? I changed everything to the IPs. I changed the version. So now I have these three files, right? Master bootstrap worker. And then all right. So now what are we going to do? We're going to I pixie our master. No, no, it doesn't matter. We're going to do all three of them at the same time, right? Well, wow. You guys, you guys, you guys do your bootstrap first. I just usually provision them all and then just go update my DNS records, cross my fingers and see what happens. Do whatever you want. I mean, it's your show. Yeah. All right, cool. I'm not telling you not to do it. Yeah, not intentionally unhelpful. Yeah. So we'll do some x86 mall. Just for fun. And then we'll do custom I pixie. Let me just make sure I can actually get to that. That you should probably start your HH proxy stuff. Well, no, I got to replace the IP addresses first, remember? That's why you don't get the IPs in order to get the IPs. Yeah. Can someone put the said for that? Paste that in the chat. If you have the said, okay, well, we'll do, I guess we could do the bootstrap first. Let me see. Yeah, let's try this. Okay. Wait a second. I feel like I was just looking at my notes and you have notes. You are such a cheater. Yeah, I have notes. I'm notes. Man, you're trying to script. It's actually a blog post, but that's why it's not working because I wrote it afterwards. And then I was trying to go back in my history of my command line. You can never do that. It's always so hard when you do something cool. And then you're like, I'm going to go and write a blog post about this. And then you have to go back through the dirty mess of whatever. Through your notes. Yeah. Oh, yeah. Notes. You took notes. Yeah, you took notes. What? Okay. So actually, I guess let's do the bootstrap first. Do we have to give that a domain name? I'm trying to think of how this works. Is there anything in DNS that's that relies on that? No. So the idea, so the host name gets set because of the reverse DNS. Okay. So it'll look up at its own IP address and whatever name it resolves to, it'll set that as its host name. Okay. So I could just literally... I don't think the host name here is important. Correct. Yeah. Yeah. Because nothing is looking at the host name for the server. Unless that's how they set up their DCP or whatever. I think it'll work. Let's give it a shot. Yeah, let's do it. Do it live. So we're going to, again, so custom IPixie. By the way, all this stuff is accessible via API too. So like all that stuff could, you could definitely blow this out with like Terraform, Ransful or whatever you want. Yeah. Christian and I are going to do that on a future Slack, or Slack, on a future Twitch. I thought you were going to do it now while I was doing this. At the end, you were going to be done writing it all. I wouldn't be done writing. Yeah. Yeah. And you're going to press the button. Okay. So that's coming up now. While that's coming up, let's just go, let's just go for it. And while we're waiting for that to come up, we'll provision the masters and the workers all at the same time. Okay. Percipeny, X1 small, custom IPixie. And we want the... Do you choose New Work because that's that's the closest data center to you? No, I just love the airport. Oh, well, you weren't kidding. I just thought... I thought you were making a joke. Definitely Dallas would be closer to him, I think. Okay. Okay. Which one? Oh, sorry. This is really probably... Or Love Field, or Love Note. What was it called? The Twitch with the fully automated UPI would already be in after-party mode by now. Mode by now, that's right. Don't rub it in. And then M1. That's my other one. Is that right? And M2. Do you guys remember? Yes, that's the one. M2. Okay. So let's go ahead and... Only because you can't switch screens or your laptop. Oh, yeah, your laptop will explode. I can't do much. If you see me looking down, I'm probably looking at my phone. Okay. So let's see. These new devices are starting up. Let's see if it works. Well, it says... Oh, no. We were unable to create your devices. The facility EWR has no provisional X1 small XC servers matching your criteria. Oh, no. I did have a problem where I had to message someone. I had to message someone at packet before because I ran out of... Like it said, we're not allowing you to provision more than five servers or something. Maybe they flagged me as a bit... No, they legitimately just might not have any. So funny story. We just had Red Hat Summit, as many folks know. We did a lot of virtual labs. We were running most of that infrastructure on AWS. And we actually got panicked phone calls from Amazon because we had used all of their M5s. Amazon is like, can you not? Because we have not... Can you not use all the M5s? Yeah. We used all the M5s. Somebody says packet has a Slack instance bugged them there. Hey, guys, you should come on our Twitch and watch us mess you up. Yeah, come on. I actually... I messaged them earlier and told them on their Slack channel that we'd be doing this if anybody wanted to watch and see my really bad sysadmin skills. Yeah. If anyone from packet is on, chime up on the chat. Let's see. I just used the mediums. Well, it's like $3 an hour. So it'll cost me an extra few bucks here. When I ran this, it was... I think it cost me $70 a day. You have no more mediums, looks like. Oh, we were unable to create your devices. Huh. This is not good. This is the one thing... I was expecting to fail, but not for this reason. This is what you get for using Newark. Oh, man, we do have a guy from packet on. He says, hey, I'm here from packet. He's checking. Oh, look at that. See exactly. Yeah. See, that's what happens when I tweet and I tag them in it. Somebody's like, hey, you should look at this. All right. So... Because I was about to... So just try one. I think the issue is just they're short on... Well, he's looking or she could be. I don't know. I don't want to assign gender. Right now, there's somebody running around with some cables. Yeah. So cables. Yeah. Remote hands is running these servers right here. He said, this person says, try T1 small. Would that be big enough for CoroS though? It might not be, but it's probably going to fail. That's good enough for a worker. Yeah. That's good enough for a worker. It would work for workers. I don't think it's going to work for masters at all. For a master, no. Or use C1. I think I just did C1, didn't I? I just... C1 large. All right. No, you did medium. So should I try S1 large? Scroll up. Is there a C1 x86? It was a C2 medium x86, which would be big enough. That's the one that I just tried and it didn't work. Are you sure that's the one that you just tried? I thought you tried a C1. That's arm. C1 large is arm. But is there another C1? So far, I tried C1 small and it said not enough and I tried C2 medium and it said not enough. You know what I wonder? If there's not enough, why even show you? You would think they would just show you what's available. C1 small, the person says, we should try again. T1 small? C1. Charlie 1. C1. Charlie 1. I think I just tried that, but it didn't work. Let's try it again. I'd feel really bad if they powered down somebody else's server just now. That would make for interesting live television. Or select a different data center, someone said. Yeah, but the problem is I... Oh, here we go. All right. Hey, thank you. Whoever did whatever. Yeah, whoever did... Yeah, whoever racked those servers. Thank you. Yeah. Okay, so we'll let those get started and then I'll just do the workers while we're at it. Yeah. Did I just do C1 small? Percipity. Yeah, how do you give them the little... Because we can give a little prize thingies now, right? I don't know. Can we? I think we can receive bits. I don't know that we can give them emotes only chat. Followers only? Slow mode? Yeah. Yeah. Emotes only chat. Followers only. Slow mode. Block terms. We can block some terms. Okay. I don't know. Intro word or phrase. W2. What's going on with the boost shop one that you made? I don't know yet. I'm too nervous to look. Just close your eyes. Is it W1? Okay, so you went with master zero? Yeah. Both with a worker. You went with one. Yeah. Okay. The workers never started zero. Interesting. Okay. They're inferior. They're inferior. That's right. All right. So last one, we'll try and boot the workers here. There's no way that this actually works on the first time, by the way. I just want to be clear. Okay. So the way you check the install is you run, what is it like? Open shift install, blah, blah, blah. Wait for bootstrap. Yeah. One of these, huh? Yeah. Wait for bootstrap complete. Yeah. Sorry. I should have copied my copy. Such file directory. I always do log level nine. Just give me everything. Is that right? That looks good to me. Oh, I don't know. Because you're already in deploy and it's trying to put stuff in Twitch. If your durr is Twitch. I went in the wrong. I was in the wrong. I was where I copied all that stuff over. So oops. If you need additional devices, next one up is M1 extra large. Oh, sorry. Zero is already life-cycled out of the cluster. What? I think we need zero. What is this? Error executed. Except zero args received one. Because you have log level info two. So what is that two at the end? There you go. That is me making a mistake. Okay. So now we're waiting, right? While we wait, we could also probably... We could wait while we wait? We could. It's an option. So, okay. What happened to my worker too? It never deployed. It never spawned up. Yeah. Maybe it ran out of Cs while we were watching. Maybe it ran out of Cs. You could try. I've had some weird things on the web UI with packet for a while, but I think part of it might just be Chrome or something. I don't know. I don't know. I don't have a lot of luck with Chrome. Maybe try to deploy it again. Yeah, I guess I should. I'm just worried that I'm missing something and it's going to all of a sudden... Awesome. Now we have people in our chat talking to the packet person about how they can use packet. Nice. There we go. I think that deserves some credits in my packet account. Yeah, that's right. Yeah, some free beer advertising money here. Yeah, exactly. Yeah, or racks some more servers or something. Yeah. Or I'll just give you my shipping ad. Yeah. Give me some of the super micros that you have lying around. W2. I don't want any servers in my home. I was going to say I waffle back and forth with putting like a server chassis, replacing my tower chassis in my basement. That's my file server with an actual server chassis. And then I'm just like, I don't even go in the basement that often, but I just don't want loud fans. Yeah. They're out of C1 Smalls. We ran a lot of C1 Smalls. All right, so just pick something random for a big one. Okay. Which one should I do? Yeah, pick the one like with eight CPUs and... We only need one of them and we're not running it for a long time. Are you chipping in on this or am I paying for this myself? What cost center are you? Hold on. Yeah, right. There's 29 and one extra largest available. Oh, shoot. I didn't update my DNS records. That's going to hose me. Yep. There we go. Okay, so let's go quick and do that. M0, can I like split screen this and make it faster? Even if it's hard for you? It looks like, well, hold on. One or more. Oh, there was an error message for you that you ran away from. Sorry. I'm just going to do this now because I think this is going to be... Oh, wow. That's an awesome rendering. Let me just... Yeah, apparently the AWS web console is not designed for mobile. Yeah. Let me... I'm just going to go change these edit records. So why is this... What's going on here? Let me just go back to this. Yeah, you chose the wrong record set. You chose SCD. Yeah. Or you chose the CNAME record, not a... The CNAME record, not the A record. Yeah. The 96 save. Okay. We'll do that one. We'll take M1. We'll change it to this. We'll change M2. And we'll change our worker. We should have the open stack guys to try and install open stack on PacketNet. I think they did. I think actually a lot of the telco team works. Did that? I think they work with them regularly. Nice. I like it. I mean, honestly, like it's been good. So this 39.3, this is going to be what my... This will be my helper. That would be the helper one. Yeah, the helper one, yeah. The API is going to be the helper node permanently or temporarily? HAProxy. So that's for HAProxy. Oh, right. Yeah. Yeah, my bad. Okay. And then this will be the workers. Yes. Assuming you have one round around each worker. Good old round, Robin, DNS. DNS is not a lifelancer. Now I shouldn't need to change everything else, am I right? Everything else looks good now. I just changed everything over. All right? In theory. Let me see here. 49, 111, 69, 27, 29. Yep. And then everything else is 131, which is the load balancer. Looks good to me. Then I need to start HAProxy. All right. I got to go change that. So SSH again. Actually, this guy. There we go. Oh, you got to edit the IPs. Yeah. Yeah. Let me just... Thanks for the follow, Robin. Okay. Miss the master here. Cool. So far so good. Moving right along. Oh, I mean, to be fair, if things are going to break, this is where they will break. Yeah. I still think you need the boot shop and the API in the config, in the API section, but we'll see. This worked for you before. I thought it did. Okay. So let me just make sure I got this right. These are going to be the three. Okay. These are the three masters, obviously. Okay. So just do this. Not bad if we could actually bang this out in a couple of hours. I mean, obviously, we've been terraforming Ansible. It'd be a lot faster. One hour, 26 minutes so far. So you're still under the couple. I'm just... If I finished... I had this blocked off till the end of the day. So you stand between me and a cocktail. And dinner. Did we decide if cocktails were allowed on our Twitch stream? I don't know. Are we between you and a cocktail? Or is it really just that you don't have the cocktail handy? That's a good question. I think it's the latter. I turn off my video for a few seconds. You know why? There you go. I don't know. Why not enjoy your cocktail while streaming? That's exactly the question that we're trying to figure out the answer to. All right. Does this look good? Does it look like I replaced everything? It looks like it. We're trying to find out. Can you look at the... I don't think it's good. Yeah, I don't think it's going to start. But let's start it. I thought HAProxy just starts. Start HAProxy ordering. Start HAProxy. System CTL action unit. This is why I... Yeah, there you go. System CTL status HAProxy. Oh, bad rapping. It's running, yeah? It says active running. You can look at its log. Yeah, look at the... You have the port 9000 open, right? So you can look at the status page if you want. I don't have the port 9000 open. Oh, you don't? Okay. You're listening on it. That's just the status, right? Yeah, that's the status page, yeah. It's on localhost. So if you... I don't know if it's a text of all... 9000? Dash, 9000 slash TCP, yeah. Or you can do an SSH boarding and connect to localhost. Oh, god, dude. Come on. That's just the IP address. And then port 9000, you said? You say? That's what I said. Boom. Boom. Have it look. So it looks bad because all your back ends are reporting down. Yeah, yeah. Back end looks down. Because they're not... Well, but they're not receiving... That's actually okay because none of them are receiving traffic yet. Like they're not... They're not up yet. Yeah, it's the install still going. Well, the install is not doing anything, I don't think. So... Well, I got an SSH to that bootstrap node, basically. Correct, yeah. Yeah, so first let me SSH... Since I will need to be on the helper node before I SSH to the bootstrap, because that's what I used where my key was, right? That's when you generated the key. Yeah, whoops. Roots. Always Roots. Wow. Always Roots. Always bare metal, always root. No stable root login? No, always bare... Not on REL7. Is it... Oh, is it Coro? What's the user name? It's Coro. It's Coro. Oh, that's right. Yeah, sorry. Oh, thanks for the follow. Although it doesn't look like Xander. But... Yeah, this is not good. Is there a console? Yeah, there's an out of band. There's an out of band. Yeah, it's like a TTY or something. Yeah. Yeah, we can get into... It'll tell you what error it is. Ooh, uh-oh. This is probably the key on your desktop, right, that you need to use. Oh, yeah. Maybe open another tab. Whoa, tiny. There we go. Wow. Not much happening here. Nope. So what are we trying to do? Debug? We're trying to figure out... Yeah, we're trying to figure out what's going on because I can't SSH to the... I wonder... Should I see if the master came up? Just for fun? Same thing. You don't have to... You don't have to open a port or something on PacketNet or something? No, it's just wide open. That's what's so awesome. So you started saying... I don't have a bare metal. Oh, I see it. I started to say, oh, because I was thinking, do I actually have permissions to download these... You're breaking up. This is actually... No, you're breaking up. Oh, my internet connection is unable. Nice. We can hear you now. Yeah, you're right. Okay, so what's the question? Sorry, I started downloading these. I thought maybe there was a permissions... I thought maybe there was a permissions issue here, but it doesn't include... Can you curl it from within that network? Because sometimes... Like, can you curl that? This looks like it's too... But look, this looks like it might be... I think there's something wrong with me. Do a... Oh, okay, I see what you're saying. If you do a colon set list, it'll tell you... Well, he's not in Vim. This is... He's looking at it in the browser right now. Well, I'm saying he's... Are you talking about the line wrappings? Like, it should all be one line? Or what do you think it is? Yeah, yeah, let me... Let's go... I don't know. I'm just guessing that there's something screwed up in my iPixie configs, right? Because it's not even... See the nicks? Normally, when I boot this, I would see like... I would watch the... He walks through that SSH. I can get to the T to buy, and then I'll watch that terminal, and I can see it come up. And it's hard to catch though on packet, because like, you'll just be getting this like text output, and then it'll... You know, if your iPixie doesn't work, you kind of wonder why didn't it work. But I'm wondering if it has something to do with either permissions or the... So what do you do, a colon, what? A colon set space list. And this one... Yeah, it'll give you the line endings. The line endings, yeah. So you don't have a file in slash R-H-C-O-S. Oh my goodness. That's what it is. There we go. So this is what happens when you plan... When you plan and you take notes, and then you copypasta your own notes. Can you set... How do you unset? Unset list? Set no list. Yeah, I mean... One word or two. Not that it matters. One word. One word. It matters. I don't like those things. Yeah, it's scaring you. You don't like to make me... It's scaring you to tell you where the line breaks are. Okay, all right. So those are in a folder called deploy, right? So I basically... I don't know. Where'd you put it? I have to basically sub... S slash R-H-C-O-S. Or is it percent? What is it? Percent space? Percent S slash... No space. R-H-C-O-S slash... That's not gonna work. Slash G? Yeah, globally. Oh no, that'll work. Yeah. No, that's bad. No, because you changed the name... Oh. So here's what you... Okay, so you ready? So colon... No, I'll get it. We'll get it. Don't worry. Colon percent S. So stop. No, no, no, stop. Stop, stop, stop. Go back. Okay, escape it. No, no. Go one more back. At. So if you lead with at instead of slash, it sets the character for the delineator for the sequence to be at. So now you can do slash R-H-C-O-S. At. At. So that's the delimiter now. Slash deploy. At G. At G. That won't work though. That's not gonna work either. No, no, no. That's the same pattern. Yeah, yeah. So put a slash at the end of R-H-C-O-S. At. Older times that. Slash deploy slash. Yeah. At G, sorry. Whoa. Undo whatever you just did. Undo. Oh, shoot. How do I get my history? Oh, there we go. There we go. At G, like that. Yeah. All right. You know what? I'm gonna copy and paste that line. Copy, pasta. More confidence. I probably need it again. Yeah. Yeah. I do. I need it twice more. And while that's before I go to the other files. Stop, stop, stop. Go back. It didn't work. Uh-oh. Undo. You. Do it again. Calling up. You need a slash. Nope. Undo. Calling up. You need a slash before that. You need a slash at the end of the deploy. Yeah. There you go. All right. All right. Brush your fingers. Can we do a lot of. How many engineers does it take? Yeah. How many, how many R-H-C-E? Can we do a, um. Is that the right? Okay. Before you do this, is that the right URL? Well, actually here. Before you do that, check the URL and see that it actually works. Yeah. But before I even do that, I'm just going to terminate all these because it takes a while. And I don't. But wait, why do you need to terminate them? Why can't you just reboot them? Um, I don't know if that will work. Let's, let's try it. What? Oh, you know why? I'll tell you why it won't work because I didn't check this little box, which I should have. Oh, checking the box. That always gets me. Let me show you what I mean by that. So the problem is, is I should have said, when I did customize Pixie, I should have said persist Pixie as the first boot option after provisioning. What was, oh, gotcha, gotcha. Because now it's considered booted. I think we could try it. Let's, let's try it. Um, oh, I knew, I knew I would do something wrong. If only we had spent six hours failing to automate this. Yeah, exactly. Okay, so we'll change these. Okay. Chris, are we breaking it? Chris asked if we were breaking anything and I said we just started to break everything. Okay. So you're saying basically make sure that this file actually exists. Yes. Well, yeah, whatever it's telling us to grab, make sure that you can actually grab it. Okay, so there you go. Is there a deploy folder on that server? Yeah, hold on. Is that the name? There is a deploy, but then let me just maximize this and see what, oh my goodness. Did I just switch completely over to Google Meet somehow? Yeah. And the heck did that happen? Sorry about that. Maybe that's why your camera wasn't working because you were on Google Meet and it captured your camera. Well, the weird thing is, no, I think I just accidentally pressed some key sequence that made that. Anyway, all right. I think it's .img, right? Or something like that or something weird. Yeah, whatever. Yeah, copy link location. Well, that's the indent ram fs or whatever. Yeah. Why is it showing me the short stuff here? Why can't I get like a, let me just go. I can never. Because that's how you have a patch. That's what a patch does. Yeah, the patch has a config. Sometimes, well, sometimes I'll see the long name and sometimes I'll see the short name. I've never figured out what. Why? Oh, it looks like they changed it to be installer. Oh, they changed the. Why was this? rxcos-443. Yeah. So this is what I just copied. The x8664 is there twice. Well, it says the a636. It looks like they have it. Oh, yeah. Now they have it. Looks like they've added this. Yeah, because now we support power, right? And Z. So different architectures. Not ARM, but that's another conversation. Okay. So basically I need a, at the end of this, I need a dash x8664. Is that right? Yeah. You could have just right clicked from the folder list in Apache and then pasted. I would probably recommend doing that. Yes, I could have. But installer and it ramfs. Installer dash kernel. Okay. Then you're sure you have the right path. All right. One is iso. Okay. So you're saying just, just delete this whole line here? Don't do that. Whatever, whatever you had just done. Don't do that. Yeah. Just hit X a few times. X, X, X, X. Oh, I would have done like, Well, V. Oh, D. Oh, like copy block. Yeah. Should I do that for all these? The initial RAM disk? Well, you need to get it to be the right thing. I don't know what thing. Yeah, it just, it seems like it seems like it's just, it's just no. It should do what I mean. Not what I said. Why can't it do exactly? Why can't it do what I want it to do? All right. And then, well, I got to do this like 20 times because I got three of these files, right? That's right. Okay. So, anyone on the chat knows the first time, then you can just where is the, is it raw.gc this, right? What's that? Yeah. Do it once and just copy the file, right? Because there's less changes afterwards. But then I have to change it from, well, from is, so like, which requires less changing. How about that? Wait a second. This would be Twitch, right? Wait, what is this bootstrap.ign? Where is it? I don't know. I think it's, Oh, it's in the manifest. So I have to copy those manifests over to make them public, right? Yes. Oh, if you didn't copy the ignition file. Yeah. If you didn't copy the ignition file, yeah, the ignition file needs to be be somewhere public. Correct. Okay. I've done this once. I'm practically an expert. Right. So we need the bootstrap ignition file that was generated by the installer when you generated the manifests. And that needs to go somewhere. It doesn't have to go to that folder. It just has to go to the right place. It's got to go to that folder because it has to be accessible by the iPixie. So you, you're getting flustered, I think. So you don't literally need a folder path called deploy bootstrap packet env. You can put bootstrap.ign wherever you want it, as long as it's accessible. Yeah, no, I know. I, yes, that, yeah. I think that's what I, I thought you said. Okay. That's what you thought that I meant, but you said something different in response. So what requires the least amount of changes is you basically copy this three times and they're just, you just change the bootstrap, right? I, I appreciate the fact that you're, you're telling me this because one, you're right. And two, you're probably getting very tired of my command line skills. Well, you're probably more tired of us picking on you. By the way, seeing, seeing someone use VI for me is frustrating because I know like all the, all the shortcuts. So it's not just you, because I know like all the shortcuts and key bindings and I'm, I'm pretty fast with it. So I just twitch, hold on. Where's my, okay. Do I need just those IGN files, ignition files, or do I need everything from that? You need the ignition files. Okay. So start out IGN. Yep. HTML. That's not gonna work because you're doing an LL. Well, it'll, it'll work, but it won't do what you wanted. It won't do what you wanted to do. That's very true. The, the, the actual. I'm never coming, I'm never coming on here again. We chased you off. Just don't rage quit halfway through because then we're going to have to start over. Yeah, yeah. You got to persevere, man. It's tough. I'm not good at talking and typing at the same time either. Oh man. It's tough. All right. Let's see. Try to talk, type and then walk like that. Like, like, yeah, on my walking desk. Yeah. And you're walking desk. So now I needed to change bootstrap that last line on the bootstrap here to be just deploy deploy. Oops. Yeah. Good. All right. Is this, is this look right now? I'll copy this. Yeah. You copy this to boot to master.boot. Should I have done it to copy master.boot to master.arriage. Just kidding. Well, straight from, straight from training. And then now you, yeah, master. There you go. Because the rest of the config is the same. So. Yeah. That seems goofy that there's not like, uh, like it, well, I guess it's an iPixie deficiency. A script to automate this. No, it's an iPixie thing where it's like, you don't, um, I guess iPixie can't be dynamic. Like it, you can't be programmatic. You can't like pass a parameter to iPixie you around. This goal, the goal of this, this whole idea is, is, uh, to kind of fix some of that. Um, all right. So let me go back to, uh, where's packet? There it is. Packet net. All right. So now I think you said, yeah, are you going to try to delete them or are you going to try and reboot them? I thought I did delete them. Didn't I? Oh, I was going to try and reboot them. All right. Let me try. Yeah. Just do them, uh, the bootstrap maybe one at a time. Yeah. You're going to have to change the IP addresses again, right? If we delete them, we will have to change the IP addresses and the DNS. I don't know if you, uh, exact. Let's do the three boots. Let's try it. And then let's get the out of band. Oh, my internet connection is unstable. Sorry about that. Out of band. It's interesting that out of band is in band. Because you can have sage to it. Okay. Let's see. Super micro. Super micro. Oh, there we go. I thought it was going to, I thought these were, these were just Xboxes or something. So what's it doing here? It's a hit F12 to get the, uh, the boot menu. I think it's, uh, I don't think that'll work very well. Because technically it didn't provision the first time. So there's nothing on the disc. So even if pixie was the second option, then it should fail. Or it's whoever used this server's last, whoever used this server last, if they didn't, uh, if they didn't, uh, yeah, I'm just kidding. They didn't degauss it. Yeah. Put, put a magnet on it. Boom. I picked. All right. So. It's your internet that's not working. Well, he's, he is trying to, I pixie. So it does, it does tend to drop in and drop out again. So it, it looks like I pixie's failing to. It says file nonexistent TFTP operation. Not supported. Is that the destination server? Wait, net zero dot seven next server one, two, 31. Is that the server you're trying to reach? Yeah. Is that the help, the helper or right? Did you have to start a service? Did you have to start a service on x pixie on something? That's, that's why I'm wondering if this won't work. No. Well, let's just, yeah, exiting Intel boot agent that, that looks like. But it tried, also it tried to I pixie, but it failed to I pixie. Don't, but, but the problem is that when I initially booted it, I gave it my I pixie file. And, and then I didn't check that box to say persist this to reboot essentially. So now when I rebooted it, it's probably trying to go to some. I pixie default, I pixie try, try one new server. And then, uh, and we'll see if it works. You think I should just try the bootstrap? I just try something. But I'm going to delete all these. All right. Do it. Yeah. Do it. Or the change DNS anyways. You are. Yeah. Uh, one of our viewers says that the helper needs to enable TFTP, but I don't think I pixie uses TFTP. It does not. I don't know what I pixie does. Does some sort of voodoo. I think it's HTT. I don't know what I'm going to ask. I've always done regular pixie with boot and all that. So I don't, I'd never. Let's see. I pixie. It's hard for me to think with all this going on. This is great. So the FAQ. What does the I and I pixie stand for? It doesn't. It doesn't. It doesn't pixie. It stands for no one. All right. Deploy bootstrap.boot. Custom I pixie. C1 small. And I am in parsypony. All the wise known is Newark. Parsypony Newark. Okay. How far is parsypony from Newark? I'm assuming they're close. No. I would think Eric would know. Eric was from there. Well, I don't remember my jersey geography. One of our viewers. Your jersey geography. Okay. He got you. So I pixie says you can boot from a web server via HTTP. So there's your answer to whether. I think sort of escape. Oh, look what it did to my terminal looks so. If you do reset, it'll it'll. I kind of like it. Yeah. You kind of you're going to leave it there. Yeah. It's romantic. That's right. With the red, the red hue. So this is where you're waiting for the server to get to a point where it's powered up enough to show us something. Is that what's going on here? Yeah. Yeah. And yeah, the hamsters have to go in and. Do you have the right IP or the right SOS URL? I mean. Yeah, I think so. I think I just copied it out of here. It's like a sauce report. Yeah. It just takes a while, right? For the hardware to come up. For the hamster or the hamster. Yeah, little L's have to go in and rack it. I forget there's actually an escape key, but I don't know what it is. I think it's still there. Oh, there you go. Okay, hold on. Oh, wait. Look at that stopping. Stopping docker stopping. This is this is. We are we are observing the there built in process right now. So this is yeah packet back and stuff. Oh, I thought I thought it was the pixie stuff starting up. Okay. No, that's what it was going down back end. Well, because it booted up and wiped itself and then it reconfigured itself and now it's rebooting again, I think. It's like, oh, someone else requested me. I wonder how how did they deal with like burning? Because like, you know, having like wiping and like cycling hardware over and over again, it must be taxed. It must be hard. I don't know. It must be hard. Yeah. All right. I mean, on the machine. Here's the Ipixie. Right. Configuring the network. Say no whammies, no whammies, no whammies. I thought I was supposed to do sync, sync, sync. Oh, that's right. Sync, sync, sync, call. Looks good. There you go. Yeah, it's getting your file. It's our kernel. There we go. Boom. There it is. There's the. All right. Should we start others? Sure. Why not? All right. I'm going to start the other three and then go change DNS because that takes a while and then we'll come back and see where this bootstraps at. Yeah. Don't forget to check the box for the thing. Oh, yeah. What's in the box? Should I? The custom pixie. Yes. It depends how you're. With the way things are going. How confident you are. I would strongly recommend it. Next, you're going to tell me to make sure I back up my files to .eridge. I would never tell you anyone that ever. All right. So master.boot will do three masters. Oh, you know what? We'll do it this way. I'm zero dot test. Let me just write down that. We'll add two more. And one and two. Good, right? Yeah. All right. I think so. Looks good. Do I have the right URL? Good enough for a town this size. I just watched the three amigos on the Cinco de Mayo. Oh, did you? There you go. I don't think you have a plethora of piñatas. Okay. Those are coming up. Let's deploy the workers. I'm taking donations, by the way, for those of you on. There you go. Watching for VI class. For VI class. VI classes. That's right. We need to send you to VI class. Yeah. I thought working with computers that was the VI class. That was the class. Yeah. The school of hard knocks is what it is. Okay. I got some. Okay. It looks like things are starting to come along. Let's fix the DNS entries. Christian's favorite. My favorite subject, DNS. We were told to use m1xlarge for the workers by another packeter. Use nano. We could use nano. Nano. It was a little easier. So the APIs, again, are the helper node, right? Because that's HAProxy. All right. Oh, so I don't have to touch those. I don't know which one of those is which because they're not. The workers, I don't have address for yet. The C1 small host name is the... That's the bootstrap? Or that's the helper? The bootstrap? X1 small is the helper. C1 small is the bootstrap. Got you. Correct. Yeah. You are correct. M1 and 2. Someone says nano rules. What about Pico, right? Isn't there a fork, Pico? These guys are not getting IP addresses over here. Maybe you exhausted their IP addresses. No. The error at the top says it's waiting for something to deprevision before proceeding. Please wait a few minutes. We're in a broad state because they're cycling things. Cool. So they'll come up when they come up. Check on the bootstrap. Maybe see how we're doing. Yeah, check on the bootstrap. Yeah, let's look at it. Oops. Is that here? It looks like it rebooted. Liberty, bibbity. Bibbity, babbity, boo. Can I just do this? Well, maybe also try to SSH into the bootstrap. Yeah. Or like type control l in that as well. Maybe it'll break the screen. Where's the bootstrap? It's this one, yeah. Yeah. Okay. What happened? Do a control... On that screen, do a control l. Uh, yeah, nothing. Nothing, huh? Maybe try redoing the SOS connection. Yeah, hold on. Sorry, I'm going to make my screen a little smaller so I could see things. Out of band info. There we go. Whoops. I guess we probably should have just left it up and running, huh? Well, you do. Yeah, but I don't... Yeah. I have no history. Yeah. It doesn't give you a buffer or anything. It just... Yeah. Hmm. We'll scroll back buffer. Does that thing... Do they reboot after they install? No, right? Yes, they do. Maybe it's rebooting. I don't think it would take that long. Yeah. Connection closed because it rebooted. Well, it closed because... Oh, it closed because... Oh, okay, got you, got you. All right. This is interesting. So, it's Apache that's serving all the Bootstrap files, right? Yes. So, what do the Apache logs say? Yeah, that would be... Hold on. Let me just think about this. If I go back to... Well, should we watch one of these? The master? Let's try it. That looks good. Not overranging. Oh, it's restarting. So, maybe it's just taking a long time to reboot this thing, the Bootstrap. Seems like it's a really long time. Oh, is it in an iPixie loop? Isn't it going to go into an iPixie loop now? It may. Should. Yeah, I checked the box to say always iPixie, right? Right. But the way that CoreOS works is that it'll iPixie and then the ignition file that it grabs, the ignition process looks at the system and goes, I've already been ignitioned once. I'm not going to do anything and then I'll just continue booting. And so it exits. Don't ignition me. Yeah, it's supposed to, at that point, exit. Okay, so this thing looks like it started rebooting the master. Cool. What about the Bootstrap? Yeah, what's the escape here? Terminate connection. It's showing you. Till the dot looks like... Yeah, it's not working for some reason. Or worm, yeah. That's the... Hey, this is my Bootstrap. The bootstrap is up. No? Yeah. Oh yeah, it's up. You have to do it from the... Yeah, there you go. Wrong key. It's up and you got a failed... Failed unit is probably better than not getting in via SSH. Oh. Is that me or Eric? It's me. Eric, Eric sounding like... So someone asked if you want to change the configuration for a CoreOS node. Do you change the ignition file? So yes and no, right? So when you first set up a CoreOS node, yes. You have to change the ignition file. Afterwards, you will use the OpenShift's machine config operator, which uses ignition like syntax in order to change. So if you want to change the SSH key or just add a configuration file, you would use the MCO for that. Yeah, so ignition is essentially like first boot. In regular Linux OS land, it only runs once. And so while what Christian was saying about ignition syntax, effectively there's always an ignition file, but we have an operator that runs on... To simplify it, because it's really complicated to explain it in detail. There's an operator that runs on all the nodes that's pulling the rendered ignition file and looking at all of the defined configuration files. And if it sees that there's a difference between the rendered ignition that it pulled and the files that are local to the host, it will rewrite the files that are local to the host to match whatever's rendered out in the ignition file that it pulled. So it uses the ignition file, it's just not the ignition process on boot. After the very first boot with iPixie and ignition, it will never do the full ignition setup, which includes partitioning disks and all this other junk. So if I remember correctly, when you're doing a cloud-based OpenJet 4 cluster, it's always iPixie booting the instances. It just doesn't actually do the full ignition initialization. It just goes, oh, I've already done this, and then it continues on to the regular boot. I believe that's what happens. I believe, I believe it. It gets even more complicated when you start talking about machine config pools and machine, but we'll say that for another... Oh, we should probably do that as a stream. Yeah, yeah, about the MCP, the MCO and Hall on the relationship. That's... I'm going back. Let's do it. So we won't know. I don't know the answer to this. I have always had different experiences. So does it care the fact that it's not named Bootstrap? It came up with the name ECR1, C1 Small, whatever. That I think will resolve that hostname though, I believe. Yeah, so I think... So yeah, that was my question. As long as it resolves, it should be fine. Yeah, as long as it resolves, it should be fine. It doesn't have to be named Bootstrap. I guess this was my question. I've never done it without naming it Bootstrap. I think that's how I did it last time and it worked, but let's, I guess we're going to find out. I don't know what happened to my terminal. Oh, you know what? When I have multiple monitors, it doesn't like it for whatever reason. Okay, so let's go back here. So I just updated all the HA proxy IP addresses in DNS in there. Let's see if I can run this. Wait for Bootstrap to complete and then also... Run the journal CTL. Do it. This looks good so far. What's this failed to create manifests mean? What am I doing? Do you know? So... It's just running in a loop wedding for EdCD and everything to come up and... I believe so. So I think... This is the Bootstrap host? Yes, the Bootstrap. These might be okay, but I don't... What are the containers that are running on this host? Do I have to do like a CRI CTL-L or something? Yes, you have to do sudo, but yeah. And I think it's podman for Bootstrap. It's... Oh, they might have changed it for 4.4. So we'll see. Podman PS, if you're trying to do that. Sorry. So it's doing something. PS-A, see if EdCD tried to start and didn't. Did a bunch of stuff. It's just not clear. Yeah, it says Bootstrap-EdCD. So I don't know if you want to take a look at that. So it's doing something teardown right now is the thing that's running. But that's why I was saying do a CRI CTL because you want to see... So podman starts the static stuff. And then I think the actual cluster is using cryo that runs on the Bootstrap host. Yeah. So CRI CTL. The same Docker command. So like PS. Oh, okay. Got it, got it, got it. So you have some container called manager that's running actively, which is probably a good sign. Actually, if many containers running. Many containers running, so that's a good, yeah. Okay, so this looks like it's doing its thing, huh? Yeah. Yeah. So the API should be responding because you have a cube API service that's running on this host. Yeah, so go to your status page. Yeah, HA proxy. Yeah. I bet you there's still... It's like 9000, right? Or whatever it was. So it's all red. So this is what I'm saying. I think you needed the Bootstrap in that OpenShift API. Well, so let's go, so hang on, right? So go back to the install terminal where it says waiting for whatever. No, no, the install terminal. Oh, gotcha. So it's waiting for the Kubernetes API at api.test, right? How is anything ever going to come up there if the Bootstrap is the API? Yeah, because it would have to be in the HA proxy config is what you're saying. I would assume at some point it needs to be. I mean, what do the docs say? You and your docs. I mean, we can sit around and poke each other in the eye until we figure it out or we could just look at the docs. You did say clown, right? I don't know why you went to look at the docs and then avoided looking at the docs. So I'm going to look at the docs. You can tell me what it says while I add it. Sure. Tell me that it won't help after I do the work. Right. Tell me that I didn't have to do it. Okay, so you're saying you got to add that Bootstrap. And the machine config server as well. What's that? And the machine config server, you're on the right track. Oh, which the machine config the one I'm on? Like the helper now? No, no, no. That's the. Oh, yeah. Oh, okay. Got, got, got. Yeah. Okay. Got, got, got. Yep. Load balancers. Before you install OCP must provision two layer for load balancers. Port 6443 machines, bootstrap and control plane. You remove the bootstrap from the load balancer after the machine initializes the cluster to 2623 bootstrap and control plane. You remove the bootstrap machine after the bootstrap machine. So 443 and 80 are the only two that don't need the bootstrap because they also don't need the master. What's, what's, what's funny is that you have your haproxy set up with see the turn green. That's a good sign. Right. Yep. It turned green. But like you, you added the workers to the haproxy even though you're not sending the the DNS to the haproxy, you're sending them directly to the workers. Oh yeah. That's true. Yeah. I guess I should change my DNS to point at the haproxy, huh? It'll, it'll work either way, but yeah. Yeah. Yeah. Yeah. Ideally, you would have a bootstrap to complete. So now we have 40 minutes to wait. Right. Well, you can, you can tell the log. We can tell the log. What's the, what's the log saying on the, on the bootstrap? Right. Because it's always, yeah. So we can, we can explain what's happening here. Um, is that tailing it right there? Yeah. Okay. Yeah. So what's happening here is the, the bootstrap sets up a temporary API server. Right. And then, um, it'll spin up an instance of SCD. Right. So it's basically we're using Kubernetes to install Kubernetes. Um, the bootstrap then will scale SCD, the scale, the SCD, um, a service, right, into the masters, and then it will scale off SCD. So then the masters now all of a sudden have SCD running on it. Um, and then at some point while all the configuration is set, um, when the bootstrap is finished setting up the masters, it'll transfer over the API to the, uh, to the master control plane. Nice. So let me, uh, go, go check out what's happening on the master. Maybe at the same time. What do you think? Yeah, let's do it. Let's see. We'll do it live. Let's get, let's get kooky. Let's get wild. Let's get kooky here. Um, nothing is happening on the master. Yeah. The master's probably waiting for, what does the out of band say for the master? That's a good question. Uh, band. Oh, sorry. It says nothing. It says nothing. Maybe it's still rebooting. That's a long reboot. Uh, it makes me want to check my, um, oops. Master makes me want to check the master.boot and make sure I didn't finger master.ign, which is there. And it's a long time I've been typing. Okay. Oh, yeah. Good. So the ignition files there, right? Looks like it. Uh, yeah. So. Huh. All right. So what is the, what's the challenge here? What's the challenge? What if we check out the other master and see what happened there? We've got so many. We've got so many masters to choose from or just wait till the bootstrap's done. Connection. Yeah. So it's probably rebooting at this point. Because the masters have to check into the machine config, um, service. Yeah. Did they time out? Uh, I don't know how long. If it does, it's a long time because, um, I remember doing it like within like 30 minutes or an hour, it'll, it'll, it'll pick, it'll pick up. Okay. Well, are the, are all of the masters doing the same thing? I just checked too. Let me check the third. I, I didn't see what's going on. So are they all doing the same thing? The first two are, I'm checking the last one. Okay. Um, oh yeah. No, this is, this is correct. So, yeah. So if my understanding is that, um, so the masters will, iPixieBoot, they grab ignition, but then ignition tells them to like get stuff from the bootstrap host, doesn't it? Yeah. From the machine config, um, machine config. Well, from whatever, right? So, so I think. Oh, I see what you're thinking. Yeah. I think the masters are actually paused waiting for the bootstrap host to be ready to get more things to do. Yeah. So the question is what, what's going on on the bootstrap host other than that, other than that is basically, so what does the install say? It's running. What is the install terminal say? Oh, you're not actually running the install. So at this point, Oh no, this is just a, this is just to watch the install, right? Oh, is it? So it's waiting for boot. Yeah. So it says, it's waiting for the bootstrap to complete. Yeah. Waiting for bootstrap to complete. This, this seems to, it seems like normal messaging. But where can we get more information about what's going on? If you do, just do a journal ctl-f, not don't look at a specific service. Yeah. Yeah. Yeah. Like the whole journal, give me everything. Actually, I wish we could like exclude bootcube.sh. What was that? That was interesting. Status for podbootstrap cluster updated to running. So I think things are starting. Yeah. It's just taking a while. Yeah, I think, it's just taking, yeah. What kind of disks are these? What size is the bootstrap server? It's, uh, yeah. Do like a free dash m or something. It's a c1 small. But even their c1 small specs are pretty decent, aren't they? They, it is, uh, c1 small. Oh, I, they don't have them. They don't have them in Dallas. In Dallas. Sorry, buffering. It's okay. Oh, he's frozen. We're back. Uh, yeah, but it's only one, it's one CPU with an unknown speed. So I don't know. It looks. Node has sufficient memory event message. That's fine. That's normal. Yeah. Okay, good. Node has sufficient. That's actually, I think it's good. Yeah. Has to be a error. There's no warning. No one has ever said warning. No error. Error success. Error success. All right. So hold on. Let's wait. Did, um, thought I saw an error went by. Oh, sorry. Let me, let me look for how to journal CTO without, I mean, you could journal CTO and then pipe it into grep to get rid of boot cube. Grip dash B. Yeah. Yeah. So the bootstrap host seems to be happy because these aren't, I don't, are any of these error messages? I don't see any error messages or nothing that stands out yet. Setting no annotation to enable my control. I'd imagine we'd be seeing some errors by now. What does the documentation say? I mean, maybe kill one of the masters and do a new one. I don't know. Maybe they came up to early. Hold on. I would say that when I did this before, it did take, you know, a significant amount of time sometimes for these to come up. Let me just try an SSH to this and see if I can then, oops. All right. So this is my helper node. So let me go from here to the. So Andrew says on the bootstrap system to look at the. Hey, it's up. Oh, the master. Hey, it's up. The master master one. Mine, we're not going to do that. Yeah. No, wait, wait, wait, wait. You're wrong, Andrew. No, he might be right. You're doing the wrong host. But the fact that it answered SSH, it means that it's up. That it's up. Yeah. I feel like I'm going crazy right now because I'm taking crazy pills. Well, you're doing this from the wrong host and you're trying to do root. That didn't work. Well, because you have to do, because you have to do no, no, no, that's the wrong host and you're trying to do root. Yeah. So you're not okay. So there you go. Core. I'm not on the wrong host. I'm on the right host. And like what Eric says, the fact that SSH is answering is a good sign. Well, that's the only sign that is important. Yeah. Hey, it's running. Right. We're just not patient. The problem is patients. Yeah. So Andrew, yes, up does not mean joined to the cluster, but the fact that it wasn't Hey, look at this means it wasn't working. Well, that means joint Andrew. This means it's up in the cluster because it's starting to, starting to turn green here. Yeah. Greenness. You're so, you're so negative, Andrew. Thanks for the follow up. I know this guy named Andrew, who's always pretty negative that I work with. I don't know if that's the same guy. It might be the same guy. It's good. They're a different Andrew. Got you. We're just kidding Andrew. Okay. Okay. So this thing is rolling, huh? So there you go. Well, so get rid of the grip V. Yeah. Yeah. So now Eric's following my my train of thought here. That's scary. Just do that. Yeah. We need less of that. Hey, look. Bootcube service exited. So bootstrap is done. Is it? Yeah. It said bootcube service exited. No, scroll up. Break out and scroll up. Break out. I saw it right here. Okay. Exited with failure. I don't know if that's good though. Yeah. Oh, timed out waiting for condition. So I did. So to answer, I guess to answer the question, it is, it did time out. So the install failed? Well, timed out. I think it's going to start again. It says scheduling restart. Oh, okay. Already exists already exists. Okay. So no, bootcube is doing something again. So this is better. It's just slow. It takes a while. It's doing it just takes a while. Yes, it did say failed. Andrew. You can just exit the masters at this point. Yeah. All right. So we should give us more information. While we watch this, while we watch this slowly move, I should have used the X1 large in hindsight. Where the bootstrap you mean? For all of them because For all of them. Because 15 minutes is definitely worth $12. That's a beer. Yeah. What? It's only like a dollar or something an hour, isn't it? Okay. It's like a dollar an hour. Yeah, but there's six of them, right? $12 beer. No, there's three masters and one bootstrap. That's four. That's $4 an hour. Those are public sporting event prices. All right. The open LA price will be $16. Complete command running anywhere. What is the output of that? So what is the install? Yes. I'm sorry? Install terminal tab say. What do you want me to do now? What is the install terminal tab say? It is now safe to remove bootstrap resources. Interesting. So should I shut it down? Well, if you want, you can. But one thing you could do would be to log in to the cluster. Yeah, yeah. Oh, well, no, wait, you do need to remove the bootstrap. Before you log in? Well, because the bootstrap process is complete, but the installation process is not, right? Isn't there something that you have to run in the installer after bootstrap? Yes. So, yeah. So here. Oh, yeah. So what do you got to run? So generating SSHT, installation program, creating the config file. Yes. So if you go into the install directory. Three node cluster, creating. Yeah, there's an auth. Yeah, there's an auth file in there. Correct, yeah. Right. But isn't there something we have to do after the wait for bootstrap complete? Yes. Yeah. So it's. You must remove the bootstrap machine from the load balancer. Must. So there's not. You must remove the bootstrap machine. Well, your bootstrap is still in the load balancer pool, but it's not part of the cluster. So you may, you have a one in four chance of garbage. Failing. So I could just take it out of the load balancer, right? Correct. Yes, that's the. That's one. And that's probably where you got, that's where you got the copy pasta from. You probably got your end, your end result. Yeah. That's a good point. You can just comment it out and then you wouldn't have copy pasta failed. I'm assuming you could put comments in HA proxy. Yeah. Future James will, will, will thank you. Not when I, not when I shut these all down in 10. Yeah, exactly. There we go. So then now go, go to the, the HA proxy setting. I just take, I just commented it out and restart HA proxy, right? Yeah, yeah, yeah. No, you, yeah, you did that. I was going to say, if you want to look at the, the, the status page just to make sure it's out, did he freeze? Oh, okay. Okay. What was it to me to move out of my way? Master, master, master. Okay. My workers still aren't there though. I got to add the workers. No. No, no, no, no. No. No, well. The workers should be coming up soon. It should come up soon with a, with a, with a router. So now go back to. The installer node. The installer node and then do an export of kube config. Export. Capital kube config equals and then whatever that has to it. Is it, I do pwd slash, uh, work. So when we booted the workers, we pointed them at the iPixie server and the iPixie server has an ignition file, but does that ignition file have the cluster information in it? Yeah. It should have, yeah. It does. Okay. Just making sure. Oh, sorry. I can't multitask, apparently. For kube config equals space. So then now if you, yeah, hit, yeah, kube config. If you do oc get nodes. Oh, I don't have the, um, where's the, can I just do a yum install oc? What is that, oc client? No. No, it's done with binary. You can, you can use the, or qBatman username and password from anywhere because this cluster is publicly accessible. Yeah. Let me just do this. Let's see if it's up. You're really worried about somebody getting the username and the password? Yeah. The photo, if their memory is, I don't want to be responsible. I don't want to be responsible for anybody's, um, anybody's bitcoin mining operation. I'm just kidding. So it would be, it's console. Console dash. Oh, here I have it right here. You can just get the routes. Yeah, I know. But I don't have the oc client. Yeah. You can, but do you have the oc client on your laptop? Yeah. So then you can log in with the, with the username and password. Yeah. You're kidding. So right now the, the router isn't up because, so you have to approve the CSRs for 4.4 now. I have never, Right. But the router's not, the router's not serving the API. Correct. Yeah. Yeah. You're right. You're right about that. Yeah. OC login. Whatever the API URL is, which is on the install page. What is the API URL? Look at your first tab in your terminal. Uh, first time in my terminal. Scroll up. Yeah, scroll up. You have a, oh, you cleared it. Yeah. Oh, you cleared the hell out of it. He, he, yeah, he cleared his buffer. Do the, um, It's the open shift install command, right? Yeah, wait for boot. It's in your DNS. Yeah, I know. But yeah, I actually, let me see. I can actually, yes, there we go. HTTPS and 6443. Yeah, uh, let me get, I am so flustered. This is really challenging. Uh, sorry. HTTPS. Yeah. Sweet. Q, bad man. What? Is it APN? No, no, no, no, it's not the issue. Oh, it's not the int one. Yeah, that's why. Yeah. Unexpected response. Yeah, you're gonna have to do this with the certificate because the API service is in up either. So the problem is that the API services, the cluster operators are, are trying to fire up. Yeah, are trying to fire up and, and. So this is new in 4.4. Yeah, um, no, it's always been this way, but I haven't, but machine config approval should happen automatically. I just haven't been able to do it. You could also just copy pasta, the kube config file to your laptop. Download the binary to the, uh, Yeah. Why don't I just install the OC client tools on my helper node? You can do it. Whatever you want. All right. Where do I get those? Try.openship.com. They were on the screen that you were on two seconds ago. This is how I get to it. The install page that you were looking at, they were there. Oh, they were there. Yeah. Oh, you were on the, yeah, there we go. Download clients or whatever it is. Oh, it's a mirror. So can I just copy the link and, and do W get? Or, uh, maybe. Yes. Yeah, there you go. Are they really that small? It wasn't small. It's 252 megabytes. There it is. Now you got OC. Sorry, 25 megabytes. So now OC get, uh, nodes, I guess first. Yeah. So now you see this is masters, but there's no workers. That's why the API hasn't scheduled yet. So now I do an OC get CSR. I just want to see it. I just want to follow the documentation to make. Oh, there you go. Finally, you finally got him. That's right. Yeah. So let's see. Get CSR dot slash OC is there. And you'll see there's a lot. There's a pending one for the nodes, right? So you need to approve those. There's a quick one liner. Uh, a little bit. You can do that. You can do them that or you can do the one liner. Yeah. Whoa. Yeah, ideally. Yeah. Ideally you wouldn't want to do the, um, the one liner. Because your XRX doesn't have, um, a dot slash OC. Yeah. Yeah. So, so the way that this works for the people who are still paying attention to this most boring of awfulness. So there is a series of, uh, yeah. So basically you don't just want to allow nodes to join the cluster. I mean, in, in certain cases you do, if you really, really know what they are, but the way that this works is essentially the worker comes up via the bootstrap and ignition process. And it basically waves it in and it says, Hey, I'm a node over here. I would like to participate in this cluster. And so that involves a certificate process. And then we're essentially doing client certificate authentication between the nodes and the masters via APIs. And so what we just did was we said, Okay, we're going to approve the certificate requests to actually finish creation of the certificate, which then allows the nodes to be authenticated. So now the API communication can finish, which means the node can finish joining the cluster. So at this point, OC get nodes should have shown that we have, if we do OC get nodes. Yeah, look, it's actually, we see that we have two running workers. And so at this point, there are operators in the cluster that are pre-configured for all of the ancillary services. CO, oh, is that cluster operators? Cluster operators, yeah. Oh, look at that. I learned something. So this was, yeah, so this will show you that. So ingress is, yeah. Ingress is the operator that controls the HAProxy router. Thank you for the follow, Charlie and steadfast. And so when the workers come up, the pods that the operators were trying to schedule can finally be scheduled. So if we do an OC describe on one of the worker nodes, dude, just put in your path. I was just thinking that. This cluster is only living for like, oh wait, OC describe. Yeah, there's no, commands don't have shortcuts. Yeah, that's, oh, OC, sorry. Do I have to do nodes? Yes, node singular or node. There you go. So now what we see is there's a bunch of pods that have been scheduled and oh look, the router is scheduled. So at this point, we should be able to visit the OpenShift console if you do an OC get route capital dash capital A. I think I have it right here. Oh, I think you got it right. There it is. All right, two hours and 30 minutes. We did it. There we go. Not days, but hours. It hasn't loaded. Well, it's like lost James again. Oh, it lost the James. Sounds like adding a server to a puppet master. Yes, it's very similar to the way that puppet does their kind of authentication approval stuff. So now August, this is for the sake of learning. Sorry, go ahead, Christian. No, no, I'll say now that this is, now that this is like, now that the cluster operators is running, now you can do what Eric was alluding to earlier was the wait for install complete command. Is that required? Is it required? No, not necessarily, no. What does it actually do? I don't think it does anything. It waits for the install to complete. The command actually does the boots. The docs don't even say to do it. It doesn't necessarily need to be run, to be honest with you. I think we're good. All right. So clusters up. This is pretty crazy. Now we can. I mean, come down. I mean, it's not. It's pretty crazy to me. The H.A. So one thing to note here is that should be all green now. Yeah, go ahead. So I'm looking at the instructions and one of the things of note is image registry removed during installation. And what it's talking about is that because we don't have object storage available by default, the operator kind of gets angry and basically turns off the built-in image registry. And so we would have to configure storage for the registry, which preferably is not NFS, but NFS does work. It's not recommended. It's definitely not ideal, but it can be used. And so was that James, why you had set up the export? Yeah, that's why I initially set that up was I basically just built a small a small bit of storage to back my registry. And then actually for, sorry, I did it for persistent volumes because I was doing this to deploy VMs via CNV. So basically via OpenShift-vert. So yeah, in fact, that's maybe what I'll do is since these are all running on small instances, maybe I'll leave this cluster up and running for a day or two and see if I can get OpenShift-vert working and import a VM or something on this one. So this is interesting. Someone in chat, yeah, it's your call center. Someone in chat asked, what about adding another worker now? Do it. What does that entail? Do it. What does that entail? Create a new worker and point it at the worker ignition file and it comes up and then I approve the CSR, right? Yep, here we go. Is that it? It should be that easy. Should we do it? Make a really big one. Should we try it? What's the biggest server they can give us? Yeah, what's the most expensive one you can do? Let's do that. Ooh, $1.70. That's gonna kill us. I don't know, man. It's all right. I'm changing call center soon, so I gotta be careful. I don't wanna... So Crutan, John, how do you balance where your control plane endpoint is? So at the very beginning, one of the things we had done was create an HAProxy load balancer running on Red Hat Enterprise Linux as a helper node to provide a bunch of the installation stuff. And so that HAProxy is acting as a load balancer in front of the three masters. And so all of the API traffic goes through that HAProxy load balancer and then gets assigned to one of the control planes. Vanilla, Cube that I run, I use keep alive D to have a VIP. We don't need keep alive D because we have more than one control plane. So we rely on HAProxy detecting endpoint health in order to handle traffic. Basically, hopefully that makes some sense. Oh, living dangerously. By the way, Crutan, John, I like your nickname. Well, why is that living dangerously? Oh, I don't know. I always put stuff in user local bin. I always leave for bin and S bin for like package install things. Yeah. While we're waiting. What do you do? Oh, corn. Oh, God. Oh, you're using... You actually use corn. I'm strangling you now. That was back in my son. So is a worker happening yet? What's going on? Oh, it's OC. Well, it's probably just booting, to be honest with you. Let's look. Oh, I got to add it into DNS. I need HAProxy. What? No, it's a worker. So this would be... It's not going to receive a router by default. Yeah, but don't I need to put it into DNS over here? You need to put it into DNS, but not into the router because... Yeah, because the router doesn't auto-scale with you adding workers. So what you've just done now... Well, like it's good practice here. This is kind of like a six to one, half a dozen of the other, right? So ideally what you would do is create nodes dedicated for... At least for routers. Use taints and tolerations to ensure that the routers only land on those nodes. And then put those nodes in the load balancer pool for the wild card. What James is doing is... It's going... You would see here... It'll never turn green. Yeah, because the router will never... So I don't know where we dropped out, but the quick recap is... If you put all the nodes into the load balancer pool for the wild card, HA proxy will eventually figure out which nodes have routers and which ones don't, and stop sending traffic to the ones that don't. And if the routers get flipped around, it will eventually figure it out and fix itself. So it's... Best practice is to create nodes just for certain things, but this works in a pinch. I'm not sure why W2 was pending... Thanks for the follow, but I... Visigot Nui. Vis... Visigot Nui? I guess, yeah. So basically waiting for this... This got to come up and then we'll be done, right? Approve the CSRs and... Wait a second, my worker never came up W3, did he? W3 did not... I don't know, reload the page maybe? Oh, yeah, it doesn't look like it. Did we get one of those... Did we get an error that we ignored? Thanks for the follow, hard work. Well, no, that's the strange thing is I actually have it here. You added an IP for it. Yeah, and now it's gone. Maybe I violated the terms of service. Oh, TOCs. Anyway, it would have worked. Trust isn't what it worked. Trust us. So what's the takeaways here? Make sure your iPixie config files work, and the contents of your iPixie files actually exist. That's important. What other mistakes did we make? Well, it's all about preparation, right? So ideally you would know all of this beforehand, like what IP addresses you're going to have, what their host names are, the Pixie configurations, all of that, you would have the load balancer configurations. Well, you're going to know your final IP addresses, right? Because in theory, if you're doing iPixie, you have to have DHCP. Now, we could have done this without... But if you want to run some bare metal and tool around with it, this is a great... This is kind of a cool setup if you just want to get some bare metal and be able to spin it up, especially if you're using the Terraform and Ansible Automation that exists. I mean, you could basically get a bare metal cluster for less than a hundred bucks a day with some pretty decent hardware behind it, you know? Yeah. And so not... You want to spin it up and bring it down. This isn't knocking packet, but in an environment where you can load ISOs, you don't need to do the Pixie bootstrap. You can actually load a boot ISO into the server, boot from the ISO, and then type in where the ignition stuff is to do your processing and provisioning of hosts. It's just that because we're doing this in packet, we have the ability to do the IPEXie. So, you know, that makes life easier. Not only that, but like if... From what I understand, packet has an on-premises bare metal management system that they sell. So if somebody has that, this gives you a good idea of how you could start to integrate those two. Obviously it's not foolproof on our side, but... It certainly isn't foolproof because these fools totally screwed it up. All right, that's like almost three hours' worth of stuff. But, you know, we did it, right? Like we started from nothing and a little bit of preparation. And we spent a large quantity of time chasing our own tails, so probably about 40 minutes of tail chasing. And we got a cluster up and running. So that's pretty cool. Yeah, thanks a lot for indulging me and letting me come on here and do this and not making fun of my VI skills too badly. Although I think there's a side chat somewhere going on about it. I'm pretty sure. Well, I'm chatting here on Twitch. Got some random Q&A. So James, I mean, feel free to bounce. We're going to... Christian and I can hang out and answer some random Q&A for the next 15 minutes. Sounds good. Sounds good. I'm going to head out. Thanks. Have a good weekend. Thank you. All right. Thanks for the content, so to speak. Yeah, see ya. Cheers. Anybody else got any last minute questions? Although I'm asking this question and it probably takes like four minutes for... Yeah, we have to wait the 30 seconds or whatever, right? Thank you. I'm actually looking at the stream now. So James is still on the stream. I'm looking at the stream right now. I got you, I got you. And James is still on. Yeah. But anyway, we'll hang out in the chat, I guess, for a minute or two. Actually, I think I probably need to bounce. But anyway, Christian, thanks so much for harassing James. Yeah, that was fun. That was fun for sure. And yeah, so we've got a lot of stuff planned for the stream. So keep an eye on the OpenShift Twitter, keep an eye on our Twitters. Do we have this stuff on the OpenShift website yet? Like the schedule? The OpenShift website. No, we're working on it. I knew it hit the blog. Yeah, we're working on it, yeah. So you definitely just have to kind of pay attention to the Twitters. Or if you're following us, you could hit the follow button. Twitch is very kind. It will send you a notification either via email or other mechanisms when we are live. So we'd appreciate the follows. And tell your friends and co-workers and, I don't know, maybe your spouses or kids or whatever if they're into technology. Yeah, everyone to get everyone on the stream. Yeah, it's mostly going to be OpenShift, but who knows, someone else may be entertained by us monkeying around. All right, we'll have a great weekend. Thanks for your time. We'll talk to you soon. All right, you too. Yeah. Cheers. Bye. All right, bye everyone. Cheers.