 Welcome to the homelab show, episode 81, Building Lab Server Templates. This is a topic me and Jay have tried to figure out how to word because it's gonna cover a few different things and well, we're just gonna throw it all in this episode because I think the question comes up a lot about, you know, how do you get your template set up? How do you get, you know, some of those, you know, simple to deploy templates because you wanna play with something and there's a couple of different methodologies we're gonna talk about and not one of them is right. It's about which one is right for you. That's what a lot of this homelab stuff, we wanna remind people of. You can all have your opinions of the way you wanna do it but the way you're most comfortable doing it because that's what I kept replying to someone in my own forums, do it the way you're most comfortable with because they didn't like my method. And I said, that's fine, you don't have to. You asked you what it was. Fair enough? Yeah, simple enough. Before we jump into this, let's talk about a company it has plenty of templates to play with. In fact, about how many different templates do you say are in Linode's store there, Jay? I have never counted. It's just this list I scroll through until I find what I want. They have a lot of stuff out there. A lot of great stuff. So sponsor of today's show is Linode and they have templating figured out. It's actually something pretty cool that, I mean, one of these times or have you done a video on that directly with some of the Linode templates, Jay? Yeah, I've done so many videos with them, like either on their channel or on my own. It's like I'm losing count at this point. So I think I covered it, but sometimes it gets a little blurry between my channel and theirs. Yeah. So there's a lot you can do with all these templates. We thank them for being a sponsor. It is a great place to run any of the different things we've talked about here. Well, most all the things we talked about here on Home Lab, there's a few exceptions. You don't run your hypervisor inside Linode. They have one they provide for you. But many of the projects you talk about, if you need to host it somewhere, Linode is a great place to host it. We have an offer code in the description down below. Thanks, Linode, for being a sponsor and let's talk about some templates now. Yeah, I wanted to start with, so basically I'm trying to think of the best place to start. And I'm thinking the best place to start is the whole idea of not having a template or image just so I could kind of address that because I know that mindset does exist. So what I mean in particular, I'm not talking about people preferring to set up everything over and over again manually, because let's be honest, that's just a waste of time. Right. Some people do and some cloud providers prefer is the concept of user data, which I don't really know where that term came from. I guess I could Google it, but either way I'm not going to agree with it because the idea is that user data is run when a instance is created in a cloud environment, whether it's OpenStack, you're running your own or it's AWS, a number of others. The idea is you just take a vanilla, Unchanged, Ubuntu, Debian, CentOS image, whatever it is and you apply user data to it, which would just have bash commands inside, maybe something like sudo apt install dash y apache2, if you want apache to be straight out of the gate, as soon as you create that particular instance. So I mean, there's still an image, right? Because the cloud provider has the image, you're just using theirs and then using a script to install packages, maybe even bootstrap your automation service. I just want to acknowledge that that exists. And one of the reasons why that some people might like this idea is they don't have to maintain an image themselves, they just cause the instance to get bootstrapped from the user data with whatever command you would normally do. If you have like 10 commands, you might just paste them in there one after another. Maybe you'll pull in an Ansible script or something. So I just want to acknowledge that. And the case can be made for no images at all, but I don't think there's any right way or wrong way. Just like you mentioned, it's whatever works best for you. Yeah, cause a lot of simplicity I try to do is I have a lot of just a bunch of different variations. I leave running that I all just designate the word template to them. There's not really anything special I've done. Other than I've done all the things. I got my SSH keys in them and things like that. And I have them running unattended upgrades. And then I stop them, fork them, play with the thing I wanted to play with, destroy them. Or sometimes I just take the template itself snapshot, it try something. And if it's not something I would want permanently added to that template, I just hit revert back in the snapshot to do it. It's all about speed at which I want to get something done or try a new thing. That's why I tell people, once you've come up with a secure methodology and a good workflow, it comes down to what you're comfortable doing. And there's some things I wanted to test. Like I wrote a MinIO installer. So I kept reverting the same template back over and over again until the installer worked because that's the fastest way to do it. Because I was like, oh, the installer failed on this part here. Right, right. You know, going forward from here, because I just wanted to get that mentioned out of the way because that's again, user data is more cloud specific. Again, though you can run OpenStack on your home lab and have user data and cloud in it can use user data. I'll talk about cloud in it later. But here on out, I'm going to use the term template interchangeably. I don't care if you're using your virtual machine provider. I don't care if it's VMware, Proxmox or physical disk images. Maybe you're going to use something like Clonzilla to take an image of the hard drive. No matter what kind of hardware it is, whether it even is hardware, the goal with templates, again interchangeable. I know it could be an image if it's physical. The goal is to have something to start from that lowers the amount of time that you're going to spend getting the instance set up for its actual purpose in life. Maybe you want some packages installed on everything. You just want those packages, you know, always there right out of the box. Maybe there could be some config files that you prefer. You want to make sure those are present. So there could be all kinds of different things that you might want to do. And in the enterprise, there's going to be the, I believe it's CISA, right? Because we had this conversation. I can sometimes remember CISA. They have a hardening image, hardening guidelines. If you type in CISA hardening guides, this is something nice. Then they've actually come a long way. We always think of the government. I think the first joke Jay made and I laughed because he's like, oh, do they work in Internet Explorer? Because the government's traditionally a little behind when it comes to technology, but CISA has actually improved a lot of the years. And, you know, they have good, really solid hardening guides and standards by which you can follow. And they're just good security guides and security practices are setting things up. Exactly. So, and that kind of brings up another point too. Like how deep into this rabbit hole do you want to go? Now, you could literally have everything you could possibly ever want or need in one image, but that's going to take a very long time. And then if you start getting into security, security is a bottomless black hole. I mean, there's just no limit. I mean, you could go, like, there's always going to be someone else that can outsmart you because security is security. So you're never going to have 100% perfect security, but there might be some things that you might want to make sure are there from a security standpoint. Companies go a lot further. A lot of them, especially if they're, you know, have some kind of business certification for compliance or something like that, they might have to follow the CISA image guidelines. Now, HomeLab, we don't even have to follow that at all. We could just install our updates if, you know, something bad does happen. Hopefully our automation and backups will, let's just drop a bunch of stuff behind my feet. Anyway, hopefully everything is just going to carry you through between your backups and your automation scripts to get everything back and running. But again, it's probably a good idea to have some security in there. I'm not recommending that, you know, you do everything in that list of guidelines because it's huge. You can explore it and look at that. I just wanted to acknowledge that it does exist if that's something that you're interested in, especially if your HomeLab was created for the purposes of getting a security certification. I know at least some people are, you know, trying to get a certification or something and that might be why they have a HomeLab because I want to work with real, you know, equipment. So you might actually follow that all the way if you want to follow the security track. But either way, it's probably outside the scope, I would say at least 75% of the audience if I had to guess. I want to make everyone aware of that. We'll have links in the show notes for that. And then from here, we can actually go even, you know, more into detail and more specific to our topic. Absolutely. Which requires some hydration. Should we just get the Windows one out of the way real quick? Yeah, because I do want to, you know, at least address that because, you know, we were kind of talking and I do agree there's going to be a smaller percentage that of people that want a Windows image for deployment because more often than not you have that Windows VM for a specific purpose. Maybe there's this app that you need to run and you might only turn on this VM because I don't know, maybe it's a Windows XP VM and you have this thing that configures another thing. It's very common legacy stuff. So you're probably not likely to want a Windows image. Maybe just taking a backup of your VM is good enough. But if you do, you know, spin up Windows images or Windows instances, maybe you're also following the Microsoft track and the Linux track. I mean, who knows what's running in anyone's home lab? As much as I know the industry has moved away from it, I still use sysprep anytime I create a Windows image which is very rare. But, you know, Microsoft has their own imaging system. I don't remember what it's called now. I know it's deployment services. What's that? Windows deployment services. Okay, so it did change its name. I'm trying to remember what I knew it as like way back in 2008, but they have their own thing and you could absolutely follow that and learn it, but for me, sysprep works. It generalizes the image enough and or the installation enough. And then you could just use that as your base. The problem with sysprep and I, it really annoys me. You have three times that you could do it. So on the third sysprep, you can't sysprep that installation again. You have to pave and reload it. So of course, as home lab people, we're going to run that reference Windows VM in a VM. We're gonna snapshot before we sysprep. So after we create our image, you know, just shut it down and take an image of it, then we're gonna revert the snapshot so we have infinite retries. It's very, very, very easy to work around Microsoft's requirements so much so that I think they should probably just patch that requirement out because it's pretty much useless. I think any first year person working with Windows deployment knows to just probably use a snapshot and circumvent that requirement or that limitation. But anyway, I wanted to address that. I think people that are deep into the Microsoft world will probably be like red faced right now. Like you're still using sysprep. You should be using their service. I know, I know, but sysprep does work well when you create images enough for home lab people and going any deeper is probably only a good idea for people that are serious about the Microsoft side. But again, just something to acknowledge that it does exist. That's the thing you can still use. Yep, and if you Google Windows setup automation, there's some different guides for it. We as a company, the business side of me, we don't use that as much and the reason why is a lot of times it's more part of their domain deployment that we're doing. So we'll have something more along those lines to maybe scripts that we deploy that push them to a certain setting on theirs. So there's definitely a lot you can do. What is that? It's called the Windows autopilot system as well for when you're joining it to a domains. There's like, they've built some automation but it's all licensed. It's not that you can't build a lot of this with some EVL licenses in your home lab, but I don't know. Let us know. And if someone gives them feedback on this, we can bring guest in that is an expert at this if there's enough interest. I don't know if there's enough interest in this and I also don't know how good it would be as a podcast other than talking about it. You know, yeah, it's a little off, but we're gonna most of this, I figured that's the Windows stuff we wanna get out of the way that we know there's some that exists. And obviously part of the problem is there's several different ways to do it in Windows because there's different options they have. So we're gonna focus a little more on some of the Linux stuff. Yeah, and one last thing about Windows, I think Sysprep is also useful, especially when I get the question, how do I move this Windows installation from A to B and A could be a physical machine, B could be a virtual machine solution or they could both be virtual machine solutions in either direction. And in that case, in my opinion, you would back up, take an image of the unchanged Windows installation, then Sysprep because Sysprep can fail and even if it fails, it still counts as an attempt. That's one of the biggest annoyances there. Yeah. So, yeah, even though it's not your fault, right? You're doing everything you're supposed to do and maybe there's something in the installation that doesn't generalize well. But anyway, you create a backup of that and then you can Sysprep it, take another image of it. I know this is the long way of doing it but it's more accurate than that Sysprep image, you could then restore that onto B, whatever the target is and have a greater chance, and I say chance because it's Windows, but you have a greater chance of having that restored properly. And then the only thing you have to figure out is the drivers because of course, the drivers will be different, but if you use Windows, you already know how to deal with that. So that could be another use case for Sysprep when you're just moving from one place to another, which that question does come up. Now, we're gonna get into the main course here now and part of this is going to include cloud init, which I used to say that Linux is a lot simpler when it comes to, when compared to Windows deployment and how you're supposed to do it on Windows with the preferred tools on Windows, I would always argue that Linux is easier. Nowadays it's been so much more over complicated that it's like, I can't make that argument anymore, especially with cloud init because now it's like, congratulations Linux community, we have officially matched Microsoft in complexity and difficulty, but I'll get to cloud init, I'm getting a little ahead of myself. So again, when it comes to templates interchangeable, the idea is to have certain things set up. You don't wanna go overboard, but just have some of the things that are more tedious, you may as well just put it in the image. Now, when it comes to Linux, the story is changing a bit with how you do this and it also depends on the distribution too because some are more easier than others. I do have a feeling that the added complexity is going to make its way to other distributions. I primarily use Ubuntu server even though I complain a little bit, I still like it a lot and that's still what I use, but even they have made some things a little bit more complicated than it needs to be. But anyway, the first goal when you create an image is to generalize it. Even on Linux, you have to do that. In the past, it was pretty much just the SSH host keys. Everything else was fine. That was the only thing you had to do. And Raspberry Pi OS, and I covered this in one of my videos, they have a system descript that is enabled in the image. And when you create a flash drive or SD card from the Raspberry Pi OS image, the first thing it's gonna do is generate new SSH host keys. If the SSH host keys are the same in your image or actually in your image, then every instance you create from your image, you're gonna have the same SSH host keys. If you go into the Etsy SSH directory and list the storage there, you'll find a bunch of files that have host in the name, those are your host keys. And those need to be different. If they're not different, then you're gonna get this annoying error message. There's something wrong here. I forgot what the verbage is when you SSH. It actually cancels your SSH attempt and tells you that there could be something going on because it's checking for a man in the middle attack. The SSH host keys are different. They're not the same keys that you've previously used. They're the same, right. They're the same on each instance, but the IP is what's different or the host name is what's different, which kind of confuses the whole system. So it's better not to have those in there, but they won't regenerate themselves. You have to have something there to do that for you, which that system descript and we'll link to it in the show notes is going to take care of that for you. That's the easy mode version. If you just want the stupid simple version, just copy that into the appropriate directory and enable it and take your image. And then you'll basically make sure that no instance created from that image will have the same host keys. And that's the first thing. And it used to be the only thing that you had to worry about, but now there's a little bit more. I threw the commands in the chat as well. It's just a matter of, like Jay said, it's delete everything under etsy ssh, ssh underscore host underscore asterisk. You'll get rid of all those. Then you just do a dpackage reconfigure open SSH server and that'll rebuild it. When you do the dpackage reconfigure, it goes, hey, look, you said reconfigure but I can't find any host keys. So it just generates them. And then you restart SSH and it implements more. You can rebuild the system. Both of those options work. Exactly, yeah, that's exactly right. Now I mentioned that that used to be the only thing you have to do and some distributions that's still gonna be the case. But one time famously, and I think this is probably a recurring thread for me, when I encounter a very frustrating problem that just annoys me to no end and it just takes me a while to figure out, you guys see the end result of that. I'll just make a video and I'll say, yep, here's how you do it. You do this, you do this and you do that. And I'm sure I sound very confident and of course I am because I've usually rehearsed like 10 different times before I even hit the record button. But what you're not seeing is a frustration that went into discovering what it is to tell you guys to do. And one example of this, and this is getting into the thing that's kind of different now is the Etsy machine ID, which is an Ubuntu and I think it's gonna be another distribution. So I think everyone that understands DHCP knows that the way that it works is a new instance comes online, it hits the network, if it's in within reach of a DHCP server, it's gonna ask for an IP address, if one's available, it's gonna get one and it's gonna present its MAC address as its identifier to that DHCP server. Now, for one reason or another, I'm not gonna get into the reasoning here. The Etsy machine ID is now a factor in the process of getting an IP address. So what happened to me is I created this image, I was pretty happy with it. I thought I had some pretty cool defaults in there, but I noticed that my SSH connection would keep dropping. I'm like, excuse me. I'm like, what's going on here? But then I noticed another instance has the same IP address. I started creating more VM instances from that template and they also got the same IP address. And it was then that I realized that the MAC address was not the factor, I made sure it's different on each of the instances than it was, especially in Proxmox. It knows to randomly generate a MAC address, so you don't have to worry about that. And then it was when I discovered the Etsy machine ID and now if that's in the image, that's gonna be a problem and you're gonna have to clear that out. Now you can't delete the file. And that's interesting because most things in Linux, if you delete a file, it'll probably just get recreated for you which is what you might assume would happen. If you delete this file, it just, it won't be there and strange things will happen. But if you empty out the file, literally truncate dash s zero Etsy machine ID, just empty out the contents, then when the machine boots the next time it's gonna say, oh, there's supposed to be a machine ID here. There isn't, so I'm gonna randomly generate one and put it there. That's basically what system D is doing nowadays. And you also have another file and I know people are probably driving and they probably won't remember any of this when it comes to paths. But var live debus machine ID is a duplicate of Etsy machine ID in most Ubuntu server installations. One has priority over the other. But what you basically do is delete the var live debus machine ID. And this is in a video of mine, if you're curious. You just sim link that to Etsy machine ID. So you only have that one file, empty that out. Next time it boots, you get a new machine ID. Except now you won't. It'll still be the same. How is it gonna be the same? Well, the SM bios serial is the machine ID. And that's actually kind of hard coded into the VM or the instance. So, but that's not gonna be a factor though. I just bring that up. If you're testing this and it keeps getting the same machine ID, that's why. As long as you create the image or template with Etsy machine ID empty and var live debus machine ID, you know, simling to that, then any new instance you create will have a different SM bios and get a different Etsy machine ID. But I mentioned this because it's a very common question. Why are all of my VMs getting the same IP address? And before we would immediately blame the MAC address. We would assume that that's probably the same in every instance that was something we used to run into. That's largely solved now, but the actual truth of the matter is now we have the Etsy machine ID to contend with. And my big issue with this is that it's not like Ubuntu made a blog post, at least none that I'm aware of. Telling everyone, by the way, this is happening. If you don't take this into account, you might have a situation where you have the same IP address. No, it's just a bunch of people that encountered this or scratching their heads, wondering why it's happening and figuring it out independently rather than the company coming out and telling everyone what they're doing. And then of course I created a blog post about this a long time ago. It got pretty popular because, you know, a lot of people are running into this, but it's yet another thing we have to take into consideration. So now we have two things that we wanna make sure are not in the image, the machine ID and the host keys. And let's validate something here real quick, Jake. Someone in the comments here had something I didn't know and maybe you didn't know either. In that plan, if you add DHCP dash identifier colon MAC under DHCP options, then we'll go buy your MAC address instead. Yeah, I did know about that. I'm a little cautious about circumventing the design of the distribution because I don't know how much or if anything they've built around the machine ID because there's always more, especially when it comes to system D. So although that's 100% valid and 100% true, I caution again, again, going against their design for the distribution because again, there could be more to the story than that. So, but then again, I mean, we're homelabbers. So let's just play around with it and see what happens, right? We could just discover first know what it does. Because obviously the default behavior is to create that machine ID, but without a better explainer as to why. And by the way, if someone knows the answer why or you happen to be the person that developed that and you would like to email the show, have at it. Feel out our contact form. Send us an email homelab2022 at the homelabshow.com. Right, and I feel like I knew the reason at one point, but maybe I was too irritated at that point to soak that in. I wouldn't be surprised if it's a security thing because I know a lot of hardware manufacturers that ship phones or something, we're catching onto the fact that people walk into a store and it's following their MAC address as their phone is reaching out, looking for a wifi access points to track them and their spending habits and where they're going and things like that. But then again, if the machine ID is the same on that device all the time, then you'd argue there's no security benefit there. So yeah, there's that. Yeah, also I just said the name wrong. Our email is the feedback2022 at the homelab.show. So if you wanna email us questions, comments, concerns, that's where that goes, feedback2022. That one's, we're gonna keep that one valid and of course we're gonna be creating one for 2023. We just wanna know if a spammer gets a hold of it so we're putting years on that. We should, we sent it before the end of the show so we're doing better. We didn't say it at the beginning. It's gotta, I gotta put that in our notes. Now, at this point, we're getting into cloud in it territory. And cloud in it is one of those things that I think is absolutely worthwhile to learn. I've even done a video about it, but that doesn't mean I'm extra generous. It doesn't mean that I'm an expert though. Anytime I create a video, it's I'm teaching everyone at the level I'm currently at. So I usually just, you know, as my knowledge grows, I'll create more videos around it. And I've had a lot of time to go deeper into cloud in it lately than I've ever done before. Now, cloud in it for those of you that aren't aware is primarily designed for cloud providers, but it's not just for them, we can use it too. And you could use it on, you know, metal. Like if you're installing a Linux distribution on a physical piece of hardware, cloud in it is basically a way of automating first run tasks or things that should happen when a machine comes online. And one of the things that cloud in it can do, and I believe it does by default, is it will generalize your SSH host keys. So if you know how to trigger cloud in it to run, basically you trigger it to run in the reference instance, shut it down, create an image. And then when you boot up or create a new instance from that image, the goal could then be getting cloud in it to run for that very first time. Now, the problem with cloud in it is useful as it is. And there's all kinds of automations that you could add in there from installing packages, creating users, whatever you want the case to be when you bring up a new instance. The documentation is good-ish at times and lacking other times. And the complexity is a little frustrating. And I think the issue here and why that is is because cloud in it seems like a secret sauce that cloud providers use. And it was developed for them primarily. So someone who has created their own company and runs servers that are creating hundreds of VMs a minute, I think that they know their way around cloud in it fairly well, you could probably argue that they don't really need the documentation. But that's not an excuse not to have documentation. I think other people need documentation, other people want to learn it. But sometimes cloud in it is just kind of off on an island and a lot of people don't understand how it works. And to make matters worse, sometimes distributions change it up a little bit and add some customizations to it, especially Ubuntu that make it work differently than the documentation. So even if the documentation was up to date, you might still run into things. But cloud in it is very useful, but I also understand if someone made the decision that it's not for me, I'm just going to add that system descript to generalize SSH host keys and just, there's nothing wrong with that. If that solution works for you, it's valid, just do that. If you have interest in cloud in it, absolutely learn it, especially if you plan on getting into OpenStack or maybe at work, you plan on getting into managing cloud stuff. It's a good idea to try to learn that. And as I learn it, I'm going to probably keep coming out with content around it. And who knows, maybe at some point, then people just send everyone to me to learn cloud in it and it won't be an issue, but there's going to be some challenges there. One of which is the metadata. Cloud in it needs metadata. It needs, for example, the default username, default password, and you have to have a server to serve that, that cloud that can then catch, call in and make the case. It'll also silently fail if it has no way of getting a hold of that, said metadata. Now, Proxmox works around this by creating what they call a cloud in it drive. And in the cloud in it drive, it, as far as I know, and I'm going to paraphrase this here, but it seems to me the way that it's working. And again, it's complex. Is, yeah, cloud in it, don't worry about looking for a server for this metadata you're looking for. It's right on this drive, just use that. And that way you're not required to spin up a web server just for that one purpose, although you can. I kind of feel like for a home labber spinning up a web service, just for that is kind of a waste of resources. It's probably just better to use a cloud in it drive. And with that box being checked, cloud in it will then run. Well, you have to run cloud in it clean to generalize it first, but that plus the existence of the cloud in it drive in the case of Proxmox or whatever, or however your individual platform makes that available, that'll give you the benefit of cloud in it. I see a few comments on this as well, that the documentation is, yeah, the machine ID and the cloud in it install and installing Ansible, which that's a very valid way to do this as well. So you can use your cloud in it to get your Ansible and then get your Ansible Pulse setup. So it pulls everything down there. Absolutely that will work. And it does work, because that's also how I do it as well, especially on Raspberry Pi. Raspberry Pi, cloud in it works very well on Raspberry Pi. It's actually kind of crazy to me how well it works. It works really, really well. And I just have this cloud in it config that I keep personal and I've only changed like two things, like the default username, and then also the command that pulls Ansible is the only other thing that I've added to it. And whenever I restore an image or Raspberry Pi, like if I download Raspberry Pi OS or Ubuntu for the Pi, I just drop that right into Etsy cloud, cloud.cfg, the one that I curate, and then it just does everything. And then I don't worry about it. I get a message in my phone, machine provisioned. I know it's been provisioned, I can log in. I do feel though, that it's complicated as cloud in it can be. It's still fun, well, maybe not to everyone, but you will get a feeling of satisfaction and also feeling like you have more power because you have more tools at your disposal for getting these images taken care of. The cloud in it also is what makes the complexity nowadays of creating deployment images for Linux machines about the same complexity as I remember Windows to be back when I used to manage Windows machines, but it's not insurmountable. It's actually probably not as hard as it seems when you learn it. And you'll probably have a mindset of, oh yeah, that doesn't seem so bad. But then again, considering some distributions add their own things to it, you might run into some things that are gonna take you a minute. Like for example, last weekend, I had an issue where every Ubuntu instance I created from a cloud in it provisioned image would have zero networking out of the box. No IP address, nothing, literally nothing. And that's when I found the hard way after looking through, again, you guys are getting the benefit not going through the pane I went through. And there was a file called, well, it's actually in a directory called clean.d inside Etsy cloud. And that directory contains scripts that'll run when you run the command cloud in it clean. Cloud in it clean again is how you generalize it an instance. If you want to utilize cloud in it, what that'll do is it'll remove the state of the instance that cloud in it has run. So at that point, the next time it boots, cloud in it is gonna say, hey, I've never run on this machine, even if it has. It's gonna think that it hasn't and it's gonna go ahead and do everything it needs to do. So you have to run that command. But when you do so, everything in that cloud, excuse me, the clean.d directory will be run as well. Now that's a pretty cool thing if you think about it because you could just put some scripts in there if you want some more things to be done to the image. But then what I found out is that Ubuntu had a command to delete your netplan.yaml file, just knock it out. And sure enough, every instance I created at CNET plan was empty. And it was hard-coded right in clean.d, right off of Ubuntu server. If this file exists and the name is, you know, zero one, no, zero zero installer.yaml, something like that, it's gonna delete it. Now, I think the reason why is because cloud providers generally have their own system for assigning network configuration and they probably want to avoid a situation where you have a default netplan file plus the cloud provider's, you know, network configuration on top of that, which could probably cause some interesting things to happen. But, you know, again, we run into an issue where cloud in it, as a project, they kind of seem to not understand that creating deployment images is an industry-wide thing, cloud or not. Sure, it has the biggest use case in the cloud, but people still provision VMs internally and they want to generalize things that needs to work for them. So I feel like this mistake that's clearing out network config is just not taking into consideration that, you know, it's used for other things. However, when Tom and I were chatting last night, they must have fixed it because I couldn't reproduce it again. So right now it's working, even if you run cloud in it clean, it works fine. I think how many times do we try that at least twice, if not three times last night to reproduce that? And it was 100% reproducible for me last weekend. Anytime you run cloud in it clean, your network config is gone every single time and now it works. So there's that. That's the thing we run into with HomeLab is that, you know, we have a process that we've gone through a lot and then one day it doesn't work, why doesn't it work? Well, there's a bug now and something that's causing it not to work, but that's one of the trade-offs that we go through. Right, and do take the time to fill out those bug reports because I thought I was doing a video on that because I have, well, I'm gonna go through my testing process probably in a video soon where I show how I find the bug. Well, the bug, you know, I kind of found me, but I'll show how I produce a lab environment where I can reproduce it. And my goal is to always give the developers the accurate answer. Like this is the system, like here's a clean system. These are the steps that reproduce that particular bug. That way, you know, they understand the scenario. We found it in a production scenario which is where we found the bug, but we're not 100% clear on it yet. So I'm actually gonna reproduce it in our lab. It's some rule creation in PF sense, but it's an edge case, but I know enough people probably have this edge case and I'll solve it for them. So always think about that when you find these bugs is making sure you file good detailed bug reports that's what gets the developers, you know, they really wanna solve the problems and not have them just lingering. Another thing that, I guess this is the last thing I'll mention is a consideration because I've already talked about, you know, making it the case possibly if this is what you want, that certain packages are installed, you know, straight out of the gate in the image or template, whatever it is. But there's some optional things you can consider. If you decide to do none of what I'm mentioning, that's fine, doesn't matter. It just matters how much do you care about some of the particulars. And specifically for me and what I'm talking about here is clearing out things like bash history. So if you have, let's just say a user you want in your image, you create that user and then you log in as that user set up user and all that, there's bash history now. And do you really want that, you know, every instance to have, you know, for that user have a bash history that's there from like before you took the image, it just seems kind of weird. So what I do is I just delete the dot bash history file for every user. And then what I'll also do is look for any file. So like find, you know, dash type F you wanna look for files in the var log directory and just pipe that into truncate that dash S zero, be careful. And that'll empty out all the logs because in my opinion, what you don't want is logs from the process of creating the image because that was from before. You want things to pretty much start out as a clean slate. So emptying out log files is not outside the norm. You probably don't wanna delete log files because I have seen some services out there strangely won't like create their log file. If it's absent, which is kind of rare, most will, but then some won't, which is the reason why I recommend truncate with the size of zero because at least then you're not gonna run into a situation where you need logs from a service just to find that it hasn't ever been logging because the file isn't there and it doesn't wanna create it. So again, rare, but it happens. Another thing you might wanna do is go into your distributions package cache locally. So I think it's like varlib, EPKG or something like that off the top of my head in Debian and Ubuntu. So anytime you run like apt install, just upgrade, it's gonna download the Debian or dev packages into that directory. You could run sudo apt clean to make sure that those get wiped out. There's just, it's just a waste of space. The image is gonna be bigger and you probably don't want that to be the case. Just wipe that out. You can always download everything later. So that's probably another thing you might want to clear. And then finally another tool I want to make everyone aware of is PyShrink, which I just love. And this is specific to people that might want to create a deployment image for Raspberry Pi. And what this will do, let's just say for example, and this has happened to me, I had a 128 gig SD card for my Kubernetes controller. And when I ran the DD command to back it up, I also had a 120 gigabyte image file. Now, forget the fact that probably two or three gigs were being used most, but DD is DD. It's gonna grab the entire thing. What PyShrink will do is it'll actually truncate the unused space that's in the image, knocking it down to a ridiculously small size. But what it'll also do is automatically trigger the service that runs to check that the installed image is taking up the entirety of the hard drive. So that way you don't have a situation where you have 128 gigabyte SD card, but like three gigabytes maximum usable space, but you're missing out on the majority of your space there. So what PyShrink will do is set it up that when it first boots, however much free space you should have, it's gonna expand the boundaries of the restored image to that so you can take advantage of that. So it's been an indispensable tool for me, especially as I back up my SD cards. And sure, you could just G-zip your image and that'll knock down the size too, but it's still gonna take a long time to restore. May as well just use PyShrink. I think it's a good tool and it's specific for Raspberry Pi purposes. Yeah, someone mentioned and I pulled the website up because it actually looks cool. So dietpi.com and it's actually supports quite a few different single board computers. It's a very minimalized install of highly optimized minimal install of Debbie and OS. So that's actually rather clever. So if you're looking for something else, thank you for that suggestion there. That was definitely great. I'm curious that the one I used to use is even still around. I haven't heard of anyone say it in a long time. I doubt it. I used to use something called, if I'm even saying this right, Minibian, like M-I-N-I-B-I-A-N. I'm not even sure. This isn't a recommendation because it's been a long time but that used to be my go-to. Nowadays, I just do things the hard way. I just have Ansible remove everything that shouldn't be there. So regardless of what it starts out as, it's gonna become what I want. But there is definitely a use for it. Jay froze for a second there. But yeah, Jay froze for a second there. But I already can finish your sentence. Jay does use Ansible very well. He's got a whole video on that. I'm using Ansible Poll and then customizing it, including not just installing the things you need but removing the extra things you don't. So kind of minimizing the install a bit. Yep, that's exactly right. So I guess all I'm saying is I'm not the use case for it, but then again, I think there is a valid use case for something like that and absolutely something that you might want to look into as an option. And if you're lazy like me, and I'll reiterate how a lot of what I do, I have an Ubuntu server 20 and a server 2022 and whatever the subversion is. Those are both running with the name template sitting in my stack of all my other hypervisors that I have running. I have unattended upgrades running. So they're always pulling whenever there's a cycle that goes through of whatever updates are needed. And it just loads, this is a template. That way anytime this week, next week, three weeks from now when I go, hey, I have this idea. I want to load Portainer. I go grab that template and maybe I'll do one more update but it's usually already up today just in case then I'll stop that template, shut it down, fork it and let it live again and then take that fork and build my idea, play with whatever I want to play with. And it's always just running it up to date. For me, and this is simplicity for people running if you want to own the whole lab, all the different more complicated ways we talked about are great. Now, if you're running a large scale dev team and things like that, yeah, you're going to want to have automation, there's even Packard, there's all kinds of different methodologies for having ready images made. But you can also just leave one running with unattended upgrades. And because it's idling, it's not doing much. It doesn't take up much CPU power. It doesn't need to have even that many cores. You can assign it like one core and a gig of RAM is usually enough. It just can sit in the background doing its thing, pulling those updates. And then you just clone with your hypervisor whether that's Proxmox or Xtp and G doesn't matter, they both have the same option. You can then do a full clone or a linked clone depending on how you want to do that. And then you have a ready template all the time. Cause I like the things in my template. We, one, not just having the package up to date but two, we have a common user at my office that we have on there that the employees log in as, it's just a lab user. And it's just already set up on those templates. Now, if I'm doing something for production, I usually don't pull from those templates. Like if I decided I'm going to do a new install of something production. I generally like to start from like a new install scratch. I'll grab the latest ISO of the distribution. And then I have my customizations I'll put on from that. I kind of like going through that raw install process. It just kind of makes me feel more engaged and involved. We don't replace production service very often. That's something it's, it's kind of like a one-off rare thing. So I like to just go through the process again on there. So there's a couple of different methodologies whichever one really you're comfortable with on there. As long as you're doing some of the important things and one of the things like Jay said, and we, you know, I threw that little a couple of commands that are resetting SSH host keys, resetting the machine ID, or as someone noted, putting a net plan to you make sure you're not pulling it so you get a different MAC address presented for DHC client. Cause you don't want to have same MAC addresses as your template that would cause drama. Make sure you've got those bases covered cause those basics will cause you some headaches when you have your template still running and all of a sudden the other thing looks just like it. And then the other thing I'll mention is probably the last thing for today for me is what do you do after? I have a rule at my company, I know it's just me, but we'll just pretend I have a whole team of people. I will eventually, but I also had this rule when I managed people before I went on my own that nothing is allowed to be accessible from the public internet unless it absolutely has to be. Well, that's pretty obvious. We've said that a million times, but also it goes through a security check. So before I allow anything that is supposed to be accessible from the public internet to be accessible, then I'm going to make sure everything is updated. You know, I understand what ports are listening, should they be listening, and I just check everything and then I will allow it to be publicly accessible. But like it's even to the point where if I'm working with someone else, which is generally going to be probably remote, maybe we're tag-teaming an instance or something, I might say, well, what's your IP address? And I'll whitelist their ability to get in. I still won't make it accessible from any IP regardless until I know that it has security standards in place. Obviously, you don't want to go too far down that rabbit hole because there's no end to it, but you definitely want to check some boxes updating being the obvious one. Closing ports should also be obvious, but then also just kind of take a look at it, make sure that everything checks out and then if you need to make it publicly available, you can, but then again, if you could get away with nothing being publicly available, even better. Yeah, someone else drew in the comments here, cloud init, space clean, tac-tac machine-id-sl. Yeah, I tried that last weekend. I'm trying to remember what happened because it didn't work, but I don't know why. I don't remember why. And this is another issue about the documentation. My understanding is, maybe this is wrong is that cloud init clean will clean everything, but if you add an option, it's going to just go after that one thing because there's also dash-logs as well. So you could wipe out the logs by adding that as an option to cloud init clean, but I'm going to have to look into that to be sure that clean is inclusive, which I thought it was, but again, maybe I'm wrong here, but if there is something that you want to clean in particular, then that's one way to do it. But again, I just wish I remembered what went wrong when I tried that. I spent all day figuring this out last weekend, so I probably am missing some details. So apologies for that. More testing needed on that. So we look forward to hear some feedback from any of you that have that. And thank you, Justin, for that suggestion on there. I've seen Justin also mentioned he used Arch. Were you trying that cloud init clean on an Arch machine or an Ubuntu machine? Jason, Jay was doing them all on Ubuntu, so. Yep, yep. I haven't run Arch on servers in a while. So right now it was just Ubuntu. And I was also running into Ubuntu-isms that were, again, specific to that distribution, not in the cloud init documentation. So that was further complicating things, too. Justin says he was using Arch, so there goes some more testing for you, Jay. There's the next video Jay can work on, cloud init on each different distribution and why it's not the same. You know, now that Arch was mentioned, I feel like there's just this stigma around rolling distributions that I don't think is fair. I would absolutely run a server on Arch. No question about it. Sure, you're gonna have a little bit more work sometimes, but the whole idea of Arch is to install as few packages as possible to serve your goal. So as long as you follow that, you don't have like 10,000 things installed, especially a bunch of things from the AUR, and it's just the things you absolutely need, especially considering you could literally just install something on your Arch server just to run a container and everything runs in containers. They don't even have any extra packages beyond that. You can absolutely have a stable rolling distribution and then benefit from literally never having to reinstall everything the next time a new Ubuntu release comes out. But again, everyone makes the assumption that rolling is bad for stability. No, just means you are in charge of the stability. If you're up to that, then you should absolutely do it because you are benefiting from not having to reinstall. And that's why we also see CentOS, CentOS Stream, Amazon Linux is, you know, semi-rolling. So it's taken off and I think it's gonna be very common. But then again, if you want something simple, Debbie and Ubuntu, there's Fedora server, there's a number of others that might just be a little bit less chaotic. Yeah, all right. Well, this was definitely a fun talk and we sent some homework for a few people. So they got a few different things they can try. And so do we. So I think there's gonna be a lot of things in here. So I haven't done a video at all on cloud on it and XCPNG and it's been kind of my two lists because to kind of talk about that, I just don't use it, but I want to. And so I've got a kind of a goal set for that. And there's been some discussion in the XCPNG forums on cloud on it. So that'll be my source materials for it. But thank you everyone for joining us. And we look forward to hearing from you. We wanna do another Q&A episode. We think we're doing, me and Jay, are we gonna do an episode we didn't talk about this for on that between Christmas and New Year's? Is it, was there any plans or? I don't, yeah, I don't think I'm gonna be available. I mean, things are a little up in the air because there may or may not be a blizzard coming. Yeah. If I knew for a fact it was coming, I'd say, yes, I'm available because I'd have nothing else to do. But I think it's probably safe to call it like next week, probably we'll be a break. Yeah, okay. We might be a break next week. Just let everyone know. That means we won't be back till after the first of the year, but we look forward to hearing from all of you. And thank you for all the supporters of the channel. Leave your comments down below or hit us up on the socials. Thank you very much. Take care.