 All right, the Homelab show is live. This is the Homelab show, episode 10, getting started with Ansible, and this is Tom Lawrence. And this is Jay LaCroy. Yes, and if you're listening to the audio version of this podcast, it's literally brought to you by Linode. One of the challenges is you people apparently like our episodes, which is wonderful, but episodes need a lot of bandwidth and Linode was happy enough to sponsor the show and provide some bandwidth. Jay's actually a big user of Linode to tell you a little bit more about it. Yep, I've been using Linode for quite some time now, like two years ago, I converted everything from the YouTube channel, LearnLinuxTV, over to that platform. I've actually been using it, you know, be even before then I met the guys at Linode years before they were a sponsor at PenguinCon, a popular conference in Southfield, Michigan. And great bunch of guys loved talking with them. And then, you know, later got in touch, they became the first sponsor of my channel and it's been going great. They give you basically access to VPS or virtual private servers. You could spin up a cloud server for as low as $5 and they have ginormous ones for, you know, that's probably overkill for most people, basically whatever you need. You can host a website on there, a Minecraft server, whatever you can probably think of. And I have, I think probably 12, I've lost count of how many servers I have on there now. So they are not only the sponsor of this channel and or this podcast in my channel, but they're the, you know, backend provider of this podcast and my channel's website. So they're definitely a great platform. And you can use Ansible on them. You can use Ansible. So it's a great idea. You could just spin up a, you know, a Linode instance and then you can basically learn Ansible, which is a great thing to learn because selfishly I love it. I'm a little biased, but I'm, you know, it's one of my favorite technologies and that's just one of the many things you can do on there. Yeah, I am upset I did lose my Linode swag shirt I got from PenguinCon. So if Linode, if you're listening, I need a new swag, I need a new swag. I only had one, I gave away the other ones and somehow it disappeared when I moved. A special URL, Linode.com slash home lab show. And what that'll do is give you $100 in credit towards a new account. And that credit is good for up to 60 days. So you can basically just get started without any bill because that'll definitely cover a few months of service for whatever crazy experiment you wanna try out on there. Yep, and I did leave that link if you didn't hear it, it's linked in the show notes. So if you need to sign up and start with Linode, that offer code is in there to get you started. Thank you very much Linode for sponsoring this podcast. You love them. All right, now once we've built our Linode server, we're sitting at a blank screen and I already see comments in there. Do we do polar push? And boy, Jay has a great series in Ansible. So a lot of the stuff I wanna talk about is also covered in Jay's series, but we kinda wanna talk through the Ansible process in this particular episode. We do, we kinda wanna start at the bottom. And we did kinda go over and overview a bit in the automation episode already. So a few of the things I'm gonna say, you're probably more than a few, are gonna be similar or the same, but I think it's important to have a little refresher, kinda create the mindset that's going to serve as a foundation for the rest of the episode, kinda just talk about what it is, why you wanna use it, how it compares to other things. And then I'm gonna go into detail. And like Tom mentioned, someone asked about a push or pull. I'm definitely gonna talk about that for sure. And I'll even mention how I use it. And it's an awesome technology. I've used Chef and Puppet, which are pretty much competitors. I say pretty much because the way they operate is very different than Ansible, which I'll get into, so I'm going to compare them. So first of all, let's talk about what Ansible is and how it compares. And what I'm gonna talk about is a general high level overview of infrastructure as code, which is a very huge topic. We could probably have a dedicated episode on, but essentially the bird's eye view of it is you have a state. You want your servers to be a certain thing. Maybe you have a Proxmox server in your home lab and you wanna set up a web server on there. Maybe you wanna set up a Minecraft server or something like that. And you don't need Ansible or any of these other solutions to do this. You could just create it and build it yourself. Unfortunately, if it breaks, hopefully you have a backup or a template, an image or something like that. But infrastructure as code is able to take an instance, a Linux instance generally speaking, but you can use Ansible on Mac OS, Windows or whatever as well. And it's gonna take that base and it's gonna build it up to what you want it to become, which is just amazing because you can literally just execute the process and just go grab lunch and then come back and your server's ready to go. It's just an amazing thing. And in the enterprise world, this becomes very important because when you have a group of servers where you need to replicate a process, there's two things that need to happen. You need to build it out a certain way and you should document the way that gets built. You can solve both of those problems with something like Ansible if you take and build your Ansible setup and then document all the steps inside of it. One, the scripts and everything that you put together inside of there, all the different function calls you make are gonna be the documentation, but of course, they'll add notes in between to what each section does. And as you kind of build these playbooks out, it's really great because now you've created something you can repeat, something you can do again. And if a server just gets messed up, then as long as you keep your data separate to the servers should always be thought of and as like ephemeral, like we can just destroy a server and rebuild it as long as the database is backed up. And usually the data, especially when you scale up to an enterprise, your data's on a separate data store. So we just rebuild the server because it broke for reasons unknown or reasons because we were playing with it. We just kick off the Ansible script again and it repeatedly builds the same server again, reconnect it to the data store, back up and running. Yeah, and a great example of that. And this isn't really specific to Ansible or configuration management, but it works, is that my Plex server is completely unimportant. I don't care if that server gets deleted right now, I just don't care. There's not one thing on my Plex server that I care to lose at all because the videos that it serves aren't even on that VM. It's like a, I wanna say 16 gigabyte VM, that's the total storage. And literally all that is is the OS and the logs. And what I do is I use auto FS, which is a solution that you could use to like, mount Samba and NFS shares. And auto FS will automatically mount it anytime you go to look at it. So if you have your data mounted in a specific folder, then that NFS mount or whatever it is, is that mounted until you access that folder, you LS against it or Plex checks it to see if you have any new movies on there. Then it mounts it so quick that Plex can't even tell that it ever wasn't mounted. So basically what that allows me to do is I could delete the VM, I could recreate it from the most recent backup and it doesn't matter. It's just gonna always mount the same NFS store. The worst case scenario is Plex has to re-scan all the media or something like that, but who cares? But that's an example of that. You want the ability for a server to just go belly up and it doesn't really bother you at all because you're doing it right. But that isn't about configuration management, but I think that is, that configuration management is part of that end goal. It gets the server to that point. Right. I think something else too, that it's not itself a containerization platform, but it can be used to orchestrate whatever you want. To even go a little bit further, it's not just limited to Linux servers. For example, lots of different switches, firewalls and things like that. If there's a way to configure them via SSH, they can also be put into Ansible to be able to run something on them. Matter of fact, if you wanted to go fetch some piece of data out of PF Sense, you could build that into Ansible. Now, PF Sense doesn't really lend itself very well to being configured from the command line, but it could be, at least if I wanted to fetch or check some status information on PF Sense, I could tie it into that or any other firewall that supports SSH admin access. So you'd be able to run these playbooks to do that. Matter of fact, FreePBX would be an example. It runs FreePBX, which is running Asterix at the end, but there are a lot of things you could do from the command line. So if you had a group of phone servers, you could write an Ansible to talk to them over SSH and say, I need to go grab all this particular information or change this setting on a group of systems. It allows that level of automation or even one of my uses I've had for Ansible is just having my server set up in a fleet where I just need the status of all these servers. I just want some type of information and pulling that information and pulling it all back and doing checks on it is a great use case for Ansible. Exactly, and I think you hit the nail on the head when you're talking about all these servers and the things that you want to do with them. I mean, early in my career, I started with Puppet and we had at this company 60 Linux servers. So going to each server one by one and executing a command is very possible. It's going to take a long time, but it's not like they had a thousand servers or 10,000 servers, but still even with 60 servers, we didn't really want to deal with that. So we implemented Puppet at the time and that was great because if I wanted to make a change to a bunch of servers, I would just implement the change in Puppet, all the servers would then get that change in one shot, which is especially useful in the enterprise if there's like a CVE, like a really big vulnerability that hackers are taking advantage of, you can just, whatever the fix happens to be the work around mitigation, you could just issue that out and then all the servers would then be protected. But the question then is, well, that's great for the enterprise. And I know that we're going to have a subset of our audience that work in the enterprise because a lot of times you use that home, which you use at work for learning, but the average home lab person might be saying, well, why do I care about updating 60 servers because I only have like five, right? But the point though is that it still has value for home lab because there's going to be some changes in settings, best practices that you want to make sure are the case on every server you spin up. So a good example of that is OpenSSH, which is probably the best thing to configure to start with because you don't want OpenSSH open to everyone, right? So you could lock down root authentication and disable that. You can use Ansible to add your key, your public key. You can, all the best practice OpenSSH settings, I'm not going to go into those because I have a whole video on it, but that's a good one to start with. And then you could be reasonably sure, just use whatever your configuration management tool of choice is on any new server you spin up or when you replace a server and then you can have those best practices already there and you don't have to go in and do that manually every time. Right, and I already see people in the comments talking about they've used it to configure their firewalls and managed systems. It is a popular way to do it. And it's one of those, even configuration backup, being able to go out and say, you build out the list of servers that you have because you want to check them regularly, get the status update or just make sure, let me just kick off a backup. But then of course, once you've done it once, you build the automation around it as a cron job. Say, every time just back this up that way as anything is happening out there, you're re-pulling everything back to a central server using Ansible or whatever the orchestration tool use. But it's part of your automation stack essentially where it's one more tool in the toolbox to be able to pull that over. So let's talk about then the competition or I shouldn't say competition, but you know what I mean. The other alternatives. The other alternatives, right? How they do it and where Ansible kind of fits in with this. And I started with Puppet and then I went to Chef. So I used both before going to Ansible, which is probably one of the reasons why I appreciate it so much. I know a lot of people are probably starting with Ansible so they wouldn't have experienced those other solutions. But with those, the idea is you have your server that contains the state. It's very common in all these solutions, Ansible, Chef and Puppet, to have your configuration management code in version control, which is a great thing to do because you don't wanna work on all that automation and lose it, especially if the server itself goes belly up. I mean, then it's like, well, I lost everything, including all the work I put into all the customizations that were in there. So it'll be in version control and you have this central server that has some kind of an index. It knows about other servers and the process is different between Puppet, Chef and Ansible. But effectively that it's a server-client relationship. The client has an agent that is using memory all the time, running and checking things. And periodically that agent is going to reach out to the server. It'll be secured by a certificate and it's going to pull the most recent instructions. If there's no recent instruction, it's just going to make sure that the state of the system is at the most recently defined state. So if you haven't updated those configurations in weeks, then the agent is still gonna look at it and say, oh, I gotta check the open SSH config file and NTP and all these other things that you've defined to make sure that no one went in and just changed it to be like a non-authorized change. So basically, if you have someone logging in and they decided, oh, I'm gonna open up open SSH to the world, then the configuration management utility looks at that and say, oh, that's not what was defined in the code. So I'm gonna just revert that back. Now that client, server-client relationship is fine, but Chef burned me on that. And full disclaimer, I'm gonna complain about that here, but I do wanna give the disclaimer in all fairness that this was my experience years ago and it might be completely invalid now, my complaints and full disclosure. But with Chef, I worked at a managed service provider at the time and one of clients would email me or message me and like, yeah, the server is hitting 100% for like 10 minutes every hour. Why? Why is it doing that? And I didn't know with Chef very well, I looked into it and I learned it and I'm like, oh, well, Chef is running and it's going through the entire server and checking everything, making sure it's perfect. And I thought that was a great answer, but the client didn't really like that. Is it not optimized? Is it, I mean, why is it using the entire CPU to do a routine check? I'm like, that's a very good question. And I don't really know how much of that was the fault of Chef because I wasn't the one that implemented it there. So it could have been the person before me might not have followed some best practices. But either way, I felt like Chef was just heavy. Just not so lightweight. And that was a concern to me. Before that, I tried Puppet at a previous company. I liked it. But what made me switch off of Puppet for personal use is I can't remember which version of Debian it was. It was Debian Stable, they just released a new version. And when it came out, Puppet didn't support it. And I'm thinking, wait, why you don't support Debian Stable? I can understand Debian testing around Stable, but you don't support the version of Debian Stable. And six months passed, believe it or not, they still didn't support the most recent version of Debian. And I remember going into a message board and maybe I was in a bad mood. I'm like, how do you guys not support the latest version of one of the most popular Linux distributions? And their response is, yeah, we'll get to it. And at the time I was using Debian on everything and I'm like, I can't use this because I don't even know when I'm going to be able to use Puppet on this distribution. So I kind of stopped it. And again, that may not be a problem. They may support all the distros on day one nowadays. I wouldn't know. But Ansible is kind of where I ended up. And I feel like that was a great solution for me and I'll tell you why in a few. I think the other side from what I like from my much more basic usage compared to Jay, because I don't actually use as much Ansible because most of our so many of our clients are running Windows so it doesn't work as well for that. But a lot of the stuff I build in my lab, I don't have to load anything on those servers. I can start with a basic server, make sure my SSH keys are in there. And then from there, if I have a couple of things I want to run with Ansible, I can do it from my system that has Ansible loaded. But because the commands are issued from my computer executed just via SSH, it makes it simple without having to load much on whatever it's talking to. This is what lends it to working on things like firewalls. You don't need to load Ansible on the firewall. You just need to have the SSH access and then you have all the commands on your system. So your system is running Ansible. And that's where we're gonna, Jay's gonna talk about that later where we talk about Ansible pull or push, but that is one of the use cases for it. And one of the reasons I like it is kind of a simplicity thing for an orchestration tool because there's no, it's agentless. It doesn't need any specialized things to be loaded on the device, which gives it that broader support. And that's kind of the interesting thing about Ansible because the elevator speech of Ansible sounds easier, lighter than the other solutions. And that's true, but that simplicity causes additional complexity in a way, which was really confusing to me when I first started with Ansible. And what I've learned is that there's a generally agreed upon way that most people implement Ansible, but there is no one right way to do it. So with Chef and Puppet, it's pretty cut and dry. You set up a server, you test the agent, you install the agent, make sure they could talk to each other. And that's how you do it. And there are other ways to implement those too, but generally speaking, when you read any book or, I mean, there's a way to do Chef and there's a way to do Puppet. But with Ansible, I liken it to someone just dumping a bunch of Legos on your lap and they're like, build what you want and how you want it. They give you the tool, but they're not going to like, the finger at you, you need to do this a certain way. Now to be fair in the forums, there's always that person, right? You do it a certain way and you're proud of it. And then the response, you're asking a question, you wanna know something. And rather than getting the answer to the question, you get instead, why are you doing it like that? And that's always kind of annoying, but it doesn't happen as often when it comes to Ansible, but it does sometimes there is a generally agreed upon way that most people implement it. And it's kind of like the server client relationship in a way where you have what Ansible calls a control server or a control system basically that executes SSH commands. So the client, which let's just say you have a client on your Proxmox server, then your control server will SSH in and execute the commands on the target server. The target server like Tom mentioned doesn't have an agent. It needs that SSH connection. So there's nothing checking, nothing running all the time. The server is making an outgoing connection to the client that it's controlling and controlling it. And you could have hundreds of servers and it's just making connections out to each one going through your playbooks as they call it. Each platform calls their config, their text files, a different thing. You know, it's playbooks here. Yeah. I'm not a sports person, but I hear that that's a play on sports. Can't confirm nor deny that. So that right there kind of sounds cool because you don't have to run an agent. It's lightweight. And like Tom mentioned, because it's SSH, it works on basically everything that you can connect to with SSH. Yeah. And kind of like Jay said, I could summarize a lot of that as the lack of an agent, the simplicity of the agent list system causes the complexity because everything has to be initiated from your server deployment side. So you have to have your playbooks and everything set up right to handle all the complicated tasks. You have to have them so they're, without an agent, you're just returning whatever value. So if I start a service, I need to also bring that value back of whether or not that service started successfully or put some type of check in there. And the command issued gets the check and then brings it back to the Ansible server to compare the value of what the response was and then it can initiate back a secondary command if the response was something different. For example, what version of software you're running? Cause Ansible can be configured to first ask that question, is this a Red Hat or a Debian or some other, like an arch? It can start asking those questions and then execute scenarios based on the responses. But that's what allows it to be not only agent list, but allow you to singularly write, depending on how good you are at writing it and diving into the complexity of it is what we're going to do. When you have that, it will then ask those questions of its agent that it has SSHX to and then execute commands based on each one of those responses. But this does keep it, you know, it sounds very complex, which it is on the server side, but this allows you to have any agent. I spin up the most basic of servers over in Linode and whatever I chose to spin up doesn't matter. It's going to ask that question, have a response and start kicking it off. And don't worry, you don't have to write this all from scratch. There are a lot of code repositories on GitHub to copy and paste from. In fact, we should probably add the repository that, excuse me, we should probably add the repository that I've added to the Learn Linux TV GitHub page that actually has a really huge example of a lot of things that you can do, which I'll talk about later. But, you know, the elevator speech kind of does sound simple, like the way Tom mentioned it, which is true, there's a lot of complexity here, but generally speaking, no agent, one less thing to worry about. That's usually the first thing that a lot of people think of. And, you know, that's true and all, but from that simplicity is where the complexity starts because with Ansible, you have an inventory file, which starts out very simple. It's a text file, one per line, you have a DNS name of a server or an IP address. You don't have to have DNS set up and you can have different roles. So you could have like a web server role, a database server role, and you could basically just put the IP or host name of the machine underneath that role, again, one per line. And that's how you determine what server is a part of what role. And then Ansible sees that when you run it, you have these 10 servers, three of them are web servers, the other are whatever they are. And with roles, you can define which playbooks run on which servers. So that way you don't have, you know, a role for a Minecraft server and then it's spinning up a web server on there because you have a web server playbook. It's intelligent to know you have this set as a web server. So I'm gonna run this playbook here only on that. And then whatever apps you have, you can separate them by roles. But taking a step back, actually the inventory file, it gets even more complex when you start reading more because one thing that a lot of people don't know is that if the inventory file, which is normally, again, a text file with a list of servers in there, if it's executable, it'll execute it instead of read it. Now, you might be thinking, why would I wanna do that? Well, think about it like this, if you're working in the enterprise and you have Amazon web services, that inventory file could be a script making API calls to Amazon and fetching in real time a list of the servers that you have there every time it runs. So rather than reading a hard-coded list of servers, it can instead use that inventory file as a way to execute a script that connects to your platform, and there's other platforms, Azure, Google Cloud, Leno, DigitalOcean, all of them, basically, to make these API calls to find out what you have, and then you no longer have to maintain that inventory file. So there's a lot of things, Ansible is one of these things that it starts very simple, but it's endlessly complex in a good way because you always have a new thing to learn. You could be in a situation where you have everything the way you want it. And then someone comes to you and says, you know, you'll save about five minutes off your run if you just implement this tweak. Oh, wow, thank you. And then it gets that much better and it just never ends again in a good way because you find all kinds of new fun things that you can use it for. So should we say it's easy to get started but a lifetime to learn? You know, let's put a cliche on it. I think it's how it is, it's kind of addicting. You get started like, hey, I added one thing, another thing, another thing. And how many years have you been working on your deployment script, Shay? I think five, I lost count. And still working on them. I love it, but I'm always, and there's always new edge cases or new apps that I discover. So sometimes it's like, I can't remember the name of it. I found a Markdown text editor. So I had to add that to my Ansible for my laptops and desktops to get that installed. And I did as I discover new apps. Now another thing that I think new users should know and maybe some of them have this question. Okay, so you have a control server and it controls whatever your servers are. But what should the control server be? Can it be a Raspberry Pi? Can it be a physical server? Can it be a Proxmox instance? Yes, it could be any of those. In fact, a control server could be your laptop even. It doesn't really matter. You don't even have to have a server per se. You could literally just have a Git repository that has your playbooks in there. And maybe your laptop is the thing that will make those connections. Maybe you will just kick off an Ansible job from your laptop. And in an enterprise, you could have a repository that engineers check out. And then on their workstation, they use that to kick off the commands or they could just spin up a dedicated server that everyone logs into and they just fire off the commands from there. That's all valid. I mean, if you have a Raspberry Pi lying around, keeping in mind, the whole process will be much slower because of the IO bottleneck, but it'll still work. You could literally just make your Raspberry Pi the control server, your laptop, whatever you want. If it runs Linux, it could be a control server. Yeah. And that's actually kind of a cool thing to think that a small little Raspberry Pi can be the orchestration server for even something large in enterprise. Because it doesn't really, I mean, it just has to do some comparative commands, do some evaluations and run the roles and playbooks as it finds those responses. So that's not a heavy lifting type thing for this type of orchestration. Right. And the way Ansible works is, the playbooks are written in YAML and each of these different solutions will have a different chosen language for their config files and scripts and whatnot. YAML is pretty easy to learn, but you don't really have to learn it necessarily. You could just learn what the individual commands, modules and syntaxes for anything related to Ansible and just stop there. YAML is not specific to Ansible at all. You'll find YAML in other places. It's getting very, very popular and surprising where I find it nowadays. It's very easy to write. So where in Chef or maybe it was Puppet, I don't remember, like installing a package via apt would be like a Ruby block of code. Whereas with Ansible, it's like apt colon and then on the next line, two spaces over, name colon and then the package name. That's it. You're done. Just those two lines, that's you install the package. And then you could basically install 10 packages off of one play as each individual instruction is called. And you could basically have one per line with a hyphen, a name of the package. And that one, in that one shot, you're installing all those packages. So there's going to be modules like apt, DNF, YUM, zipper, Pac-Man, whatever your package manager is for whatever your dip drill is. So if you're running Ubuntu, then you'll use the apt module, same with Debian, zipper for open SUSE, Pac-Man for ARCH and those are all modules. You just call Pac-Man, you just call apt and tell it what you want to do. But another thing is that there's a generic module called package. So where you would normally say apt, you say package. And at that point, you don't care if it's Fedora, CentOS, Debian or whatever. It knows what the package manager is on the target. You don't have to tell it. It'll just install the package of this name with whatever the package manager happens to be. The problem, of course, is that not all distributions, name the package is the same. So if you're trying to install Apache 2 and Debian, that's fine, because that's what it's called. But if you try to install that package name on CentOS, it's not going to work because the package there is HTTPD. So there are some, of course, limitations there. But if the package is the same name on all the servers that you manage, you could literally just use the package module and not even write anything that's distro specific at that point. And I've gone as far as to make the package name a variable and then it checks the distribution and I just fill in what the package name is for the various distros and I could still have one play for installing packages for all my servers. So like I mentioned, you could keep on simplifying it, consolidating it and it just keeps growing. In an example of this, and someone says, do you run this on a schedule or do you just run it on it as needed? There's a couple of different school of thought there. If you, for example, I find another package I want to add to all my servers and this has happened before. Like I don't, I've done a video on Elnav. It's a really cool log navigation system. I love the way it highlights. It has RegEx built in neat thing. I'd like this on all my fleet of servers. So, you know, without something that's automation, I have to log into each server because there are existing stands already stood up and running. I can log in each one. Obviously an app to get install it because it's in a repository or you add it to your Ansible. And then because I've made a change to my Ansible, I can then push that change out and say go ahead and check all these packages. And it has all the previous packages I've installed on the servers and the new one I added or you put things in the queuing system. And the concept of that be, I added it. I don't need it right now but I'd like it to be there in the future. So I update my Ansible system on that server and say update it. And I know it runs every night at 11 and it goes and updates those packages. So it kind of depends. You know, you can always force run it but it's nice when you put things on a schedule because maybe there's downtime when you want these packages to do something else or your script does other things like go ahead and check to make sure everything's updating fine and returns the value. I know you can standalone each server to like unattended upgrades or it's equivalent and non-WM based distributions but you also want an answer back of did those updates fail or for some reason you have a server with DNS failure. So it actually never sees any new packages. This is where the Ansible automation can be checking this all the time for you because it's checking it and logging in that server and bringing you back the response and then you can kick off a trigger to say this one thinks there's no package updates but all the other ones do and kick you off for investigation. So it kind of depends on your base usage of whether or not you want these to run all the time or you want them to run some of the time. And this is really HomeLab too. If you're just running a few servers you still want to create those up to date and you still want those things too especially if you're making anything public facing in your HomeLab. You want to make sure all those servers aren't just saying they're updating you want some verification they are. When I first started with HomeLab configuration management there were a few things that I wanted to make sure that every server had. And one of those was obviously open SSH settings which I've already mentioned. Since I do a lot of file synchronization then I wanted to make sure NTP was installed. And I apologize for my sporadic allergies. I live in Michigan and it's springtime so there's that but anyway, the fact is I mean you could just implement this any way you want intelligently but it always has the same starting point. So again for me as open SSH and NTP I learned the hard way quite a long time ago when you are syncing files between computers and servers if the clock is not synced on any one of them it's gonna throw all the other ones off because you could have old versions of files that it thinks are the newer one. So NTP obviously one of the first things I wanted to make sure that the NTP package was installed that it's running that's one of the things that it can do and then from there just kinda grew. And what I ended up with and this is kind of like the and this is part of it I'll get into the more advanced side of it you can have whatever layout you want. The layout that makes sense for me is to have three roles and you could define whatever roles you want a Minecraft role, a web server role, a Plex role whatever what I did was I have a base role the base role is applied to everything there's no separation at all whatsoever. The base role are the things that I feel like every server needs to have examples my bash config when I log into a server I want my bash RC on there and I want it the same on each so that's on the base role because that's not system specific and open SSH tweaks NTP tweaks that those are all in the base role because they're not really specific to any use case so the base role gets applied to everyone and then the other roles I have our server and workstation workstation laptops and desktops and server obviously servers so that's when it starts to branch out I'm not going to install steam on my server that's for my workstation role to do I don't want that on my server so that's why I have that separation there and you could be as granular as you want maybe you're instead of having one server role you just have a bunch of different kinds of servers that you can create I even at one time had it checked the CPU type and if it's ARM, well, it's a Raspberry Pi so there's going to be some Raspberry Pi specific things that I want to do here so you could basically lay it out any way you want whatever makes sense for you. Yeah, and I would say LNAV needs to be part of all the servers or even workstations because it's just great for log navigation I'll give them a plug it's a cool free open source project and it's in the repository I should probably check that out Yeah, I recommend it, Jay. Yeah, I hear that I'll definitely check that out Now it started to become a problem for me though because the issue is that well actually it's not an issue the first thing I did was I implemented messaging so I actually hooked it into a Telegram bot so anytime Ansible ran it would fire off a message to Telegram to let me know that it did that, which you can do and that worked out great but the problem was I also have it running on laptops and desktops and I don't want to waste power so I'm going to suspend my computers when I'm not using them maybe my laptop is in my bag at the moment and it's in suspend mode maybe my desktop hits suspend because after 30 minutes, that's what it does so I would start getting errors Ansible though, the control server would send me errors like over and over again because I had it on a Cron it was running, can't reach your laptop can't reach your laptop can't reach your desktop I know it's in my bag I don't care right now it doesn't need to always be available but the control server is just trying to hit your machines during the time you tell it to hit your machines and I came up with a very hacky solution to fix this problem which I'm not even going to mention because it's not the way that I recommend to do it we've all been there, right? We have a band-aid approach to something Wait, I'm still there Yeah so I discovered Ansible pool and that's what changed everything for me and that is my favorite way of doing it and I don't feel like it gets enough recognition almost everyone uses the control server approach and there's nothing wrong with that if that works for you, that's awesome especially if you only manage servers that are on 24 seven it's especially easy but I wanted it to be the reverse and that's what Ansible pool is it pulls from a Git repository and runs it local host so instead of a server reaching out to your web server your web server is reaching out to your Git server pulling down the playbooks and running them against itself the problem is you might think that you can't really differentiate roles at this point because every server is named local host at this point so if you are matching on name and your web server name is web server one but it's gonna report as local host because Ansible pool is running local host so what do you do? You could still actually use a, I think it's dash I if I'm not mistaken and then you just set that equal to the host name variable in the shell which is just dollar sign host name and it'll match on the host name still even with Ansible pool so you can still use roles at this point and that's kind of the thing that makes this all work for me so every single computer basically checks out a Git repository and that's what Ansible pool does the Ansible pool syntax is ansible-pull dash capital U and then it expects a URL to a Git repository and it supports other repositories too inside that repository it expects to find a playbook with the exact name of local.yml if it doesn't find it then it expects you to give it the name of the playbook if it's something else but if you don't give it the name of the playbook as an argument after the URL then it's going to expect to find a local.yml file in there and then for me that's an index that pulls in all the other things so when I first start up my desktop the next time the cron hits it's going to check out the code it's going to run at localhost but here's where it gets even better each machine when I run Ansible the first time it adds its own cron job to itself now let's think about that for a minute in Ansible I have a playbook called cron.yml and it's adding the cron job that will be used to check out any further runs of Ansible that going forward so after Ansible runs the first time the code will add the cron job to that machine so all subsequent times it's going to check it's going to do it on its own you only have to do it manually that one time and then from there it's going to pull it itself and that's kind of like the glue that I think makes this work because run it once then you're done and then you have the problem though of silently failing because what if you think like everything's okay you don't have it set to email you when it runs or maybe you do and you just haven't seen an email from one of your servers in a long time but you know you're not thinking about that you know life is happening you don't really care it's just one left email but then wait a minute I haven't heard from that server in a long time oh it hasn't been running for three months because something's broken what you can do is use a service called healthchecks.io which is not specific to Ansible that you can add you can add it to the end of a cron job line and it's just like making essentially a ping request basically and you can set that if you're a member of that service you sign up for it you have a special key that you give it and you could have one for each cron job one for each server and you could say if you don't see this respond in 30 days or a week or whatever you think the timeout should be email me let me know and it'll literally say that this cron job hasn't ran in a long time so if you look at that you could say oh well that's my secondary laptop and I haven't used it in over a week so that's expected but if it's your web server oh yeah well I need to see what's going on there because that should have ran by now so Ansible pull gives you a lot of flexibility but again it's the reverse rather than a control server using SSH to log into your machine and run commands on it it's just doing essentially a get pull and then it's running Ansible local host sounds simple I also added in the show notes it's healthchecks.io just so we're clear on that and the other one was lnav.org lnav.org so we make sure we're clear on those but those are the healthchecks is actually a pretty cool one for being able to set up free cron job monitoring is a really simple service on there Yep and one thing that I think is universal regardless of what solution you use for configuration management the bootstrapping process so as Tom mentioned earlier this is kind of self-documenting because if you have good code you know what it's doing I mean when you first start out if you're like me you have a text file for your Plex server whatever it is with all the commands that you used to build that server one after another and if it goes down you just execute them all and then you can put that in documentation so if you need to rebuild the server you can do that but if you have configuration management you don't need to have all the steps written out anywhere in documentation because it's in the config management code however what about the bootstrapping process meaning if you're using Puppet how do you get Puppet installed if you're using Chef how do you get that started you have a new machine you want to get Ansible running on it the first time well what do you do to get that going obviously is apt install Ansible or yum install dnf install whatever you could document that but the bootstrap process gets interesting because that's where I have a little bit of fun now what most people will probably do they'll have like an Ansible dot text file somewhere maybe in their documentation system they'll say something like when you first create a server run these commands to get Ansible going even if you're using Ansible pull you're gonna have some commands you need to run it the first time what command do you use are you gonna memorize that you put that in the documentation or you could just make a bash script and put the bootstrap commands in there and you could just copy the bash script over to the server run it delete it but I decided to go a different direction because I always do let's be honest I always have to be weird about everything but this is cool I set up a web server locally on my land that serves a bash script via Apache now normally I don't really like it when people freely just curl URL script the URL to a script and then pipe it to sudo bash as so many websites will tell you to do when you actually go to install something and people don't even think about it they just oh yeah copy and paste the command done and what if someone in the middle put something malicious in there well in this case my web server for this purpose is not available anywhere else it's on my land so unless you are on my network you can't get to this and what I could do on any machine in the house is I could do curl deploy slash bootstrap and then pipe sudo bash because I have a server web server named deploy the script is bootstrap that's the name of the script that's being served so literally that one command because I use Ansible pull kicks off the first Ansible run because it's just a bash script so it's gonna effectively do like in my case it's checking if it's Arch Linux W and Ubuntu and then running whatever command to install Ansible and then it's kicking off the first provision that it runs and that one command is all I need to do but then the elephant in the room is what if it's Linode that's not on my land well that's where zero tier comes in because now you could bridge that gap or wire guard or whatever you could connect an outside source to your local source so it too can benefit from that local bash script that your Apache server is serving and you could still bootstrap a new computer with that one command regardless of where it is so that's literally all I ever do on anything I just install the you know I just basically install the distribution and then I run that one command and as far as what I'm using Ansible for oh boy where do I begin because I mentioned workstation server and base so base is everything my workstation role sets up flat packs it hooks into the decomp config for gnome so it sets my wallpaper, my keyboard shortcuts the defaults for the terminal emulator the defaults for g-edit because I have everything in there and you can literally watch when this runs it just is so cool like the wallpaper will change in front of you as Ansible runs for the first time and you'll see the desktop your desktop kind of come to life so regardless if it's a server or desktop I still have the same code or the same repository and it has roles for each and even the mate desktop I hook into the settings there and I customize those steam I have a playbook for steam sync thing, virtual box you name it like literally every app I have it down to a true and false if steam is true it gets installed and it knows what's a workstation and what's a server so I don't have to worry about those server apps or those workstation apps ending up on a server I'm not saying anyone should go that crazy with this don't get me wrong I know that it's orders of magnitude more work than anyone should probably put into this but also keep in mind where this has come from is my YouTube channel I was constantly wiping my laptop and installing a different distribution to do a review and I got tired of rebuilding my computer literally every week I'm not even kidding you so I started building automation to get my laptop back to how I like it nowadays I have a dedicated studio laptop that I do all my reviews on now so I don't have that problem anymore but that's kind of where this started and like Tom mentioned earlier I've been revving on this for at least five years if not longer and this is one of those things where this, you know people ask the question a lot even of me like how do you back up your Linux desktops, you know and I'm like, I don't need to I only care about the config and the applications on there in terms of making sure I know what was on there so you build it as something that you can automatically reload again that way once again my data is real time synced and backed up on other servers and I use TrueNAS and things like that for my data side of things but as far as the system, yeah I mean, I would not be thrilled if it decided not to boot or the hard drive melted down and all the magic smoke came out but the goal is be able to drop a new drive in kick off those bootstrap scripts have everything back to the way it was you know even the little things like having my GitHub setup so I can on any system not only my system but any of my lab systems that I do demos on for my YouTube channel quickly go in here I grab the scripts that put the things together such as my command prompt the way I want it load the different tools and things you want so like kick this off have these here and this is what Ansible lends itself to be like very good for that's true and I need to find some sunglasses because I'm gonna pull a Morpheus moment right now and say what if I told you that I could eliminate the bootstrap process I'll think about that for a minute eliminate the bootstrap process how do you do that because you have to get Ansible installed, right? Well, I think I'm experimenting with here and I've done this on the Raspberry Pi super easy there's a an Ubuntu version for the Raspberry Pi Ubuntu server and it has cloud config or excuse me cloud init built into it so what I did was I put the Ansible pull command with the repository URL in cloud init so that way when I flashed that image to a new Raspberry Pi cloud init has that command in there and it's told to run it when it's first pervit when that's first set up when I first put that image on the SD card there's no bootstrap process now literally I just take that image I just create the image from the SD card where I put that tweak in there in cloud init put that image on another SD card put that SD card into a Raspberry Pi power it on make sure there's a network cable connected since the Ansible pull command is built in I'll literally flash the SD card walk away and I'll get a message on my phone that the provision finished and I had to do no bootstrap at all Yeah, and we did discuss a little bit of this in a Homeless Show episode eight automation and you would say so cloud init is not as well documented of a project as it could be No So I guess one thing I need to ask people about some of these technologies is like how's your anxiety and frustration levels in general? Are you the type of person that's really chill and easygoing and nothing really bothers you or are you quick to get upset? I mean, we're all unique, right? In cloud init kind of frustrated me it's a great solution there is documentation out there but not as much as you would have an Ansible Ansible is, oh man anything you could ever think to do someone's documented it somewhere I mean, you could even just go into GitHub and search people's Ansible configs and probably find anything you'd wanna do I have done that before but cloud init, which is one of those things I wanna do a tutorial on some day to kind of help people not be frustrated there isn't as much documentation and I think at least this is a theory I think why that is is because cloud init is often used by cloud providers when they roll out a distribution image for their customers so you have high level engineers that are just doing cloud init configs all day, every day they're always using this so for them it's probably like all the they don't need documentation they use it every day they probably have their own documentation but for us when we wanna use that most of the time people don't use cloud init because it's for like I mean, they don't really have the mindset often to know that it is for AWS and Google Cloud most of the time in what use case would this have for me at home but if you can actually learn this and you don't have to learn it well just a few things you can really benefit from cloud init and one of the things I did was I made sure that my normal user account is already there and the cloud init config file so normally the username is Ubuntu and this is kind of where I got frustrated because I changed the name and I followed the documentation the little documentation I could find on how to change the name of the default user I wanted it to be J instead of Ubuntu because I didn't wanna create that user and I never really fixed this but if you change the username in the Ubuntu image you will never log in it'll never happen I can't explain this I added the SSH key to it I made sure that the user was created properly the password hash all of that I promise you it was right but it doesn't work now what does work I take that same configuration and put it in a different section not the default user section but a separate section it works fine Ubuntu really doesn't like its default user customized but I found a way to basically tell it don't create that user and then in the separate section create this other user but who has patience for that? Like most people would it actually I did kind of give up for a week or two on this when I couldn't get that user working and I came back to it and it worked fine but I'm not saying everyone should use cloud in it maybe after I do a video on it I'll be able to endorse it because then I can give people something to use to learn it but I do think if you do use cloud in it the documentation is there whether I create the video or not there's benefit to the homelab people because you could basically create the defaults there and you can hook into things like Ansible and have that automatically run and then you don't need to do a strap process anymore. Yeah, those are like I said, I remember that was a big discussion point and other comments on that particular video podcast episode because that is the challenge and I get that a lot of people have asked me to do some videos on it as well and the reason for asking is that so I just wanted to make sure it was clear if you get frustrated you're sharing that frustration with us as well trust me we are not magicians when it comes to that I've only played loosely with it Jay spent obviously much more time than I have on it and still did not find it to be intuitive so if you're stuck there you're not alone you're stuck actually with people who do this for a living and when I don't hit the record button there are some very interesting words that come out of my mouth where I get these things working so don't think for a minute that we're above frustration or anything like that we're human too believe it or not we're not sylons I promise a couple other things about Ansible I should probably talk about briefly at least the Galaxy Ansible Galaxy that's kind of like your place to get things that other people have written so the mindset to have with configuration management is if someone else has solved the problem maybe you might consider using their work if it does what you want it to do and you could do that Ansible Galaxy you could pull in things there that someone else has written and then in addition to that we have basically an entire server application that you can use optionally with Ansible and it's called AWX which is actually the open source version of Ansible Tower which is what Red Hat calls it if you're at enterprise you can pay for the Ansible Tower server which gives you a GUI in the browser that you can use maybe as in place of a control center and basically execute those commands to all of your servers AWX is something that you can use for free and you have to install it on you have to install it yourself because it's free the onus is on you getting it installed it's not the easiest thing to install it's one of the more frustrating things I mean they want you to use Ansible Tower they're not gonna come out and say that but let's be honest that Red Hat is a company they make money they have bills to pay so it's not that hard you just have to be patient and go through the instructions but lately since I've set it up there's been some great articles written that have come out that will walk you through the process of doing that Yeah, so that's very helpful before we wind down towards the end of the show what's some of the most basic commands that people can get started with other than install Ansible where's that first couple things that they can do what's that first thing you recommend or maybe even a series of things that you have a video on Yeah, I have a whole video series on that and we'll have that linked in the show notes but if you go to learnlinux.tv the name of my channel for those that don't already know is the same name as the website so it makes it kind of easy to know where to get it and there's a section on there at the top I forgot what the menu is called I think it's tutorials if you hover over that go under there, there's Ansible and you can definitely check that out it'll walk you through it my series will take you from the very beginning all the way up to I'm not gonna say advanced but you're gonna be comfortable with it then I have dedicated videos on Ansible Vault which is how you can encrypt things so if you are going to be using Ansible pull against a repository and you don't want private info visible you can use Ansible Vault to encrypt things in that repository to keep those things private I have a video on that and Ansible pull as well but as far as the beginning command ping is always the one it's like dash m is the option and then it's ping but what's interesting is it's not actually a ping and I don't really like that they call it a ping because when we think of ping most of us is the ICMP ping where you get a response it's the ICMP so it's not that what ping actually does in Ansible is it's making a real TCP connection via SSH to the target and you could give it an IP address of a server as an argument and it's going to do what it's gonna basically look at the stats of that machine like what distro are you and so on it's getting some information about that machine which is not what ping does everyone that has used ping they know what that does it doesn't do that it doesn't give you information other than hopefully a ping response but the Ansible version of ping that's more than just a ping it's not only just telling you that the host is on the network and is accessible it's proving that it can actually interact with it by pulling some information and sending that to you there's also modules you could use to run a command so you could test it with I forgot what the option was but give it the module apt basically and install a package for example you don't have to have a server for that and that's one of the benefits is that you can just use Ansible however you want that's why it's complex in a way but it's also simple at the same time it's a gift that keeps on giving I guess is the best way to describe it Yeah someone asked in the chat and I don't know the answer to this does Ansible support HashiCort Vault so we know Ansible has its own vault but does it support the HashiCort Vault as well or is that a we don't know? I don't know I've been meaning to look into HashiCort Vault I don't know if it's changed but when I did look at it I thought that it was extremely overly complicated not that that's ever stopped me before because at one point I just decided to set up my own mail server because I guess I have thrill issues or something but I do plan on looking into it so I could talk more intelligently about it because complex doesn't mean I'm not going to look at it I'm definitely gonna look at it but for the use case that I had at the time that wasn't appropriate at that moment but it has its own vault like I mentioned does it hook into Ansible Vault? I wouldn't be surprised I'd probably be more surprised if it didn't but speaking of HashiCort the logical question a lot of people that are more familiar with the different tools will ask is where does something like Terraform and Ansible begin and for those that don't know Terraform is also by HashiCort it can basically provision a machine or an entire installation for example, if you have Amazon Web Services it could set up like 10 servers your load balancer and your networking rules all of that you can script all of that which is great and you can use Terraform to maintain changes and most people do this but I find Terraform to be very messy after something exists so I usually tell people Terraform is to make things exist Ansible is to take things that already exist and keep them up to date with whatever the configuration should be that's where I draw the line personally Yeah, and someone earlier in the chat had said Terraform provisions the VM and Ansible configures the VM so it's kind of there between provisioning and configuring and XCPNG recently announced a lot more Terraform support that's an example of we're gonna use Terraform to build these VMs off of these templates in terms of this distro with this template and then once you have them at the ready then you're gonna kick off either whichever method you choose like Jay mentioned the bootstrap on there and kick it off and start pulling your Ansible configs to actually make the VM dance and get all your things loaded so it has to exist before Ansible starts I mean technically if you wanted to get really crafty because XCPNG is all API driven you could even write Ansible code to talk to his answer to build some of the templates but that's outside of it Terraform's the more ideal product to use that with so that's kind of the dividing line for it And one thing about dividing lines with Ansible is you also have to be careful with that too because anytime you say Ansible doesn't do X which is probably logical someone's gonna say yeah it does that it didn't even know it so for example it can spin up VMs it can hook into an API of a hypervisor solution it can create containers for you you'd give it the image and I mean you can use Ansible for that I just like Terraform for it but going back to Cloud in it our favorite topic now Noah mentioned that he had the same problem with Raspberry Pi where you try to change the username from the default of Ubuntu to whatever you want it to be and you can never log in I don't really know why that happens but the general idea that I'm trying to remember off the top of my head what I did again video will come eventually I'm fairly certain what I did was I told it no do not create a default user just don't but then later on there's a section where you can put the user that you do want to create obviously you have really big problems if you're telling it not to create any user at all and you just have root you can't even SSH it's even worse well you can't SSH anyway if you change the username for whatever reason so I just said no no default user and then later on I was very careful to tell it yeah create a user for me with this SSHP or the password hash and make it sudo but in a specific section outside of the default user because some weird breakage happens when you try to change the name of the default user in Ubuntu again I don't know why but just say no I'm pretty sure that's what I did and then later on in a separate section you can create that user and then there was some module again I'm kind of new to cloud in it in full transparency but there was a option in there where you could tell to run commands and then that's where I put the initial ansible pull in that section and everything was great it worked out pretty well so at some point I'll do a video on cloud in it but I can't promise when because the unfortunate truth is I have to learn it before I can teach it and I don't know it enough yet to teach it but once I have it far enough along to where I think I can even if I'm intermediate at that point even if the entry level stuff might help someone I'll start doing videos on it at that point so just stay subscribed or subscribe if you haven't already done so it'll show up eventually I promise yeah and we didn't mention earlier but I'm maybe mispronouncing it but I believe it said idempitance and it's the it's the yeah it's it's used in mathematics and ansible I that's why I've heard the term most frequently idempotence before I don't even know if I'm saying that right isn't it sad the person that wrote a book on something is mispronouncing things that happens to everyone yeah it was us you know and before I ever went to a Linux conference I mispronounced many things before I met people in person because I've only ever read them online but it is a concept someone brought it up and we didn't mention it but you know it's making sure to all this always gives you the same result these servers are all the same it's reaching that level in there so it is a popular term you may hear we I just didn't hear us mention it earlier and someone mentioned it in the chat you need to kind of mention it for you know really quick it's really important in ansible because for example there's several modules you can use to basically take a string that's in a text file replace it with something else now where the problem comes in is if you do that wrong it's going to not only replace that string that you told it to but every time it runs it's going to append that line to the same file over and over and over again and the biggest thing here that will tell you that this is happening is if you run ansible and it's telling you that there's a change but you didn't make a change it's always saying that there's a change why is it doing that and not always but often the problem is that it's not idempotent if I'm saying that right basically what that means is that you think that you're doing one thing one time but it's doing it every time that's that's one thing there and find and replace text you have to put the carrot symbol at the beginning for example when you're replacing a line otherwise it's just going to keep doing it over and over again and there's other options we could probably do a whole tutorial series on this but I have but you have done a tutorial series so the more yeah we try not to get too much of things that would have been very that lend themselves to more visual tutorials on the podcast here that's why we do reference some of the videos that we've done because they're easily accessible you can just go watch them on YouTube and that gives you that better visual reference for these type of things we talk about so you can watch because it is important that you know which characters go in which order when you're typing commands because you don't want that to just keep replacing a line or adding another line you want it to replace versus concatenate the line so those little nuances are easier expressed within the tutorial videos not on the podcast and speaking of topics and diving into things my challenge for our audience is on the YouTube version of this to in the comments mentioned something that you want us to cover we're not gonna promise that we'll cover anything in particular but let us know in the comments there what you think we should cover in a future episode and you never know it might just become a topic that we'll cover here and you guys could just put those in the comments and if we get a lot of people asking for a particular thing that's gonna potentially put it up higher on the list if it's something that we feel is in scope. Yeah, when the challenge we're having we have so many ideas when we want to actually favor the ones that you maybe pick out of those first if not don't worry we will have content if you have no ideas or not sure where to start that's fine too because we do that's actually our discussion we have all the time it's trying to nail down which is the next step which is the next thing we're gonna start So for us it's the order that's the problem not the content because I could list off 10 different things every time we do one episode I'll have 10 more ideas for just speaking for myself but then the challenge comes to be is it like, is it time to talk about this yet or is that too soon? That's usually what the discussion is being but yeah, like Tom said we'll prioritize things higher in the list that people want more. And of course you can find all this at the homelab.show that's where we have the show notes we leave the YouTube up as well so whether or not you wanna join us for a live stream or you wanna add comments here we've offered all the different ways to listen to us and it is brought to you by the node sponsor of the day and in the most literal sense when you click the download button you're pulling it from a Linoad server that is hosting the site. So if you didn't know if you're on the site now and you didn't know you're using Linoad surprise you're using Linoad right now you're using it already you didn't even know it. And we've made these available as MP3 used to download so you can download direct or any podcast app that you want we've tried to publish in all of them if you find one we're not published and let us know and we'll do our best to get it over there as well and spread this podcast to everyone posted on your socials tell your friends and colleagues about it if people ask how can we give back that's the best way to give back just tell everyone about it spread the knowledge that's how that would really help out. You know, share us on your socials put us on the Twitters or whichever platforms you're using and we want to see more people get in the homeland that's a big goal we have we want to see more people in tech this is it's really daunting to get started and that's kind of our goal to get you started in tech and maybe you want to pursue a career in this it starts with that. I was just on another show and it's funny because the hiring manager, my friend Bob he said that's one of his frequent questions when they hire techs at his IT company is he says, you know, what's in your home lab is just a general question to ask because it's not always people people don't always take the time to listen to a resume but even hiring managers are going, hey, this is something we want to know we want to know what you're actually working on at home are you learning this because maybe those skills are transferable to a job so these are, you know, real world things we want to help people with. And that's a topic on my list that I want to cover at some point is what are we running in our home lab but the trick is I have some people I want to see if they want to be a guest I'm not going to say who because I don't want to promise anything yeah, but at some point I think it'd be a fun episode to have a guest and have a topic around like, what are we running? Yeah, there's so much fun stuff. Oh yeah. Anyways, thank you all for joining us we're is 151 of you so like I said, share this out that's a big thank you for us so happy to have all of you here me and Jay are going to go back to I don't know, whatever it is else we're doing we're not doing this. We're saving time. One server at a time. One server at a time. This is Tom Lawrence and Jay LaCroix and I'll see you guys next week probably next week. All right.