 and welcome to the Home Lab Show, episode 48, and it's another Dev Random. Me and Jay have fun with these episodes because we have so many topics and we're like, it's hard to bring this topic to be an entire episode. And we think there's a lot of value in all these little tools and tips we're gonna talk about that help you optimize building your lab and little tricks and software tools and things like that. So if we don't want to just call all of them, just one little thing or a bunch of miscellaneous. So we've just chose Dev Random as kind of our theme. I think that as long as everyone likes it, we're gonna keep going with it. Yeah, every now and then we'll do one. Yeah, they're fun. Yeah, we just build up the process in the backend. Here's we're looking at tools and using them all the time and breaking them down. I also thought about a Dev Update episode idea. So let us know what you think about this. And I didn't bring it up to Jay, he's learning right now. So I wanna see his reaction to it. The tools we wanna revisit because there's new version released and it's kind of the changes maybe major day, maybe minor, grabbing a handful of products and doing Dev Update and revisiting them. So you can learn about the base of the product in our previous episodes, but the update, recently XCPNG has a new version out. TrueNAS scale has been released since we talked about it. So there's been changes. These are fun things. It's fundamental of the product is the same, but hey, let's get excited about those new features because a lot of them are exciting. So we might do some episodes like that. Now, one thing we're gonna do right away is thank a sponsor of the episodes so we can jump in because we have a lot of random things to talk about and educate people on today. And this episode is sponsored by Linode. You guys are shocked, hoping another name would pop up maybe. I know someone said they wish we had a bigger variety but Linode has been such a great sponsor. And so many of the tools aligned with this. It's easy to set up Linode. It's easy and DevOps friendly to build things and build on that platform. We think they're just a good alignment for the people that listen to this because well, it's a product we use. And don't tell Linode this, but we probably still use them even if they didn't sponsor the show because we were using them before they sponsored the show if you didn't know. We just came to be in alignment. This is actually where all this stuff has been hosted pretty much since day one because Jay's longer than me has been a Linode user but I am as well. And the offer code we have to you is head over to linode.com slash homelab show if you wanna get started and lots of little things, tips and tools that we're gonna be talking about are things you can run on Linode and test out on Linode. So if you don't wanna run all these servers at home or for whatever restrictions that's not plausible for you to do or possible for you to do, then yes, Linode is a great place to run on. Use our offer code to get started and let's jump into this. Yeah, let's do that. So quite a few things here. So the first several things are going to build on another one another. It's basically about network organization. And I think that this is really important because... Yes. You know, I remember, believe it or not when you had like one computer in the house and you had like maybe one Palm Pilot or whatever it was back then. So you really didn't have much on your network. So like a standard slash 24, I mean was overkill. Like who's gonna fill that? And then next thing you know, you have Roku's and you have video game systems that are online now. Every room has a television. You might have a laptop in addition to a desktop and everyone in your house has their own depending on your status. But at this point, it gets to a point where you have a lot to maintain here. And when I talk about Roku's and all this other stuff unless you're using Plex it really doesn't have anything to do with HomeLab. But it kind of does because if you're maintaining the network there's a lot of overlap. And if you don't have a theme or a scheme it can get kind of confusing. So like I said, these things are gonna build on one another. And the first thing is an IP layout. Now I'm not talking about subnetting your VLANs. If you have VLANs implemented, awesome. But I do understand that a lot of people don't because they don't have managed switches that can do that. But they might just have a commercial off the shelf router or residential router or whatever and they have a limited feature set. So if that's you, then you could come out you can come up with a scheme and there's not gonna be any segment or segregation here but it's just like a abstract layout type of thing. So you could have for example let's just say .2 through .10 of your IP scheme could be like network devices and you could have something like .30 through .40 servers .50 through .60 desktops then DHCP could be like the last 50 IP addresses. It really doesn't matter how you carve it up. There's a reason why you might wanna do that. We'll get to it. But it kind of just makes everything make sense because then you have your servers one after another in your DHCP table, your desktop laptops together maybe your internet of things hopefully you have them segregated but their IPs are fairly close. So depending on what the IP is you have a reasonable idea what it should be if you have DNS even better but that's kind of where we start. I know that's not particularly exciting but it kind of helps with what's gonna come after this. And coming up with a naming team can be a struggle. I used to do planets for a long time but then that was 20 years ago and it quickly exceeded it then we went with some Star Trek names. I'm still a fan of naming servers after different Star Trek things. It's hard to have a good scheme but it's fun to set that up. The other thing that not to underestimate I see people popping in about VLANs being a little bit difficult to manage there are some challenges that come with it but don't forget to what extent you can you can frequently with many services whether they're Windows or Linux lock them down to a access level like these devices have their own firewalls so you can lock down the access levels to say not everything can talk to this device so even though you don't it's not like you're eliminating security by not segmenting your network you can still have a reasonable level of security because even for us with our specifically server more protected LAN where we have more critical things running in my network those systems don't talk to each other and specifically are not allowed they only have implicit allow rules on the firewall for what they are allowed to talk to it's one of those things just practicing principles of least privilege so even as a home lab user yes you can offer mitigations if things are on a network you may not be able to control that IoT device but you can set rules frequently in the firewall of your device to say you know what don't talk to those IoT devices they may not be friendlies. Exactly and naming scheme is the second thing on the list actually the first was just all about having an IP layout and the next thing that builds on top of that is having the naming scheme like you mentioned and I you know I've mentioned this several times on the YouTube channel that I use Final Fantasy 6 Espers as my names for my devices because that I'm obsessed with that game I can't count how many times I've beaten it you know every character level 99 like multiple times since I was 13 so it became a natural fit to name everything out after Espers but then it came to a point where I had way too many devices more than there's Espers in the game so what I've done is I just continue to use that naming scheme for laptops but everything else I have a different naming scheme for and this is where you can get a little clever because one way that you could do this is to make it such that if you I don't know export a list of your machines and alphabetize it in a spreadsheet that they are able to be alphabetized if you come up with the right name naming scheme so for example it could be something hyphen something hyphen something for example net for your network devices like if you have managed switches or access points or something like that then just start the name with net and then hyphen this is what I do by the way and then hyphen what type of device it is like for example a switch could be SW so net hyphen SW hyphen then the location office then you know kitchen whatever and then if there's more than one hyphen one hyphen two so you might have like net hyphen AP for access point hyphen upstairs for the upstairs access point or if it's a storage server it could be SRV for server hyphen NAS hyphen one you know something like that so you just come up with your own naming scheme that makes sense and I also have one IOT hyphen SP for smart plug or I can't remember what I have for bulb oh SP for smart bulb is what I have so basically that's just how you can name your devices and then that leads right into the next thing which is DACP reservations for everything everything everything and it sounds like a lot of work and it really is like I'm not gonna sugarcoat this okay so people might be wondering why the heck would I wanna go through that I have like 20 IOT devices five game systems like six Roku's a bunch of TVs like who's got time for that but once you actually get that implemented though it's so great because then if you have anything that comes in with a you know non-reserved IP for example it's within your DCP you know realm then okay that's a new device because if it wasn't a new device I would have had a reservation for it downside is you have to remember to create a reservation if you buy a new device but then it makes it really simple to see like a new device was added to your network because it's outside of that reservation and the beauty of this is that you can turn on ARP watch if you have access to that like PF Sense has this where you could actually get an email alert that something was added to your network and when you go to the DCP reserve or the lease table in PF Sense you can sort it by most recent lease and your new devices are gonna show up right at the top so somebody added a new device not only is it gonna have an IP address that's predictable it's not a reservation it's not gonna have the right name it you know ARP Watch is gonna email you so if somebody's trying to circumvent not that anyone do this on your network but if they're trying to add a new device or someone breaks in and adds a new device you know I mean you got all these layers of things that are gonna let you know this is a new device either you haven't configured it yet or you weren't expecting it at all so all these things together kind of just make it's a lot of work at first but once you get it done oh it's great it is so great. It's also really helpful when you decide to start growing your network or wanna rebuild it or re-segment it with a different range you know even myself I took all the stuff that I was using in my studio different things that are IP connected and combined them with different things I have when I move my studio to the new location and I just you can do this in granted PF sense makes it really easy but there's other firewalls too that do this. When you're everything reserved we just grab the reservations and you can do a selective restore to restore the reservations to a DHCP and you're done like great we just restored all the reservations so I didn't have to redo them and as I brought them all over to a completely separate network they all aligned right where I wanted them to be and because the files XML I could just do a search and replace to line it up in the XML file as I needed with PF sense. Now we also do this for our businesses because we do a lot of camera deployments and people ask well you know how do you do this and everything and we have we have a couple of them have like four or 500 cameras at some school networks we have everything set in DHCP reservation tables so this completely scales directly to business and then anytime we replace cameras if we have a we're making this up because this is literally what my team is doing this morning is replacing cameras at school we pre-program their reservations with the new replacement cameras so they have the same IP address at the same location because of the way the camera system works so it's gonna pull that you know this IP address is expected to be in this particular hallway we just swap out the reservations as we swap out the cameras and it becomes really easy to do that way when you send something over to another network it gets that IP address you don't have to plug it in find it you know and everything else we pre-program it get it set up so this scales out to a good proper way to do things across even these business platforms. Yeah I agree with that so one of the best things about this is that no matter and this is pretty much the same as what you're saying but if you think about it if you are reloading a server or a computer let's just say you boot into a Linux Live disk or something if you're using a static IP that you set manually by hand then obviously the Live image is not going to get that IP it's just gonna get a DCP IP but if you have a DCP reservation then that device is gonna get the same IP no matter what so Live instance regardless of what it is it always has the same IP and it may not be super popular of an opinion but I really can't stand static IPs that are done by hand I just don't see why they're popular I don't think it's a good idea I think that it's a bad practice now don't get me wrong I think that or actually I know that there's some devices out there that don't give you an option like some apps I can't remember which ones are the top of my head will expect you to make a static IP and type everything in and won't even give you an option for a DCP because their mentality is this is a server people should be using static IP but I disagree with that because not only is it better I mean it's just I mean it makes everything better you go into your router or your firewall you'll see a list of all your devices and IPs it's so easy to audit I mean, if you think about it you go in there oh I'm not using that device anymore I'm not using that device anymore you have one central location for all of your IPs you know what they are you don't have to go into a server to look at the config file for it you don't have to set the config file you set everything to DCP and be done with it I don't care if you're a business I don't care if you're not I just don't like static IPs I think it's time to let that go and let DCP reservations aka static leases take over And kind of added to that one of the interesting things that happens we have some of those different services that really want a static IP but we still put a reservation in for it and the reason why if we ever have to work on that machine or boot it from a live CD it always gets the same IP address because it's based on the MAC address so even when we start a reload or some process change on it that may break some settings or booting it from a live CD to a recovery the MAC address doesn't change so it gets the same address that it had to be assigned statically so even the static assigned ones I still put a reservation in that way in the off chance that server has to be rebuilt or something happens the interim of it not having a static IP means it still has the same IP Yep And someone in the chat room I'm glad they mentioned this they mentioned how there's some sort of UID or UUID for networking because most of the time it's the MAC address that's presented to the DHCP server DHCP server sees the MAC address oh yeah that's Jay's laptop so it gets this IP but there's that's actually now the Etsy machine ID it's slash Etsy slash machine hyphen ID I know that Ubuntu server switched over to it Debbie and may have I haven't looked into how deep this goes yet but I know that Ubuntu at least and that is a whole new topic altogether I have covered it on another video I just can't remember which one but if you're out of net video that yep it's in there too I think I wrote a blog post at some point about this as well so yeah that's definitely a valid thing that you have to work around I'm not really fond of that to be honest I know there's reasons for that but that does make it a little harder for static lease if you don't know what the machine ID is going to be because when you boot a live instance it's going to have a different machine ID it's just not the greatest thing but that's yeah that would take way too long to cover here but if there's enough interest I might do a video on it if the cloud in it video people feel like that didn't go deep enough so yeah it's it's a confusing I was I mean I've talked with you on it that's how I remember it was in the cloud in it video I remember the discussion around it but I remember being a confusing topic so that'll be a later topic and out of scope for today's DevRandom yep out of scope but what's not out of scope is the next idea so I've met again I've mentioned this before and some of the stuff I've mentioned before but I just wanted to kind of get some syntax out there and some information around it but the next idea is having a central config management slash work server what I mean by this is a stationary machine it could be a Raspberry Pi could be a virtual machine could be a physical machine doesn't matter but if it's a physical machine hopefully you don't pay too much because this would be a waste because of this is kind of like a disposable kind of instance but you have this central machine where you use you know USSH into it you could use Tmux which is preferred and recommended you have your session going you can have a different session for every project so if you're implementing Ansible you have a session for that if you're updating your server configs you have a session for that and what's cool about this is that let's just say you start your work on your desktop but you know you want to relax you still want to work on your servers but you don't really want to be in your office chair anymore you kind of want to be on the couch with your laptop so you don't want to have to reopen all your apps and things and it just gets really kind of time consuming but if you have everything in a Tmux session in a central available server again Raspberry Pi, VM doesn't matter then you could just use Tmux restore your session and you could take your workflow with you wherever you go and this is actually even more powerful with Mosh which is a wrapper over top of SSH and what Mosh does is it automatically restores your connection so imagine this like normally you have Tmux running on your Raspberry Pi you just open up your laptop you SSH in to the Raspberry Pi Tmux Space A that gets your session back and that works great but if it's Mosh you eliminate most of those steps you just open your laptop and your session is found and automatically attached like you do nothing but just like when you're connected to the network so for example, let's just say I don't know you're configuring your servers you take your laptop with you to work so you close your lid, you go to work and then you open your lid it has your session there that you were working on but it's not connected anymore because you're not on the same network as your Raspberry Pi, you're not even at home but then when you get back home and you reconnect to your Wi-Fi it'll reconnect to that session because it notices now you're on the same network so it just reconnects you to it now the caveat with Mosh is that it was not made security first as far as I know I don't know if it's been audited do not make it publicly available at all don't consider that it's not for that I don't think it was developed for that purpose if you have something internal on your LAN that you can't get to from the outside like you can only get to from the inside that's the way to do it because then when you get back home you reconnect Mosh is there it uses UDP I believe port 60,000 through 61,000 uses a random port so you might have to open something up for that to work but once you get that going and as long as you don't expose it externally it just makes it so amazing paired with Tmux that your sessions just restore and you just continue working like nothing happened Yeah It took me a long time I seen someone mention they still use screen screen's not bad once you get to use Tmux and all the split screen options boy is just reattaching to a Tmux session especially when you have updates you're running or something you're monitoring you want to leave there and you're switching computers you're moving around and that's still a thing I do you know between my main computer and going I have I have my laptop in my kitchen just you know got to stay connected while I eat but being able to jump back in and grab that session with even the Tmux just makes it so much more convenient it's worth the time to learn it both me and Jay have videos if you search our channels several videos on using Tmux and talking about the keyboard shortcuts for it I'm a big fan of leaving them at default I don't know if you are Jay I do that because that way anytime I run into Tmux I know where the keys are I thought Jay was on the opposite program to the keys either way works I really don't yeah that's I can understand that I really don't like the defaults in Tmux because I feel like it's hard to work with in my opinion because you know it's control B is the shortcut key to do a keyboard shortcut you start with control B or whatever the prefix is that's what they call it and then you type the command but holding control and then B you're stretching you know you have your pinky finger and control you're stretching your pointy pointer finger all the way over to the B it's hard I use control F and control J because your fingers are already there anyway and you could have a secondary prefix so that way if your left hand is pressing another button your right hand is free to do the prefix or your left hand is free to do the prefix regardless so it lets you do no more than two you can have a primary prefix or the secondary one and that makes it a lot easier for me but also the default shortcuts where it's like prefix and then I think percent symbol and then double quote to do a split I do I change it to V and H H for horizontal V for vertical so prefix H horizontal prefix V vertical it's super easy to remember then I map shift with the left and right arrows to the different tabs and that seems to work a lot better in my opinion yeah well what definitely doesn't work is it is harder and I did the hard way on my channel of just showing the actual commands and what is I have on my muscle memory but I think it's like percent splits one way and then the other I have to look at it from muscle memory to do it so I get completely J's reasoning for wanting to change it to a more rational I don't understand how they came up with the keys on there for a long time when I learned Tmux you can look up Tmux cheat sheet and there's a cheat sheet that has a list of the common commands you can use on there so yep so that yeah those are just a few Tmux tips where we're not which we're not actually part of it but I'm glad you brought that up because check our videos out we will show you how to config that you know change Tmux to work for you and it's so amazing the different types of things you could do with Tmux beyond the default so don't just use the defaults and then think that's it now screen is fine like you said at some point they did they did patch in splitting and doing pains though I don't remember what I had to do then I didn't know you could do that screen not every version because I think it might depend on and this was a long time ago so maybe every distribution has it now okay last I looked it kind of depended on if that patch was present and it may or may not be and I don't think it's quite the same like it's I mean the future exists but I think it's a little bit different not quite as good but you know screen is still perfectly valid yeah I don't use anything wrong with using it the split screen just helped me a lot when you're doing a lot of testing so I would actually sometimes split my screen and then each Tmux session would be a different server and that way I'm doing you know IPer for different transfers and settings file moving I can look at them from both screens especially when I've got a copy and paste things they're just on one screen to be copied and pasted by the way turning on mouse control in Tmux if you use my Tmux settings which are available on GitHub I have all the mouse controls turned on which adds some context menus for right click they also allow you to click around to the different panes so if you click on a specific pane you can start in that particular pane yep there's extra context is actually added when you're running in a terminal like that is one of the cool things I think about Tmux is that they have these context aware menus and right click menus for doing and launching different things so it can get real extensive really quick it really can and it's really fun another one this is fairly simple and I've mentioned this several times I haven't really mentioned it and I don't think I've mentioned it by itself I may have but worst case scenario it's the second third or possibly 10th time I've mentioned this healthchecks.io yes it's awesome because here's the problem let's just say you have some jobs running via Cron we all do right and you have to check them every now and then make sure they actually ran maybe you have them send an email to let you know they've run so you expect that email but with healthchecks.io you can actually ping a very special URL and that basically means it ran that's all you set up I can't remember how many checks you can have by default for free but basically if your script doesn't run and you have your check set that it I don't know maybe expects a response every seven days it doesn't get one then healthchecks.io will email you and let you know hey this didn't run I didn't get anything for this which could be so great because that way you have something watching your Cron jobs to make sure they've run now a best practice here don't make the healthchecks.io ping or I don't remember if it's a post or a curl whatever don't put that at the beginning of the script okay because the problem is that'll always work the script will always start unless you really mess it up like because then the check is pretty much invalid at that point what you want to do is check the exit code of your script if it's exit code zero meaning it's fine do the health check submit that over there they'll get the message and it ran but if the exit code is not zero then healthchecks.io gets nothing then you know something happened because something went wrong and they could check the logs or whatever to make sure that it works that is I used the heck out of this because I have I don't even know how many Cron jobs I have if they're all managed by Ansible but I still want to make sure that they've run so I basically set them to seven days with a one day grace period it could be a few hours after seven days but definitely not more than eight and then I'll start getting emails if something doesn't run and it's saved me quite a few times it's kind of nice just having the simplicity of that that's it's been around for a long time too yeah I think I'd heard about it first last year the year before that so I highly recommend you check it out and by the way none of these are sponsors by the way that we're mentioning oh yeah it's just tools we use there's tools I mean use a free account like if you want an offer code can't help you I don't know just go on the site or maybe Google for someone else's offer code but you know we just like these tools yeah I mean the alternative and someone mentioned it is you can send everything to logging servers and then grep those logging servers and I have a whole video on gray log which has triggering options so you can send notifications there's other ways to do it but we started with the simplest way certainly the more complicated way is fun but hence more complicated yeah and also I mean it even if you do have that setup it doesn't hurt to have something else watching too you know who watches the watchers exactly or if you have like two things watching the watcher that's fine if nothing else then using that might be something you could recommend to someone else if you like it if they're looking for something like this so there you go yep absolutely the next thing I recommend that everybody run every now and then which is completely free is called Shields Up and it's done by GRC you know Steve Gibson he's the security now person I love that podcast Shields Up is great because you run it from within your network so if you want to check your home network be at home you know don't run this at a coffee shop okay that's not what it's for run it from where you are or where you where the network is that you want to check what it'll do is it'll actually tell you what ports the website was able to get a response from so if you think oh well I have this port open but I don't have anything else open shoot then okay great glad you said that but let's actually let's actually see this let's let's put your money where your mouth is run this test and if you have any ports open that you know you didn't know about or something like that then it'll show up and you'll know about it that's really all it does I mean there's some other things that it doesn't get into and GRC has some awesome other tools as well but shields up is great because again it just shows you what ports are open so you can audit yourself basically it's not a thorough port scan or anything like that so you know if you have port knocking it probably won't find it I don't think but don't like over exaggerate the value I mean it tells you that a port is super easy low hanging fruit but if a port needs a little bit more work it's not going to tell you about that because it's not a deep scan but it will tell you like low hanging fruit you have port 22 open to the world oops maybe you should close that down yeah there is I tweeted out an interview with a CISO so go all the way to the enterprise again and it it was about them getting a very large company that got ransomware intact and it kind of came clean talked about the whole process now the technical details of this are not interesting oops somebody left an RDP port open completely by accident they knew better than to do this a mistake was made it was left open someone got in and this is what you're trying to avoid is I've had people where they just accidentally do something they were testing something they whoops I made a port that opened up on accident and because of an experience with the firewall or an oversight or some test that was being done that you just didn't realize would lead to these results whatever the reasoning there's many of them just just check your ports that's one of the really critical things to take a look at it and I'll give kind of a second shout out is Shodan is another option as well they have a paid service but it's a one time this is a really you if you go to the pricing on Shodan it's not in their initial pricing scroll down to the bottom and there's basically a almost like ideal for this audience here like a the homelab users it's one time fee of $49 a year they let you monitor a few different IP addresses they give you a limited number of queries but they give you access to some of the more feature rich things you can do with Shodan now one time $49 not per month not per year one time and I think sometimes it even goes on sale below that when they run specials I think Shodan has more than once on a Black Friday special oddly enough but the important part is you can set up and throw your IP addresses in there I believe you can do it via DNS so if you have a dynamic IP address you can yeah you can also it gives you access to the Shodan script so you can do things like tell it to win the query your system you only get like a hundred queries a month but I don't think you need to check your own IPs more than a few times a month maybe you have a couple others you want to keep an eye on either way I think this is a really interesting tool that allows you to even build some automation to have an alert sent to you if it goes hey look what port we just discovered in your network and you're like huh how'd that happen so it's just one of those little things to keep an eye on and keep check because even is large and obviously the problem actually gets harder at some of these larger companies despite people saying oh they should just close the ports to never open RDP yes they can agree with you but things happen and this is why you have a layered approach to security and also why you have checks and balances to the things you do such as just monitoring for open ports because it could also be a sign of you being infiltrated where all the sudden ports are opening up and you're like I don't remember opening any of those why are they all open now and then you can find out that's another indicator that someone's trying to exfiltrate data out of your network because they got access to your firewall and started opening up the ports that you didn't intend to open these are just a few tools though that help keep you a little bit safer by doing these and they're relatively easy to do shields up as free show dance really inexpensive at a one time fee of $49 yep yeah that that those are all great points and you know just keep an eye on your networks and what ports you might have open and even if you you know have an open to port I mean that doesn't mean that something else didn't like you said especially installing a service or updating something who knows what could happen we would love our systems to always have the same state always but you know besides if you know everyone else is using you know show it in against your network so why not join them right I mean if they're using it against your network you may as well do it yourself as well so yeah so using a VPN is a good solution I see people asking but you have to have a portal for VPN yes but now you've created a choke point and VPNs generally speaking specifically I should say open VPN and wire guard to really popular ones are well vetted protocols that have been both gone through a level of security auditing that I feel comfortable opening up them on my on the network because it takes a really solid cryptographically secure transmission and challenge response in order to get in via those VPN so granted yes they are if there was some major flaw found in your VPN then there's a potential but if you're using something like straight open VPN and straight wire guard they're pretty well vetted and within reason of security I mean see if you want to be the most secure just unplug everything just throw it all in your closet throw it all in the closet it goes somewhere where there's no internet and you're a little safer yeah on a deserted island yeah you probably nobody will hack you yeah but that's not reasonable so back to reality well I should kind of would like to be on a remote island sometimes sometimes so the next thing is you know back to the whole thing about you know watching the watcher I use status cake I don't want this to sound like a personal like recommendation for status cake because I feel like there's a bunch more that do the exact same thing so I don't really care what it is that you use for this this topic if you do want to use this status cake was like the first one I found which is literally the only reason why I chose it it's been fine I've had no reason to look elsewhere you could get a free account they have a paid account well anyway what it lets you do is just make sure that your site is actually up so for example if it's not getting a normal response via ACDP okay I mean your server can be up but that doesn't mean your web server doesn't have a problem right so what this will help you do oh the website's down you get an email and I'm not sure if this is a paid feature or not but if it takes too long to load you can also get an email now how valuable this is to you I mean home lab may it may or may not be all that important but if you do have a site that you want to get some kind of notification that it's down it's one way to do it and it doesn't have to be status cake I feel like there's like a bunch of these I just can't remember the other names for these just go with whatever one's the cheapest and that's probably fine I think there's I can't remember the name of it but there's one that you can self host as well I just don't remember the name with that I do but then again it's another issue you're self hosting your watcher which is watching your self host stuff I don't know how well I how good I feel about that unless it's in a different account but yeah it's just a quick mention not a personal recommendation for status cake I really don't care which one you use we don't they're not a sponsor like I said none of these guys are but that's just something to consider so the one Jay mentioned I threw both these links in in here which is status cake dot com and the other one that's free to host is phpservermonitor.org the phpservermonitor something is really clever obviously one of the challenges is if you're trying to monitor your own infrastructure with it you need to do it so externally good news is we have an offer code from Linode you can set up a Linode server for this so but phpservermonitor is a really clever tool they've really done a nice job it's got a interface if you haven't looked at it in a while it's been updated since I remember when they changed the interface on it they made it a lot prettier I should say and added a lot more features but it has different ways of sending you notifications several different outputs it's a clever system for being able to monitor your infrastructure set up a you know like I said a Linode instance and make that your device and then you can set up another phpservermonitor at your home to monitor your Linode instance so one monitors the other and you monitor all the different sites and I do recommend throwing something in there that's not part of yours so you can make those determinations and correlation data of what was up or down did the because if you ping something like 9.9.9 and like quad 9 you can have that data and then compare it to your data if both of them go offline at the same time odd coincidence you're probably not in the same network as 9.9.9 but if you monitor multiple things that once you see if they all go down at once or if just one particular thing it's a way to create the correlation data with it but it's easy to set up PHP monitors is a pretty cool software now granted you have to set up the full stack you have to have a web server and I believe it uses a database backend but hey you probably already have that it's one more thing to add into the tool arsenal if not the free service that you get with status cake let you monitor a handful of I think it's up to 10 monitors for free as of right now in March of 2022 Yep but now another tip and I'm just I just remembered this off the top of my head I actually didn't have this in my notes but it's a very unusual tip but it works so well it's so funny and that the tip is going to sound you know not so great at first but it's just you know bear with me on this and the tip is to use Twitter okay what do you mean like you start tweeting about my home network start posting no no no so here's a funny story that happened to me and then I'll tell you how this applies to tech so it was I think 2012 I'm pretty sure and I'm in Michigan for people that don't know and the I was working you know in person back then and the building kind of vibrated a little bit like what's going on and I felt kind of queasy I'm like this is weird a weird feeling so I go on all the local news sites and there's nothing now within seconds though I go on Twitter and I search for Detroit earthquake and within seconds it's just flooded with messages from people that are you know experiencing the same thing so I know it's not just me the point of this is that it keeps happening to where Twitter is tweeting about things before the news sites the tech news sites can even get ahold of it so how does this apply to HomeLab so if you're let's just say Comcast or whatever your connection is goes down one easy way to know if it's you or anywhere everyone in your area is you just go to Twitter and then search for latest tweets Comcast down and if it's a bunch of other people are like a system-wide issue outside your home then you'll see a bunch of people complaining about this you'll know right then and there now just take that a step further you can follow down detector on Twitter as well which is you know going to talk about these exact types of things but it doesn't matter I mean if you're trying to go to your favorite gaming website or your internet provider isn't doesn't seem to be working right you could pretty quickly find out if it's just you or if it's everyone by just going on Twitter and searching latest tweets or looking at something like down detector there's also the website isup.me isup.m is and Michael he's and Edward is up.me and it you basically just type in a URL there and it'll tell you it's just you it's fine or it's not just you other people are having problem that's another way that you could find out pretty quickly as well yeah there's also down for me or everyone or just me there's a few of them out there but the one of the things I've seen let's sort of come back to the Twitter the challenge can be with some of these is obviously Twitter's full of garbage because any term you put easy platform people to post their thoughts they do that those thoughts are not always great and when you focus though on the tech aspect of Twitter is one of the reasons I use it a lot I'm looking for things like is Office 365 down because that's a business tool that we provide to a lot of our clients you know or manage I should say for them and Microsoft allegedly provides it Office 365 is the name almost implies it would be up 365 days a year but is not anyone who uses it we've jokingly we knock it down each time and go to three I think we got a way down on a 352 last year I was going to say like if it's down for 23 hours out of the day technically it was still up that day at one point yeah so nonetheless it's been very helpful to figure out when something goes down it was actually when you use any services you can sometimes see those trending hashtags for that name of the service and go oh okay this service has gone down you know Amazon had a couple major outages not last year but I think the year before where they were relatively broad and you know so did Facebook Facebook became pretty infamous for their outage there and this becomes kind of the question you just want to know is it a problem with my network do I have a DNS problem or is there a problem that's beyond me that I can just tell people or you know and sometimes your support if you're a homelab is the people within your house going hey dad the internet's not working I don't know I'm online well no the thing you're trying to do isn't working it's like when I remember when the one of the Christmas is when I think was at the PlayStation or the Nintendo network that went down for Christmas so I think that's happened a few times on both but then there was also the famous Google outage I can't remember what year this was and and literally everyone's like the internet's down the internet's down no Google's down the internet's fine it's just Google everything else is fine but to them you know it's their start page so their start page doesn't load they don't really know the difference between that and you know a website being down or their connection having a problem so for someone who's not tech savvy that could be a very confusing situation to walk through yeah so having a little bit of extra knowledge it's a place to get the knowledge in Twitter's pretty of the social media companies relatively in not too invasive they don't ask a lot of questions about you to set up an account so just to have an account there just to monitor a few things and follow like down detector it also means when you go to Twitter you won't be distracted by whatever dumb things people are saying on Twitter which I always have to ignore so exactly exactly yep just follow the good stuff they're there's still a use case for it I know it sounds odd like Jay said but he's not wrong so um you know when it comes to your data and what I mean data I'm not talking about your picture collection although yeah that's important but app data right you're running a Linux server or a container and you have an app running on that container or VM and you know if that VM goes down the data goes down everything's in you know self-containing that VM if you can get away from that if your you know infrastructure is good enough to where you have like extra RAM and cores and things splitting out those services is better if you have like a you know true NAS or or Synology or something like that some kind of NAS solution you could do this so the idea is you make the VM or container disposable and technically containers are already disposable so I'm not going to talk so much about that part of it but it's it's essentially an idea like trading a VM like a container so let's just say you have an app and it's running inside your virtual machine and if you if you want to set it up to where your data is not there to where it's disposable just think about this scenario for example your database server could be external the app directory where the app is installed could be an NFS mount the app data could be an NFS mount so for example if it's a web server you could have bar www HTML mounted via NFS to your storage server obviously you have to have a decent network for this because when you start doing this like crazy and you have just one gigabit you can start to really see some problems here but this is like a solution that's not going to be for everyone just like you would at a company you know someone said that in our chat room actually I think it was Luke who mentioned that even if you don't really have much of a use case doing it the right way is great because you get used to that and you do it the right way in the your working environment at work too so think about it like this if your instance gets hosed for whatever reason you just delete it you know bring it up via a backup and if it doesn't have any data because it's using an external database server the app directory and config directories are all mounted then there's nothing that you care to lose on that VM obviously you have to have your NFS server backed up but the idea is is that the VM is disposable at that point you're treating it like a container because it's mounting everything outside and the VM itself you know if it let's just say it gets hosed with malware or something I mean don't even you don't even want to clean it I never recommend cleaning malware because you never know if it can come back just delete the whole instance and recreate it and if your data is not even on the instance anyway then it makes it that much easier because the instance is completely disposable worst case scenario if you have to run like apt upgrade or something that when you bring it back online that's fine but aside from that it's if you can achieve it it's best to treat your VMs as disposable whenever you can and if you have the data mounted on your NFS you could have like version like under version control like the snapshots and such on your NAS solution so if you have a config that goes wonky and you want to go back you can just revert it and right there in your NAS and you're done I brought this up when I was discussing the migration if you wanted to go this way from True NAS core to True NAS scale because people are saying well what happens to these free BST IO cage containers slash jails versus the Docker and Debian mix that's over and True NAS scale and like honestly you know for example use NextCloud NextCloud is a great product you have your data hopefully separated though from the container that just means delete the container delete the jail in your True NAS core no big deal your data should be stored somewhere on one of the data sets with the migration rebuild reattach done it's not that big of a deal provided I mean the nuances are going to be different because you're using IO cage in one and that plugin system in core versus Docker so there's a little bit different interface to get them set up but honestly it shouldn't really matter you'll just reattach your data reattach the data store I know a lot of people will do this they load these different plugins and then store the data within the plugins and that's where you can find yourself a little bit of trouble so you want to make sure you've separated compute and data essentially this is how it's done in the larger enterprises and this makes it a lot easier for you to migrate back and forth between products once you have a good understanding of that yep absolutely so a couple more things so I'm going to go over fairly quickly because I'm coming up on time on my end here so and these are all about you know mainly company defaults but I really think you should get into the habit of this and for example use partitions wisely I think every Linux user has heard if they've been using you know Linux for a week to put their home directory on another partition and that's true because you know that goes without saying if you've been using Linux a week you know because if you've someone's probably mentioned that to you but you could reinstall your Linux distribution as long as you don't format slash home and you have your data that's great but when it comes to servers I feel like you should be doing that as well and I'll even go as far as to say bar log should also be on another partition and the reason for that is especially at companies I've seen this all the time right you have someone who let's just well actually a server let's just say you have 20 gigabytes free but this isn't a storage server that's way more storage and you'll ever need because this server might be serving something kind of you know small and then somebody wants to work on that server so they copy a tar ball to their home directory that takes up the entire space right the whole system comes crashing down it's a lot better when you have these directories that could get full in a different partition because in the server isn't going to get full so for example if you know you have everything in one file system and somebody fills up slash home well that's on the same file system there's no space on the entirety of the server it can't even log anymore at this point and it's probably going to crash it's probably just going to come crashing down but if you know slash home is on a different partition it's not going to fill up the same partition that var log is using and if var log is on a different partition even better because then if in an app you know logs like crazy excessively and this happens where just logs the same message over and over again like 10 times a second and the next thing you know your file system is full that's less likely to happen if it's on another partition you can't always predict which directories are going to get full but I would I would say slash home absolutely is a very common one var log is also common when things get out of control if you have a mail server which I'm not going to talk about that because you know that's not something that I recommend but if you do have a mail server that you maintain you probably don't want those mail files to be on the main file system either because if there's a issue sending and it gets queued up it's just going to fill up the file system so I'll also pair this with LVM logical volume manager I have a whole video on this definitely use that I recommend using LVM on everything always at work at home doesn't matter it gives you the ability to resize file systems on the fly it gives you the ability for snapshots and worst case scenario if you never used LVM it's not going to hurt you to have it implemented but when you do have a use case for LVM or something that LVM helps you with then you you will be thankful that you set up LVM so I do LVM on everything I might not use it that much in practice but whenever I do use it I'm thankful that I I've set that up so I would just get in the habit of LVM all the things and pick up my video about this topic if you want to learn more about what LVM is and why you might want to use it the video just tells you everything you need to know so there's probably nothing here that I could say that's not said there but LVM is great and and set up different partitions yeah definitely handy when you've got to do some of those things do we have time for any more tools Jay or are we at the end of the show here we're pretty much at the end I do have a few more but I will save this because I think this is something that I'd like to do and we were talking before the show started that I think the problem is things become muscle memory that I do on a daily basis I think this is true for others as well so I have to kind of recondition my thought process all these little things I do so quickly we should probably talk about it on the show because it's beneficial for other people so I'm going to pay special attention to things that I do regularly services that I use apps tools and things like that and then come back with an even greater DevRandom episode for next time yeah there's just so many fun little networking tools and things like that and that's one of what we might do is just a whole dedicated one to different tools you can use like IP traffic in some of those how fun we did do that already didn't we I think we did a bunch of fun networking tools but I have to go over the list in that one to see which ones we didn't mention because I know we didn't get to all of them that's that's the thing I think that one might be when we mentioned and I can't remember if we mentioned Elnav a lot so for log navigator but man I love that tool Elnav that's good I've got a whole video on Elnav and I it's still one of my favorite tools for log navigation because it's just it's so handy supports really good searching and grepping functions and it'll do live loading of your logs combine it with Tmux and watch or not you can do it where you can put multiple log logs together at once or open up multiple Tmux screens run Elnav and different Tmux streams to watch two different log streams and then run the commands and see the output like when you're troubleshooting a crazy storage problem I've literally used it for some weird issues before and it's just really helpful because it can do highlighting and stuff like that and by the way logging is something I almost always ask for for those of you that head over to my forums you'll notice there's a lot of replies that sometimes get dead air I ask people for the logs to what led up to their events and they go oh and sometimes there's not a reply because they say oh I looked at the logs and solved it because the error message was literally in the log I'm like yes especially with SSH and check the off log or whatever your distro uses for that or bar log secure something like that Yeah in me and Jay when we were talking about CrowdSec how did I know there was a problem of CrowdSec without talking to CrowdSec and figuring out why it wasn't working it said port in use it was right in the logs easy enough ports in use something must be using that port so Yeah maybe the ports in use I don't know couldn't be that it says that Yeah it says that which is weird because I looked to see if a port was in use it wasn't but then we turned out as me and Jay discovered it was by another service even though it wasn't showing up but the first way I looked at it but either way the log told me the answer this is so much of the troubleshooting and what keeps me from posting in the forums as much so speak and asking a lot of questions that the noobs ask is just learning how to log so it's allows you to answer better ask better questions and also those logs are frequently as people who do help a lot of people in noobs out I'm always I'm never not kind to them I'm always asking for logs so that's spend some time thinking about that when you do a post get in just don't dump the whole logs dump the part you think is relevant do simply google searching too on those log files just a little helpful tips though in general when you're posting things on reddit or any forums not just mine any forums having that little bit of extra context and it just lets you go oh I've seen this before it says you have a bad cipher you know I've seen that with people troubleshooting open VPN for example so that's that yep absolutely so highly recommended you know definitely take advantage of logging it's really great it's there for a reason it's not there to just fill up your hard drive for no reason it has actual value so definitely take a look at it mm-hmm yeah and frequently you can crank the logging level up a little higher sometimes if it's not providing what you're looking for so yes all right all right well awesome and I see someone post a dd do you say a dmessage dmesg oh I see it okay I see it that way too see I take these things all the time I don't know how they're pronounced I don't I don't want to even if it is I don't want to say damesg damesg yes dmesg it's supreme general control definitely solve any problems just because they'll have the outputs on there so yeah awesome all right well thank you all for joining us it was great having you and looking forward to seeing you next week thank you