 Welcome to the Home Lab Show, episode 82, Server Monitoring. How you doing, Jay? I'm doing well, how are you? Great, I checked my server monitoring and everything was up and we still managed to start a few minutes late because I forgot to monitor how much water was in my cup before I started, so. Oh boy. It's always those details, but this is a topic that we were both shocked we haven't covered. Me and Jay were scratching our heads when we said, we've covered server monitoring, certainly we have, and we searched for that term and could not find it, so we said, you know, we should because there's a lot of different fun tools to talk about and that's what we're gonna be doing today. Now, if you're looking for a place to host that server monitoring, our sponsor of the show today is Linode and it's a good place to host some of these monitoring tools for monitoring things externally. Now, this is a self-hosted show, so yes, we'd like to talk about things you self-host, but hey, sometimes you gotta put those things in the cloud and Linode, the sponsor of the show is a great cloud to put them in. So we all know the cloud someone else's computer, so why not make it Linode's computers that you're using for all that stuff. Definitely a great service. We thank them for being a sponsor of the show. We got an offer code of the Home Lab show and it's also linked in the description down below if you'd like to get started with them and let's jump into our monitoring topic. Yep, let's dive right in. Dive right in. What's the first one we're gonna talk about? Because we have a few. Yeah, we have more than a few and some of these are more full-featured than others and I do wanna acknowledge that there's a ginormous number of these out there, so I'm not going to even pretend like I've tried them all. Right. Yeah, so for me, I use Nagios. I know a certain subset of the audience is like, you still use Nagios because there's that mentality that Nagios is old, you gotta move off of it, which I can understand why people feel that way, but at the same time, there's nothing wrong with it other than the fact that it literally looks like a GeoCities website from 1997 and I don't think anybody will argue with me on that when they see the interface of Nagios, but it's the classic, right? I mean, it monitors whatever you wanna monitor for the most part, you can write scripts to monitor the things that it doesn't by default. It's not something I would recommend people start with because I feel like there's a lot better solutions that are easier to get started nowadays, but it's something that I set up like so many years ago and it works, so I just keep maintaining it. But Nagios is fairly standard in the industry. You can self host, obviously. I don't, I think I've only met one person that paid for the Nagios hosted version in my entire career, just that one person. So basically you could download it, install it, there's setup guides out there, but again, I'm not going to pretend like it's the easiest, most straightforward installation you'll ever perform. Yeah, so a couple other things about it. I mean, the default monitors, I mean, Nagios is kind of like the Swiss Army knife. So it'll monitor CPU, swap, SSH processes, whether or not they're running. So basically all of the key things that you might want it to monitor, it basically does that out of the box. And I think the thing I like about it the most, and I also think others do this as well, you could basically write any script and then the exit code is going to then determine whether or not the check was a success or failure. So if you're good at writing scripts, you could basically extend it and push it even further. Yeah, and we're starting out the high end side of it because Nagios, I'll admit, I think there's quite a few network services that it can tie into as well. From what I'm mistaken, even PF Sense has a Nagios plugin. Are you using that or? I am, yep, I actually am. So it's monitoring everything. And I think I probably should have started with what to monitor because I think that was probably a better place to start because I think that's also a place where there's some confusion. For example, if a server is, I don't know, 75% busy most of the time or 60% or whatever, a lot of people might think, well, what's wrong with the server? The CPU is always kind of high. And I think, well, it's doing work. You have a server and you have it to perform some sort of job or some sort of purpose for existing. But then again, when the CPU gets up to 100 and then you have your load average that gets a lot higher, it's time to step in. So monitoring CPU obviously is something to monitor as well. There's also the argument between memory and swap. If you have swap on your systems, then some people will just monitor swap because they figure, well, when memory is empty it starts hitting up swap, but I'm a fan of monitoring memory beforehand. I think there's a certain art to finding out what to monitor, but basically the way I look at it anything that you want to have running should be monitored, whether it's a process, an entire server, something that you want to be informed of. On my end, I'm monitoring the access points, the switches, PF sensors, which is how you reminded me to mention that. And I basically know the internet is down before anyone else because I still have a wired connection between everything in the outside world. So if an access point drops, I already know about it and I get the alert on my phone. So finding out what to monitor is always the first thing, but then Nagios, yeah, just like you mentioned that is going to be more full featured. It just monitors everything, but the kitchen sink and it might be overkill for some people. Yeah, and let's make a couple of things clear. There's monitoring, such as Nagios. There's monitoring and metrics, which is something Nagios will do as well. So there's some of the tools we're going to talk about are just monitoring like status is it up versus diving a little deeper. And then there's monitoring metrics and maybe an action that needs to occur and an action can be more than just a notice that something's down. So Nagios goes into the advanced. So we're starting on the advanced side of it because she has some cool scripts that will do the ability to restart a service, stop something, make an action happen based on a Nagios result. So I think that's an important distinction that Nagios has. There's one of the reasons it's more complicated to set up is because it has a lot more features. I think the same for ZabX, which is on the other side of it. It's a little further down the alphabet, but it's one of them that I've been seeing for a little while. I think ZabX is great, open source, fully functional. I've done a video on it, but unfortunately, I haven't had time to do a new video. So my video has a more dated looking dashboard. They have updated the dashboard. ZabX is one of those, it's kind of easy to get started but it's gonna take you a whole lot longer to get good at type of thing. And ZabX is nice because I generally, we're talking a lot about Linux servers on here. ZabX goes much deeper. ZabX will monitor, do the metrics, do the actions, even has an entire troubleshooting system inside of it where you can say, hey, who resolved this like as a ticket and have an action on there and monitors Windows servers as well. So you can integrate this. This is all agent based. Now, you can use some SNMP features with ZabX, but they're not gonna be as rich as if you did it with the ZabX agent. That being said, they do have ZabX agents for a lot of different platforms, not just PF sense, but many of them. And as I stated with Windows, you can even load ZabX right into, they have a normally compiled binary for Windows. There's no special weird way you gotta set it up and start gathering metrics on Windows servers. It's really slick because over time and having used ZabX for years, it's kinda cool to see just the history of all the data and transfers and bandwidth that I'm getting like out of PF sense. There's also a large, rich community for ZabX of templates for all kinds of different things, which is important that way if you go, hey, I have this Cisco switch here. I wonder if ZabX has a template. Well, for a lot of the big brand name equipment, it does. ZabX also scales very well. There's a video you can find of a large IT provider that manages, I wanna say they were in the 100,000 roughly devices managed in the ZabX instance. It has the ability to load balance and cluster. So it's got some abilities to scale, but it can be done. It can scale to some pretty impressive numbers to monitor large amounts of infrastructure. So it's definitely on the higher side of it, but definitely a really cool tool just like Nagios. I would say nine ZabX compete with each other in a way. They're in that same similar category. Yeah, I've used both. So I've always used Nagios in my home lab, but I worked at a job where ZabX was used quite heavily, actually a couple of jobs now. And I have to say overall, I think I might like ZabX better. On my end, I mean, I'm so into Nagios to just be time consuming to replace it and it works just fine, so I replace it. But I'm not sure how far you can extend ZabX. Nagios I feel like is gonna be on a higher level of complexity, which is basically what we just said. I'll give you an example. So at one job I worked at where we used Nagios, I actually worked with another person. We created a Python script with an API call. It was able to log into the company's AWS account through an inventory of what instances were there, apply a host template to each and every single one and dynamically like Nagios would grow and expand to match whatever we had, which is not how it works. Like everyone that uses Nagios, they know that it's a little tedious. You have to have a host config file, a text file per server. And it's not hard because you can just copy an existing one and just change the values. It's pretty easy to do, but it's still a little a lot more complex, but the fact that I was able to do that, I'm sure ZabX probably has plugins for AWS that makes it a lot easier than it is in Nagios. So I think when you compare the two together, there's no right or wrong answer. Whatever one has the feature set that you're looking for is probably the best one to go with. Yeah, both of them are good. And you kind of mentioned, I think he's got a couple of write-ups on it. Hostify, my friend Riley, who owns that company, he built his infrastructure out to manage his thousands of servers and create a self-healing system because he hosts unified controllers. And anytime there's a problem with any of the Mongo instances because they use MongoDB in the backend, he's actually got ZabX monitoring it, letting them know what's going on and restarting or stopping services or making modifications kind of as needed. So there are some great examples out there that people have for good use cases for scaled installs of ZabX. So before we go any further, I wanna talk about a side subject that's directly related. And someone brought this up, Learn Blue Iris brought it up in our chat room asking if I use SMS. And the answer is no. I have never used SMS. I mean, I think I've used it for a week. I did configure it and set it up just to see how you do that. But I just got to a point where I didn't like that for whatever reason, I don't remember why. But you do kind of need a way of getting alerts to whatever device you wanna receive those alerts on because the whole point of a monitoring system is to let you know when there's a problem. But how does it let you know? How do you get that notification? And early on, I had set up email alerts via Nagios, I had an email server, bad idea, I'm not doing that again. But it would allow me to get the alerts via email, but then the problem becomes emails kind of get lost in all the other emails in my inbox that I have yet to answer, which is not a good place for your alerts to be in my opinion. So I use a system or a service called Pushover. It's a paid service. So I know it's not gonna be like the top of anyone's list, so to speak. But it's effective. It's an app on your phone. You have a kind of like a hash key or a API key almost. And you just basically have all of your services reach out to it, whether it's Nagios, Ansible in my case, they all go to this app. And you could snooze if you don't care for an hour, like you're just trying to have dinner and you want the alerts to leave you alone for an hour or something like that. But it's very easy to set up and very flexible. You don't have to maintain an email server. In that case, you don't have to buy like SMTP access from a third party. It's still paid, just getting that out there. I forgot how much it was, it's not much money. But if you have no other solution, Pushover is something they consider. But then again, if there's a self-hosted option that does the same thing, then that's probably also worth considering. But you need a mechanism through which to get those alerts when something goes wrong. Yeah, and I'm partial to, because we use it at work and a lot of times the reality of using these is pushing them into Slack messages. But a combination of Slack or email is usually adequate for what I want to do. But if you know, if HomeLab people, you're not gonna find anything for free, just hone it out there. SMS is gonna be a popular option because a lot of people want to push right to their phone because they have an overwhelming number of Slack messages and emails. So they're like, that particular pipeline is full of noise, please send it to this lesser used pipeline. But I don't know if it's lesser used because I get it, well, Google has a good job of filtering it, but I still get a decent amount of text spam, so. Yeah, it depends on the Slack account because a lot of them that I've been on and I kind of avoid Slack for this reason because it's just like notification overkill because you can mute the notifications in different rooms that aren't something you're really actively following, but then anytime anyone mentions at here, you're gonna get the alert. And then just to find out it's a bunch of memes or something. So I really like Push Over being completely separate from anything else. It's a dedicated app just for alerts. It doesn't do your taxes. It's not gonna help with your spreadsheet skills. It's literally just that one thing. It does that one thing only. So it's almost like the Unix philosophy, do one thing and do one thing well. So you can have this dedicated app that is just there for that reason. So when you see an alert from that app, there's nothing else it can be other than an alert. That's the only thing that it does. You don't have to separate the politician spam during an election that goes into your SMS messages just for example. And we all get a lot of those and I'm glad that's over. But it's good to have it in a dedicated app in my opinion when you can. Yeah, I see people mentioning Discord and Telegram. I mean, the downside can be with any of these Slack being that it's more business oriented. I'm gonna say there's extremely few Slack outages. Discord, oddly, I don't use it for much but I don't recall a lot of Discord outages. I use it to talk to my son and a few of my YouTube friends but I can't think of too many outages with Discord. Telegram, it's very third party. I remember being a huge fan of Telegram but I know a lot of people seem to like it so it comes down to what you wanna use but it's food for thought to think that now that the more services you have to rely on or the more complicated those external services are could be a pipelining problem to get the data to you. Just something to consider on there. There's not an easy answer for that one. It's really where is your attention going to be drawn to is where you want that to be notified. So what works for you is what that comes down to in your workflow. Another tool that I wanna bring up and someone mentioned this in the chat room and that person reminded me I can't find the message now but smoke ping is something that I also use in addition to Nagios which is going to measure network latency as long as you make it a point to always add new network host to your smoke ping configuration. I mean, if you keep forgetting to add things it's not gonna be accurate but if anytime you have some latency in your network you could pretty much track it down right to the device. Now granted, I could do that in PF sense too with some of the meters in there but it's not at the level that smoke ping is in. Now honestly, I really don't like the UI theme in smoke ping because I think the graphs are a little hard to read. I mean, it's not that hard. It's just they don't look all that great and there's not a lot of themes out there that are available for putting a UI theme or a CSS theme around it. But if you just let that slide it basically just does that one thing. Again, if you just wanna see network latency if you get a message from your learning system that something is taking too long to load, for example which is another thing you can monitor then you can go into smoke ping and find out is it like the network VLAN that device is on is being saturated by something is that device itself the only thing and you can kind of narrow it down that way. So sometimes having a mix of tools is a really good idea because it can help you hone in on things that your monitoring solution itself may not do like the best job. Like Nagios has meters and graphs too but it's still not gonna be on the same level that smoke ping might show you the details. Yeah, yep. The next one, should we talk about uptime robot? Yeah, so we talked about this off camera a bit because I was kind of in a hurry and I hate it when this happens when I make decisions in a hurry because you know, content schedules and things like that. So I was using status cake and I thought it was really expensive but before I get into that, what's the point here? So there's a term where you have to watch the watcher. You know, if you're monitoring if you don't get an alert on your phone or your Slack channel, whatever happens to be in a long time, you're probably thinking, man, my homelab is awesome. There's not a single server that's down. Everything is up, everything is great, which might be true, but it also might not be if you're not paying attention to whether or not your monitoring server went down if that goes down, then yeah, you're gonna think everything is fine when it's not. So I like to have something on the outside that does simple uptime monitoring, basically is your website online, is learnlinux.tv online, that kind of thing is the homelab.show online. And yeah, I'm monitoring all of that to Nagios too, but what if Nagios goes down? I want something else to kind of watch it for me. And at first I was using status cake just out of habit. I used to use that at work. It's super expensive. I'm not gonna recommend that to anyone because that's kind of why I switched away from it because I had more things to monitor and then to move up a level in the paid plan, I just felt like the price was like astronomical for such a small operation. I'm not like Microsoft, I don't have like a zillion servers. I just have the servers I need, a dozen or so. But to move up to that level is a lot more money. So I looked up an alternative and uptime robot was highly rated and I do have a paid plan. It is cheaper than status cake. I don't remember by how much, but I did save some money by switching to it, but then when we talked last night, I learned that there's uptime kuma. That's a thing that exists apparently that I did not know about when I paid for my year plan for uptime robot. So I do like uptime robot and it does what it says it does. I'll just keep using it, but when the subscription is over in July, then maybe I'll look into uptime kuma. Yeah, now uptime kuma, I just had a video I posted and I posted this on Twitter as well, talking about using uptime kuma and it doesn't have a really high system requirement. I took an old Raspberry Pi 3, 3D printed a case do just a short video up on YouTube and asking if people wanted a more in depth video, but uptime kuma is wonderful. And the inspiration for uptime kuma and why it looks so much like uptime robot, which a lot of people are familiar with is literally the developer said, hey, I want to try to make something that looks like uptime robot, but free. Now the integrations it has are interesting because it doesn't just do ping monitoring, it can do monitoring via TCP ports, look for specific statuses. So you can actually say, hey, I want to make sure that this particular status exists when I pull this HTTP or HTTP ads or any particular TCP ports. There's a lot of fine tuning to monitor things more than just ping. They can give you the uptime status of those things. It can create public facing dashboards so people can just go jump in and look right at the dashboard of the thing they want to see. And for a demo, I set up all the different internal production servers and our lab servers all in a one single interface so the staff internally can just see the entire list. Now, the way they notify you in it is fascinating to me because I can't believe how long the list is. The things we mentioned like SMS and Telegram, they also have signal, they also have Slack, they also have just a really wide range of things. They also have Cloudflare tunneling built in. So if you want this to be public facing, you can actually just throw your Cloudflare tunnel token in there and I did a video the other day, you can find on my channel about how Cloudflare tunnels work and the integrations for uptime Kuma is just like wild. Like how much you can do with it is pretty interesting. Yeah, I'm hearing a lot, I'm seeing a lot of texts in our chat room here about uptime Kuma, so it seems to be well loved. So I really like seeing that it's kind of like, there's no question of what I'm going to try. If that many people in our audience are all about it, then there must be something to it. So I definitely plan on checking that out, I just don't know when, but it sounds great from what I'm hearing. Right, so I definitely, that's one of the easy ones to set up. Now, this is not metrics, it may have some stats, like, hey, it pings that's many milliseconds, et cetera, but it's really not a metrics gathering system. It's not going to be as extensive what we mentioned with Nagios, it's not going to be as extensive as Xabix in terms of having process for usage because there's no agent you're installing on here, you're just taking it from its position wherever you put it inside, outside your network, you're giving it a list of things to talk to and give you the status of whether or not those things are talking and send you a notice is an option if that thing is down. So it's really, really simplistic. It makes it one of the easier things to get going. It's, as I said though, not where you're going to get that in-depth data. You're not going to go, hey, the server's still pinging. Yeah, but the load average on it's so high, it's not useful. So up in the working are not necessarily the same things. That's why the more complicated things for metrics management come in. Right, another one quick one I want to bring up, and I brought this up like probably 15,000 times throughout the series so far, but I think it's fitting. So I'll just sneak in one more, healthchecks.io. And this is very, very specific. And it's not necessarily monitoring, it kind of is. So if you have a cron job or multiple cron jobs, how do you know that those ran? You could absolutely set up a script that's going to email you or hook into whatever your system to let you know. But healthchecks.io is really cool because it gives you like a hash key that you add to the end of your cron job. Well, you could add it at the beginning if you want, but that'd be silly because you want this to run at the end of the job. And what it does is it resets a counter. So if you set, I don't know, let's just say seven days, you have to get a ping to this URL within seven days. And if that doesn't happen, it'll email you. So at the end of the script, you basically just have it reach out to that URL and as the last step, and it'll reset that counter. But if the counter doesn't reset, that means that the cron job did not run. So in that case, you need to go in there and look at that. So if you have cron jobs that you need to make sure have run, then there's a service for that. Healthchecks.io is fantastic. And there's probably other use cases for it as well. I personally use it to make sure that my Ansible jobs run because I use Ansible pull, which is kind of like the inverse of the Ansible model where it runs local host on the server rather than the server reaching out to other servers. But the consequence of that is, did it run? It's running by cron, I don't know. So healthchecks.io added that to my provision script and it needs to run within seven days. If it doesn't, it gives me an alert. Yeah, that's a really easy service to use. Now, we wanna probably address the elephant in a room here with a complicated elephant in a room because me and Jay talked about this last night during our conversation. It's using the whole Prometheus, Grafani and Loki. TechnoChimp, he's an easy example of someone who really likes that. And we have another friend, Phil. He's not a YouTuber, but he also really likes it. The two things those people have in common, they do very complicated developer things. And it seems like people who do complicated developer things really love all the cool stuff you can do with Grafana. I have not played with Grafana in a while and Jay has had some business experience with it as well. The problem I've always run into is the layers of complexity. Don't get me wrong. Once you get it set up, it's beautiful, it looks pretty. I'll give it to them in terms of their ability to collect all the metrics and display them in a great way. Awesome. The downside is when someone's like, oh, it's easy. And then I watch them just spill script out of how they wrote it. And I'm like, oh, it's not really that I know of and correct me if I'm misinformed on a lot of this. But last time I looked at it, it was a little bit harder to set up than any of the ones we mentioned. And I've set up Xabix, he set up Nagios, we still found Grafana still a bit more complex to kind of get tuned and going and doing. So I think that was the challenges we've seen with them is this, and I imagine it's become more popular to documentations getting better, but I didn't find the best documentation when I looked. Grant, it's probably been about a year or two. I think this is an example of the homelab community actually improving the environment for everybody. Like we're actually heroes at this because when I first set up Grafana, it was actually me and an employee of mine when I was back when I was working at a place other than my main company here, when I went solo. And we found that there was just no documentation at the time and my colleague literally engaged with the person that was in the process of writing a book just to get the process or project done faster by collaborating because there just wasn't any information. But nowadays I feel like with the homelab movement, we're creating documentation, we're making videos and there's probably a lot more out there right now about it. But what I would say though is I feel like Grafana is a better fit for an MSP that is hosting clients and CTOs from those businesses that are being hosted need the graphs to show their upper management that everything is running. And I know I'm kind of oversimplifying this and I'm not saying homelab people don't have a use for this. We absolutely do, there's a presence. But I feel like there's another solution which I think you're probably gonna mention after this that I think is a better fit for homelab people because again, although Grafana is absolutely a great choice and very effective for homelab people, I do think it's overkill and I do think it's a better solution for MSPs and the developers and DevOps people within that that are trying to show CTOs that their server is running fine. Now, one of the things that I'll mention for swinging it over to the opposite of Grafana in terms of simplicity, but still has really, really pretty metric presentation for monitoring and that's net data. Net data, I've loved this project for a long time and it just keeps getting better. Every time I look at it and I've talked to the developers there, they're just doing cool things and the latest iterations of it with the way they do the consolidated dashboard, the way they do the installer, they've covered everything from Docker to managing ZFS, managing Redis, managing MongoDB. It gives you metrics on everything it finds and as far as setup goes, it's extremely automated. All the different what are called plugins, I guess if you will that you run on it, you don't have to do any configuring of them. So that's why I say kind of call plugins. When you set up net data, it figures out all the things that it has access to on that particular system you set it up on and just automatically loads all those different modules, gives you really, really good insights and the insights are really kind of interesting because when you start taking all of your servers and grouping them together and then tie them to like the net data dashboard, you can actually grab a time slice of a specific time when something happened and then pivot that across all the different servers and their kickstart install script, just like a one liner. The only prerequisite and this is when you learn when you do YouTube demo videos because usually you end up with hosts with the same name, turns out if you have the hosting and clone a host and you don't give it a new host name when you clone it, net data won't install and you'll scratch your head for a minute but now they have a fail error that you can find because it says, hey, this ID is not unique. So outside of that, not a problem anyone's likely to run into but it's not a big deal. You just change the host name of that system to add a one after it, simple things like that. And then you can get in there but I can't believe how easy they've done it. And that's one of those things when I look at any project, if the project is so easy to set up, so easy to get running, that tells me how good the team on the back end is. If you have a good team developing a product, it's gonna be easy to set up, easy to use, easy to update and easy to get all of your data out of. That is definitely a definition that the net data people have done. It's one of the reasons I like it. I do enough complicated things. I don't need to spend and don't get me wrong. I've watched some videos on Grafana and said that looks really cool but I looked at net data and go, this also looks really cool and it was a one liner to get it going. Yeah, that's something I plan on trying soon when I get a chance. It definitely sounds pretty good. I have seen it in action so I do understand it. It does look pretty cool. Yeah, the team keeps adding more and more things to it and it's fully open source. Now, granted they do have a business model behind it they're doing so they will have an alerting that goes up to their dashboard. Their dashboard is free but there are enhancements you can do the dashboard for long-term data storage, et cetera that are upsells that there's options for. So there's a business model behind it but you don't have to use their dashboard it's not required to get it going. You can monitor individual servers, you can just load it. It, I think it's port 19999, don't quote me on that but it's in a documentation. You can just go to that port on any server on your network that you've loaded it on. It does not have to go out to anything to do any of the monitoring. It can all be done and you can then start looking at it from that internal point of view. It's just kind of cool when you throw it on there for the what do you call that for the dashboard but even internally it does offer one of the new things they added this year is what prompted me to do a new video on it was anomaly detection. I'm not good at tuning that yet because they're still building the documentation for it but it looks for your baseline average of that server over time and says, hey, the server did a thing and then has some monitoring slash notices it can give you so you can focus on any anomalies and then trying to figure out what happened when that anomaly occurred. So it's just kind of a neat way to unravel all those resources in how they're being used on your servers. Yeah, that sounds like a great way to do it. All right, what was the next one that we have? I think that was the main one we had on the list here. I see a few people ask about Libre NMS and Observium. I have not really used those ones. Libre M&S, much like Nagios, unless they've updated something recently, always had an old and ugly kind of looking interface and things that aren't agent-based that try to reach out and monitor servers are never gonna be as good as something like Xavix or Nagios that has an agent that can actually gather a lot more of the data. I mean, if you like them and they work for your use case and I don't see any reason not to use them, they just wanna fit when I looked at things for the use cases we were doing for monitoring things. Yeah, another person asked about UPS monitoring. Nut is a popular solution for that. And actually, I'm kind of boring in this regard because I don't use anything. I actually have the plug-in for PF Sense, the APC plug-in that it comes with or is available out of the box. You can install the plug-in and it just alerts my phone if the power is out and since I have a VPN client installed on my laptop wherever I may be and also on my phone, then I don't really care so much about Nut. I know that's the better way to do it. You absolutely should do it that way and I'm not making a case for over simplifying things here but my mindset was if I get an alert on my phone that my power's out, I always have a VPN client that's within reach. I can just log in and shut things down myself. Granted, that's a lot of work because there's a lot of things to shut down but thankfully power outages here are extremely rare which is the polar opposite of how it used to be where I would have several power outages a month but here I think it's happened twice since I moved out this way. So I probably should set up Nut on my end. I just haven't had a chance to do it but then the counter argument is how infrequent it is in the fact that I can just SSH in and shut things down but if you have the time to set up something like Nut, I've heard some great things about it and it comes highly recommended by quite a few people. Yeah, and Techno Tim has a deep dive video on that so if you type in Techno Tim Nut, oddly that was gonna give you the video you want. And he's got like an hour long video on all the details that's setting up on Nut. I wish I had a better name but it doesn't so. Sometimes you feel like a nut, sometimes you don't. Yep, sometimes you feel like a nut, sometimes you don't. But yeah, it is the de facto standard. You'll actually find, there's actually a few interesting things you can do with the one built in the PF Sense as well. You can actually plug it in to some of the USBs and have, if I'm not mistaken, PF Sense has the ability to act as a nut server to send the shutdown on there but anything that you'll find when it comes to UPS, I don't even think outside of vendor proprietary software, Nut is the only one I'm aware of that has the broad range of support of so many different UPS vendors and you can use Nut as a server to then send out the signal to the other servers to tell them to do actions. So it's definitely, it's really extensive. It's a great tool. It's pretty much though the only tool. So it's great because I don't have a better answer because I don't think there's another one out there. Yeah, no, I mean, there's the APC specific tool and I think other vendors have a specific tool for their devices. I think it's better to have something like Nut anyway because what if you use like me APC but then later on you switch to something else then you'd probably have to switch your solution over as well and reconfigure all your servers and things but having something, I don't know much about Nut but I'm assuming based on what you said you could just continue using that and not have to change as much as you'd have to change if you went from one vendor to another. Right, that's very true. You then you would just change a Nut server and the thing it does to talk to all the other servers can remain the same. So you can swap out vendors without too much pain. So that's probably a good point of having an advantage of running it. I believe Tim's got one about running it on a Pi because you can then make the Pi the Nut server. So it's because Pi's by the way putting a UPS on a Pi actually pretty easy. You just take a U.S. because they're powered over USB you just go grab one of those battery banks you can have a Pi last for days. I really can too. You have to watch those because a lot of those don't have the correct voltage and the correct amps when it comes to the output. But as long as you have that taken care of I totally agree. I would go a step further and say if you have access to a Raspberry Pi I'm too scared to look at the prices and availability right now. Oh, that's true too. But if you do have access to one I like to run a utility server which is this is how I do it. My utility server has Nagios on there smoke peeing it used to run my Unify controller not anymore though. That's a little complicated to get Unify working on a Raspberry Pi. You absolutely can do it. There's update issues I won't get into but for the most part the idea is you're monitoring softwares and solutions and things you can have on this Pi you can image the SD card really easily. So if something goes wrong you could just re-image it and everything's back to normal. And that's a good way to do it as well. I still recommend something watching the watcher but I think there's something to be said of having a utility server and then at that point running something like Nut on that server would actually be a good fit. And you can just have that be the thing that monitors all the things. Absolutely. Well, I think we've hopefully educated people on some of the options out there maybe narrowed it down for some or maybe expanded the options for others and gave you some more choices and more projects to play with. There's bigger which end of the spectrum I want to start with I won't lie between NetDate and up Tankumo the ones you mentioned those two are just going to be the dead simple easy to get going and set up ones. So go ahead and check those ones out and if you are feeling a little bit confident after getting those up and running dive into Xavix or Nagios or any one of the other ones you talk about or even the Grafana and fun that is with Grafana and setting the complexities of that up. So. All right. All right. I'll see you guys next week. Thank you. All right. See you guys later.