 Welcome to the homelab show episode 106. It's time to audit your homelab. Whether you know it or not You should definitely audit it, right Jay? You absolutely should. I think I'm one of the few people that do this regularly Probably because I I should probably have more hobbies in my life, but that's a total different podcast We should probably stick to the auditing. Yeah. Well, we you know because this is our hobby We do have this as a podcast because we're a little bit You know the reality is no one wants a backup that works They want to restore that works and I've said this a lot, you know untested backups are just wishful thinking so Stop thinking wishfully and actually test your backups and all that fun stuff And that's what the topic of today is well not just testing backups it's about how to audit and validate and Unfortunately, a lot of people don't realize they've lost data or are about to lose data until a system fails and you go Oh, I guess my data that I thought was being backed up or I updated a doctor container Assumed my data was mounted properly separately no longer is it's a recurring theme So we kind of walked through some of the processes of how we look at that I figured you know me and Jay talk about this a lot amongst ourselves because we we're double-checking ourselves to go Hey, you know, what are what are other ways? I should have some perspective on this and I'm like wait a minute This is a great idea for an episode It really is and I'm kind of curious about this I'll ask a question to our audience Am I the only one that just regularly, you know regularly audits the homelab for no apparent reason because if you think about it You have enterprise companies that you really can't get them to do a good job of this And then they they're in the news because they lose data or something That's a big company with an IT team and they're still not doing a good job of it But sometimes I wonder if homelabbers are so in tune with their technology because like you said it's a hobby, right? So when it's a hobby, we're gonna pay special attention to it So it's kind of like it's not as tedious as it might be for someone else who's not a you know That works in the industry. They're not a huge fan of it, but they work in it via, you know us I mean, did does anyone in the audience just every couple of weeks or couple of months just randomly start auditing things I mean, that's what I do, but I'm just curious if I'm the only one. I don't think so though Guess we'll find out. Yeah, absolutely I'm finding out that I need to turn off, you know, you get more things that beep and then I have to turn off more things that beep Yeah, the beeping things are driving me crazy like I have to probably buy a new fridge because that things beeping because the door won't Shut properly and then this thing beeping then in here. I have this and that and the other thing Yeah, no reason audio is just tough. Well, this one was my fault. It's my watch that was beeping I forgot to turn off notices on the watch So I've been watching my health and that also has led to hey, it'll send me notices and then sometimes you Don't want them nonetheless Let's segue into the sponsor today show and that is Lennon We have renewed our sponsorship with them where they will continue supporting the show and we think it's a great place to run the cloud Matter of fact, I just did a video on hosting your own unified controller I used a Lennon instance for it They have just a really slick and easy interface for getting things spun up And it seems like a pretty good thing to host in the cloud is your unified controller instance So check out that video if you're interested in setting up a unified controller and check out Lennon If you're looking for a good place in the cloud to host that and many of the other projects that we talked about on this channel Thank you for being a sponsor the show and there's an offer and a link down below to get you a deal and get started with the Node. All right, so let's get started with auditing Yeah, let's get started with this. So I you know every now and then I feel like there's a topic with my home lab That's just staring me right in the face and I'm just blind and I can't see it and then you know Sometimes I see it and there's a million of them But the auditing thing came up for me For one based on your comment and I'll let you say it again I forgot the situation but something about the data. I thought the data was there in the container What was that again? Because that's kind of one of the things that it helped inspire this idea So this is a you know true nests has Unfortunately been the subject way too frequently of a problem says they moved over to true nests scale and they're using well a presentation of kubernetes and Docker all combined together and then you have the waters being muddied by places like true charts And I say muddied because they prefer some different types of storage and it's not as clear where that data is Which has led to because we often consulting not just me seeing posts in the forums But an unfortunate number of people reaching out because they have updated an app or broke something and lost the data It wasn't putting the data where it bought it was and this is one of the first things We really want to just consider especially with hey docker's awesome It's a magic incantation to some people that will simply spin up an instant loaded application without all that You know pesky work of setting up and dependencies and configuration while that's good I always am a little hesitant to people who don't take the time to understand Exactly where their storage and more specifically where their data is living and this is why before now I admit like I've used that type of storage inside of true nests where I don't care because I'm spinning up an app to go Does this app work? Cool? It works. Okay. Can I spin it up again? Delete app? Where'd the data go? Oh, hey, I should probably go through and set up these hosts pass properly and validate how you attach to a matter of fact I even filed a bug report because their first version of next cloud was very buggy And even though you specified where the data was going they had a variable mismatch Your specified location didn't actually put the data exactly where you thought it was which led to of course people when they would load updates Losing data and that is obviously something you really want to avoid So this is why this topic came up and kind of auditing that and yeah, it's kind of where this topic led We and Jay had a pretty long discussion about your nests last night. Yeah, we did and I think without going on a tangent I'll just say based on what you've told me and you know disclaimer. I have Very little experience with true nests scale. I use true nests core currently and I'm just kind of waiting for the Interesting conversations to die down before I go to true nests scale But but as you described to me, you know the kubernetes implementation I keep thinking what or what are they doing? Like I don't know if it's just maybe something I'm missing and it's fine But based on what you describe it's just always this weird Topic but then you know, of course we get into the my My data was there and now I'm getting like this Screen that's like the first screen you get when something isn't is not set up You know name your app or put the settings in but I'd set all that where to go And that that kind of led to this but also in parallel I I just habitually audit my network and I think this might be a homelab thing like I just check things and sometimes it's a matter of I want to Consolidate code. Maybe I found a way to Do something in ansible With fewer lines of code and this is kind of how it is when you learn So you go back to your older stuff and then you could shrink things down because you found a more efficient way to do it And that's not really auditing. It's kind of code auditing But it's consolidating code is fine, but that leads to things you're like, yeah, but this implementation I haven't checked on that in a while. I haven't checked on this in a while And I keep and then eventually became a habit. I just like usually every other weekend. I'm just checking the backups and The backup chain as we're going into and then the most interesting thing and the most frustrating thing for me Is that question? How did this ever work in the first place? Like you have something that You you implemented and you're like you're set you sit back and like in awe of how great this is Like this this solution this application that you're running whatever it happens to be and it's working It's fine. You're like, wow, I created this thing and it's just working fine I'm I have my data my data in there and then Eventually it stops working and then you find the problem And then you're like, well, how did it ever work because it can't work with this setting But it was working but now it's it you never know right and I keep running into weird things like that and I feel like it reinforces the auditing persona because you know, you keep finding things that you're wondering how it worked in the first place and Things you could do better next thing, you know, you keep making your home lab more and more and more efficient But I'm not sure that everyone does this because you know, we get comments about You know losing data and things like that. So with attention to detail, it's harder to lose data Not impossible. So it kind of makes me think that maybe I Could be one of the few people that do this regularly Maybe most most people find this tedious and they want to avoid it, which is fine It's not the most fun thing I've ever done. But now we have the topic Yeah Now one of the first things I want to mention and this actually isn't on Jay's list So I'll bring it up before we jump down our punch list that we have here is When you're thinking about this, this is a good what it referred to as a tabletop exercise Just think about bare metal restores For example, you may have a password manager. It can be an external one like bitward That makes it kind of easy But it also could be an internal one that you're using key pass is really popular in a home lab And it's a great system But what are you doing to make sure that your key pass database is where you need it to be? Because what if you lost your primary nas and you're like, oh no problem. I backed it up I encrypted it before backup So I have it securely stored in an encrypted form in the cloud and like great How do you get into that cloud service? How do you? Get that encryption key you need for it This is something you should always walk through is what does a bare metal restore look like step through it? And you don't have to do this. It's more fun Of course, if you do rebuild it on some other piece of technology to see if you can get things up But this is where especially from businesses. This is a problem I've run into a lot when we take in over it Especially when they had kind of their friend helping the it We had a client who stored all of the API keys they needed and all of the encryption keys on the server that they were backing up Like oh, it's all encrypted at rest in the cloud fully compliant and when the server had failed This is how we met the client. Um, they're like, yeah, we have the encryption in the cloud But no one backed up those API keys and no one knows the really long string That was made that was saved in a notepad On the desktop of that system and it was this fire in the building So things were melted in a way that you couldn't do data recovery And we could not recover that data and it came down to there all their backups are fine Matter of fact, all the logs from the email said, oh, yeah, and we could tell there was all the data was in the cloud Without that one key and no one ever thought to have that key anywhere else But the desktop of that computer they was like, yeah, right and they had done reshorter tests They actually had a pretty what they thought was a thorough process that all came down to Where are those critical security keys and you have to think about that from a very big picture Sit down and draw this out on paper step one. Where are my security keys? What is the risk of those because it's awesome that you're encrypting everything? I think it's a great idea, but it's with risk and that risk is making sure that you've mitigated the risk by Do you have it? I'll also kind of reference and I didn't read the whole article because I was aggravated I was wondering if you're referencing what I think you're referencing but continue I I seen this article come by last night when I'm flipping through things So I watched a lot of some of the video stuff and I believe it was probably vice news one of the big news Organization vice or vox beans with v to my youtube subscription somewhere They were complaining about having lost data on one of the sandis Because it failed, but I'm also like you shouldn't just have a single piece of media That has all the critical data on there even myself I do have backups on encrypted drives of my ssh keys and it's in a plural because my ssh keys are my key to getting back into many systems I also have an offline copy of all my passwords I export out a bit warden and I keep them on multiple keys and I audit those keys and every now and then like on a schedule that's set in the calendar So I I say every now and then there's a timing. I do this on the first of each month I run through the process and I'd read download and re export them and are multiple keys And I re verify that these keys are readable So I've tried to mitigate it in every possible way I can So I know if you were to somehow magically wipe out my computer systems I have access to and there's more than one of them and you wiped out my ssh keys As well, I have a methodology by which I can get back And this is the part that people you know, you really got to think about that bare metal nuclear Thing that happened and be able to table top your way back into okay How would I get into it? What are the things that are going to take it out? This is something we help with business continuity planning with the business world But it applies directly to the homelab world And it comes back to you're hopefully taking that knowledge if you work in the commercial space and applying that knowledge all the way up It's funny that so so you weren't referencing what I thought you were but it's interesting that you're talking about or that you've mentioned Um, you know your the keys in the cloud When my you know azure has a big issue with keys right now And it's actually part of the topic of the next podcast I'm going to be on right after this one, but then also it's interesting and this is kind of funny So I'll just mention this real quick. Um, you said something about a Sand disc and I thought you said sand disc, you know the brand of Sand disc is the and I was like I'm today years old when I realized the brand sand disc could be read as storage area network disc I hope I'm not the only one that never put that together Until literally just this episode when he said that a few minutes ago I thought that'd be amusing to bring up. Um for those watching and I'll describe it so because those watching I'm holding in my hand a Actual ruggedized sand disc that I use these are pretty nice I have a one terabyte one, but I don't trust it by itself It's always an extra copy of things not the singular and that's where I thought that vice vox news because they were talking about They lost a bunch of footage they were storing on this and I'm like Yeah, this is a challenge when you shoot critical footage for whatever and for me and j We can always rerecord something in the lab But that gets exponentially more expensive if you're on site doing something and you did an interview with someone You would let on to lose that data and I see a lot of photographers and videographers. Oh, yeah We love these rugged sand discs. I don't think they're badners a few competitors as well But it's a copy. I always like to see On site. We've actually designed this. We helped the company design an rv full of storage servers So Everything goes on arrays because they've lost individual discs So they record on the cameras and bring it right to the arrays that are available in the rv And um, that way they they don't like because they're on site You're like, yeah, we we filmed something at the bonneville salt full flats We copied the data it instantly went to a raid array with redundancy And then it drives away in an rv to go to the back to the production house. Pretty cool But that redundancy is what you want to be thinking about Exactly. So so one of the first things I'll mention is to Understand when you're auditing your home lab and you don't even realize it because most of the time it's a conscious effort I'll use a store a fun story from you know back when I used to You know work for someone else that is kind of timely and I'll tell you how it ties into this I had a situation where you know, it came up that a need Um, we were I don't know. I don't know if it's an audit thing But either way the requirement was that we're auditing backups quarterly, which is reasonable. So Um, you know, one of my employees came to me like how do we set this up? How are we gonna do this? Like we like what's the best way to? Um, you know to regularly audit the backups the the images of the customers virtual machines And I'm like, well, there's you you've you already do that regularly and he's like What no, I don't like I do the the client has a contract where Um, they also get a quarterly upgrade of the piece of software that we're hosting for them So when when that time comes, what do we do? We grab a their most recent backup We restore it in isolated non publicly available Environment to make sure you know to kind of see what would happen when we upgrade it So that way we kind of know what Things we might run into during the process. So when we go to the customer, we're like, okay We're ready to do the upgrade. Is it okay to have maintenance window? We've already like rehearsed the upgrade a few times and then we implement it now the interesting thing about that is my employee didn't realize by giving the client a quarterly upgrade And because of the fact he had to use the most recent backup to do this He's testing the backups quarterly because he's actually restoring the image and he's booting it up And upgrading the software and you know playing around with it just to make sure he knows Check we're we're auditing that Clients backups quarterly literally In the home lab you could think of this like how often do you spin up machines? Right if you're spinning up a machine to play around with something once a week You're technically if you're using a template you're auditing your template once a week right there done You know it works because you use it regularly So there's going to be some things that you do regularly you may not tie into auditing But it does check the box if you are legitimately Restoring something or testing something And that'll help eliminate any double work that you might do and then leave the things you're not auditing regularly as the things to focus on Yes You have to stay hydrated So going down the list then Um One of the things I like to do is version control all the things and this is something that I've mentioned on the podcast before and When I say this I don't mean that you should upload your Private config files or anything to a public repository Obviously when it comes to a git repository the first thing we think of is there's a remote we're going to push to but You know have to you could keep a git directory local and never push it anywhere But what but you can commit changes all the same And when you want to test changes or find out what's changed you could just do a git status If the you know everything's under version control you could find out exactly what worked or what's been changed But more importantly if something is broken Get status what's changed. You know what has changed from the known working config and you could also Tag the config when it's known working So you have a known working git tag to go back to for the config so you could put your You're a patchy web directory in version control and I've done this on one website I managed for myself. It was the least important site And eventually someone did get in and I didn't really care because I always think about You know getting rid of that website anyway, and I eventually did but When I realized that someone had gotten in and that something was going on I went to the you know wordpress directory git status I know exactly what files they they touched and I know exactly which files to undo And because the dot git directory is owned by root Not by the web server user You know someone gets into the machine via the web server user And they don't have a way to escalate their permissions to root They can't alter that git database Now if they did have an ability to escalate to root I have a bigger problem Okay, git is not going to help you with that if they're able to get to root There's nothing that that is safe at that point, but luckily for whatever reason they didn't get that far, but The dot git directory was fine git status find out was changed revert revert revert revert done go about your business That is to say if the malware is you know rooted deep in the system That's not going to help you but the point is with version control that may or may not help you But it definitely won't help you if you're not using it if you're not Because someday you might be thankful that you did this and it's possible It may never pay off and it might be a complete waste of your time but it doesn't Really heard anything to just drop a git init command into a directory and you know set that up And that could be like the like just one way of auditing what's changed because you can go through the commit history And you know exactly what's what's happened since the last time you audited that configuration You absolutely Um, you know Jane tell you tell me if this is still viable Is there's still some good tooling out there because it's been a long time since I looked at it years and years ago I used I think it was called trip wire, which I believe was an open source one. Yeah Yeah ages I know um, there were kind of a few home. They would notify you Of changes and I think that's kind of neat I you know, that's a very good. Um, I thought net data for a future show I guess I think I think it might be but Um, doesn't that data do that or am I mistaken? I thought no, no there's something I saw in there But I maybe I'm mistaken. Um, the problem I mentioned wazoo Wazoo is another one that would probably do something like that They do have the ability to watch for some changes Wazoo is a fork of ossec with a interface on it by elastic. So it's kind of a neat product Yeah, yeah, I'm sure there is it's one of those things where I'm so used to it being a corporate Crap thing where it's just like, oh, yeah, if you want this, uh, you know trip wire thing, you're gonna have to Sign a statement to work. That's why I didn't know if they worked for I think they went commercial. So what I think there's probably some Options out there. Maybe that's a future homelab show. We'll end up doing so I think I think it's interesting too because there's going to be some people who's you know at work at their day job If they work in it might be undergoing an audit or maybe they hear that They're the company wants to get certified iso sock two or whatever And maybe they're just starting out this process if you look at the requirements It's like I mean the requirements for sock two and iso and all these things you can get a hold of that Even without looking at the work thing and then just say, oh, what are they going to look for? Oh, maybe I should implement this and then next thing. You know, you have a sock two certifiable homelab which would be hilarious if someone put the Six figure dollar amount into a you know company to audit your homelab But if you have like a bunch of money and you just don't know what to do with it And you want the ultimate bragging rights. I'm not saying anyone should do this It's a complete waste of time But if you wanted to practice for an audit that would absolutely be one thing that you could do with that But the biggest problem in my mind for homelabbers is going to be silent failure And I feel like yes, it's going to be worse than enterprise because The situation is like this, you know when you have a situation in a company You know, you get your people together you figure it out But at home, it's like you have a personal life if I assume you do I mean you have something going on other than your homelab I would hope and that's going to take you away. I mean you might have family You're going to spend time with or things happen or maybe it's the holidays I don't know whatever it is and you just can't seem to get to check everything But you keep getting sidetracked because you're at home with your network And that's totally natural happens to everyone But when it does The likelihood of silent failure just in my mind rises exponentially at that point like like you Prop it's not going to be uncommon for someone just to not have time It's just the way it is. So when you do set time aside or you have some kind of a schedule it works But silent failure should be something I feel like everyone should focus on first Because you like we talked about earlier We want to know that our data is going to be there and that it's safe We don't want to assume that we want to know that and this is the only way that we're going to know that By actually auditing things that could silently fail for example a backup chain Maybe you have something maybe something like me. I've synced thing and it goes to a nas and then that goes to Um an upstream backup server for off-site and then maybe there's another local copy I recently like Built this automation that automate or that actually synchronizes my true nas to my synology So it is working fine. All my data was there and then a week later I find out it's failing I didn't make any changes and that leads me to how did this work in the first place because I find the error I'm like, there's no way that worked But it did my data is indeed there and the only way it could have gotten there is if the script ran But you know these weird things come up and then I figure it out I fix it and then I get the synchronization going and then you know that that was a silent failure I didn't know that it was that this was going on until I looked for it And that's just something to keep in mind. I mean this could I mean imagine losing family photos or something because you assume the backup chain was working and it's not I mean that would be horrible Yes, that's that whole um untested backups are wishful thinking I see someone I love saying that and I see someone repeated it It's exactly the case and it's you know, but part of this The part that's really hard for us and I think it's hard for everyone There's some challenges that go into this too. Like Do you have time to look at every single photo regularly to make sure there's no corruption? I don't think anyone does and worse if you Want to audit your plex movies. This is where we start to get into a major challenge You don't have time unless your plex movie collection is like down to a few and you have like a puny movie collection Which is fine You can audit that you could watch those movies every year and know and know that they're working But if you're like a lot of people and you have a bunch of movies on a plex server You don't have time like it's literally impossible For you to regularly watch all of your content. It just can't happen. There's no time in your life for that um So what do you do and that's where you start to get into? um some challenges and One of the things that I feel you'll agree with because I think this is exactly what you said last night is If it's important to me, it has to be on zfs now I am before zfs. Okay. I had a pearl script That a friend and I wrote he did most of the work because he knows pearl and I don't but we came up with this thing And it regularly creates an md5 Some file of every single thing on the server If the second time it generates an md5 if it's different, it'll email me. Oh that file has changed and I didn't change it Of course, it's a mess because things you changed on purpose will be there in the list and it just was a confusing mess But it worked. I was able to find out what's changed but that's just so I mean that my home lab was tiny back then I could never do that now But zfs you have scrubbing that could help with this and then you get into a territory of ecc or not ecc, which would probably be a podcast episode of itself of its own because that's Been a topic of debate lately, but having the ability to scrub at least gives you some kind of um piece of mind that it's more unlikely you should still Like pull some random files out of your backup just because you can't watch all of your video content that you have saved doesn't mean you just say Oh, I can't audit that there's nothing wrong with pulling down a random video Or even just creating a script that pulls a random video from your You know collection and then copies it locally Then you watch it make sure it works and just look at your data I mean that's the least you can do but Silent failure is definitely going to be a big problem and it also hits us with crown jobs We have we have automations going. But how do we know that the crown job worked? Right, um, we could set it to email us So we know if we got the email that it works But are you going to remember that if you did not get that email that you should have gotten it? Or is it just going to be filtered out in your mind and you're just going to go about your day Then there you have a silent failure even though you have an email system So that's when I start to look at things like health checks.io Where you can attach a view uid to your crown jobs And it'll ping health checks.io with that uuid And that'll clear it and you you set how many days You're willing to go without that being pinged and then health checks.io is going to email you say, hey Look, I haven't seen anything from your crown job here in a while. I think there's a problem um So of course that helps out too with when it comes to crown jobs at least you can know that they ran And as you go through this you audit all the things but at the same time Don't overdo it because like I mentioned earlier if you are regularly using templates to spin up new vms You're auditing your templates. You don't have to audit those anymore. You're using them regularly. It's totally fine Um, so you got to try to know what to audit what not to One more thing I in the words you didn't use was bit rot. That's what we're talking about one You can't necessarily view all your media and that's one of the reasons I lean so heavy on zfs As long as you have a zfs array array as an array of drives a bunch of them together where they're Away plural you can do it zfs on its own can't do anything with a scrub if it goes Hey, look, I found a rotten bit here and we've lost part of this image part of that video in your library But um, we don't have the parity to fix it When you have things on a zfs array when you run those scrubs if it finds something it'll go Huh, look this bit is wrong on this particular drive But no worries if you have like a z2 we have two more copies of it If you have a z3 we have three more copies of that data so we can rebuild the integrity of that file So you don't have to worry about bit rot. This is something we've run into very much so in the world of Movie and film where they film a lot of things and we seen some big failures on very large commercial Quarter million dollar servers that didn't have good integrity checking and they've now moved to zfs And this problem has just gone away now It really is under the hood some of these companies run their own versions of zfs But they load all their proprietary garbage on top of it right That doesn't always create the best in my opinion use your experience compared to using something like true nas And zfs. It's one of the reasons I'm so you know Well as I've been told occult on the zfs, but it's just where I trust my data to live Well, it works and you know, it's funny. It's it's like, um, I think tlc was wrong based on what you're seeing scrubs are good So that's that's the takeaway. No, um little uh late or early 2000s music humor there, but um a couple things to Mention too. I definitely want to throw in here Um, I recommend running shields up from grc every now and again And it's so fun like like you run this and it'll let you know what ports are open Obviously, you can find out what ports are open yourself But what shields up will do is it goes outside your network and comes back can it hit your network and come back? Well, if it can't then that's pretty good You you don't want it to have a two-way conversation, right? But um and of course I might be oversimplifying or possibly slightly incorrect about how it works Because I don't really know what it does in the background But what I do know is you run this and it'll tell you which ports are open And the reason why you do this is you want to make sure something didn't open up that wasn't open before You know, maybe you installed something new And didn't catch the fact that it's publicly available on a certain port This is something that will help you determine this and find out if if something That's not a obviously a situation where if you pass this test your bullet proof It just means you have no low hanging fruit when it comes to ports doesn't mean you're invincible means that You have a good starting point, but at least you know when something opens up and um showdan is a good solution too if you have a domain and you want to um Have some kind of an alerting on that because you and you taught me this you could have an email sent to you that A new service opened up as well if you have a domain aesthetic ip and all that Yeah, just aesthetic ip is all that's needed of note to keep an eye on the I believe they still do this almost they've done the last few years black friday sales You can get like these Long time subscriptions and get like five ip's monitored for a really flat price On showdan. They're really inexpensive When they have their sales for things they encourage it on like the home lab and basic user stuff to get a paid account And that paid account comes with that monitoring of ip's which I think is really cool And I also want to mention linus l y and is yes, you know because it's we're not talking about linus tech tips We're not talking about linus torvolz. It's the third linus the third wheel here, you know the It's a great linus. It's a great linus. So even though it's a third wheel. It's actually really good um, so this one is Is you basically get download the community community edition i pay for it because i'm a business You get more things, but What what you could do with linus is it it gives you a ridiculous amount of detail about the security of your system And you want this i mean you want something that's really going to look at every freaking detail to the point where The length of this list is annoying how many things it finds no matter how good you are. It's going to find some things Um, I I thought I had somebody in my audience hit 70 percent or something and that's pretty impressive actually Yeah, um, so you look at this list and it's going to complain about everything like the message of the day down to kernel parameters and tuning and um potential things it's it's obviously not going to make you 100 percent bulletproof nothing will but This tool and I have a video about this it really dives in deep So if you want to do a security audit on your systems and it's a per system thing unless you have their Cloud offering But you just run it on your systems and you get this report and there's a way to generate an html report if you want You can just scp that back to your desktop Open it in your in your browser and you just scroll through a list of things that it's complaining about some of the things You might not care about maybe you don't care if you have a message of the day like whatever That's up to you, but there's going to be some things that you should care about that's going to be more egregious for example You know this setting in your Apache or nginx might be insecure might want to change this. What are your ciphers? Things like that it's a really good tool and you could check out the video if you want to find out how to use it But it's something that I make sure is installed on everything every single time. How is it spelled l? lynys ynys i'm putting that to make sure i wanted to make sure we had the Spelling exact for people googling it So lynys i'm going to put a link to the video in the Description as well Yep, and and that'll teach you basically how to use it and everything It's it's just a good idea to it's it's fast. You can scroll through it find out You know whatever is like rated like super high as far as like a vulnerability is concerned Focus on that. I mean the lowest scoring things you could fix if you want to but at least fix the egregious things that Will probably be the most likely to make you have a very long day in the home lab that you didn't want That that's not for the right reason not because you Discovered a great web app you are fixing things because someone broke it on you and you don't want that so It's just try your best to avoid that would be a great thing to do Obviously if we get into security, we already have topics that are shows on that We can mention crowdsack and all these other things But I won't because we you know talked about that before But that's just a great tool to audit and I highly recommend it and it's been added to the notes So it'll be in the it'll be in the description and of course that anything in our description follows over to our show notes So ly nis and from here It's it's I think we get to a realm where it's impossible for me to give any further advice because I feel like after this You get into the territory of it depends on what you're running Okay, and that depends that determines what you should be auditing If you are running containers, well, obviously you should audit that I'm a big fan of When you're implementing things and we both do this Before it enters production break it Implemented again break it Implemented again tear it down build it back up until you have it down to the you know Fewest steps to get it running. That's what we do for videos because we rehearse this over and over again I mean you might be seeing anywhere from the second attempt to the 15th attempt by the time we film it That's why it looks so good because we've we've gone through this over and over and over again And we've we got it down to the smallest number of steps But you don't have to create video content to have that mindset You can absolutely do that in your home lab because once you have it down to that You could implement it and know exactly what's required to make it work You want to automate it later later? You have the smaller number of steps or smallest number of steps So you also have the bare minimum automation to get it running at that point You you make sure your data is good that's backed up and secure tear it down Can I build back up? Are you have have you restored something from your templates lately? Like spin up a new vm. Why not just just try something new in a vm You have to spin up from your template anyway So you're testing your template and after that it just again depends on what you're running because i'm not going to give you advice on Auditing next cloud if you're not running it because at that point it doesn't really matter So then we you know like I said it just gets into the personal preference at that point Yep So from the beginning like we said table top it out for what it looks like for disaster recovery for you That's really important take some time to rebuild any of those applications And you know in the end get a little more secure run these tools like Linus and Kind of audit some of your things and you can go further It's just I know people mentioned like wazoo and some of those But you those are a little bit more complicated So it's not like you just like hey one click install wazoo and it gave me all the information I needed There's a little bit more in depth and of course There's also the results that come out of wazoo that will take some more in depth on there It depends on your skill of although. I think it's a great project absolutely I think you actually should dive into it But uh is it a lot of scope of what we're talking about today on the simple side? Yes But hey, I always encourage people dig deeper. There's good documentation on that particular thing Take take a look to it some of the ossec documentation because I I used to run ossec forever ago And it's definitely a neat project how it works and how it manages things and I it's just really cool You should also avoid time negative situations and the and you kind of reminded me of this for some reason where You know, it's okay. If something is too overwhelming to learn right now I mean some people get to a point where they're like I must be stupid. I can't learn this. No, you're not stupid You're human Excuse me, and I'm human too because my allergies are creeping up on me again Excuse me but Don't just force yourself to learn something because you feel like you have to um In that same amount of time that you're trying to force yourself to learn something that might just be like You know way over your skill level and it's okay for some to be over your skill level It probably would have been better just to do it manually at that point because the same amount of time that You're trying to force yourself to learn something that same time could have been spent manually doing it And it's okay to put something you want to learn on the shelf and come back to it in a year When you've gained more skills So it's not like an end-all thing when it comes to a complicated solution But at least don't be so enthused about learning something to audit that you don't audit because that defeats the whole point. Yeah All right We have hopefully dropped you a little bit more knowledge a little bit more things in a add a tier to do list because this is definitely Um lots to do go check all that make sure your homelab is up and secure You know where all your data is and test those backups folks because untested backups are wishful thinking That's it. All right Check out some of the releasing videos Jay's got a whole just ton of tutorials He's been dropping for us on linux So if you uh want to do some learning of linux learn linux cv is your place for that Check out my channels. I got a few new videos on unify and pf sense So, you know more comparisons and I have a few more coming always love hearing from you So engage with us on the socials that you find We love hearing back from you from the feedback feedback at the homelab show dot com I'm sorry at the homelab dot show Right. Yeah, so We're old school like I know old school man So nonetheless and we'll see you next time take care