 Welcome to the Home Lab Show, episode 44, Dev Random. We had so many ideas that we were flinging all over the place and I was out of the office last week. So was Jay, we both had some things. I was building a new studio. Jay had some personal things going on. So we didn't quite have a show, but then we had too many ideas almost like. Yeah. It was hard to land on. So you know, let's do a Dev Random and run through some of these. And yeah, it was a lot of fun. We have a good list. We're going to cap the show at an hour because I think the list is exceeding what we may talk about an hour. We're not sure in any of these topics could be something we dive into deeper later. So it's, this is a new experiment with us, Dev Random. This is kind of like the Q&A episodes. Me and Jay want to do a few of them. Yeah. It's just basically things that each individual item wouldn't probably make an episode of them by itself. So there's some things we want to talk about, but it's like, do we want to have a five minute episode for this one topic that is just super quick? No. Let's just kind of just go through all these different things. Yeah. And like I said, a lot of this leads to further discussion. We do listen greatly to the audience so we can make sure we're covering topics that are always relevant to them. So, hey, feel free to throw some things in the chat. I'll hit us up on a feedback form if you listen to this, not during the live. And I should turn down things that make noise. So. Yeah. So I should probably do the same. Yeah. I just realized I have a couple of things on. So we're going to do that. Those, you know, technical details, you don't think about it in a new studio, but hey, here we are. So, yeah. That's a good problem to have, I think. Good problem to have. And this is the first time we're doing this show from my new studio. So bear with us if there's any technical difficulties. But I think I'm all sorted out. Now, where there has not been any technical difficulties is with our sponsor. We've been working with Linode since the beginning and they want to continue sponsoring this show. They've been a great host and they were a great sponsor. And yeah, they are just a great place to host all the different magic that they have in their system. If you're looking for places, and someone actually posted in my forums and I thought this was a clever and maybe we should do an episode breaking this down. And it's a way to run a proxy in Linode that proxies in front of your servers in your home lab and ways to get connectivity to it. I think this might be a fun Linode and Linode was talking about, hey, is there any other, you know, includes you could do. And that might be a fun episode. This is something you can absolutely run in Linode. This would keep your public IP from being exposed but still allow you to run services, learn about how things work and never worry about someone, you know, attacking your public IP or revealing who you are through it. I think this might be kind of clever. They're gonna know your Linode IP but that is something you can always, you know, build, stand up the server in Linode, tear it back down and move it over to another location if you want. It creates some interesting scenarios. And this is pretty easily facilitated with some of the, you know, auto ways to build it in Linode but of course in this show we dive into the manual way to build it because I wanna do a whole site to site video and I'm thinking that might be a fun one as well where you for especially people stuck behind CGNAT. I think these are some fun ideas and absolutely all these can be run with our sponsor Linode. We have an offer code to get you started. You'll find that in the show notes there and absolutely great sponsor. We thank them for continued support of the show and that's what keeps us going on this stuff. You know, gotta pay the bills. Someone said we have a different sponsor because we keep having the same one. I, you know, I like the simplicity of having one sponsor especially one that's easy to work with and is centered, there's a big crossover. A lot of you people probably already used Linode for some of these projects but now we gotta dive into the dev random of things. So that, what was the first thing on our list? That was it. Oh, Jeep hearted. Yep, Jeep hearted live. Yep, and a couple of these are tools that I think should be in everyone's toolkit, so to speak. Jeep hearted live is just one of those. It's not something that I need every day, every week or even every month. Sometimes I can go several months without needing it but it's always good to have it on hand whenever I do. And it could be something as simple as, you know, some hard drives that were previously used with LVM or ZFS and I want to just, you know, purge everything. There's many different tools that allow you to do that. Jeep hearted live is not the only way. It's not even the primary reason I would use it but it is one, I just want to nuke everything. I could do that. Another use case for it is sometimes Linux installation utilities, they don't really give you the option to partition the way that you want to. So sometimes it makes sense to use Jeep hearted live before you install Linux because they'll let you tell it where the mount points are that you want for your installation but they might not facilitate creating the partitions. So you could use Jeep hearted live to create the partitions the way you want them. And then launch a Linux installer and get that installed. So that's actually one way I can use LVM with Ubuntu because I still don't know why this is the case. You use LVM with Ubuntu through the official installer and it uses up 100% of the available space. And I do understand most people want that but then that eliminates LVM snapshots. They don't give you an option like Debian does to say like, I want 80% of that, not 100% of it. I want some wiggle room here. And with Jeep hearted live, you can create the LVM config beforehand just the way you want it and then install your distro. So it's one of those things I just always love to have around. It's also really claim. I'm trying to remember the Linux command. Maybe you can help me out with this. Windows has a tool called Estelit that you can do online but there's an offline tool you use in Linux to basically shrink a virtual machine that's taken up to a space when you have it then provisioned. It's built into Jeep hearted. What's that? I can't remember the name of the tool. Resized 2FS. Resized 2FS I think is what it is. Yeah, and what it does, it's also does writes out a bunch of zeros to the drive as well. It's like a, yeah, it's a way you can shrink your VM. The resize 2FS is allowed to increase the VM but Jeep hearted it's got all the utilities you need for doing things like that when you've run into some of those problems. Some of the VM can get kind of over time a little bit bigger than it should be on a thin provision and you can shut it down. Windows, I know it's Estelit. There's an equivalent command in there. Estelit's not part of Windows. It's something though that's from Microsoft part of the tool set, but Jeep hearted becomes very handy for that, very handy for fixing when Tom goofs up a logical volume or maybe a couple of broken times. You know how I learned about Grubb by breaking it. You know, and there's some other things too that are that kind of annoy me sometimes. Like it's like these problems that happen every once in a while. So you don't quite remember, but you'd remember it being a pain where you, you know, again, I'll use the example of LVM or ZFS where you have to reload something in the kernel just to get it to recognize the fact that you've deleted all the partitions. And some installers will give you the option to delete partitions, but they'll air out when it tries to do so because they don't like they try to automatically mount and activate LVM, but that might not be what you want. So now the kernel sees your disk as LVM. It's a previous installation and is trying to safeguard you but you're just trying to like delete everything. And then it could become a pain. So yeah, it's just a great tool to have. Pretty much, I consider it the Swiss Army knife of partitioning basically. Yeah, and it's for recovery and things like that one. Things have broken. It just, it works well. We'll just say that it's one of those handy dandy keep that in there and yes. Yep. Yep. Oh, sorry, go ahead. Yeah, I would say let's go to the next one. No, memtest. Yes, there's a really big reason why I mentioned this because I don't know what this is. And I think it's because, and this is a valid reason that homelabbers and Linux enthusiasts are more likely, in my opinion, to be using older hardware, nothing wrong with that because that server that served a huge company like 15 years ago, that server was great for them back then, but now it's kind of too slow and we can't really get as much use out of it. But for a homelab, it's great because you're just, one person or a handful of people, but it's hand-me-down hardware, which is usually a good idea, you save a lot of money. Same with desktops, laptops, why buy something new? You could, and I'll talk about this later, you could buy something that's not quite new but still better than what's available for new, for cheap. But the problem is often that your memory will be bad. And this happens a lot more than people think. And I can't count how many times I've seen people try to install Linux and it doesn't work well or at all and it doesn't make sense, but Windows worked fine, for example, and then I'll mention, you should try to check your memory. Then the rebuttal is usually well, but it was fine with Windows. So I don't think that's the problem. Well, the truth is each operating system handles or doesn't handle memory issues completely differently. So one operating system might be more tolerant to a specific type of hardware failure than another one. So the argument it worked on my previous OS doesn't work on this one, isn't really a valid argument. I would say memtest86, you should test your memory once a year on everything, your servers, the physical servers obviously don't run it in your VMs, there's no reason to do that. Run it in your physical machines and your desktop laptop servers just run it. Usually I just run it for 15 minutes. I haven't seen a case personally, I'm not saying this doesn't happen where you let it run longer than 15 minutes and it finds errors. Usually if it's gonna find errors, it's going to run, I mean, it's gonna show those errors like pretty much within the first five to 10 minutes. So for the last 15 minutes, you're probably in good shape. And you'll know when it finds errors, like the screen turns red, it looks hideous. It's pretty in your face the last time I used this. So it's so common memory issues and things like that. And I think sometimes people will get upset at Linux for not working on their hardware when all along they had bad memory. And sometimes memory issues are, they're kind of weird to be honest. Like I would love to say this is the symptom, but it's always something different, something strange. I remember one time, and I still don't really understand why this happened, but I used to be in game development. I was never good at it. Just something I was doing for fun with friends. I would just design a game and send it to a friend that's like one level or something, maybe as many as 10. And one game I developed had a save feature so you could continue where you left off, which is cool. Worked fine for me. So I send it off to a friend and they save and resume their game and they end up in a random level. I'm like, what the heck? So then I compile it, I look at the code and I can't find an error. And it turns out my memory was bad. It's on the desktop I was using to develop the game on. And the resulting executable was corrupt, even though on my system it was working just fine. I still can't understand that, but there's just all these weird things from Linux installations failing to development errors, file errors, all these different things. So just test your memory and just make it a point to do that as regularly as you can. It's not the most fun thing in the world to do, but it's pretty important. Now I have seen it take longer and we've had to have some machines, the real mystery ones that just don't do it very frequently. You'll have to run numerous iterations of mem tests to get the result. And it kind of goes to the randomness of a memory problem, especially when it's a very infrequent one. And they're the worst problems because they're not easily repeatable. It's the best way to describe it. You end up with a problem that is like, yeah, it almost is a repeatable because we know it happens, we see the result, but making it consistently happen under certain conditions becomes very, very difficult. And sometimes you just have to take parts of memory out. I will warn, there's occasionally, and mem tests has been updated over the years to fix some of this, but occasionally you run into some configurations that mem tests just fails on immediately. Not because there's a problem, it's because of an incompatibility in it. So you have to look at what the error messages are, throw them into a Google search, and really, it's challenging at times, but it's definitely very handy tool, especially with, I find it very compatible with laptops and sometimes have helped us solve some of those laptop troubleshooting issues specifically running mem test code. Oh yeah, it looks like some of the memory is not well in this. Swap it out and away you go. Saves you so much, especially when Linux can be memory efficient, it's only when you run the bigger applications that it hits the section of memory where those bits are a little bit less stable and it does not hold the memory. And then flipping a bit, if it's not important bit, you didn't notice it. If it's an important bit to a running process, well, and that process wasn't set to auto restart or loses or ends up in an unknown state, you have random problems and a lot of head scratching to do. So. Yep, and yeah, I totally agree. And similar to that, I'm going to go kind of off away from the order that we were going to go over the topics in because I think one kind of leads into the other, but I'm going to throw in one that wasn't on the list. And this is super simple. Have a flashlight available. What, why do you need a flashlight? Okay, hear me out on this because this just reminded me talking about mem test. I've had people bring me computers before and this happened like two or three times because it takes me that long to learn a lesson, apparently, where I couldn't get the operating system to install it all. And I could have ran mem test, but in this situation, after a few hours of cursing and fighting with it, I just opened the chassis, get a flashlight, shine on the capacitors and notice that they're leaking. Oh, there's the problem. That's why I can't, you know, you don't really think of that because it's not very common. I think I've seen it maybe five times in my entire career. So it's not like super common, but when you're buying a server from somebody like secondhand, absolutely look at the capacitors before you take it home. You'll save yourself a lot of trouble because if you didn't know to look for that, you might be, you know, troubleshooting that server for hours, trying to figure out why it won't work. Just to find out physical layer, there's some leaky or bulging capacitors, there's your problem. So that's another thing. And the other thing I'll throw in there too, because I'm about to talk about buying workstations or laptops or whatever, because we need a good workstation or laptop to modify our home lab, right? We want a decent machine to do all that config and all that. One of the things that I recommend people do, like if you're going at a, going to a used computer store, let's say, and they have a desktop on display or a laptop and you're thinking about buying it, it's good price and it's a solid machine. Usually it's gonna run Windows at first, go to the event viewer and filter for system errors or critical errors and look at that before you buy it. I don't care if it's like a server, if they have a monitor plugged into a server, I don't know if they will, or a desktop's laptops especially, look at the event viewer because if you have like a failing hard drive, it's just gonna flood the Windows event log with errors and other things are gonna flood the Windows event log with errors and you'll know right then and there that, okay, there's something going on with this machine so I really shouldn't take this one home. And it's so easy to do once you learn how to do it to go into the event viewer and that's not gonna be like a foolproof test. This is definitely a great machine because there's no errors. There's going to be errors because Windows always has errors so don't be alarmed that there's a few but you'll know the critical ones when you see them and that'll just give you an idea of what not to buy which is kind of like one of my tricks that I go through. But getting into buying laptops and desktops, my tip is to consider used more often than most do like I love System 76, I love Tuxedo computers and Lenovo and all those, they're great. Buying a new computer is awesome. If you have the money, I mean, what's better than that? I have a brand new rig, that's just so cool. But in reality, we can't afford a machine that's over a grand often. So sometimes you only have two or $300 extra. So what do you do? So that's why I wanted to bring this up because business class laptops and desktops are absolutely the way to go. I personally, I know this is an extreme opinion, I hate consumer level laptops because yeah, you can go to Walmart, yeah, you can go to Walmart or Mayer or whatever your local store is, you could get a brand new computer, not even a Chromebook, an actual PC laptop for $200 or $300, you absolutely can. And it's going to be fast. It's going to be efficient because hardware is cheaper now. You're going to think, man, this is a great machine but I've repaired these things. And I've literally told people, like if they cracked their screen because I'd replace it, I need you to buy a bezel too because I know how this is going to go. It's all going to crack. It's all going to fall apart. The minute I take it apart because the chassis is just bogus. There's a reason why it's $200 or $300, trust me on this but the $200 or $300 that you might have to buy a brand new machine, you could use that same $200 or $300 to go on eBay and buy a business class laptop, for example, or a business class workstation. That's a few generations old which is why it's $200 or $300. It's not the newest one. However, I would argue that the build quality is going to be better than anything you'll buy new at that same price. Plus, you know, business class laptops just last forever and they're often really fast. Like the processes are decent because processors are not changing in my opinion as often as they used to. So throw a solid state drive in an older, you know two generation old computer that thing will fly and you'll definitely have a better machine and absolutely research Linux compatibility first if that's what you want to run. But I would highly recommend people go that direction instead of buying new if their budget is really low because I think they'll get a better machine for their money. They last a lot longer. They have, I'm assuming everyone here is going to reload them right away. So the bloatware problem is less of an issue but they generally ship with less bloatware on them with the business class ones. The other thing too, and I've done a video on this before if you dig around on my channel I probably five or six years ago I had a couple of taken apart laptops and I was discussing that exact thing the difference between the higher quality ones and that hasn't changed in five or six years. You know, I haven't really readdressed that video but it also doesn't really need to be readdressed. If you look at the build quality on them it's substantially different. The kind of flimsy floppy nature of these how thin can we get it and everything else is just not as good because they're trying to produce it for the consumers in the mass market so they cut every corner possible to make them, you know, not as high quality. You take something like one of the Lenovo ThinkPad series which I've been a fan of for a while they're just a pretty solid build and a pretty solid feel. But if you can go new, don't get me wrong especially if you're going I want an absolute Linux compatible one look at something like the system 76. Now, and I have not got my hands on one I have not ordered one because I just don't have a need for a new laptop but I definitely, I know there's been a lot of positive feedback from several friends I've had and other YouTube reviewers on the framework laptops. I think that's a great idea. I like where they're going with it. I don't personally have any review experience with it but my overall feeling from the people who have reviewed it that they're being honest and it's a pretty quality product. So I would say yes, a lot of good stuff there if you are looking at some of the alternatives but we know it's a home lab show that's why we mentioned the budget conscious stuff first. All right, right. Not everyone's got the budget just to run out and buying stuff or get stuff sent to them because they do reviews sometimes. Yeah, it's just the best job perk ever but what we understand it's not a perk that very many people have where the few, right? We're the 1% I guess. But specifically I would say look for Dell Latitude for the Lenovo T-Series, Pinkpads because just because it's Lenovo doesn't mean they're all good because their consumer level machines are every bit as bad in my opinion as anyone else is like. They're business class Lenovo's or business class Dell's not the stuff that Dell sells that the retail stores is often so. Yep, T-Series. Although I think the model naming changed in Lenovo's Thinkpads series now I can't remember what it is now but I did it because I'm not I think it is it still a T-Series because that's what I've always known it as and then there's the latitudes have pretty much been latitudes forever. Those are some to look at. Absolutely. All right, what's on the next? So system rescue CD is really quick because it's just a quick mention system rescue CD downloaded put it on your flash drive have it ready. You can use it for cloning hard drives like clonezilla for example it actually supports that but there's other things that could do too it has a graphical user interface you can use it for file recovery or things of that nature. So just throwing a quick mention out there if you've ever used one of those what I call a multi rescue tool CD or whatever it is they have all these everything with the kitchen sink in there there's several of these out there I like system rescue CD it's treated me very well so I think it's just one of those things to have available if you need to recover files or do some hardware testing clone hard drive another quick mention and I'm gonna make this very quick because we had I think we had an episode about this about clonezilla. Yes. Now if you've seen that episode you know that it's gonna help you clone a hard drive but what you might not know is that if you have a failed hard drive or I should say a failing hard drive this doesn't always work but if you really wanna get that machine going again you could try keyword try to take an image of it with clonezilla but activate the advanced options and tell it to continue if it reaches a bad sector which means it'll still take an image of the hard drive but it won't be the best image because there's gonna be a couple of errors unfortunately this is for people that are really desperate obviously reloading is better if you have it scripted or automated do that instead but if you just couldn't get to backing it up or automating anything and you put weeks of work into this thing and you really don't wanna lose it you could just clone the hard drive activate that option to continue when it encounters bad sectors and then restore that image onto a good hard drive it might work because I have seen this happen several times where people were thinking oh man it's the end I've lost everything like I can't get this thing to boot and then I do that and then you'll still probably need to do like a file system check no matter what because there's errors but after you do that it might fix it but it's just something to try something to keep in mind if nothing else it might let you pull files off the hard drive too so I just wanted to throw a mention out there for that Yep List the next one on here CrowdSec and Fail2Ban Yep, yep We list those together because they're very similar in fact Fail2Ban inspired CrowdSec and the way they handle this is kind of similar because what they aim to do is offer a layer of protection I don't wanna say it's gonna protect your server but it's gonna obviously increase your security nothing's 100% but if configured properly Fail2Ban for example will look for certain things in the log for example it could be watching the SSH log and then seeing that there's someone just trying and trying and trying to get into your server it can block them by adding a firewall rule you set it for how many attempts you're willing to let it have because I don't know maybe you mess up the password four or five times because you're like me and you're a klutz but you never mess it up seven times so if someone has just set it to seven if it goes beyond seven it can basically create a firewall rule to block them you could put a white list to add your own IP in there so you don't block yourself ever and it's not just SSH you could configure it for Apache and GenX there's just a huge list of different things and you could also create your own jail that's what they're called jails for something else just tell it what to look for in the log file that constitutes a concerned view and if it sees it it can go ahead and block it excuse me and CrowdSec is basically very similar to fail to ban I don't know if you could do custom I think you could do custom jails so to speak they're not quite called jails on the CrowdSec side but CrowdSec also creates firewall rules when it sees some activity that it is not appropriate like a brute force attempt or some other kind of security thing that it notices it'll block the IP but another thing that it can do is it downloads a block list from CrowdSec which is paying attention to this and seeing these attacks happen so if one IP is getting attacked by another that it knows well that IP is doing some shady stuff so we need to keep an eye on that and it could let all the other clients know that IP is bad so that way it helps increase security by leveraging the crowd or their users and helps enhance it that way that's the trade off though because if you use CrowdSec it's going to send activity it sees negative activity like intrusion attempts up to the mothership so that way it could let all the other ones know so you have to be okay with that I don't see why it wouldn't be it's just trying to grab IPs but that's a trade off and I do believe if you don't like that you could pay for it it's free otherwise you could install it on 10,000 machines and it's fine for free but if you want to opt out out of that information sharing then that's I believe when there's a cost associated with it but it's so easy to install fail to ban I would say is a bit harder but it's not hard it's just CrowdSec is more of a turnkey it's not completely turnkey but it's closer to that fail to ban if I remember correctly bans or excuse me it watches SSH by default but it won't watch anything else unless you go in and enable the other different things that it has in the config file that you want it to watch for I just think it's really good to at least use fail to ban consider using CrowdSec if you want to have a global knowledge power at your disposal there for that but one or the other is fine you should definitely run one or the other especially on machines that are forward facing to the internet Yeah, and the thing I like about CrowdSec and fail to ban is great for the individual and I see actually someone through in the comment and you can get tricky with it taking the ban list that fail to ban creates unless say you're running a cluster of servers create a list and automatically do it but now you're getting into building a lot of engineering one the reality is you are not the only one with something open on the internet that's just going to be the way the internet works so the concept of CrowdSec I really like how they're doing it because this really opens up an opportunity and there are places like greynoise.io where you can get IP list from and there's a lot of threat intelligence data out there here's alien data has a free app feed where you can go through and look for bad reputation IPs this is kind of a cat and mouse game but CrowdSec really takes it to the next level of not only building the lists in real time having you help to contribute to those building of the list then combine that with you know you can end up with these IPs blocks so it doesn't even fill up your logs with noise so if right away in log4j was an easy example of this they were sorting out and greynoise was listing them and so were other places like these are the known threat actors attacking these type of devices that are you know maybe unpatched and already having that list and CrowdSec protects you in case you didn't patch but I know all of us and everyone listen that's already patches their system right away as soon as something comes up but on that off the hats that you know a person and they're not doing it and that's for a friend that's for a friend here it's nice to have an automated system that can roll out those block lists and it can buy you a little bit of time it's not a replacement for security at all but everything especially when there's a zero day out there someone gets hit that's how we learn about zero days that's how we learn about these active attacks sometimes it's only through the attack that we learn what was broken and this is an example that happened in some of the WordPress sites if in CrowdSec does integrate right into WordPress and the web applications this is really cool because as soon as someone started dropping all these attacks on these WordPress sites right away they're like whoa and it's not like there's a thousand IPs even doing the attack it's like here's a couple IPs that were initiating these attacks and exploiting this once that gets into a block list at CrowdSec cool and then the people that wrote the different plugins for WordPress that were at fault well now we gotta get those updated and we've now bought a little bit of time because we didn't even know about the problem so it's another layer on your security I think I really like what they're doing with it me and Jay talked about maybe we'll have them on to dive a little bit more in depth on their whole platform how their business model works because it's really a cool open source intelligence tool I just really like where they've gone with it and it's something I'm diving into more and Jay's got a video on and I plan to dive in a little bit more I've set up some demo servers I've been poking out with it so and I'm hoping they work on a they have a PF sense integration but I think it's pretty manual right now I'm hoping at some point they'll get a more in depth PF sense integration too because I think it'd be an awesome addition to PF sense because it sits at the perimeter of your network it can listen to the noise it can report back and then automatically add block list so it's gonna be something that I think we're gonna see a whole lot more of in the future field demands a cool independent tool but I like the full extensibility because it's putting security as a community effort just makes a lot of sense to me that they hit the right nail on the head with it and their background is also in managing hosting servers at scale the developers had a hosting company that's where they came up with the idea for all this and then their idea was well why not do it for everyone not just us and that's where they've been really pushing and expanding on it so I'm excited I'm gonna talk to them about that Yeah, for sure Yeah, maybe that'll be something we can set up Yep, someone mentioned earlier I just wanna give a shout out to Ventoy it is a way to set up the multiple boot USBs I've never used it, but my staff has it's a way to take some of those things like system rescue CD, memtest 86 and combine a few different ISOs I believe into one I believe that's the right tool name I've seen someone mention in there so Yeah, something like a Ventool, Ventoy, something like that Yeah, it's a great one I haven't used it myself but I've heard a lot of people talking about it I've read about it I've seen, you know or I've read how to documents about it I probably should be using it because I have like 50 USB keys in the little bucket that I kind of sort through every time I'm looking for something it probably makes that a lot better for me maybe I should give that a shot Yep, all right now related directly to the tools that was offered by CrowdSecond failed to ban or you can even kind of throw in PF Blocker those are all tools that identify things and then have block list different but can completely coexist at the same time is Saracota and Snort those are traffic analysis tools and the rule sets based that they use are looking for specific sequence of events or attacks that can come through now the efficacy of these products can be hampered by encryption so it kind of depends on whether or not they have the ability to see into the traffic that's passing through them typically you're going to run Saracota or Snort on an edge device and it is built into PF Sense and one of the things I like about the way PF Sense does it they call it Saracota they call it Snort they don't call it just IDS a lot of other companies are using Saracota in the back end they just call it Intrusion Detection System or Intrusion Prevention System you know that's but they don't tell you what it is and what the rules are around it so it's kind of interesting that PF Sense is kind of a you know not the only one but one of the companies that really exposes you to it and gives you everything to tune with it now what these actually do is they're looking for things like they can even look at the way DNS queries go out and you can say you know hey here's a rule set if these DNS queries go out or they look like something bad or looks like it's resolving something go ahead and have a block I managed to block myself the other day on accident I actually thought my VPN went down from my home to my office and it turned out Saracota see me doing something with SSH and I wasn't thinking about it but Saracota cause that's not the way that should have been done because I was hammering something on SSH as a test and then it blocked me and once you get on a block list I couldn't even log back into my own firewall cause I was banned so yeah that happens to all of us at some point cause let's be honest at some time you're like oh that's right I shouldn't have done that but nonetheless I was doing some testing and those are really good tools to use though and there's a cat and mouse game though those rule sets that are being constantly updated so it's another layer of security so that's why I refer each one of these they're not an end all solution they do coexist and they're best actually to coexist with our solutions now do you need Saracota and Snort on your home user system if you have no ports open I would put it only in detection mode so you can just kind of see things that might be interesting to you they do take and I have a video about tuning the rules and it is a constant effort so sometimes there's a new rule that comes up that it misidentifies something or the rule identifies a specific attack pattern but also by the way this other tool looks like that attack pattern I think in the early days syncing the transport layer of syncing triggered something unrelated all together so at first you're like wait why is this thing attacking oh looks like syncing triggers this this is where you get the false positives and because it's a pattern based system and sometimes it's looking at traffic that it may not have full insight into it can be really difficult to absolutely nail down that so it's not a hands off type of thing and a lot of times when you see it in like the Unified Dream Machine has Saracota running under the hood they don't give you as many tuning options for it and you're kind of just relying upstream on whatever the vendor pushes out for the tuning rules they give you such a good, better best or something I think they have a really generic way you turn up how secure do you want it and you crank it up to best so everything starts breaking and then you slide it down a little bit I don't know how much more tuning options they've added in Unified Dream Machine but when you do something in PF Sense it's absolutely showing you the raw data so you can get a better understanding of what rules flagged and what rules you want to tune and maybe why you're tuning it which usually, hey, highlight the phrase that it found in there or the rules that it found in there of what it told you it blocked copy, paste into your favorite search engine and you'll probably find a result in a discussion form about the rule why it exists and what it may be a false positive of so it actually helps you a lot with the research of why something's not working and yes, even people who work in high levels at security and sim places that's actually what they do they'll sit there and look at a rule that's weird to them and they're like, I don't know this one even with all my experience and yes, they use Google search that's for any of you that's thinking about getting into the enterprise market people are still on stack exchange right to the enterprise market don't work They really are I'm kind of surprised at the highest security level just doesn't disable every network interface in every device That makes the most sense right? Just break them all break them all Just disable everything related just kill the entire TCP IP stack 100% just to annihilate the whole thing and then yeah you'll be way more secure at that point Yep probably useless but secured nonetheless All right we got get for security Yeah, this is a fun trick but there's gonna have to be some asterisks here because I don't want to oversell this as like a really good idea in every use case but it does help sometimes if your server gets owned but it's not owned by something like full system level for example let's say you have a website and you have some HTML files that you're serving just something simple and someone just gets in there and they just corrupt the files they put the base 64 encoding in there and your PHP files and next thing you know there's a crypto miner if you have if you make your HTML directory a Git repository and I'm not telling you to upload that to GitHub because you can have a local Git repository that doesn't go anywhere there's no Git server you could just have a local Git repository that doesn't even leave that particular server anywhere and you can get the website or whatever the web app working the way you want it and then just commit right then and there and if someone does break in and they do some bad things there you could just do like a Git status you see all the files that they've modified and then you could do Git checkout and everything and it's like they never did anything now the asterisk is that obviously that's not always going to work because what I do is I make the Git repository owned by root so you can't commit unless you root so even if someone breaks in as the web server user they can't get into the Git directory and lock that up unless there's a privilege escalation trick that they use to get up to root or there's vulnerability chaining yes they absolutely can corrupt the Git database so I don't want this to seem like a end all be all security trick but if it's a lower tier intrusion that does not go beyond that it can help and it's so easy to do because it's a local Git repository just do Git init inside the directory and then Git commit everything there and then from that point on you at least have that starting point it doesn't take the place of backups doesn't take the place of automation this is just a I only have five minutes to fix this problem and I got to get to work kind of thing right I don't want to oversell it but I have seen this happen where someone just does Git checkout and everything and then it just makes everything like nothing ever happened as long as they didn't get outside that directory and infects the rest of the system somehow then that might actually help you out yeah and I want to make sure there's that clarification because there gets to be gets to be a lot of confusion in this Git Hub is a service which is currently owned by Microsoft and Git is a tool and a lot of people think of Git and GitHub is one and the same but they're just an opportunistic company that said hey we can throw a hub at the end of this we have this domain and we're going to offer it as a Git repository hosting now it's probably good that a lot of people use it it's been a really popular place even since Microsoft purchased it for hosting a lot of open source projects and allows you to easily get cloned throw the name in there and pull down information on there but they are separate things I just want to make sure that's always very clear there's, because you can also because you're using GitLab is it Jay? Yep, GitLab they're an absolute competitor that also uses Git of GitHub you know I use GitHub, Jay uses GitLab I was going to switch because we were all going to protest everyone said they were going to protest Jay actually protested because I think you moved from GitHub over to GitLab right? Yeah, but I ended up opening up GitHub for the YouTube channel it's not where I keep my configs for the business but GitHub is now where I want things to be like internet facing if I go over something in a video GitHub is a good thing to learn so I put some of my stuff out there so people can fork it or maybe put in a pull request or whatever but yeah, I did switch to GitLab for all of my production stuff on my end and I like it, it's really cool if I take Microsoft out of the equation I like GitHub and GitLab both I also like GitT which is something that I want to cover soon I don't have a video about it right now but that's a roll your own Git server that you could use instead of either GitLab or GitHub and just have your own you could put it on a note instance for example run GitT over there and you have your own Git server that's your own in your control so there's definitely options you can get the code out there if you want to but I think one of the beauties of having Git as a local tool that's not connected to a service is you can have local repositories for version control even if you have like, I don't know another family member that's also into IT and stuff and they're also learning with you and then something breaks you can just do a Git diff what did you do? Yeah, you modified this file why did you do that? Right You have some traceability there especially if you work in an IT department you can even have people committing changes there it just becomes a really awesome way to modify or excuse me to maintain server configs and in this case you probably shouldn't upload that to any of those Git services just keep it local this isn't like your pet Python project this is your app you want to keep that local but it could give you some additional options that may help as long as that intrusion wasn't like a higher level that broke out of the jail, so to speak and infected your entire system Absolutely Jay, you have a weird run on the list Yes, I do, don't I? You're probably surprised to see that one, weren't you? I was surprised to see this next one and I think you're talking about ripping CDs, right? I am Yes because I was thinking about I'm not going to get into the drama and the politics about Spotify and streaming services and all that but there's one universal criticism that I have which is these services these cloud music services are good because I don't have to have like a two terabyte hard drive with me everywhere I go I could just stream as needed but the problem is when they what happens when your favorite artist loses a license or they can't negotiate a license or something like that for the longest time tool wasn't even on Spotify and that really made me mad because I want to listen to tool so what do I do? And I feel like with Google music going away it's now YouTube music used to be able to buy MP3s and things and I used to have this Python script I would use to actually purge the hidden comment that Google put in MP3 files that would have traceability I just purged that and I had my own I own this content I can I know the music industry would disagree with me when I say I own it but I do feel like I bought it I own it, it's mine I'm going to listen to this until the end of time but now that's gone so what do we do? So I do kind of feel like ripping CDs is an important thing because you could set up like Plex Amp or one of the other hosted music players or Volumeo for example on a Raspberry Pi and create a jukebox and because I know this isn't initially like a homelab topic but it becomes a homelab topic because it's not much different than running a Plex server and watching the Avengers, right? Sometimes you don't want to watch a movie just want to jam out to some Metallica or something and it's good to have something to do that with and when it comes to ripping CDs what would you use nowadays because most people don't rip CDs anymore and for all I know CDs can go away all together but there's something about having a physical copy of something that if I Fubar the MP3s or whatever I'd actually rip an org format personally but I can recreate that content infinitely and it doesn't depend on a record company's negotiations to whether I can do that and the tool that I'm going to recommend because I'm sure this is not something that people are thinking about right now but I'm just putting the idea back in everyone's head about ripping CDs is called Asunder A-S-U-N-D-E-R has been my tool of choice for so many years, I can't even remember now. So I've been using that to rip CDs and have that digital copy and then I put it into Volumio. I also have Plex pointed to my music share as well so I can use Plex amp when I'm on the go to listen to my music collection. I think I have over like 150 artists in my collection right now and probably over 10,000 songs. So it's definitely a lot to keep track of but it's really awesome to have that control and until they discontinue audio CDs all together which let's be honest, you know they will. We have this ability now to just go to a flea market and buy a stack of CDs for a dollar and throw it in our music collection. Yeah, it's such an interesting concept and having some younger kids they some think about it a little bit but it's not as prevalent to them that we used to own all of our media. It wasn't the inconvenience of CDs. I mean granted, yeah, there's inconveniences swapping out audio disc or anything like that and CDs, obviously I've never been the biggest fan because they have delicate nature of them. And I remember forever ago, you know, yes scratch one, you're like, ah, I lost the music I could buy another one of those and there was places I used to, you know especially go to places to buy all the used audio but right now so many people in the controversies around media rights when you subscribe to some service and that service can't negotiate with the artist you like but they were able to before. This is a really interesting thing that is bringing back the rights talk and I'm really happy about that because I don't like the concept like that I pay an artist on a subscription. No, no, no, I pay that's not, that's a concept a lot of people seem to be confused by. No, the artist gets paid and has the ability to sell their same work when it comes to music in a digital form multiple times. That's a really cool thing. Now the artist has an opportunity to make a lot of money by having a popular song and each one of us pay for the right to use that song. I don't think that's an unreasonable request but I can play that at my leisure because I paid for it. Whenever I decide I want to play that and I think it's a lot, it's really annoying when you think about it because I wanted to watch something that I knew used to be on Netflix and now it's not. And I'm like, oh man, I wanted to watch it again or share it with someone and they're like, no, you can't. I'm like, wow, I can't even have friends over to rewatch this movie we wanted to talk about from a number of years ago, but it used to be and a licensing deal moved it to another platform. And yeah, it bothers me that we're getting away from it. I think hopefully some of these big controversies that come up bring people back to it because you pay for the media and I don't mind. And obviously it's different when you're paying for it as a service. When I pay for one of those music services, I don't get the rights. I get temporary rights as a service is the way I would describe that via streaming but I'm fine to go back to the days of purchasing them. I don't mind and I do this already and unfortunately, and I get lazy and out of convenience, I'll buy something through like the Google store and buy a movie so I can watch it whenever I want but there's still a concern that if something were to change with that movie I bought some licensing deal, if I bought it through Google or any service changes, could I lose access to it? That bothers me. And even worse, I mean, they could have an alternate version where Han Solo didn't shoot first and then you wouldn't have any control because it changed on you, right? Exactly, I need to know. Yeah, I need the version where Han Solo shot first not the other one that other people have out there. I mean, this could get bootlegged and changed and you know, I think it's such a huge conversation that may not really be the best fit for this but I podcast completely, but it kind of is where we really need to have a conversation about rights and digital media because it seems to me that people are in general, no, I'm not judging anyone in particular here but in general, the public is okay with losing things and it'd be one thing where I have a music CD and I accidentally step on it, okay, that's my fault, right? I shouldn't have done that, I broke it, I'll go replace it but if that CD that's on my desk was just disappeared because the record company didn't think I needed it anymore, what? I paid for that, are you gonna give me a refund? So if I went into a music service and I'll use this one example, it's the first time I've ever noticed this, Walmart, this is so long ago, used to actually sell MP3s a long time ago, so long ago in fact, I bet most people don't even remember that and I bought maybe one or two albums from them, I don't even know why and I got an email some time after saying, yeah, we're discontinuing that service so download your stuff while you still have a chance well, at least they allowed you to download it again but still it's like if you have your right to access media in someone else's data center they will shut that down, Nintendo is a good example of this, you buy games on the Wii, guess what? If that Wii breaks, you're done, all of your investment is completely gone, they're not going to keep that data for you to re-download because they don't feel like they need to and people should be fighting this because if you're going to gain control and hold the keys to my content then you need to keep it available to me or give me my money back but that's unfortunately not how it works at least with CDs and ripping DVDs and blueberries and things like that, you have a physical copy that if you did accidentally delete them or your hard drive died you could actually recreate the media infinitely you have control and I think that's something that I feel like a lot of people should really consider again in 2022 that yeah, it's nice to have everything in these media company servers available to us until it's not available anymore, that's not really fun. And I think it probably goes to even when you do the downloaded games on the Switch and things like that you're talking about, yeah, the same concept where you don't actually own the game anymore and it's not like Jay has a lot of the physical cartridges from the old-school games and things like that and yeah, you can play the Final Fantasy II anytime you want. Exactly, but the thing is, I hope I'm wrong but someday, like the Nintendo Switch is great, guess what, at some point Nintendo Switch is going to be considered retro, it's gonna be several Nintendo systems beyond that and you won't be able to download those games if that SD card fails that you downloaded those games to worse the system failed because it's locked to the system you will not be able to play those games again you cannot get those games back on the system again because they are some kind of serialization to the system that you're playing it on which is fine, now you call customer service my switch broke, I bought a new one and to transfer everything they'll absolutely help you out with that and I've had them do that for me but what happens when the Switch isn't really a thing that's being sold anymore, guess what, they don't care and there goes your investment so I think there needs to be like a bill of rights so to speak for if you're gonna sell me digital content then these are the expectations that you need to make it available to me if I need it again in the future or I think that's the least we could ask Yeah, absolutely, I think this one will stop with the list we have we have actually a few more and we'll save them for the next show and I'll throw out a couple answers in here because I've seen people asking about this and this is great directly related to the media topic and it's always a point of confusion someone says, hey, Tom, how's your phone talked to you and I use MB, I was using Plex I moved over to MB, I liked the interface and MB a little better and MB just worked better for movies I don't know why I was having weird stuttering issues and stopping issues with Plex it just would stop sending data, I don't know why I loaded MB, it works on the same server, no problem it's not a resource issue because neither one's using up that many resources on that system, but I did a video and me and Jay talked about this with VLANs the phone is an IoT device and so the Chromecast by which it will broadcast to I have the MB player loaded on my phone I have MB on the IoT network with my phone on the IoT network with my Chromecast and the IoT network those are all on the same network so they can easily talk to each other because the number of forum posts I have of, oh, I'm trying to get these things to bridge across multiple networks and keep the security is a headache and it doesn't always work well the portion where how do you get data over to MB or Plex is the question someone asked and it's really easy the NAS has multiple legs on it, so to speak we have a network interface that has a windows share and SMB share, if you will CIFS and the Samba share is on the secure network I drop the data over to the Samba share the Samba share is not available because it doesn't need to be on the IoT network because you're not sharing these over Samba you're sharing them through the program MB so MB is attached to the IoT network but the backend of where the data is stored within MB which is on my NAS is attached via a Samba share that's successful on a secure network it's as simple as that there's not, it's a nice secure way to do it and it makes your life a lot easier because you're not trying to figure out how to get all the devices to talk there and it makes MB stream perfectly fine there I can use the app on my phone the apps all look for local discovery on the same subnet so as long as you're all in the same subnet they all play happy together so hopefully that makes sense yeah there's all kinds of clever hacks and we'll definitely get to them I'm sure another DevRandom episode will have like quality of life tips for how to get your data to one place to the other sync this to that or whatever clever combination I think that's kind of the most fun thing about HomeLab it's like we have all these different things how can we combine them to have the most badass configuration we could possibly get in combination of services so yeah yeah so hopefully that makes sense and the last question I'll mention because it's just a really quick one and it rolls back to the very first thing we talked about our sponsor Linode someone says hey how can I look at my network externally I mean you can use things like Steve Gibson's shields up and look for open ports but if you wanted to dive into doing it yourself you can go ahead and sign up with Linode get an account going and use a tool like Nmap on a Linode instance to scan backwards towards your IP address because it's your IP address assigned to you from your ISP because someone asked well how do I know if I have any ports open and things like that yeah run a full port scan with Nmap externally is a way you can kind of test for your security internally testing for your security is a little different if you want to test application security Kali Linux is pretty much the go-to for having a complete set of tools on there runner up is going to be Parrot Linux as well that's for internal testing I mean you could run that in a cloud instance as well but it's a little bit harder to pull it back and you may as well if you're testing something locally a local application you may as well run it local for speed and things like that but just you know spin up a Linode instance to do an Nmap back to you easy way to tell if your ports are open so Yeah the way I see it there's a lot of people running Nmap against your IP why not join them why not join them why not do it yourself I mean everyone else is doing that right you might as well see what they're seeing Yeah absolutely so thank you very much for everyone who joined us today this was definitely a fun episode and we should listen Nmap is one of the tools in there because it's definitely you know what ports are open on this server what can I see from this it did my firewall rules actually work Nmap there's a easy way to do the testing in it so even built into PF Sense you can actually test from PF Sense and look at different sections of your network with it so Yeah you can do OS fingerprinting too if you don't know what the heck is this device that's connected to my DCP server Yeah try to do an OS fingerprint oh that's a that's a game system that's that's what that is or hopefully you know what it is after you run it Yeah and you can go I mean boy there's a lot of you know I haven't messed with as much but there's a lot of Nmap scripting out there that allows for some testing it's not as it's not a full vulnerability scanner but it will go a little further than parts open there's different Nmap scripts that can be added to it for some further insight so I don't know I don't know if we that's another tool that I don't know if there's enough to talk about it for a full episode but definitely a great tool overall to have in your toolkit That's right All right well thanks everyone and send us over some more feedback we always look forward to that we love doing Q&A episodes and right now everything's on schedule for next week we only had one skip so conversions of happenings sometimes happens with life Good news is both me and Jay had something on the same thing that caused it so we both had our day off at the same one Yeah some dev random caused us to take a week off and now we're doing an episode called dev random Yep exactly it's a full circle here All right thanks everyone, take care