 Welcome to the Home Lab Show, episode 84, choosing a Linux distro versus an appliance. Now we're not going to cover every distro. That's something we want to do next week. Well, we can't cover every distro. We don't have enough time to make an episode that long, but we'll cover some of the major distributions and why you want to do them next week. This week we want to start with the question of, that I think is a before question of, do I use an appliance or a distro? And the answer is yes, but it depends. And it goes both ways. This is a topic that is discussed quite a bit because, well, there's some great distributions out there. There's some great build it yourself that we like talking about. There's also having things like TrueNAS and just having it work and not having to do all the work it gets to build something like that. So there's lots of pros and cons we're going to discuss today and hopefully give you a few ideas of a few different things to try before we jump into that. We're going to take a sponsor this show and that is LaNode. LaNode has been a sponsor show pretty much since the beginning and I've been really happy with them. It's a great place to host all of your HomeLab projects that maybe don't belong in your lab, but maybe you want it in theirs. If you want something public facing, you want to run your own VPNs or any services that, well, maybe not suited or are going to get too much traffic for you to handle in your HomeLab, it's still a lab in the cloud and you can use the LaNodes cloud by using the offer code down below. We thank them for being a sponsor show and let's get started on this, Jay. Yeah, let's get started. So I think we should define appliance versus distro because you could argue they're the same thing because an appliance is a distro as well, but it's not a distro in the way that we think of one. So when we talk about Linux distributions, we talk about the common ones like Debian, Ubuntu, Fedora, Red, I mean, I can go on, there's like 10 million of these. These are obviously distributions that are essentially an operating system you can install on your server, you can install on whatever. They're general purpose. So I don't care if it's Debian, Fedora, CentOS, you can install Apache, Nginx, whatever your flavor is there, whatever your flavor database server is, it's general purpose. You can make it into a desktop, a server, power and IoT device. You could do whatever you want. And people do, obviously. And that's why we have appliances where it's really hard to start your own distribution. It's immensely difficult. Yeah. And it makes more sense if a distribution exists that's close enough or is closer than others to your goal, you're going to maybe consider one of those as your starting point. And when we talk about appliance, it's an application that is installed bare metal. Its focus is being an application, not a general purpose distribution. So XCPNG, for example, would be an appliance, even though it is based on a Linux distribution under the hood. And you can make that argument that there's no difference. The difference is in the intent, not necessarily, you know, the fact that it's a distribution, they all are, but sometimes the intent is to be general purpose. Sometimes it's just to solve one goal and be a turnkey solution. Yeah. And this is where some distros do blur the line a little bit. I think OpenMediaVault is one of them that I've not really used it, but Jay has. And it's interesting because they have two options where you can take and convert your distribution to run OpenMediaVault, or you can download an image of a MediaVault and set it up. So there are blurring of the lines, too, which, you know, makes definition harder for them. So you kind of have to really think about these on a case-by-case basis, which is why we're going to run through a list of them until some of the pros and cons of running them, the good, the bad, of, you know, to give you some data points when you're making a decision on which one you want to choose. Right. And I'll add even more confusion to that. Proxmox, add that to the list, too, because you could download Proxmox, obviously, in an ISO file, and you could also install Debian and then manually install the repositories and everything in the packages. And the instructions are right on their website. So it's not like you're doing something you're not technically supposed to do. It's in their documentation. So it's, you know, you could absolutely do that. But another question is, why do we want to draw the line between them? Right. And I think that's something we're going to get to, because I don't want to get, you know, jump to the end of the thing here. But just a little teaser for what we're going to do, because we can go over some of the distributions as examples of what or which ones are distributions, which ones are, you know, appliances. And then once we have that, we can talk about why is that distinction even necessary? Yeah. So let's talk about the popular ones really quick here. It's going to be Red Hat and, well, what used to be CentOS, Rest in Peace CentOS. Yeah. Well, CentOS Stream is the new normal, but not everyone wants to move to it. But I totally understand. It's not for everyone. Yeah. And then we have Rocky, which is, can you describe Rocky for me a little bit? Well, I like to lump all of these into Enterprise Linux. It's just what I call them. Rocky Linux and Alma Linux, they serve the same goal. And some people ask me, which one do you prefer? Both of them. I mean, they're both great. I don't prefer one over the other. So essentially CentOS, what it used to be is a recompile of Red Hat Enterprise Linux, which requires a subscription to use, but the source is open. So CentOS was, you know, famously that recompilation of the distribution to make it something you could download for free. And later on, of course, it got purchased in the rest of history. CentOS moved to a different format instead of being a direct recompile. It's more in the middle. It's not quite streaming, but it, I mean, it's not quite a rolling distribution, but it's a semi-rolling distribution. I'm not going to quite get into that. Yeah, that'll be next week's topic. It's complicated, right? But the goal of Rocky Linux and Alma Linux is to be what CentOS used to be. And give people that back, that had that in the past. And they're both solid choices. I'm not going to tell anyone to use one over the other. You could just, just flip a coin for all I care. They're both good. And, you know, you could add Oracle to the list too, but I don't, I don't have any experience with that. So I'm not going to speak to that, but my understanding is that it's also essentially the same thing. But I like to lump those into Enterprise Linux. So Red Hat, not CentOS now, obviously, Rocky and Alma Linux. And that's going to be great for your Enterprise, you know, situation or HomeLab, if you prefer to use that in your HomeLab. The reason I brought up CentOS is because it's a little foreshadowing when we get to XP and G, because this is where appliances can confuse you of, we just said CentOS is dead, long live CentOS. So, right. Well, yeah, this, we knew it, but it's been reborn into something else. Yep. Then we have your BSD, which I only bring it up, not as much as a big talking point, but just, you know, we're going to talk about TrueNAS and PF Sense, which both have a TrueNAS core being BSD-based and PF Sense being BSD-based. So those two are, you know, just another popular distribution series. Now BSD is obviously different than Linux, but you know, nonetheless, it still lends itself to some really good appliances. Yeah, it really does. And we have Arch Linux on the list and wait, what? Arch Linux? We're talking about servers. Actually, Arch Linux can legitimately be a great server OS. You have to go about it a bit differently though. Again, that's for another episode. But we have the new version of SteamOS, I believe, is Arch Linux. I haven't dived deep into it yet, but that's my understanding. And I fully support that. I think it might be strange for some people, a rolling distribution is the bait. Yeah, I mean, because you don't want to have to re-spin everything up, again, every six months. Again, being ahead of myself. But it is used in an appliance. Yeah, for Steam Deck to be considered an appliance. Yeah, I mean, people are installing all kinds of things on their Steam Deck that you could probably argue doesn't belong. But if you can, well, you can, right? Or can you? We'll talk about that later. But anyway, yeah, Arch Linux is the foundation for some things as well, believe it or not. So if you didn't already know that about the Steam Deck, well, now you do. Yep. So this covers, from there, you have your Ubuntu-based distributions. Or I should say Debian-based, but there is some variation because Ubuntu does do a lot of contribution. But I was grouped together as things that use apt-get. It's the apt-get family of distributions. That's fair. Yeah, it's an easy way for me to try to distinguish them of things that are using that package manager, which I've been a huge fan of. And the Debian world is pretty reliable, been around for a long time. But Ubuntu, being a popular one, because Ubuntu goes a little bit further, where they add some of the things in that maybe Debian doesn't, like better support for hardware and things like that. And Ubuntu has not been afraid to let us have a little checkbox, a little couple proprietary drivers and things like that. Doesn't mean they don't do controversial things, but they're a popular distribution from that. Now it seems like, to me, not as many appliances are based on Ubuntu. They usually go back and base it off with Debian instead of Ubuntu. Yeah, I think that's because of the, there's a little bit of a non-predictable nature. It's predictable, non-predictable at the same time it comes to Ubuntu. So it's important to remember, it's run by a company. It's going to serve company interests. Even though it's a community project, that's just what it is. So with Debian not being backed by a company, then there's less chance that there's going to be change that's going to disrupt everything. When Ubuntu finishes snapping all the things, for example, if someone that's developing something doesn't want to go down that path, or any other path that Canonical might choose, then they might stick with Debian. And I think that's probably why Debian is more popular for that. But then again, you could even argue, it could just be down to being based on the thing that Ubuntu is based on. Don't be based on the thing that's based on the thing, just be based on the thing. It creates a lot of confusion on there. Not confusing at all. Not confusing at all. All right, now let's start talking about some of the different appliances and why you may want to use each one of them. So we're going to run down the list here and we'll start with PF Sense. Now, this is where I have actually met someone who did this. I was impressed. You can build your own firewall. You can take BSD and the PF filter is a very powerful filtering system that's been around for a long time in BSD. BSDs had a long enterprise history because of that. And PF Sense was originally MonoWall, which forked into PF Sense. And it's just been a solid performing with lots of options, firewall. All these things you can do for yourself from the command line, I won't argue with people who do. But one of the consulting jobs we had that was interesting is they were having a hard time hiring someone because this person hand built an incredible firewall for a very large company. And they were the only ones who knew how to do it. And they kind of realized they trapped themselves into being the only firewall manager. So this is where things like a firewall distribution like PF Sense become a very popular choice because it makes ease of management and not happen to learn everything from the command line or all the little switches it may take to set up a IPsec VPN from the command line or anything else. It puts them all in an IGEY. It also makes it really easy to have a technician if you need to rebuild a firewall. And this goes for a lot of appliances. The ability to load the appliance software, it's customized very specifically. They trim all the fat. They're going to put into the appliance just what needs to be there. Then they usually throw a way to manage it. Most of the time it's going to be a web UI on there in this most common way. And then this is the best part like with PF Sense, a single XML file that has all my settings in it. So if I need to reload it, I just need the latest download of an ISO. I need the XML file. Those are the two things I need. And I can have a almost junior level tech reload and rebuild a firewall. But if you were to build a firewall by hand, you would also have to build the backup by hand, the restore procedure by hand. You'd have to teach someone how to load the OS, set the packages up that need to be there, and then start putting all the config files back where they belong. You could also write an entire automation for that, but you can see why how much of your time should be spent on one thing versus another comes down to whether or not you want to run an appliance or not. And PF Sense is like a pretty low hanging fruit one that's an easy example of probably best to run your firewall as an appliance, not build it. Building if you plan to get into learning how firewalls work from an extreme fundamental level and really want to understand how to build firewall rules or NAT or anything inside of the command line, that's great. But generally for, especially our audience here in HomeLab, you're going to want to run it as a distribution. Now, I could throw out there that untangles another one. I'm not positive they have a free version anymore. And I think there's a few other Linux-based firewalls that are still floating around out there, but five absolute huge margins. PF Sense is one of the most popular firewalls out there that's open source, that's free. You know, even their PF Sense Plus, I know someone will point that out. They give you a license for it, but you can still get in the 2.6 and 2.7 versions of PF Sense for free. There's still continuing development on that version. So despite everyone tells me all the time that they're not, which always makes me laugh. Every version's been the last version since 2.4, according to the comments on my YouTube section. Well, there you go. But yeah, that's, you know, the firewalls probably said low hanging fruit for one of the, just shows out there. What about Proxmox, Shay? Let's talk about that one versus just running, because it runs what KVM and underhood, right? Yeah, QEMU, KVM, Combo. So Proxmox most often is going to be downloaded as an ISO that you install bare metal. And I think everything on our list, well, most of the things on our list will be installable that way. But like I was alluding to earlier, you could build it yourself. If you have a Debian server already, you could just start going through the documentation on their website and they'll tell you how to do that. So that's almost like a hybrid approach, because it is both. You know, it is a distribution of its own, but it's also built on top of Debian proper. And the difference is some of these distributions that are based on other distributions have their own repository. So something can be based on Debian, as in grabbing Debian's packages, putting it in their own repositories, curating only the components that they absolutely need for that product. And that's what you can apt install, not anything outside of that. Or you have, you know, like in the case of Proxmox, the proper Debian repositories, and you can install whatever you want, which also means you can install some games if you wanted to. You could install GNOME if you wanted to. I'm not saying you should do that, but given the fact that it has all of Debian at its disposal, it can, you could do that if you've decided to do that for some reason. So that's why it's more of a hybrid. Instead of like, you know, just curating their own packages only, they, you know, have pretty much everything. So yeah, it's kind of, it's kind of nice. I mean, you can load all this stuff, but Proxmox makes it a lot easier to load it, give you a nice web UI on top of it to manage all of it. So plus one for that. And the, just being able to apt-get install whatever you want makes it very flexible. And I understand completely, even though I'm not a big Proxmox user, I understand completely why people would like to use it in their home labs, because that flexibility is awesome. So I definitely, definitely like that, but, and this was a potential consulting that came in today, and not the first time I've run into this. I don't know what they did. They installed a lot of other things, as usually the result of these consulting jobs, but someone says, Hey, my Proxmox cluster is not performing the way it should. I've got all these problems. I got all these weird issues that randomly reboots. And I have done consulting where people, and we try to, we don't do much consulting on it, because it's like, I need to know what you customize. And the answer is I don't know, or the person before me changed all kinds of settings. This is a big downside to allowing people to do it, and then anyone may or may not do it. So you kind of kind of take that into consideration. It's a lot of the danger that comes out there. And yeah, it's just one of those big warnings that come with these appliance-based distributions is just because you can customize them, really be conscious of what you're customizing and that allows you to, that may allow you to break the performance of it or just cause some weird incompatibility with the package you loaded. Yeah. And I think that's the main point here. I mean, there's some things that are very reasonable. So when it comes to Ansible, I manage everything with Ansible. There's some things I don't. The appliance is I don't. If it's going to be a system I built, sure. But if it's a true NAS, it's not that you can't manage it with Ansible. Sure, you can. You can manage anything with Ansible. Should you? Well, maybe, but what if you have a change in your Ansible config for your other servers that accidentally finds its way in a different role and then next thing you know, you break your entire system? So that's kind of the thing. But a more, I think, agreeable change in my mind. I don't think anyone, even the support forums I have a problem with is what I do, for example, with Proxmox is I'll install the NRPE package for Nagio so that it can monitor things. I think that's reasonable. I'm not reinventing anything. I just want the monitoring system on there. And I don't think anyone's going to ever have a problem with that. But if I think the main issue is sometimes you'll have people in forums, oh, I might buy Proxmox isn't working. I'm getting this error and then somebody responds. I've never seen that before. How did you manage to get that error? Then like three pages down, everybody's trying to troubleshoot it. Nobody could figure it out. And like, wait a minute, you installed Proxmox on Ubuntu. How did you do that one? And number two, why didn't you mention that at the beginning of this whole thing because we don't support that? And it's kind of how it goes, right? If you decide to take an appliance into a different direction, then I know some people disagree with me, but I really feel very firm on this. You are the support at that point. Don't go to the forums and ask them to support your custom Ubuntu Proxmox. The Proxmox forums is there to support Proxmox proper. At least tell them before you start going on. Let them know that because that can be a little frustrating for the people that are trying to help. And you have to draw the line somewhere. I've seen people post on some forums, like maybe they'll post on the Arch Linux forums and they're using Manjaro. Maybe they're feeling like they didn't get a response fast enough on Manjaro's forums. So they ask the question over on the Arch Linux forums and you'll have, and these people are smart. They'll look for your question in other forums. So, but getting back to the point, when you take an appliance and you just add all these customizations to it, it's kind of like, at what point is it no longer the appliance? It's kind of like that ship experiment, right? When you replace one board at a time, at what point does that become a different boat? At some point it does because you've replaced every single part, but it operates the same thing. It looks like a boat. It floats like a boat. But is it actually the same boat? No. And at some point, it becomes a completely different thing. And I think that's one, a decision you have to make. Is that something that you want to do? And, you know, is that something you can get support on because you might be kind of making it hard for yourself? Yes. And let's switch over to XCPNG, which takes a different approach. Now, this is where CentOS is going to come back because people, and I've had a misunderstanding, I should say with people, where they go, oh, man, I have to get rid of all my XCPNG because it's based on CentOS and CentOS and supported. And there are numerous blog posts and very detailed of exactly how this works in the blogs for VATES and XCPNG specifically. So VATES is the company behind it. VATES manages the repositories that drive XCPNG. So even though it does have YAM, it's CentOS based so we can do YAM install, you can't install anything. They didn't put in their repository. End of story. They've crafted, because this question comes up, is someone will say, why is this version of this in here? And because I want the next version of it or something else. And he said, this is the version that we know works. It's about stability. So they craft each thing. They craft each kernel that goes into it. So even though it's CentOS, you're like, well, doesn't CentOS not get updates? There's regular updates almost every month for XCPNG for new features, new versions, and sometimes OS updates. That's because when a company takes the time to exclude all other repositories and have their own upstream, PF Sense does this as well. You're not pulling from BSD. You're pulling from their crafted repositories to give this. This gives you a more predictable experience because it would take a great deal, not a great deal, I should say. It takes a lot more effort though to add all the extras in there to break something. This makes support costs, so to speak, lower. The number of questions in there. And like Jay said, if you do anything like that, you've decided to enable repositories that are going to pull in packages, I think it will just break if you do that because once you start mixing repositories, it'll go, I'm not sure which version to use, so let's just grab the new ones. And if their custom-based appliance wasn't designed to do that, it can break quite a bit. But back to XCPNG, one of the nice things about it is once you have, and anytime there's a new release, they also spin a new ISO of it. So you can always grab that ISO and quickly, if you have a server go down or as we have a lot of clients sometimes, I think we have one of them has like 30 or 40 of these servers. They can replace any one of them at any given time because it's just an ISO load. Once again, it goes back to a service ability. I can have a pretty basic instruction set like grab ISO, load OS, wipe the machine, join back to pool. All the settings will come back because they're held within the pool as a collective. So all the settings come right back, done. Now we don't have to monkey with anything getting it set back up. I don't have our long install procedure. But XCPNG is nice because they allow you to build the server where it doesn't matter. Like the server is very replaceable, very quickly replaceable. You're not worried about a hardware dying. You're not worried about reloading it or joining more to it. It's not this labor intensive process. So XCPNG, I like the approach they take of customizing a very focused repository without a lot of fluff in it. Makes the install very small, keeps you from wandering off the path without a lot of effort. So it should just work out of the box as expected. And you are getting support and updates. The team's very clear about that. So even though, yes, you can look at the OS version and scratch your head going, well, it's not a currently supported version, but you can trust that it is currently supported. And I'll mention this back again with PF Sense. The 2301 PF Sense is out in terms of release candidate. And so is 2.7 is close as well. Both of those are based on FreeBSD 14. Now PF Sense is also customizing your repositories as an appliance, which means even though FreeBSD 14 doesn't have an expected release to later this year, you can still have a stable firewall because the people at NetGate are hand crafting the kernel for PF Sense and customizing their appliance to work with a stable version of it, even though the release of the full operating system, but you can't remember, it's not a full operating system. So once again, you got to kind of think about that when you're looking at appliances. Yeah, I agree. And also what I said earlier about basically, you are your own support if you, I want to say fork a product or because essentially you're forking something, if you take it like a turnkey appliance, distro, and you add repositories, you forked it. I mean, that's what you've done. You've created your own thing. And then you are the support for it if you decide to do that. You have every right to decide to do that. You are free to do that if you want to do that, but you are the support. But the same thing goes to companies as well. If they decide to base on a distribution and build something out of it, even a distribution that's not supported by the upstream vendor, they are the support. In your example, XCPNG is the support. They are responsible for applying security patches if there's a security issue. They're taking that on. And that's not, I'm not saying that's a bad thing. That's a decision that they make. And from what I understand, they're doing a pretty good job of that. But whoever forks something, if they break it, they keep both pieces. But just because a company is based on something that might be end of life, then that doesn't really mean it's end of life in that product. And famously Ubuntu has many times in the past released LTS distributions without, or end of life kernels from day one, but they support it. They put the patches in there. They cover the CVEs. Even though it's a lot of work and I don't think they should have done that, it's their right to do that. And as long as they're keeping up on security, vulnerabilities, that doesn't mean that you're good to use that kernel and any other distribution. It's end of life. But if Ubuntu is taking that on and their security team is keeping eyes on it, then there's no problem with that. Yeah, the other fun one to talk about next is going to be the topic of storage servers. I think this is where the arguments go crazy. So there's good ones on this. I go both ways when it comes to when we set these up because it comes down to it depends. And I need to do some more updated video on what it depends on because there's more things to talk about. So you have the Houston OS, which makes managing ZFS a lot easier when you have large drives. This is from the team over 45 drives. It's open source. Any of you can use it on any Linux. I think it works on a multiple of Linux distros. They base it off a Debbie and when you get a 45 drive system, but they're flexible. The same will work, I believe, fine on CentOS or several others. It's based on cockpit. But what this does is gives you a nice web UI for managing your storage. And there's some good advantages to it. I just did a project where we have, we needed 300 terabytes of beam storage. It's all going to go to S3 buckets. And I don't really need to run TrueNAS to run that. We actually just ran a whole thing with the 45 drives Linux space. The people we built this for, their internal administrators, very familiar with Linux. They like Linux. They are happy with it. They actually threw some ansible things on it to monitor it. They've got like, it just integrates into what they're already doing. And that's a great excuse to build a storage server without going through building a TrueNAS appliance. They were a little sketchy going, TrueNAS seems like we wouldn't be able to do what we wanted to do. And they were not wrong about that. Because the TrueNAS distribution is going to be more turnkey. Now, when you move to something like TrueNAS, first we'll split between core and scale. So core is going to be your free BSD based one. It's tried and true. It's great performance. The Jail system is okay. But there's not a ton of apps for the Jail system. But when we use that, TrueNAS core, like our choices for that, are pretty much like someone wants a dedicated, often for virtual machine or large scale storage server with ZFS that's reliable, easy, and very simple to do your updates, very simple to do replacement. You just grab the config file. That's it. Load the appliance, load the config file. It's great. Everything's just there. Turnkey, you have backups, you have ZFS replication, all of that's really nice in the UI. And all those things you set in UI, all download to one single config file, making it extremely easy to build and replace. And you don't want to spend too much time customizing it because it's just not designed to, because TrueNAS curates the feed that goes into the updates in there. TrueNAS scale, same answer. TrueNAS scale now has support. They're using like a customized version of Kubernetes and Docker, a lot of custom implementation here. So you can run more containers on it, jails is called in the core series, or they're essentially just Docker containers. And they're very customized. It's not exactly native Docker. They went a little different there. But what they are doing is, once again, creating a turnkey appliance that is great for managing ZFS, great for having your shares set up, your Samba settings, all those little tunables, having a web interface, so you can set up ISCA's real quick or an NFS share or a WebDAF share, any of that's all done through the web interface. So there's a good argument to use those because I want to just create a turnkey solution that I drop in and I don't want to think about how do you actually write all the Samba stuff? How do you build all that? So it kind of goes back and forth of it depends on which one of those is the right path for you. If you want to have lots of customization, you definitely want to go with a Linux rod on there. And I think the huge deny is nice because it will allow you to just start customizing it. But kind of as Jay alluded to earlier, you're very on your own for support and going through forums because you're really diving into a custom solution of doing it. The tools, the web interface at Houston United does make setting up Samba a lot easier or setting up some of the replication and snapshot features, which completely you can learn how to do it from the command line. You can do it all with cron jobs. Do you want to? That's the question you're asking yourself when it comes to storage. You can do it. Is that where you want to spend your time? Exactly. I mean, that's a great way to put it. And then another hybrid approach we kind of touched on is open media vault. I mean, same thing, right? We're talking about storage. And I feel like people asking for support in open media vault forums with a manual system is going to be way more common because they actually offer a script. You can just run on a Debian system. So they wouldn't create a script if they didn't want you to use it. But also you could still run in issues all the same. I mean, if you custom compile some things and then their script tries to install an app or a dev package, essentially, for that same thing and there's a conflict there that can happen because that's the nature of customization. You could do whatever you want to do. And will the script work? Most likely, but it can't account for anything, although I found it to be very stable. But it is a hybrid approach. You could just install it on an existing Debian server and maybe that's a good fit for somebody who already has data on their Debian server. Maybe they have a bunch of files on there and they manually did everything. And then, oh, maybe I would like to just convert it over to an open media vault server. What very few do well is having a direct sync between the config files on the file system and the UI, meaning it's like this unicorn that we wish everyone could do where you could just hand edit a file. And then the next time you check the UI and the web console, it has the most recent settings from those files. And if you change it there, it's changed on the underlying files too. That's hard to do. So there's always going to be, or most of the time, a disconnect there. But that's just the nature of the beast. But then sometimes, if you hand edit files, then why did you install a web console? Because now they're out of sync, and then what do you do at that point? So, but open media vault is the hybrid approach that's kind of expected. I haven't tested that as much as I've tested the other ones, but I have used it. I've done videos on it. And that's, again, an example of a hybrid approach. It's both a distribution. It's based on Debian. You could install it on Debian, but you could also download an image and run it on Raspberry Pi if you wanted to. Yeah. And I think I've seen someone through this in the comments I was going to mention. Cockpit's interesting because it is a hybrid because it doesn't save everything to a database. Every time you edit, if you're editing a Samba config, it's actually editing the sd slash Samba. When you edit different facets, it's pulling from there and doing a pretty good job of writing it back in the proper way. I'll throw out there. If you've ever used Webman, Webman will read config files. It writes them not always in the right way. It's been a while since I've used it. I remember it really making a mess out of anything you do with Apache. Like once you've edited an Apache config file in Webman, it's not very clear. It doesn't do a great job of formatting the file that it actually writes back out for that. So if you ever want to go back to the command line one, I remember having to support stuff like, oh, you use Webman to set this up, didn't you? Because it's just a whole bunch of series of numbers, no logical naming, no nice format. So there's kind of a back and forth. Maybe it's gotten better, but yeah. The appliances generally don't like you munking with the cockpit kind of being the exception, but you generally don't. You don't see cockpit running on an appliance. Right, right. That's true. And I think that's a point, that there's some exceptions. But I would like to see a future where those things are more in sync because for the enterprise, you're going to have a larger enterprise, people of different experience levels, or even just preferences. Someone just might want to use the command line because that's what they're comfortable with, and someone might prefer to use the Web UI. It's fine. But if you have two different mindsets in the same team, then which one wins? Do you cater to the person who's just starting out, or do you teach them command line stuff? If they could both do their thing the way they like to do it, that's even better. But having that consistency between the Web UI and config files, that's really hard to do. Yeah. All right, what's the... We had another one on the list here. This is... I'll mention this because there's a few variations of it, and I can't remember all of them, but people ask me about this, and it's kind of interesting. ClearOS is one of them that's out there. Now, ClearOS was an independent tool that kind of... The idea was to replace the Windows small business server with it so you could have like a turnkey operating system that did all the things. It's late... I haven't tested in years. HP has since bought it, and they've kept... I'm not saying they still have a free version, a community edition that you can download. It's also based on CentOS. It's kind of a neat distribution. I'm always mixed on those because they go back to that... You want something turnkey that will just do all the things for you. It will manage your storage and all your settings and be like a centralized server for all of your users and everything else, but that can be a little bit... I've seen a break. That's how my interactions with it are usually from people who call us because it seemed to be popular in schools for a little while that couldn't afford or didn't want to spend the money on some of the standard Microsoft Active Directory stuff to do all the management of all the students and things like that because of licenses and Cales. And I get it, but usually it broke because people tried to color outside the lines a little bit. They started customizing and adding more features on it. Those issues are kind of an interesting one out there from appliance standpoint, but I've never seen any of them that I would say is robust enough. Like they always have these lofty goals of throwing everything possible into one megabox that does all of it, but I've never seen those hold up in some of them. And I can't wait to see if Clearhouse is one of them and features. They also throw the firewall in with all of it as well. And I'm like, in concept, that sounds good, but because I can have one box because they're even their slogan right here, one platform, one interface. That's kind of a lot. I wish. Yeah, that's like the ultimate appliance. So it's got cloud apps on it, network apps, gateway apps, server apps, system apps. It's the everything. It's been a while since I've played with it, but that is that's where I will say, in the reason I have it, the reason you don't see me running around seeing in praise of these, I've never seen these work extremely well. They try to promise too much. They try to put a lot into it. It takes a lot to manage that because you think about how hard it is just to manage the PF Sense project as a firewall. Think about doing a firewall and a storage server and then user management and then VPNs and everything in one single box. Once you start getting into it, I use clear as an example, they're not the only one out there doing this. I always shied away from a lot of those. It's also scary that your edge device, if you're using as a firewall, is also where your storage and users live. Insert your favorite movie where in, we broke the firewall. But now they didn't get into your network and have to figure out how to get into your NAS. They're in both. To me, it's just at some point, as much as you have a lot of stuff you want to put in your firewall, there comes a point where maybe my storage server shouldn't. My data is really what you're after. If you're trying to get in, whether you're trying to encrypt it through some ransomware, you want to exfiltrate it for whatever reason because the data is the gold, whether it's to encrypt it and extort me for it or take it because it has some value. Having it on a separate server means if they get by the firewall, if they impersonate a user to the VPN, there's still one more, hopefully, defense that you have good passwords on however you control user management on your storage so you can not have them in it. So I've always shied away from some of those all-in-one mega boxes, I would call them. Always a locked goal. Then they will sing the praises. It reduces your IT costs because everything is just one server. Right. We should, I think they should rename it UnicornOS. Yeah. I think that'd be a better name for it. Maybe I'll just call these Unicorn distros I mean, I hope that's something that someone's able to do but I look at these like from the standpoint, I've seen so many of these come and go. All these promises of compatibility with this and with that, I think that's why the Steam Deck is the true Unicorn because it works very well. Then again, that's a single-purpose device. These all-in-one things. It's like, well, that'd be awesome but do we want that? Is it a good idea? I think Ian Meltham said something like, if you didn't even bother to think if you should, you could. Don't know if you should but I'm not trying to say that everyone on this project should just give up. I mean, if they feel like they could pull this off, they should continue working on that for sure but I just get a little, I don't know. I've just seen so many of these come and go that it's just hard to impress me at this point unless they do it, then that'd be great. Yeah. It's like I applaud the effort. But it's not something I'm ready for. Honestly, when you look at the overall cost, if you're thinking from a business perspective, outside of some power concerns, because I know European power is more expensive and I know I have a European audience, so I feel that for you. But the compute power is relatively cheap now. Firewall appliances are not incredibly expensive. Even the NETgate ones and things like that, I know we're going to have some import prices but that's not usually where the biggest time sink is. So having some of these as separate devices, might just make more sense and it's not like, the cost savings isn't there when it used to be. Hardware used to be substantially more expensive when you started adjusting to today's rate. I mean, especially you go back far enough, it's almost incredible that anyone had a Commodore 64. I mean, you're talking about a computer that costs thousands in today's dollars that did very little. Now we can afford these really inexpensive, small mini PCs. You can find used equipment for cheap. So it's not the problem that I think it used to be when it comes to cost. The technology costs, economies of scale, popularity of these things, they've driven way, way down. So having these things separate might make a little bit more sense. And hey, if you have a storage server update that you need to push, you push it and have the users offline. But when you have it all as one, you're also rebooting your Firewall to reboot your storage server. So it just seems like everything can go wrong there. Another aspect is security. Because what if company ACME comes out with a project or whatever comes out with a distribution that is all the things. And everything does work 100% and it's great. They've succeeded, but then they also have like a couple hundred CVEs that they've created by wrangling all that together. And then next thing you know, reputation goes down because it's something that people are able to easily break into. Then I mean, there's a tug of war here. I mean, you add custom code and you keep doing that. But all the custom code you add can create CVEs, which is why I squirm a bit when someone uses this insert name of browser here that has like two developers. And I make the comment, well, it's probably not secure, but it's based on an open source browsing engine. That's great for the browsing engine, but the UI is custom and there's a possibility that CVEs can be created or exposed through custom code. So hopefully I'm wrong in the projects. That are great, but it's just again, it's really hard seeing what I've seen when you get to this point in your career. You have the eager cadets and then you have the general that's been there, done that, right? It's kind of like the same thing here. It's like, oh boy, here we go again. Yeah, it's funny before we started this show, I ran over to Distro Watch to take a run through memory lane on some of the distros. But that'll be our topic for next week. But I rolled it all the way back to 2002 and go, oh yeah, those distros are trending then. I mean, my Linux career started a long time ago just like Jay's. So we've been doing this for a minute. We've seen a lot come and go. We started at the point where we thought Distro Watch was a clear answer to which distros are the most famous and then you graduate to later in your career realizing it doesn't mean anything. All these different things you learned, I think Distro Watch is a really good metric because how you think of Distro Watch is pretty much identical to where you are in your Linux career, basically. Yep, I will answer a question someone had in there. With ZFS, yes, you can actually move with ZFS from appliance to distro and back provided that you aren't on the wrong versions of ZFS. ZFS can be imported to different appliances. So there actually is some fun you can have with storage servers like that of trying things and then bringing your pool in, but you have to make sure you're matching versions. You can't go forward. So if your ZFS pool is built on a newer version of ZFS, then the appliance has available. It's easy enough to do if you're doing it from a distro. You're going to grab the latest, but if you're doing it from appliance, I'd make sure the appliance is probably what you want to build with first because you can ZFS is backwards compatible or not forward. So you can have some problems importing pools. So I'll mention that for people that want to play because I think that's a pretty relative question because it's one of the exceptions because for firewalls, you're not necessarily going to take PF Sense. There's no exporting config files because PF Sense says everything in a XML file. So you're not grabbing the config files and building your own PF Sense firewall off of what it wrote in slash etsy slash whatever. You're building it by hand again. There's not much interoperability between firewalls, but hey, there is for ZFS. Your data is separate from the operating system if you're doing it right. So I think ZFS is a unicorn that actually exists. ZFS is a unicorn. It is magical for sure. But it actually exists though. That's the difference. It does exist. I've sung the praises of ZFS. That's why I have shirts because I've been since being a cult member of ZFS. So I said, yes, we're a cult with integrity. So I do have those. That's good. I like that one. I like that one a lot. So I do have those shirts. People want to be part of the ZFS cult with integrity because I don't know how else to say it. Yeah. All right. Well, thank you for joining us. And we left it up, but we forgot to say at the beginning, but we do have feedback at the HomeLab show. So that's feedback at the HomeLab.show. We love hearing from you. We've been trying to get more of these Q&A episodes going. We want to make sure we're engaging with the community as best we can. So drop us a line, drop us an email. We like reading the questions. And it gives us some more things to talk about and maybe help some people along their path and their HomeLab journey. So it all started with us. And we're trying to pass it on. We know a lot of people are just now getting into it. We want those people who are just getting started. Let's ramp you up faster. And I'll get you started in this HomeLab fun. Oh, yeah. Fun times. Fun times. Yeah. It's a man. There's so much to talk about, but we'll save that for next week because I'm excited to talk about the distro one. Do you see me listing it? He's very, very excited to talk about it this week. I'm like, oh, no, that's all of next week's topic. So. Oh, yeah. Distro hopping. That's what we should call it, man. Distro hopping. Oh, yeah. Yeah. I think there's value in doing it. And we'll talk about that too. We could just call it the distro hopper starter kit. Distro hopper starter kit. All right. That'll be fun. See you everyone next week. Love hearing from you. And if you have a favorite distro, bring it up because maybe there's something we haven't heard of or that you like for some particular reason. Let us know your favorite distro. Feedback at the homelab.show and thanks.