 Welcome to the homelab show episode 11. We're gonna dive into true nas today, and I'm here Tom Lawrence and with J. LaCroix. Yes We've talked about storage servers in general and we're gonna be driving down Specifically to true nas and today is June 2nd and on June 1st 12 true nas 12.0 u4 was released So yeah, I'm excited because I was already reading all the you know every time there's a new release There's more exciting new things a few new fixes and things like that and at least one new bug to report But either way we're excited to dive into this topic. It's really popular We know among especially those of you that hoard data You need a place to put it and true nas is a popular choice Before we dive into this episode, we do need to think a sponsor of the show And that's gonna be the node and we even have the link right now because I In the very first time we had we had it wrong. We got it right We did fix it. So if you clicked on the link and we're wondering about it. It's correctly in the show notes right now, so But this show is literally brought to you by Linode. We host our website on there That's where all of this is if you're downloading the podcast and it's pulling from there And Jay's gonna tell you a little more about Linode and the offer they have Yeah, so I've been using Linode for I don't even know how long I want to say two years But I don't know I lose track of time especially nowadays So, yeah, like Tom said the website for this this podcast is on there is hosted by Linode So like you said, it's literally brought to you by Linode and we mean that if you are on the website You are using Linode and we chose Linode because both Tom and I like it I probably had a little bit more time on it because it's been the official Sponsor and actually the first sponsor for learn Linux TV and the infrastructure provider for my YouTube channel for a couple of years now And my rule on my YouTube channel is I only have sponsors that I actually like that I actually use so I literally Use them on everything and it's been a great experience So with the offer code you get it's actually $100 in credit towards a new account and that's good for 60 days I believe so you can do all kinds of crazy home lab Experiments on the node and I do consider Linode home lab even though it's not in your home Because as a home lab person you make a decision What do you want to host on your hardware and what do you want to you know? Put on the cloud or make externally available and you could argue that it might even help security by Excuse me giving you the option to make things publicly available that are not in your home lab So you're you're not exposing your home lab to the public internet So there's all kinds of cool things that you could do with it Yeah hosting servers and all that stuff if you don't want it on your public IP address put it on their public IP address So I want to thank Linode for sponsoring the show and it's it's been a great service They've been great to work with it. They've been nice people too. That's that's all their side of it That's behind the scenes Yeah, we met on my penguin con. Was it three years ago now? I want to say it's been a while Wow, then it feel time flies and at least one year feels like it didn't happen, but Well, but something did happen in the last year and it's the convergence of free nas and true nas and That's actually been really cool. So going back on that project is you know free has been around a long time and they also maintain a commercial version which is still the open source code but They distinguish them So if you wanted to buy enterprise level support you got their true nas system and you can buy it on the true Nas hardware and they support it all the fancy enterprise things like service level agreements and monitoring Etc things that the enterprise business world wants but in reality is this the same software that runs on free nas and This created kind of a problem for them because maintaining two almost identical code bases To distinguish it to was kind of a pain and it took a long time for them to merge it all together And that is what true nas is the name they decided on and so unfortunately to drop the free nas name and unfortunately They got rid of the shark logo, but I think there's still a checkbox. You can make the shark logo come back So with an updated logo and a converged code base any converged Documentation base which of course is important as well. We have the true nas system. So free nas didn't die It kind of merged. It's all the same team it's still the same great open source product and That's where we are today and as I said in the beginning the your latest version 12.0 u4 was just released on June 1st of 2021 but before you get started with true nas the first question is what do I run it on and that's Where the will wander all over the place because I won't say it'll run on anything, but it'll run on most anything I mean you can it's pretty based on free BST You have the ability to run it on a wide variety of hardware The support is relatively broad not as much as in the Linux community And I bring that out because someone will say what about true nas scale because are you guys going to talk about that? That's still an alpha. I look at all the latest updates We will probably do that as a completely separate episode once that gets into at least a beta once it comes out of Alpha and into beta. I think it deserves it. It's basically they took and modularized the Interface for true nas and said alright, we're gonna have one that's based on Linux Eventually, which is gonna be true nas scale for some slightly different use case And we're gonna have the tried and true BSD base of true nas core And that's the one we're gonna be talking about today because it is in full production release Yep, and Tom and I both use it. He's been using it longer than me But I can't remember when I switched to it. I think it's whenever you did a I think you did a talk at Penguin con so here I am talking about penguin again But basically before that I was rolling my own I mean literally I think it was Debian that was my my storage server and I wanted to create a share I would just SSH in I would create the config file restart a service or whatever I had to do depending on what it is I'm sharing with and That was okay with me, but then it got to a point where you know I don't really have so much time to manually tweak everything and true nas for me It was just a really easy way to get going and there's a lot of quality of life improvements They put on top of it like being able to back up and restore your entire Configuration you can mirror the boot volume to another drive. So, you know, the primary one goes down In my case, I can clone Zilla the you know boot drive if I wanted to have an image backup of it You can you know put you can basically have it send all of your data off-site to a External provider for backup and you can of course encrypt it locally before you send it up there There's a lot of great features. Yeah, and you know, I learned a lot doing Samba manually because I did the same thing I rolled my own Debian file servers and you know, that's how I did it before I got in the Free NAS when it was you know, I originally started the into the project But yeah, it's um It's so much easier in some ways and it just consolidates everything in a one nice You know web interface to be able to control a lot of advanced features And those features have only gotten more and more advanced as the project has progressed over the years the first question I think though is what hardware should I buy for it and This is not just a true NAS question. This is also a ZFS Concern want to have whatever hardware you're using this in has to have direct set direct access to the hard drives That is a whether you run ZFS yourself on whatever role your own hardware Or you choose to do it with true NAS that same parameter as applied that we need direct access as in we don't want a raid controller There's a misconception when you're starting out in homelab a lot of times that any time There's a card that controls 20 hard drives. It must be a raid controller, but it's not that would be a Controller, but if it just passes through the drives for connectivity Then it's not a raid controller and that's fine That's what you're looking for now some this is where you can go down the rabbit hole of firmware and there's a cup There's a eBay seller who has a YouTube channel I can't remember his name at the moment, but he offers like how to get these Certain Dell ones reflashed into different modes. That's cool We won't go spend too much time on that basically However, you get to that point where it's direct access hard drives Or you just buy a board that has a handful of SATA ports away you go and though That's at least something I just can't stress enough for problems We've helped people through in forum posts that are all over the place of I can't get it working with this You know insert name of used server day found, but they also have a Setup as a raid controller, and they can't figure out why I can't see the drives Yeah, I think the lesson is that it does work on you know, not everything but quite a few things but don't impulsively buy something just making the assumption that it'll work because I Was burned on that too. I found out the hard way about the raid situation and I found someone on eBay like Tom mentioned, you know This is a thing and I literally searched I thought I searched for eBay for a raid card Freenas it was Freenas at the time and the first result was just an individual here in the United States That all that person does on eBay for what I could see is they just flash the raid cards And resell them and yes you could do that yourself, but I was looking at it like I had a lot going on I'm like I want I want to press the easy button. So I just ordered it It didn't cost me very much put it in and it was fine Right. The other cool thing is if you spend a few minutes in our forums they they have a category called will it Freenas and Much like the name implies will it Freenas is a hardware discussion of what is some good supported hardware? So if you spend just a few minutes researching ahead of time You can find a good set of hardware plenty of compatible things matter of fact If you go over to like eBay if you're looking really on a budget and there's some good You know commodity servers out there usually by super micro But if you type in Freenas or TrueNAS into your eBay search You'll actually find systems that are already set up in the mode to work for Setting it up with TrueNAS. So there's a lot of good ways to find hardware on there It's just one of the things that you try to before you just run out and buy something because it's got a dozen drives And at a good price double-check that little bit of compatibility save yourself a lot of unfortunate headache Is those things the shipping cost quite a bit and you don't want to have to ship it back? Now right the next question usually is can I just virtualize it and I would say in a lab environment Yes, in a advanced use case that you're really willing to put the time into troubleshoot some of the problems that may come with Virtualizing it yes, but once again that direct access to hard drive dovetails into this because when you virtualize it you want to make sure you are passing through the Controller so whatever you're using as a hypervisor, and you go hey, I want to virtualize the install Okay, you can virtualize the boot device and then you pass through into that VM the controller now there has been a recently some Bugs that have been fixed in that area. So they've been getting better I think one of my friends had some trouble with XC PNG and They've went in the BSD community found some bugs and some of the drivers that happened when you virtualize it So there's always extra because you're adding an extra layer that you can cause a lot of problems So my general answer is don't do it unless it's just because you are curious and want to see what the latest version of TrueNAS scale looks like you want to play with it Yeah, I virtualized it for that not in production or you're someone who's willing to put the time in to Solve all the extra parameters and problems, but if you're brand new to TrueNAS. Yeah, you don't want to start with I don't know this product where is it my lack of knowledge on this product? Or is it all the other problems that came with virtualizing it that are causing all my drama right now of getting this to work So I just like to remind people on That part of our virtualization on there Yeah, I feel like my opinion is I'm not a fan of Like a central point of failure to where you virtualize everything on one server and I totally understand it's convenient You have one server right, but if that one server goes and everything on there goes, too But I also understand I think a lot of people with homelab They're setting it up for the first time. They don't really have the money to buy three or four servers or a dedicated server I would still recommend a dedicated server like Tom because if you don't have the money save up for it I think it's better to do that than it is to just force things to work in virtualization Some people have success with that I know there's going to be people in the comments that'll say it works fine for me Well, yeah, it probably does but I think that it's going to work a whole lot better on a dedicated server So if that's not something in your budget, I always say, you know, just just save up a little bit more Just wait you don't need it today, you know, just just just put some money in a jar or something And then when you have enough just go to that section of the forums that Tom mentioned about and let them know You're thinking about buying make sure you check the raid card. Do your research also also check out the noise from some of these servers Wow. Yeah, you could buy a Server and then it ends up just ends up being super loud Sometimes it's just a matter of going in the bio settings because in the enterprise, you know Where these things are from they really don't care about noise. They're behind a locked door anyway So they might have the fan on the highest possible speed so maybe you can get that down but but check that as well and I think you'll be fine with a dedicated machine But just do your research and also power usage check that too Because if you live in an area where there's, you know, very expensive electricity Might want to factor in how much power usage a particular server might actually utilize. Yeah These are things we brought up It's all these little parameters because as much as I say, hey Go ahead and type freenass over on eBay and you'll find usually Unix surplus is one of the companies and we bought from them Before that have them that have specifically systems that have all the drives accessible via whatever LSI raid card and they know because they know which ones are popular for free now So once again, you're buying something specifically purpose, but the other side of that is you know, we've had some of them And we're like wow that thing is not quiet because it wasn't when they were designing it They engineered the fans to cool the system not to keep your living room quiet or wherever you've decided to make your So definitely take that into consideration before you run out and buy one. So Next thing is I would say installing the drives versus the data drives This is where you can Get a little bit confused you want to set up and you can set them up in a raid like a mirror Which is adequate for boot drives I've seen people insist on using MVME for boot drives that feels kind of like a waste MVME The way the system works is it in SSD works perfectly fine for the boot drive You take it you boot it off that boot drive. You can use a USB thumb drive that is acceptable But it can wear a little bit faster and end up in a less good situation Serve the home is good at keeping up to date some of their recommendations And we'll have a link in the show notes for them Basically, they call it their free NAS or true NAS buyers guide and they talk about what good boot drives to use because the boot drives and data drive are separate you don't if you have a Large drive, it's not really worth it You may as well just get a small drive for the boot drive because you can't use the boot drive for anything other than Booting the operating system. So if you have a one terabyte SSD, you can't parse it up going well It's only going to use like 32 megs or 128 megs for boot. Can I use the rest for? you know storage like no the boot drive is the boot drive and That's why I recommend don't I've seen a lot of people run out and buy like a pair of MVMEs for boot I'm like, yeah, it boots a little bit faster, but you're not rebooting it often So save the money use the MVMEs for caching. We'll get to that in a second for special drive purposes But the boot drives go right, you know reliable is important. This is one of the reasons serve the home recommends some of the commercial Intel's and a few others that they recommend but You can put them in a pair and kind of like Jay said It's easy to back it up because all the config file is one file. So if you ever lose the boot drive It's it's annoying. Yes but if you have a single drive because that's what your budget allows and As long as you have a backup you can reload true nas to a new boot drive if it fails and then Restore the config restart and all your stuff is there because the data drives are a separate piece I'm gonna play devil's advocate actually on some of that because I I do agree about the USB flash drive thing It's it was problematic for me I think I I think I had two in a mirror and I had one go every month for like three or four months It's probably a coincidence Maybe I should have did more research and bought a better set of flash drives, but it is what it is So I went to M2 SSDs for the boot volume myself Keeping in mind that I bought the cheapest possible one I think I paid 20 or $30 which isn't that much different than a flash drive and It might be the placebo effect The interface feels faster to me when I'm clicking around the interface I don't really care about the boot speed because honestly on servers They're gonna boot slow anyway because the memory test alone at the beginning is gonna dwarf any improvement in boot that you could possibly hope to get Yeah For me, I think that there's no wrong answer if you have a really decent flash drive or a cheap SSD It's fine for me. I mean I've had fewer failures I've had none knock on wood so far, but again, it could have been just because I bought like, you know Not the best flash drives, but I I think that the SSD idea can work But like Tom said don't buy a terabyte SSD you are wasting a lot of space You get the smallest one you can if you're 32 gig 64 or something like that and just all the day And they're so inexpensive now to get those they cost as much they do make if you dive into a little bit They do make high endurance Writing thumb drives those ones be more acceptable and that's why I mentioned the server the home article kind of dives into some of those topics Because it's not that you can't use them But there are certain ones that are better because they're designed for a higher endurance thing your average thumb drive So they're kind of not really on a recommended list because the price on the high endurance ones comes back up to Being closer to the SSDs. So it's worth mentioning on there now when it comes to buying the boot drives This is well right now in June of 2021 is apparently really tricky because The hard drive prices are a little high and a little out of stock because apparently someone thought of a way to Create some type of cryptocurrency off of the hard drives. So that can be a little bit of pain but still one of the tricks that is still viable and this goes back to eBay is buying a handful of enterprise drives that are Depends on your storage needs of course new is always the best But buying a bunch of enterprise drives and getting them off eBay sometimes can be a pretty good deal and you can get a decent amount of storage for a reasonable price and Because those drives when you're buying them in bulk if you build an array of 10 drives for example Buy a couple extra in case one goes bad, you know If you're buying new great buy what your budget can afford and maybe one hot spare if you want or if the drives readily available use Amazon Prime as your spare which is frequently what a lot of people do because they know they can get it fast enough that like Well if one goes bad We can decide that but the storage capacity will be dictated and just type in ZFS calculator in Google You can figure this out. You have different levels of raid easy ways to remember as let's say we talk about a ZFS And in this tool we could occupy the entire show with different ways to set it up But we're gonna keep it simple and assume you have one VDev and we'll get to the hierarchy in a second But if you have a ZFS standard ZFS level raid, you're gonna be able to we'll call it ZFS one You let's say you have six drives You're gonna be able to survive a drive failing without losing data Then you have ZFS version two or ZFS two now You're gonna be able to have raid Z two that will allow you to have up to two drives And then raid Z three ZFS three is gonna be the Three up to three drives failure now where this gets a little bit tricky is obviously it's not just like oh We'll just go for the best where we can have a three drive failure each one costs more storage You lose storage efficiency as you gain redundancy across these so how critical your data is where does your budget fall? You should always have it backed up because raid is not a backup rate is resiliency But for a lot of people, you know They want to maximize storage because they have it duplicated somewhere else Just go on with ZFS or raid Z one is going to be probably adequate And I had an interesting experience and I don't know what the Scenario is like about ZFS now because I think this is a couple years ago. I remember I had four terabytes total Well actually I had four terabytes worth of drives And I was running up on the space there and I read that at that time That if I was to change, you know each drive to be bigger drives That's how I used to do it back in the day, you know when I it's a risk, of course But if I want to extend a Linux raid, I would just Fail out one drive replace it with a bigger one repeat for each and then resize the file system But I was expecting to happen when I did this with ZFS. I did the same thing I failed to drive replace the drive I was expecting to not have any additional space because what I had read was that they're working on Having it be extendable, but it wasn't that today the day I was doing this But what ended up happening when I replaced that last drive? Is it actually utilized all of the available space? Yes through the ZFS and I'm think I just read that this isn't possible And I also read that they're working on this for a future inclusion in a newer version of ZFS, but it worked So is that still a problem today? Is there still some is there still confusion about that now? Yes, now this is one of the reasons that the base of how you set it up is so important because One of the things about ZFS is you can't expand it, but you can under certain parameters So it's easier to say no because of the complexities because it's a but and here's all the parameters You have to meet let's say we have six drives all these drives are one terabyte and one of them goes bad Well, there's no more one TV drive. So we replace with a two TV Once we've replaced every drive with a two TV it can expand to the larger drive set, but not until then This is one of those kind of exceptions to it That's one way to do it and right now It's probably a good time to talk about ZFS hierarchy because this plays into how you expand the drives There is a couple different ways to do this So the hierarchy starts like this you start with the drives And I've always liked Wendell from level one text the way he references the drives that they really are an amazing Rube Goldberg machine of mechanical engineering and spinning rust and it is amazing that they work as well as they do But this is the groupings of these are important because this is how you get your redundancy The first base is you take the hard drives you group them into a Vdev virtual device Now the Vdev you can have one Vdev and have that Vdev be let's say we'll just use ZFS RAID Z2 as an example. So we have five drives We're gonna put them in RAID Z2 Then we take these five drives and then we want to build them into another Vdev So we have another five drives each Vdev has to be Symmetrical so if you have two Vdevs three Vdevs four Vdevs However me you have they all have to match if you built the first one with five drives The next one has to build five drives and you're probably wondering why would I even build them like this? the reason for it is If you build them like this the rights have to go to both Vdevs But the reads can basically go in between them and pull from each one kind of like how RAID 10 works So you can start looking at this hierarchical structure For ways you can build these Vdevs So you can actually achieve faster reads because putting all the drives in one pool as they call it making a Wide Vdev. Let's say, you know, I only have 10 drives. It's more Speed efficient to take 10 drives and break them into two five drive Vdevs now that RAID Z2 Where you can have up the two drives fail applies to each individual Vdev So technically I can lose more drives as long as it's not all in the same Vdev This is where that it gets kind of complex But I do have another video which I'll link to that I dive into More visually how to describe all the different ways That the Vdevs can be tied together the next couple pieces that are really important is the Vdevs have to be symmetrical But they don't have to be the same size. They have to be symmetrical in architecture But different size and drives and I have a video where I explain how unbalanced Vdevs work and the way that would work is let's say the first grouping of drives in these Vdevs We have five drives and a total is 10 terabytes. Then we get five more drives and a total is 20 terabytes It will actually Interplete back and forth between the Vdevs and for every one right on the 10 terabyte drive two writes of the data Go to the 20 terabytes. So it'll actually balance and unbalance Vdev So there's all kinds of little trickiness to it and You can take a pool and expand and add another Vdev to it So technically if I started out with let's say The 45 drives case is a probably a great example. It holds 30 drives. Let's say I bought 20 of them and built two 10 Drive Vdevs and later I buy 10 more drives As long as I put in 10 more drives to keep the Vdevs symmetrical in design and architecture of the other ones I can actually add them later and expand but there's no contraction So this is this is where a lot of people get confused. This is a zfs thing. Not a true nasty This is part of the parameters of the way zfs is done, but it's one of those Really important concepts to understand because some people, you know, they look at some of the more Generalized like, you know raid that you'll find in linux and go well I can just expand that or let's talk about a synology and you could just add some drives and expand an existing raid pool Traditional raid works that way Zfs because of the extra features and everything that is wonderful about zfs Which by the way for anyone that wants to leave a comment Yes, I'm part of the apparent cult of zfs because I'm a big fan of it like many other people are I've always I always loved it when people said I'm I'm too into it but it has so many enterprise features for The things it does their sacrifices that had to be made such as not being able to willy-nilly add a drive whenever you want And expand it dynamically But you get all these cool features for sacrificing that expandability This is why so much planning has to go into a consulting That goes into how do I build out my storage servers in you have to really think about them putting a plan together for that Yeah, make sure you buy as much as possible everything the way you want it all in one shot You know by your drives, you know plan for future growth if you have like four terabytes of data You know don't say I think I'm good on six. Are you I mean, what's what's your growth like? What do you think you're going to be using in in three four years? And I I argue a higher upfront cost of drives is better in the long term because it just makes it last all that much longer And if you buy everything the way you need it the hardware and everything And put some research into that then you won't even really run into any of these things because you already have everything you need now One of the other ways people get away with this and I I haven't done a video on this but you can find the Someone will post this quite a bit. It's an old article But it's it's a fair assessment of you want the ultimate expandability You do everything as mirrored sets you build you can build instead of just using standard gfs You can actually mirror each pair of drives and some people say well do it that way because that way It's easier to expand because you can always buy just two drives at a time versus If you put everything in blocks of 10 your next block has to be 10 drives you buy So budget concerns the downside of course of this is Go back to that zfs calculator when you start planning things out like that you'll run into the problem of storage efficiency It's an less efficient way to store the data. So you're getting a lesser utilization of the drives so while it's good in the way of You know easier expandability you're going to sacrifice a lot of space Which of course then pushes you to needing drive sooner if storage is your goal You can see how that cycle of that is a problem. It's it's There's no Wonderful exact what's the everyone starts with the first question I want to build a free NAS or true NAS ever what's the best way to lay out the drives and watch that thread explode of Everyone has an opinion how to do it, but you have to ultimately decide do you want storage efficiency? Do you want the most speed? Do you want a level of expandability? Pick one can't pick them all Yep definitely Yeah, there's one of those well those little things to really think about I wanted to make sure we hammered that home Because those are always the questions that come up in every forum post either my forums and true NAS forums on there Okay, now Going on now that we understand or hopefully understand or at least nowhere to go read some more So you can learn about vdebs once you group these vdebs into the pool The pool is the top of it that pools actually what you interact with so to speak be where you actually start creating your data stores And there's two types of things you can create you can either create a data set Or a z-val the difference is a data set You can think of more like just a folder just a place to store the data and the data sets where you're going to create Let's say a samba share because I want to share it with a bunch of windows systems or even Linux systems It also is where you'll create an nfs share Let's see you're using it for virtualization as a storage target And you have a vmware server or a xcp ng or proxmox and you want to have an nfs share You can set up an nfs share matter of fact The there's a lot of different little things you can do with them You can also use data sets for jails which we'll get to later So data sets are essentially just where you store data and from a command line They're just look like part of the directory hierarchy of things you go into z-viles are the special use case, but a really cool one z-viles are block devices And the most popular use of z-viles is going to be for an ice cozy target What this allows you to do is if you want to set up ice cozy on there You attach it to a z-vile and it's presented as far as ice cozy works in a way ice cozy presents It's like having a hard drive even though it lives on the array of drives within the pool So z-viles are those special use case where ice cozy being probably the most popular thing I can think of it's used with them, but this creates a virtual block device that allows you to manage it You don't get it from the command line see anything of the vial system But it still gets all the benefits of zfs like snapshots and replication But it's presented as a block device So it has a different type of usage efficiency and primarily you're going to use it for things like ice cozy the next piece is how the When the data set or z-vile the big benefit is the scrubbing the bit rot That you're avoiding the way it does the data scrubbing and the way a copy on right file system works This is the bottom line of why people are so enthusiastic about zfs Especially when you're doing large large-scale storage the slow Rotting away of your data because a couple of bits here there got flipped is a real concern when you're doing mass amounts of storage So once you start down that path of being a data hoarder for example And maybe you have those photos like I do that date all the way back to my first sony mavica camera I still have all the photos that I cared to keep from it And you know that thing I think was late 90s. I got that camera the one that stored on floppy's Maybe some of you have uh go back that far But the worry you have is of course the data. I'm not looking at those photos too often They're all just stored on a server somewhere They're stored on a ray to ray currently matter of fact very specifically they're stored on a true nas server And having that data integrity checked is a tricky business And this is what zfs excels at being able to do that level of integrity checking So you know that if I wanted to access that really old data that I haven't touched That's been sitting on there that the data will be available Something we do on the commercial side the enterprise side is consulting with a lot of movie studios And they have run into problems with other storage servers of having some of this bit rot Bit rot is a silent problem because the data appears to be there, but it's actually been corrupted in some way This is one of the big things that the enterprise market really likes zfs for is If they ever go back to and want to access any archival footage that they know it's there They know it's available to them and they can trust that anytime they want it they can get to it This is this is a huge factor when it comes to the data scrubbing that's on zfs I think j is running to this before and it's a hard problem to explain It is I think it's The first thing to get out of the way And this is just to adjust the mindset Every single time we mention this and I could totally understand no judgment at all We'll have someone mention, but I've never had a bit rot problem I've never had a file get corrupted and then my counter to that is are you sure because To put it into perspective. I have last I checked 10,000 photos. I do photography I'm not good at it, but I still want to keep my photos because every time I get that lucky shot, you know, it's great but um That was 10,000 photos last I looked at it. I have family photos on there too And I'm not going to be able to look at every photo Regularly to make sure it's not corrupted because you could argue You know if a file is corrupted you have backups I hope you could go to a you know time before it was corrupted pull it back and replace the current one Well, that's time one But again, you're not going to listen to every mp3 in your collection You're not going to watch every movie in your collection because there's just not enough time To even to do that or even just look at all your photos. So I was going through my photos one time And a couple of them I think it was like back in 2003 were corrupted. You just couldn't read them So I can't I can't tell you at what point those photos got corrupted You know, they could have became corrupted a year ago You know five five years ago at the time I was looking at it. I have no idea. So I I don't believe that there's ever a 100% anything but I certainly want to do whatever I can to make sure that things just um You know are just mysteriously going away and I even had a another strange issue on my retro pies because Your retro pie is an emulation system But that's backed by true nas too in my case Because if I add a new game It's added to a data store in true nas and it you know copies that via sync thing to all my retro pies And I had a super mario all stars would not run very well It was just sluggish and horrible and I just put it down to maybe you know Emulation just doesn't handle that well A couple years. I didn't even bother playing that game just to find out at some point that rom got corrupted I redownloaded it. It was perfectly fine. So um, you know, yeah Do you have a problem? Maybe not but you never have a problem until you do That's why you kind of want something to back you in case Something could happen that there's a chance that it won't Yeah, it's it's one of those little things now The other side of this too. I mentioned copy on right. I'm not going to dwell on it too much But there is a book by michael lucas called zfs mastery. I believe is what he's called that one He he didn't just write the book on zfs. He wrote two of them So If you ever want to dive in deep It's a really intricate look at how the file system works And it's also been referred to many times as the billion dollar file system One thing specific about the whole zfs is there's been a long time research and development on it And it just helps so much with that You know avoiding bit rot and all the other potential problems because of the scrubbing The other thing it doesn't have is a check disk because it doesn't really need it Because that's one of the advantages once again of copy on right file systems They're very very fault tolerant to keep from losing data All right, i'm going to quit singing appraises of it and talk about one of the other features that a lot of people like And this comes into the age of ransomware um snapshotting Snapshotting is a way to grab Each data set and say I want to have a snapshot in time of what this data set looked like Now this is actually kind of cool because this even ties back into something you can do with samba And something you can do with active directory is like shadow copies This creates immutable shadow copies if you tie it to your active directory systems And this is a popular use in the business world where they have it tied to ad But you can just in general have these snapshots So this is where my data was and I bring up ransomware because ransomware goes to encrypt everything Well with a zfs snapshot you just say roll it back and they roll back with amazing speed You can jump backwards to the snapshot in time You can also kind of create a fork of it too if you wanted to fork a data set because You don't want to overwrite your existing one as in roll it back to A snapshot in time of what that data looked like at that point in time You can also create a fork and create a new data set based on a previous snapshot. These are really handy ways to Being able to manage all of your storage Especially if you're going to make some mass change to a bunch of documents or maybe reorganize something Just quickly snapshot ahead of time or even better you snapshot it on a schedule And that's how we have you know, we just set this up as an automation both in our home labs or for businesses We're on a regular schedule. It just does a snapshot that way if there's anything bad that happens or a vm That gets corrupted that's running on an nfs share. I have all these different snapshots I can just jump backwards to or because we have so many vms running in one nfs share If I had one of them that somehow became broken in an unusual way I could always fork that snapshot copy that file back over where it needs to be and away we go This is a great tool because each snapshot only takes the differential and change between The current data and the snapshot So if you're only changing 100 megs a day, then your snapshots are only 100 megs Which is really cool in the case of storage though you expire them So you keep let's say a parameter that says I want to keep them for two weeks or three weeks But if you're once again only doing 100 megs a day And you have a daily snapshot You're only keeping that 100 megs times 14 days and that's how much of the extra data it uses So it still takes a little bit of storage planning, but they're relatively efficient and zfs has Built-in compression on there to keep the storage efficiency high on this So this is a really great way to be able to especially when you're experimenting in the home lab Where you may goof something up be able to roll it back really fast. So this is a you know, definitely a big feature on there Now directly related to snapshots is what if you have a second true nas? I seen someone ask that in the questions because yes, you can have more than one pool on a singular true nas But even better would be e2 have another true nas and have another pool on there and that's where you get zfs replication Zfs replication you can replicate the entire pool or any specific data set and it does it at the block level So it's not even looking at the files. You can have thousands of little files or one big file It kind of doesn't care it looks at the differential at the block level Because that's the same way snapshots work snapshots not looking at the files It works at the bottom level of the file system and goes i'm creating this snapshot in time It also then can send that snapshot over to another server to replicate it So you can actually have all your data at the block level Replicated to another onsite true nas server or even an off-site one This is where you can get that level of redundancy and confidence when you're Moving your data and in case one of them suffers an absolute catastrophic meltdown You know all your data has been replicated over there and true nas has done a great job of making this easy of You pointed at server a you give it the credentials for server b And you tell it to replicate on the schedule you set I've got plenty of tutorials on how to do it It they've made it so easy because all you're doing is dropping in the parameters and giving it the ip address of the other server or You know a domain name or however you have it set up you just drop it in there Now they replicate now all your data is in two places life is good That's it's it saves you so much when you're doing that now On that note some especially in the home lab. We realized not everyone can afford um A system of The be having more than one system on there Then you actually go through and do something else like r-sync because you can't afford to true nas all right What are you gonna do and set well you just go over to r-sync and From there you can say all right talk r-sync over to Uh, whatever other server you have this is a lot of other protocols are built into true nas That allow you to do both so you can go in there and say I want to back it up to my Non true nas server or like in j's case if you had rolled your own server you still have that server then you finally got around to Uh, you know building a true nas server you can still sync the data with non true nas equipment They don't lock you into a zfs ecosystem by the way also you can also talk Zfs to other systems and do zfs replication to non true nas systems They're using open zfs on the base of this you can load open zfs in a bunch of sue If you wanted to talk zfs replication and kind of learn how it worked from a command line There's no lock in it's all open source. You're using standard protocols So they kind of give you a good overall for getting your data. I call it, uh You know very much you can deliberate your data from it. They're not locking you into this weird ecosystem or anything like that Yeah, I think that's an awesome thing. I'm I'm actually syncing it to a synology now Which is actually kind of challenging and not because of true nas but Surprisingly because of a synology not really facilitating our sync as well as I would like them to but anyway You have options. You can you can copy your data to another device another true nas or another nas of some sort So, um, it's recommended. I mean why why not have more copies? Yep, and then let's get on to the cloud servers because you know, I just like to hammer home the backup side and Backing up to the cloud. We'd mentioned back plays. Um They're probably still one of the best price being for the buck me and j are both fans of back plays They're not a sponsor yet of the show But we so I have no offer good for you on there But back plays is a great service But there's a long list of cloud services Whether you roll your own and target it at your own service or use one of the ones built in They do have that option and because we shouldn't just trust these cloud providers. They're not necessarily You know, they're they're not infallible They may accidentally leak some data. So true nas offers all the encryption So the encryption on true nas before you do back up to a cloud provider You can encrypt it prior to sending it so you don't have to trust the cloud provider You just have to hopefully trust that they are going to have your data As long as you're holding on to the decryption keys You can do it and once again It's one more thing that true nas is relatively made easy in there Yep Now let's talk about the part that I know j's going to have some opinion on and so do I let's talk about jails and Jails are is jail is a containerization They're using specifically the underlying technology the jail and bsd that they're using true nas is called io cage io cage is a pretty cool system It is very similar to the way containerization works in linux. You're sharing the kernel space but It's different because you're sharing the bsd kernel space and this is where where's a little bit of a sigh And I know there's going to be so much more excitement when you get over to Playing with uh docker and kubernetes and everything else that that is all getting integrated into the true nas scale because it's based on linux Because it's based on bsd These plugins and the jails if you build them yourself build build your own bsd jail That's completely possible I've got some tutorials on how to do that and sometimes it's a better route to go and the reason for that is The plugins sometimes just lag in updates. I don't think they have enough maintainers for some projects So people sometimes get upset it that they their project they want is several versions behind in some of the jails This is something they're really trying to work on the community with but it just comes down to developers developers developers as was famously once said We need more people in the bsd side to maintain these packages But you know linux is the hotness now. So there's more developers to you know update you can probably find a newer Docker container for something before you'll find the newest version in the bsd jail ecosystem And I use the um the sync thing plugin. That's the only one I really use in It's kind of like you have you know, you have plugins, which I think I mean correct me if i'm wrong. I think every nas provider has some sort of plugin system nowadays I mean synology you name it you could just go in their little store or whatever they decide to call it And click on something and install it. So true nas has that And when you install a plugin it's going to create a jail for that plugin to run in And that's kind of like the one-to-one match there That we have you could create a jail in the command line without creating a plugin You could just decide I want to create a jail and it doesn't have to be for a plugin But if you were to go in there into the GUI install a plugin you're gonna you're gonna get a jail And that's not really a problem necessarily. It's just the underlying underlining stuff, but Like tom said, it's just sometimes I go for the latest version of a thing and it's not there And there's several schools of thought. Yeah, we need more developers now I could probably go in there and you know put in a pull request to make that happen But then it's like when do I stop right? I mean, I hope someone does that and I certainly probably should be doing that but The idea is that people have jobs and it's hard for them to Even know how to give back let alone give back, but it's still recommended if you have some time It would really help true nas kind of keep things updated because they can't themselves update everything I think they I think whoever maintains syncing Last I checked the version. They seem to be doing a pretty decent job Yeah far so next cloud I haven't looked at it in over a year, but I know that was there and that was severely out of date I hope that's not the case now. So it's it's hit or miss Yeah, and syncing is kind of a cool one because I've done this where you can build your own jail with syncing I really love that tool and Obviously jay's a big fan of it as well One of the cool things is syncing has its own Updater and because it doesn't rely on a lot of dependencies It's a relatively easy system to set up inside there And even when you launch it from their system, it's not hard to Get it to auto update from its own syncing because the way syncing is compiled They compile their bsd version at the same time they compile their linux version and windows versions all together It just pulls from its own update server So it's easy for that particular plugin to be maintained Some of the other ones are like that But some of the ones that get a little bit trickier to maintain is when they have dependencies that are more Linux leaning I should say where there's a bsd equivalent, but it may not be as popular So this is where you can get into the nuance of trying to figure out how to maintain and update that because Well, bsd and linux are not the same despite what someone in the comments may have already said There are some fundamental differences to the way they work and then that little nuance difference between them Does mean the difference between that plugin working and not working The way you expect it So that that is one of the so to speak a little bit downside you can run into with the jail system on there It's still a good system. It's still a good way to run certain things Um within there and you can set up There's plenty of instructions how to run plex being one of the really popular ones that you can run inside there And it's better in my opinion to run plex on a separate server running linux because it's easier to update But absolutely if you put the time into it you can learn and get the latest version of plex up and running Or it's not too terrible to be a version behind a little bit of a version behind on Plex when you're running it through like the jail system. So it's workable But with warnings, uh in very brief mention here I don't use the virtualization the beehive virtualization system in churnass anytime. I've tested it I found it to be buggy. I don't think jay's ever tested it. I have a few friends it It's the one thing that I feel very it's very undeveloped in there So my my recommendation is always don't use it. It's unless you just like playing with it or um want to take the time to really learn it Yeah, I mean you have the tools available with churnass to Basically do this however you want. I mean if if you don't have a version available of something you want to run That's natively available in churnass You could put it on a linux server like tom mentioned or even a container because you know Churnass exposes nfs and I like to use auto fs Because that mounts everything on demand. That's how I have plex set up. I think I've mentioned this in a previous episode Those that didn't hear it Um, I have this disposable plex vm and all it does is mount via auto fs Which is nfs this uh share and it's it's it's like anytime plex looks at the video share It mounts it so quick that plex can't realize that it was ever not mounted So that kind of helps avoid the nfs locks doesn't 100 avoid them, but it does make it better And then you can have churnass as the storage back end and then you can have containers or vms To host the individual applications that you want to connect to those uh back in storage Devices and that works for me. Except you're being synced thing I just really like that being there on the churnass server since churnass for me is where all my data goes It just makes sense for syncing which handles data synchronization to also be where the data is natively And not run that separate But um other than that I generally try to run everything outside of churnass and just Leave churnass as a network storage device and not an app store. Although, you know I do agree that having the app store so to speak is pretty cool Yeah, it's it's one of those Little nuances and the way you see it use and I know I get that the love in the home lab world to put Everything in one box because that makes more sense. But unfortunately zfsb can I'm sorry Churnass can be a tricky box to do it Which is of course, and I've already seen it rolling in the comments why so many people are trying scale By the way, I did mention churnass scales alpha But it is because of all of you and many of you I should say that are listening right now I'm running it that help report these bugs in churnass scale That's what helps the developers understand use cases and brings it from beta or alpha right now to beta to production So I'm actually really excited to see a lot of you in here posting it. You're already running scale You know, this really helps anytime you can help troubleshoot problems and Participate in the bug reporting of the product to improve it. I love seeing that I know a lot of people are doing it So I'm not I'm not telling people not to do it But I'm saying it may not be the best approach if you if you're not familiar with it But if you're already familiar like I am with churnass and you want to play with it. Yes, I'm playing with it too So Yeah, we have a lot of enthusiasts they they want to try the latest things I mean just as an aside I have a couple of pine phones in the studio right now And they barely work at all like almost everything that I try is is an epic failure But it's it's still fun because you could watch the progress or the project grow and mature You can submit the bug requests or bug reports And you can let people know about the problems you're having or wishlist items things you wish it had that it doesn't And be a part of the process and let's be honest Usually a lot of companies they don't make you part of that process. You just get whatever they produce for better or worse Yeah, you can engage quite a bit with the community on there now One of the last things I wanted to cover is something I didn't think as much about but apparently This can be a problem for some of the homelab people that want to create A portal for people to change their own passwords and things like that now you can tie The churnass system into active directory for authentication and we do this for a lot of businesses common use case But for home users, I didn't really think as much about this But I've realized it's come up a little bit more You can set users it has a user system It has a granular permission system and access control is system And I've done several videos on how to set all the parameters and things like that What it doesn't have is a way for each user. It doesn't have its own ad server It doesn't have a way for users to go in and tie it to there That was something they built in a long time ago But I don't think it was ever well developed and they dropped it later in churnass So it can't act as its own ad server like Some systems can I think synology for example has like its own ad server here It can tie into an ad server, but it doesn't act as one So it's not a replacement drop-in. It has to be a companion to a windows ad server It's well as little details, but some people That is something important where they want to be able to you know replace their active directory I don't need this windows server. I want to replace it with churnass Well, you can create all the usernames and set up their passwords on there But it doesn't give users their own portal when they change it and synchronize with their, you know Windows machines if that's something you're doing. I think it's worth mentioning on that But as far as if you want to tie it to some other authentication server, absolutely you can Yeah, and also it's important to understand for a lot of people out there that don't already know Ad is not LDAP and LDAP is not ad. Yes there you could argue that they're the same technology that you know There's there's similarities and one's built on the other and so on but you know microsoft they put new stuff in there And they're not going to hesitate. You know if they want to put a new feature in active directory today They're going to do it today. They're not going to spend time and update the samba project and update all these other ones Unless they have a a company that has a partnership with microsoft Then yeah, probably but i've seen situations where people get active directory working And then it just breaks for some reason I would say if that's something you want to do I would look at LDAP in my opinion if you don't have an enterprise use case That might work better But that also depends on if you have a mixed system environment, which is a whole another episode altogether Yeah Yeah, getting all those systems at dying together becomes It's my only fun thing Yeah But I you know you you do have the ability to do groups and users and things like that But you the admin i'm speaking to all of you homeland people are admin any systems You will be the one setting it up. You won't have some you know delegated portal or a similar way that is handled It's not in parity. So to speak with the way permissions are handled and let's say a windows domain controller environment So I thought it was worth mentioning on that though because this is those little things other than that it's of course Everything that we said and more because we didn't touch all the features it has but I think this is a good way to get the Idea I mean you you can go crazy with it. It works as an s3 target. Yes, it can emulate amazon s3 protocol So you can use it for all kinds of expanded enterprise things and in the serve the home article We didn't talk much about it, but obviously network card interfaces being able to support 10 gig 25 gig 100 gig connections. Yes. Yes. Yes And serve the home does have some of that I didn't mention at the beginning I think it's worth mentioning too if you're going to connect your storage server You want it to be fast and one gig just ain't going to cut it and 10 gig Is now very affordable and 25 gig and 100 gig stuff is becoming more I would say home lab before 10 10 gig is almost a no-brainer in a home lab with some of these inexpensive 10 gig connections and SFp cards that are well supported and true nasty You can buy them for you know, you can buy a dual sfp 10 gig card for like an intel da5 20 I think are probably around 50 60 bucks on ebay There's all kinds of deals to be had when you're doing these so it's it's getting affordable to do 10 gig in the home so I think i'm going to go back to 10 gig. I probably still have some of the same equipment as I um had before so I'll re-implement that um And I I really wish that somebody would just make videos about true nasty, but that'd be great. Oh wait a minute Oh wait, we'll have a link All my tutorials curated in a link that will leave in the show notes as well for all my latest true nasty videos to Cover a lot of a wide variety of topics on how to get things done in true nasty including one that just discusses um The whole argument of what's the best way to set up my true nasty all the ways Yep By the way, that sends you down a rabbit hole of a lot of reading by some of the engineers that have wrote some Good articles I curated all those articles and it's I've spent so much time reading them And they're still not the best answer But there's a lot of good answers and I didn't even have time in this episode to dive into special drives like l2 arc caching read cache or Zill intention log caches or even the new ones wendell man I should leave a link to wendell's video because he did uh a couple zfs videos including one of them on A metadata drives That's a really cool one and wendell does a deep dive because he's got some magic he put together Of how to build uh metadata drives and ways you can adjust them because zfs is Made has so many little knobs you can turn to tune it Including on the fly you can change some of the block size and storage size of different data sets on a per data set basis to fine tune What files you're going to store in there for the fastest way to access those files? There's a lot of tuning parameters that can go in there once again. These are all those little things that uh You can really you can spend a easy time getting this set up and have a system up and running But the next thing you know you spent two weeks just reading articles and tuning each little parameter for each data set Based on the level of storage and then dynamically adding things like metadata drives and like I said Wendell's dive into that was one of my favorites. I'll make sure I link that in there He talks a lot about a wonderful level one text about uh And it's their use case because he's building a new storage array with zfs and it says zfs talk about how they have So many files from so many videos that they've created over so many years that they've now had to create metadata drives Which is basically high-speed drives to better and faster indexing of millions of files that they have That sounds exciting. Yeah, it's it's one of those things. It's uh, it's pretty extensible of what he has He's his new raid server. He's been doing some videos on has 54 hard drives in it. So It's uh, that's it's a man. We've seen some pretty big systems out there built with not your ass. It's it's impressive for zfs in general so Ah, anything else we should add. I think we I think I covered it as much as we can squeeze into an hour The only thing I would add is just a tip, you know after you have a true nest setup Um before you go crazy copying your data to it just spend some time Thinking about the logical arrangement of your data sets like the higher level the lower level for example, I might I have a data set for media and then a Data set underneath that called music and then one under, you know, also underneath media called, you know videos or I think Movies or something like that, but you just look at the logical You know, how you want to carve out your storage You just kind of map that out ahead of time and then create the data stores to Abide by that layout that you've come up with you'll have an easier time because the you know What you might end up doing instead is oh, I should have put more thought into that Then you're really copying data from one to the other and it's copying data between data sets It's not like it's on the same file system in this instant I mean you are literally copying from a different file system from one to the next and it takes some time So just keep in mind Of just lay that out first Just just get a piece of paper write it down how you want to carve out your storage Have a good plan in place and I think that'll avoid that being a problem in the future Yep Oh, and I just because someone mentioned the comments I should mention this to you it supports full levels of encryption for data at rest so that's that's an important parameter with Especially in compliance level things that we deal with in the enterprise market They you want to make sure that anytime you have a system Let's say someone were to physically remove that system Is it encrypted and this is a concern with homeland people? I care about my privacy if someone were to come in and take a server I want to make sure they can't gain the date off it and that's something easily and well supported on True nas as well as full data at rest encryption. I just see a few comments combined I'm like, yep, I should have mentioned it earlier But absolutely a supported thing not only do you encrypt the stuff before you send it to the Cloud for backup, but you can also have it encrypted So if someone were to physically take a plug out of it, sorry without the passwords, you're not getting any further That's important. Yep. Hey, I encrypt everything So that's uh I just encrypt all the things it's my default answer for it because it just that way I never worry about did I encrypt it? It's always my first step now anything I build whether it's the desktop I'm sitting at right now, you know, you load it up a pop o s you set up their encryption Whenever I build my zfs of that encryption box always check the encryption box when you when you're building out the drives Just do it now, even if you're not sure if you need it do it now Yeah, save that password in a in a very very secure place Yeah, in case you lose it and don't put your password in a you know somewhere where it's easy to Get to buy someone you don't want to have access to it. Just keep it handy if you lose that you're done It's important to keep in mind So all right, well Thank you guys for joining us on this episode diving into true ass As always we love seeing the feedback leave the comments below on the video Or if you listen to a podcast, you know, there's ways to reach out to us and connect with me or j All the show notes will be down below and thank you Thank you for listening