 Tom here from Orange Systems, and even if you have not experienced this, I can guarantee you hard drives fail, well, not frequently, but frequent enough. Enough that I wouldn't trust all my data without a backup. Now, RAID is not a backup, but RAID is resiliency. Backup is one way to mitigate things, but ideally, you don't want to always have to restore everything, so we put multiple drives into systems. And my favorite system to use spend true NAS for storage with ZFS, but that comes with a lot of confusion and understanding the trade-offs of creating this resiliency. The first problem being if you start using these drives and you have three drives, you would have the cumulative storage of all three drives minus the ones you dedicated to parity. This is a formula that's really simple when you're holding three drives in your hand or three spinning rust or three solid state or six drives or other groupings, but it starts to get a little bit more complicated as the layups get larger, deciding how many you want to dedicate to parity and how many you want to have per vdev. Now, once you get into these larger storage projects, there's a lot of thought process and trade-offs that have to be made. And you have to first think about what you're optimizing for storage efficiency, which we're going to break down, or are you optimizing for speed? And there's ways to set this up to be faster at the expense of storage. So it's not a simple yes or no, how should I lay out my drives, but a thought process on the different layouts. Now, what I want to do in this video is explain briefly what a ZFS layout looks like, what vdevs are, what vdevs can be matched and why you don't want too wide of a vdev and what a wide vdev or a narrow vdev even means. We're going to get down to the details in there. But first, if you'd like to learn more about me and my company, head over to laughtonsystems.com, click the hireshare project, such a storage consulting, there's a hires button right at the top. If you want to support this channel in other ways, there's affiliate links down below to get you deals and discounts on products and services we talk about on this channel. All right, I'm going to use diagrams.net, which I reviewed before, and we'll leave a link to below to draw some pictures here and kind of give you an idea of how ZFS works. This is not a deep dive, but at least I want to cover some of the fundamentals. I will leave links to further readings so you can dive deeper into this topic if you're interested. In the basics here, your data sets is where your file storage would be and your Zvol is your block storage, two different storage types that are in the ZFS pools. Then we have the ZFS pool itself. Now data sets and Zvols do not know where their data is stored, but it is stored within the vdevs. This is what the mechanics of the pool handle is the grouping together of all the vdevs as part of the layout. And we'll get to that later in a video showing you actually how this works inside a true NAS. Now you can have a single vdev. You can have a single drive and have single drive vdevs, but that would offer you no redundancy. And a data vdev of RAID Z1 with these four drives right here means four drives and one of them is dedicated to parity, which allows and is actually spread across all of them. So it's not like you assign a drive. Any one of these four drives can fail and you have enough data to rebuild. If you went RAID Z2, any two of these drives can fail and you could still rebuild by replacing the failed drive. Now the vdevs, when you're doing something simple, there's not a lot of thought that goes into it. Usually RAID Z1 is fine. You back up all your data elsewhere. And if one fails or is giving you some indications of failure, you find the failing drive and replace it. The data rebuilds, everybody's happy. When you start having a few more vdevs, because you can have more than one vdev for the pool, as long as the data vdevs are symmetrical, then now we still have plenty of resiliency. And this is where the decision making comes in. Should I build two vdevs of RAID Z1, or should I put all these drives in one? Well, up until 10 drives, putting them all in one pool is actually a pretty safe bet. And when you put them in multiple pools, there's a storage efficiency cost that we're going to get to of having them all broken down in the pool to a series of vdevs has a high cost of storage efficiency versus putting them in what they referred to as a wide vdev. This would be a narrower vdev with only four drives. But if you were to lay this out with 10 drives, now you have a wide vdev. Now why stop at 10? Why not go 60 with this whole 60 drive layout we're going to get to towards the end? And that's because once you start exceeding 10 drives, you run into a right problem. Now what I'm referring to as a rate problem is if you have a series of drives laid out 10 wide. Now 10 is a good number. You can go more. There's a little wiggle room in there, but let's just say we have 10 because that's the layout we're using. And those 10 drives all are doing rewrite operations because, well, everything's in operation and one of the drives failed and we need to replace it. The resilvering process means is going to pull the parity it has from the other drives to rebuild the drive that you replace in it. The problem becomes that is not static information. If there is no activity on the system, yes, it is static information because the drives are sitting there and they can take the time to send all the data to rebuild the one drive. But if the system is in use and there's constantly new data being written to the other good drives, the resilvering process is now slowed down a bit. There's a lot of engineering behind the scenes to keep it going, keep it working and kind of pick up where it left off. So it didn't have to restart the resilver process, but it does slow it down. And also simultaneously, if you are trying to write to these drives, you will notice the performance loss with these really wide vdevs because it's taking longer to resilver. They're being disrupted. The whole engineering that's going on behind there. So this can kind of create a problem. This is why wider than 10 is seems like a pretty good rule of thumb. There's going to be someone out there that has a super wide one that they were happy with the performance they got resilvering when they had 20 drive wide vdevs and a drive failed. Some people are going to say I was unhappy with the performance with only five and how it failed. So there's a lot of difference opinions in there. So these are just some good guidelines at least to go off for you understanding of why you don't necessarily want to lay out a 60 drive wide vdev pool. Now, once you get these larger pools, I highly recommend at least Z2. And the reason for that is if one of these drives fails and you go to do a rebuild during rebuild process is a very critical moment because if a drive has failed and the rebuild does not complete on the drive you replace, that would mean that all that vdev fails. And when one vdev fails, you have lost all your data. So even if we eliminated just this one, well, I didn't grab all of them, but if we eliminated just this one right here and we lost this vdev, all the data is gone because all the vdevs are where all the data is distributed evenly across them. That's why we have to have the resiliency at the vdev level. And having them at the vdev level means that we have to decide that we want probably, as I said, RAID Z2. But why is that not just the easy answer? And why is it getting a little bit more complicated than that? Now I will mention just briefly down here before we flip to the next slide, there are other special cases. That's why I say data vdevs have to be symmetrical that don't have to be. These are for another video. I'm just doing the graphics now, log, zil and cache metadata. These apply to the pool, but not to the vdevs. And they're a separate type. So only the data vdevs have to be symmetrical. Now let's dive into the reason why we can't just lay everything out because it just answers the question of put them all at 10 wide. And this comes down to storage efficiency. So let's say we build a RAID Z1, which gives us a 90% storage efficiency when you have 10 drives. That sounds great because I have a lot of storage, but that problem I mentioned of, if a drive fails, that critical moment of rebuilding the drive, when the other drives are all working a way to try to get their new found friend in the pool up to date and sink and re-silvered, this is a critical point that could cause a failure. So let's just go to Z2. That brings us down to 80% storage efficiency, which means 20% of the drive is committed to having the parody so the drive pool can reboot. And as I said, the parody is not dedicated to a single or even two drives. It's spread across evenly. So any one of the 10 drives can fail or any two of the 10 drives. So let's say one drive fails, you replace it and bad luck happens. And another drive fails. Yes, the rebuild can continue and you can even replace that other failed drive within there and still survive. Or, you know, it's really critical. Even though we have a backup, it's going to take too long to restore all this data. Go ahead and go RAID Z3, but then now we're down to 30% of our storage per VDEV is dedicated. And I say per VDEV because that's what matters. When you start looking at these larger layouts, if we have this RAID Z2, and as we said, our storage efficiency is only 80%, that means we're not losing 80% just in one pool. Every piece of the pool, every VDEV within this pool has an 80% storage efficiency. So now you have a little bit less drive space available to you because each VDEV, as we said, we don't want them wider than 10. And this would break down that storage efficiency to be not quite what you were hoping for, maybe, but we have that resiliency and that is what we want. And let's go back over here to storage efficiency and think about other layouts. What if we just broke these out and we have 60 drives? Having 60 drives means there would be, if we laid them all out in groups of 10 in the VDEV, that would mean 6 VDEVs, 10 drives each totaling 60 drives. You could also do this differently. You could lay it out in a series of VDEVs that all have 5 drives each. But that means a RAID Z1 has a storage efficiency of only 80% versus a RAID Z2 here has a storage efficiency of 80%. So now you actually are sacrificing storage, which back to, well, isn't it just make more sense to lay them all out in the 10 wide? And this is where the benchmarks come in because this is a real thing that will change is the performance of the drive system. And let's actually take a look at that. So I built each of these drives with the 10 drives per VDEV RAID Z2, 5 drives per VDEV RAID Z2, and then mirrors. Now using the mirrors, of course, gives us the most performance. And it's a common layout suggestion that people have had. But that absolutely comes at the highest cost in terms of how much storage you have available. The mirrors leave us with 475 terabytes of storage. The 5 drives per VDEV is 589 terabytes and the 10 drives per VDEV allows us 785 terabytes of storage. So these differences between them are somewhat substantial. You're talking about quite a bit of storage being thrown out in the sake of parity and performance. And the real performance hit comes down to random rights. The synchronization when you have to write to a 10 wide VDEV brings it down quite a bit because all 10 drives and the parity information that has to be carried on a per VDEV basis all has to be written and committed before the system can say, all right, we're good, we have this data on this particular VDEV. And of course, that problem, as I said, is amplified. If one of these drives in that VDEV goes bad, that's 10 wide, it's going to take longer to rebuild versus it'll take less time to rebuild the one grouped at five. And when you're dealing with mirrors, each pair of drives is mirrored. So it just has to copy the data from one to the other. Of course, the risk in the mirrors is if you lost both drives of the same mirror, you're in big trouble on there. But it does offer the highest level of performance. But when you look at the sequential read performance, they're all neck and neck with each other barely any difference the same with the random read. Reading the data back is not a problem at all. The penalties come at the right. There's a lot more bench results here for people that want to dive into everything. I'll leave a link because this is TDC small to read on the screen. But all those stats for those of you that are really want to dive and nerd out on this, hey, link down below for that. Now I'll show you real quick how simple it is to lay these out inside of TrueNAS. This is TrueNAS core 12.0 u six, we're going to go over here to storage pools, add the pool, create a pool, let's call it test. And then we can say encryption because you should always use encryption. Go ahead and do that. You can't do encryption after you have to do it before. And now we can choose the layout. Now if we're going to go with like the five layout, we choose one, two, three, four, five, and you add the data vdobs, you set the raid type, raid Z, or raid Z two comes down to back to some of the opinions of how you want to do things, go to repeat, and it will repeat this and lay it out and give us all these different drives laid out. Now what if we wanted to do the mirror layout? Well, let's go to the mirror layout and we say one, two drives over here, set them up to be a mirror, repeat 29 more times. And now we have the mirror layout. And you can see that estimated raw data capacity that is available here. Now let's go ahead and reset layout again. And we're going to go do a 10 drive layout, you would just say one, two, three, four, five, six, seven, eight, nine, 10. And the same thing, repeat five more times, giving us the six separate vdobs with 10 drives each, repeat vdev. Actually, I'd have the first changes to raid Z three, whatever you set these two down here, raid Z two, or raid Z three, or just raid Z is how it will then lay out once you hit repeat. So repeating this vdev. As I said in a future video, I'll cover some of these other vdevs in here. Now I'm hoping this helps when people say, how shall I lay out my drives? And it says it depends. Now you have a better idea of what that depends on. It is how you want to lay it out. Do you mind the potential slower, re random rights at the better storage efficiency and better use of your drives? And if you're doing something like storing lots of videos would be a great example. That's probably not as big of a deal because the sequential right is not that much of a penalty on the larger layouts. The random read rights, what you're doing is for a bunch of virtual machines and you really want that random read rate performance. Yeah, you may want to look at one of the other layouts, but then again, maybe you have a storage need for lots of VMs and you can deal with a little bit slower rate performance because you need more storage. This is that back and forth of how do you want to choose to have all this data laid out and there's not just one right answer. But my goal is of course to tell you the different options, what they look like when you benchmark them so you can make that decision more well informed because once you set all this out and get all your data on there, it's not fun moving your data elsewhere to relay these out in a different methodology. But I a lot of fun doing this and I will see you guys over in the forums. If you want to have a more in depth discussion on it, also leave some comments down below, I try to reply to all of them and thank you. And thank you for making it all the way to the end of this video. If you've enjoyed the content, please give us a thumbs up. If you would like to see more content from this channel, hit the subscribe button and the bell icon. If you'd like to hire a short project, head over to LawrenceSystems.com and click the hires button right at the top. To help this channel out in other ways, there's a join button here for YouTube and a Patreon page where your support is greatly appreciated. For deals, discounts and offers, check out our affiliate links in the description of all of our videos, including a link to our shirt store where we have a wide variety of shirts that we sell and designs come out well randomly, so check back frequently. And finally our forums. Forums.LauranceSystems.com is where you can have a more in depth discussion about this video and other tech topics covered on this channel. Thanks again for watching and look forward to hearing from you.