 Time here for more systems. We're going to talk about replacing failed drives inside of TrueNAS And we're going to forcibly fail a drive for this particular demonstration now the failed drive scenario is Based on the resilience level of your system now It goes out of scope and I have other videos that dive deeper into different configurations for ZFS arrays But this particular array can suffer a drive loss without a problem and we can rebuild it There are different scenarios for whether or not it can suffer one drive to drive three drive etc Or when you build certain type of mirrors goes like I said a little oscope of this I'll leave a link for those of you that want to dive deeper into all the different ZFS raid configurations that are out there Which there are a few other notes This is a 320 gig hard drive and these are 320 gig hard drives If you use a larger drive than the pool was built with you can but it will not use the full extent of that Particular drive so these being 320 if I use anything other than a 320 something bigger You can't go smaller that means a terabyte drive would still only use 320 of it so you end up with a bunch of dead space on the drive But sometimes when you're dealing with older drives We've run this problem before you have only one drive of a failed array and it's hard to find that older smaller drive So it's not a big deal to put a larger drive in it just won't take full advantage of it now Let's look at the system and dive into how we're going to fail this system over now. We start with it works It's all up and running we have a green online checkbox And there's no problems with the system and the first thing we're going to do is show you what happens Just when you have something like a loose cable scenario, which has actually happened to me a couple times So we're going to switch over to the overhead and look at the drives We're just going to take and unplug the cable now when you unplug it and go back over over to here And I'm going to go to here And we're going to go to Status so we can see the drives right away. It does let me know a drive has been removed and degraded now Sometimes before a drive fails not always you'll get a read and write or check some errors because you'll have some Indicators that the drive is going bad, but that's a maybe it is not a guarantee But we have had and we've seen this in one of the enterprise systems We were dealing with we had a sled that wasn't snapped in all the way that would cause a drive to kind of come And go sometimes We're not exactly sure we moved it to another slot on the drive And by the way ZFS doesn't care what slot things are in and the problem seemed to go away We just didn't we still have another drive to put in the other slot We don't know if that slot's bad, but we know that drive is really happy in another spot The good news is even though it's degraded right now and actually let's go in and Move some data around and this is where some test data is Yeah, let's go ahead RM dash RF This folder here now the reason I'm doing this with that drive Disconnected out of the array is so some data has changed since this drive became disconnected Because you can just reconnect it and it will just join back in and how fast it joins back in is going to be dependent on How much of a difference there is on this particular drive from when it was attached? So now that we've removed some data Let's go back over here. It's still in a degraded mode All right drives plug back in let's hit refresh online It's already started syncing it and pretty much happens really fast The system is back online and its drive is now synced up with the other drives So those type of scenarios are actually really easy. So if you remove a drive, you know, even as production It's only got to re-sync that drive based on whatever data changes happened while it was in disconnect mode That's why I made some minor changes But they were so minor it really it synced up really fast because there wasn't much in that particular folder What about the next scenario when the drive fails and it's not just a loose cable. It has a catastrophic failure So same thing again here. We're just going ahead and unplug the drive now. I actually have here another drive So let's plug this one in So we'll take this drive and I got to get some power over to it All right, now we have a different drive plugged in and we'll do a refresh over here Now it sees that that drive was removed and even though another drive was plugged into the same cable ZFS isn't looking at physically what cable something was plugged into in order to be able to replace a drive CFS actually goes and determines like if this was part of a pool and it wasn't so it says well You added a drive, but it wasn't a drive We expected therefore this is where manual intervention comes in to actually replace this drive So we go here We're going to choose replace Only disc that's extra in there now I'm using the force command because I think there's data on this drive force means don't ask questions Don't do anything just erase it just I'm not worried about the data on the drive But if it does detect or something else on there, that's why it has a force option. So we're going ahead and hit replace disc replacing formatting success for replace disc and Well, it took just a very short amount of time Scanning that was going to go through the re-silvering process the re-silvering process. Well, that's it It's now taken the data from the other drives to rebuild the pieces of data that need to be on this particular drive And we have successfully replaced the drive into this pool now This question seems to come up quite a bit of can you expand the pool and all that that's out of scope of this I have a video I'll link to about expanding V devs where you can add another V dev But no you can't like as I stated before just keep popping more drives And you can only replace the existing ones now one more note These other two drives happen to be the boot pool and I'll show briefly where those are I've covered how to add into an existing in another video about backing up free NASA I'll leave a link to of how to replace one of the boot pool or add to a boot pool It doesn't have one but the concepts work exactly the same But when you look at the boot pool, it's under system boot and not under the ZFS pool And you go to actions and you'll say boot pool status, but the commands are the same You can remove one of course because it's just a mirror, but there's the replace You would choose the extra member disc and submit and that's how you would replace one if it was one of the Boot pools that had gone bad. So kind of the same concept just different location for those you wondering how to boot pool So hopefully this is helpful and hopefully you're not watching this in a panic attack wondering what to do It's actually Provided you have your race up and with enough resilience that losing a drive isn't that big of a deal It's really not that big of a deal to replace a drive. You can go in there Get the drive replaced now one thing of note This particular board is older and doesn't always like hot swapping So it sometimes requires a reboot when I swap drives around but it's inconsistent Most modern boards are going to support hot swapping Especially if you're using any enterprise equipment and when they do support hot swapping is no big deal All this can be done live without any real, you know, takedown So if you're actually working on a production system and you don't have the well Let's see don't have the availability to shut it all down. Yes, this can be done in real time That's one of the reasons the hot swappable racks are so popular And because it tries to fail especially under heavy load or just over time or just for no reason at all occasionally It feels like that's why they usually have the you know, easy to get two bays in the front All right, and thanks and thank you for making it to the end of the video If you like this video, please give it a thumbs up If you'd like to see more content from the channel hit the subscribe button and hit the bell icon If you like YouTube to notify you when new videos come out If you'd like to hire us head over to Lawrence systems calm fill out our contact page and Let us know what we can help you with and what projects you'd like us to work together on If you want to carry on the discussion head over to forums that Lawrence systems calm Or we can carry on the discussion about this video other videos or other tech topics in general even suggestions for new videos They're accepted right there on our forums, which are free Also, if you like to help the channel in other ways head over to our affiliate page We have a lot of great tech offers for you and once again, thanks for watching and see you next time