 So we did all the review of the TrueNAS server. Now we want to do some testing of the TrueNAS server We've actually been doing this for the last couple weeks I have been randomly unplugging, replugging it in, working the drives, loading data, changing configurations, and Making sure that the failover works. So these are some of the failover tests And we're going to, you know, I did a lot of testing to make sure I understood how the system works So any of the problems are, if there's a problem, I should say, with the system and not with the testing method or with My misunderstanding of the system. And I got to admit, it's been very trouble-free. Now, I will note, one, We had the technicians from TrueNAS help set this up And that's something when you buy a TrueNAS that you get is help from their technicians on the setup, on the cart failover That way when you deploy this in your enterprise environment, there's no problems. Second, It's really interesting to me how it works and it took a lot of discovery But there's a lot of neat things I discovered about this So it's been kind of a journey of learning and kind of amazement of how resilient it is. Now the usual things work like Pulling the plug out and that's going to create a status alarm and let you know that one of the power Units have failed. No problem. That's kind of expected. That's common in enterprise hardware But where it gets really interesting is how the failover between two motherboards that can both read a Array at the same time That's where things get really clever and how it does the failover while doing data transfers That's really neat and I'm gonna put the power back in now Also, I wanted people to ask how loud this is and so I figured the microphone is just at the edge of my fingertip here It's running you can have a conversation over this. It's also not a ton of heat I mean it's saying obviously with all the hard drives and whether or not you go from solid or spinning drives to solid states is going to create a heat difference But we have almost all spinning drives in there except for the two on the bottom Where you see the orange stickers one set up for zill and one set up for a cash drive Those are both solid states so they don't generate quite as much heat But overall really impressed with the resilience of this system And it's been it's been kind of fun to play with and like see what happens when you unplug things including We're gonna cover what happens for a network failure and walk you through the process of when one of the motherboards Loses connectivity on the network. What happens to the drives and your data in flight being written there? So a couple things they have set up is it's doing a file transfer right now and doing some load testing You probably see some Lincoln lights up front here, and it's also At the same time The dual failover is completely like in place and we're gonna show what it looks like for a live switch between these And show the process of how it migrates from over here to over here And we also have an ice guzzly running windows in a virtual machine attached through the network Showing what happens when you lose connectivity and it switches over to the motherboard And we're not using the luai specifically wanted to do it with the virtual IP Versus if you are familiar with ice because the lua support would be I can attach it to both IPs at once the individual IPs Instead of the virtual IPs for the failover So that's a test mode. We're gonna do this in but this does support the lua So if you're doing ice guzzly you actually can set up multiple IP Connections for it and one to each motherboard. So that is another failover option We're testing everything on the virtual IP for the purpose of this to show you what happens when a virtual IP fails Now all this is over gigabit. So we're not going to be performance testing I don't have any copper 10 gig connections I only have SFP and this particular model was shipped to me with copper 10 gig not a RG 45 10 gig I didn't have SFP on this pick your model. So I'm not going to be speed test So let's get into some of the testing and some of the details and show you what happens when we do things like pop drives out For a quick basic test, we're going to pop a hard drive out of a live volume Now this is obviously not a real hardcore test But it's to show you the resilience of the system running ZFS the lights blinking are one continuous volume The one's not blinking is another volume, but nothing's really going on and what we're gonna do is swap positions Inside of here and show you that ZFS one doesn't care only pauses for a second when there's Volume removed so you don't really have any lost time there Now that drive was removed and now marked as failed in the system and an alert was sent But by popping the drive back into another slot it gets added right back to the volume Group now it doesn't matter what slot it goes into it starts reabsorbing it. You see this is in real time It's going to start blinking and there we go. It's blinking again and writing data as if nothing happened These are one of the ways you can do of rearranging the hard drives is kind of neat because it doesn't involve breaking anything So let's kind of dig into the system here and show you what that looks like So I have one called tank sill and one called tank cash now this volume has a zill storage We're gonna talk about this in a few minutes But this one has a zill storage this one has a cash storage and so this is the Arrangement of them the one we pulled the drive out happened to be the tank sill storage now There's no errors on here now because when you put the drive back in it actually will clear the air So if it's a temporary error it logs it it alerts you but it does not it'll clear the error because oh look everything's back So you still see this data is here as healthy inside of the free NAS system now Let's talk a little bit about the arrangement of this So you see this as IP address 192.168.3.249. We can also go to 245 and Log in there which will and we'll show you the other one 247 This one's on standby. This is the second motherboard. So the active IP is gonna be 249 So the arrangement of this is 249 is our virtual IP address Shared by the two instances the two separate motherboards that make up the true NAS box This one is the inactive. It's in standby mode waiting and this one's the active one but when you're logging in for a sense of What you always log into and what you point all of your shares at and your eyes because he target? I'm doing everything at 192.168.3.249 now It doesn't necessarily tell me which one I'm logged into is it doesn't really matter unless you look to the host name One is true NAS and one's true NAS B now I've got him plugged into a unified switch to make it easier to show how things are hooked up and how things Work in here. So here's true NAS and there is true NAS B So there's a 247 one and a 245 one that's logged in now And here's where they're plugged into the boards port 1 and port 2 now You can see port 1 hardly any data going across here port 2 lots of data about 182 gigs when across here So it's dumping files back and forth and this is where the ice cut he's actually connected is here And let me show you where the ice cuz he is now. This was running in the background while we were doing the test This is a windows and I'll show you the storage says, you know win 10 Zen on iDrive and it was on this right here for storage. I'll double-click it so you can kind of see how the storage works 10 on zill storage and when we go back over here to zill storage It's an ice cuz he target of dot two four nine So just so you have an idea of the testing Environment that we have set up that both of these have these virtual IPs of Or have real IPs of two four five two four seven, but through the failover system that is called Karp. It has a Virtual IP of two four nine So as you ever have to think about is just having the one IP address and you don't have to go Which one is it? It'll switch off make we tune them and that's the fun stuff that we're gonna show how that works This is really clever So obviously taking drives out like I said kind of the boring way to do it So what happens and we're we still leave this running in the background? I still got file transfers going on I got just a file copy from my local system to this system and Data flowing here. So let's show you what happens with the file transfer and with the ice cuz you're running when we disconnect A port so we know right now this one here on port two is active Because we know this because of all the data going across you can say unify switch port two two four five So there's the active one We're gonna open up a ping here. So there's two four five pinging away and Everything's connected to it. It's the current active system. So here. This one's on standby now Let's walk through the process of actually disconnecting it. So we're gonna go ahead and click on this port here We're gonna choose to disable this port apply it's gonna take a second and We stopped pinging and when you stop pinging Two four seven which was the standby it pauses it's being elected now And we'll refresh again it takes a few seconds here and now this nodes active There's about a six second delay So it waits six seconds to see that one timed out and it switches over now. Let's go back over here Because it technically is a different system. It does make me log back in But as you can see our storage Still sitting there the same even though we switched We'll go in the background here Here's my windows 10 stayed running. I go to the council. It's perfectly fine Doing what it did now granted during those six seconds. There was a pause of data That was not going across the network things weren't working, right? But it never disrupted my devices attached across the ice because he my file transfer in the background pause for just a second and The machine stayed up and running. This is kind of the resilience of having the fail over with carp and the dual motherboards This is how it looks in action So pretty much the system stays completely up and running without really any more than this might pause and disruption All right, so let's plug the network cable back into that particular one. So we'll go back over here We'll go to the unified switch and we're gonna edit this switch and change it from disabled back to enabled and it's provisioning that port right now It's ping and again And refresh the page on this and now this IP the two four five became the standby IP So what happens when you're doing this is the system goes from one to the other when the failover occurs There's a pause while it holds an election essentially and switches over but It does not switch back to that IP not neither one is primary nor secondary to each other You can see like the host name is changed to the true NASB local the to work in unison So once failover is complete and we can actually read it here You can follow the whole process of everything happen. It detects that it's missing detects that There's a changeover. It begins the volume imports importing tank cash importing tank sales. So there's our Imports then service restarts complete allowing network traffic synchronizing Child process terminated starting up failover event complete. So there was an alert sent. There's a failover Done it works and this is really cool because these are obviously the worst-case scenarios is you know losing a network interface Or losing a motherboard because we're gonna cover that next of what happens when a motherboard failure because this was a pretty nice Handoff we just disconnected network cable. It goes. I don't see this no more and it just failed over So it's a less aggressive one and We want to show what happens when you have a real failure now It's very not advisable while the machine's powered on to pull the motherboard But ideally these are going to be in high availability environments where you can do things like that You can actually say I want to replace a power supply no problem But that's easy a lot of enterprise stuff easily replace a power supply live They slide in and out from the back as long as you get access to the back You can replace a power supply and stay up and running But how many devices can replace a motherboard while you're doing it? That becomes really impressive because there's the ability for these to talk directly to all of the hard drives at the same time and Stay in sync with each other is is a pretty Industrial feat that works really well in here. So let's show you what happens when we remove a motherboard So you can see that this is the active motherboard because it's blinking away And this is the one that's currently assigned active So what we're going to do is we're going to eject this motherboard slide it out Everything's going to stay up and running. I still have a file transfer going. I'm still running ice because he attached to virtual IP And it's out. So now that the motherboards out, let's take a look. We do have to log back in So I'm gonna log back in over here because I made sure to disconnect the active one Which is b now we're back on the one. It's just called true nas not true nas b And without missing a step. Here's our true nas b. We're logged into here's all the storage Of course, we have a warning Failed to check failover status with the other node no route to host so We didn't just fail over this time. We realized we disconnected the internal parts of the failover Because there's a separate network that runs internal besides the external visible network There's an internal network that ties these two devices together to keep Changes between the two free nas boxes in sync with each other So when a failover occurs all the information is always the same So if I make a change to a share make a change to the way the storage configuration or any configuration changed to the free nas system The internal network makes sure everything synced up between these two devices That way if one of them fails the other one can take over So now we didn't just break the external network connection. We removed the whole motherboard On the one that was active and running and without a beat all of these drives still have all their data The data transfers are going and we're going to go back over here Here's our windows box. We'll open up firefox. We opened up google last time So we can do some grinding and open something but uh, the box is still working granted It's not wicked fast, but you know it works And uh, here's this firefox session running. Well, I actually said yes to something So now google's gonna or windows ends can open up this But you can see other than the six seconds again that it paused Between these we're able to survive an entire motherboard failover In real time on this that's just a really impressive setup for how this works So once again everything's up and running and this is what you get when you get an enterprise grade system That can handle things like, you know yanking a motherboard out while it's live Uh, they gave me permission to do that I asked them if that was okay to test this uh when they sent me this unit It's like you care if I pulled a motherboard out in real time I mean I want to simulate a real failure on this thing And I thought that was kind of a novel way to do it Now I've pulled the plug already now we've pulled the motherboard During a read write operation and uh, just to give you some ideas of how things work Like I said, it's it's really neat to see that during a read write losing a motherboard did not corrupt files I can still go see all the files on it and actually pull some of those up over here Open up a window for you and show you so here's a whole bunch of video files I have from when I was just in new york And uh all the video files are Perfectly fine I can keep playing them and uh watching videos in that while I'm You know swapping motherboards on my enterprise NAS box because this thing is very very resilient on there And this is kind of neat because now I can put the motherboard back in now I have no idea because I'm doing this as I'm recording this I have no idea what happens when you put a motherboard back in on these things While it's on there So that's that's a different experiment. You probably should turn it off But um, let's keep the tests going because we want to make sure we're thorough on this All right after a short boot up time. I'm able to ping that device that was down, which was the 247 It's now back in standby mode and we can see from here the Uh master backup more freaking advertising received Maintain route deletion failed interface is the internal interface when it noticed it failed But this is the ntb interface But now it's back up and running which means these devices are back in sync now. I still have a critical I haven't cleared well actually I just have them refresh the screen now that I clicked on it You notice it's healthy, which means everything's back to normal and everything's working once again I still have a file transfer running and we'll show you the in the background here This is still working fine back here the windows box is able to open and do things And all the drives are still showing healthy here without any issues at all. Everything's online So you can not only remove a motherboard you can replace and put any motherboard here I mean granted provided the configuration is same But I imagine in a failover situation if you had to get a new motherboard in one of these have Failed the people over at true nas ix systems would be able to ship you one out And you just copy the configuration files over from the live running one And copy them over to the other one and you can get this thing back up and running Now the last couple things we showed removing a volume while it was working But it was just one of the regular volume. So let me give you a breakdown of everything in here Here is the zill tank And here's the cash tank now the difference is with zill and cash. Let's talk about that for a second Now i'm not going to read you the entirety of these articles, but first let's talk real quick about a s log or zill how zfs uses this and kind of a quick overview So isn't zill just zfs name for our right cash a right cash So you kind of think of them as the same thing but this causes some confusion understanding how it works And the best can figure first of all zill is more accurately referred to as a log whose main purpose is actually for data integrity Now this is kind of the Details of the zfs file system and you like I said, there's an entire they break everything down about how this works So here's our tank with zill There's the drive So here's our raid z2 and here's the single drive we have dedicated to that now Because of the way zfs works and we're dumping data to the zill and then read ready synchronous You worry. What if that one drive failed now? You can put these into their own Raid ray, but this is obviously single standalone drive. So let's just talk real quick of what happens if you lose that log Now this comes back to the way zfs works and some of the details of the file system But if you do lose this uh zill drive right here Obviously that seems concerning because there's only one of them. So let's go ahead and disconnect it So the hard drive with the z sticker is the zill drive You can see all the active writing and because of the way the zill works It only blinks a little compared to the rest of the drives in the zill ray Or as I call it tank zill. So all the other drives in there are blinking away loading up the data And you're only doing that in a couple logging so you get a blip here and there on that drive now once we eject it You can see that the failure goes here. It says uh the serial number of the drive was detached The peripheral was destroyed uh kernel Exit so what it did was Going over here We can see that the drive has been removed now we can replace it and put another drive in its place But we do have a failure integrated mode, but on the other side of that We didn't end up with any file system corruption. Matter of fact The delay happens almost instantaneously that it removes this drive. So there's really nothing going on here So all of this working just fine over here and it's really none the wiser Obviously the performance has been changed and we don't we no longer have that Zill drive in there But in from a functionality standpoint the system is working perfectly fine Our ice because the attachments are the same. I can zip around through here So we're none the wiser and we can you know get replacement drive on here Now one you know thing I mentioned before in the original review of truenas is the ability For them to see this as well So you can if you have one of their higher levels of proactive support and this has not turned on for this demo machine But it can actually alert them when a drive goes bad so they can get one ready for you So if you're busy and not doing it this system actually can Notify the truenas support engineers with your warranty and they can realize when a drive has failed or has a failure mode problem Or maybe when you yanked out a motherboard So they're very active in this and able to do this able to see it in real time Which is just a really cool feature. So now we've degraded this one But as you can see no big deal everything stays up and running So we've survived that level of fault challenge as well Now the last thing we're going to talk about here is the cash one So cash drives work a little bit different. So let me explain those real quick All right, so I pushed in the drive for the zill drive and it's back in the system and it's back being happy I cleared the error. You can see we are showing healthy. We look over here And no problem everything's you know back up running like we wanted it to be So now let's talk about caching and we've got that set up over here. So I moved from the windows 10 one here Let me force a shutdown. I don't feel like waiting for this And I creatively named this one windows 10 cash money volume So this is a another instance. I just duplicated it and this is the true-nash cash volume same one here Same storage device on the dot 249, but it's going to a different volume And real quick how I did that was services Go over to your dice because he this one Is the cash volume. This is the zill volume. This one's on tank zill This one saying tank cash and when we look at our storage Tank zill tank cash pretty straightforward. So now I'm switched over to this tank cash one And I got a bunch of data dumped to it. So This is l2 arc is your level 2 adaptive Replacement cash So once all the space in the arc is utilized zfs cfs plays the most recent and frequently used data in the level 2 adaptive cash So what this is is first you have your memory read cash That is obviously really fast, but you know, you can only put so much ram and a device before that gets kind of fracy So in between you can put an ssd. So all the frequently used files Can you get migrated over to the ssd as an l2 arc? Now when you're looking at free nasa how that's set up We go over here to the volume And here's the Raid volume itself and here's the cash Now this is a little tricky because getting all the data over there and you only have a gigabit connection to it It's not going to put a ton of data on there But some of the data will go there Especially running like the ice cozy. So let's go over here General storage i'm going to pull up my nyc trip again Find a big file And if I open up a couple of these big files This is just before I left it was me. I'm playing in the gopro We'll pump this one So now I've opened up a few files. So some things should be pushed to cash And of course while this is running I have the Ice because you're running on the same drive and what I'm doing here is showing you from the command line Here's the pool called tank cash. Here's the cash drive and Looks like we got about 272 gigs of data on there and it's kind of reading back and forth So sometimes we have a bunch of reads and writes going. Let's jump back over here to our ice cozy box Go over to the council on this one Let's launch a couple things So it's doing some more reading from that drive And you can see the cash hits here and now they're jumping up a little bit because we have a couple Reads and writes as it pulls data back and forth to and fro on the cash now while it's doing this. Let's eject the cash All right with the cash drive out. You see everything has now dropped to zeros on there So let's see what's going on with our windows box over here I don't want to use firefox as my default browser on this one Let's google some things. We're doing that. Let's browse some files over here Yeah, I can still open up all these folders. I can open up websites. So this Machine hasn't broke at all Let's go back and open up that video Yep, that played reasonably flash opened up a new video out of there. Some of the other videos we had previously played Now pull you back over to here and i'm jumping on screens But so still see no cash all zeros here The cash is just not working. So let's go back over to our free NAS box or to our true NAS box and you can see Perf invalidated std task requiet It basically failed it out per peripheral invalidated peripheral destroyed We have a critical to let me know that a hard drive was removed the volume tank cash is online But one or more devices been removed by the administrator Sufficient replicas exist in a pool to continue functioning in a great state Basically, what it's like, you know is it failed but It's okay because the system can keep working So let's go back over to the volume status and there we can see that it's removed Let me clear the air so I don't have a little red light anymore So now the cash is missing, but as you noticed We didn't miss a beat. We can still read and write to the drive We did lose that fast cash from the l2 arc, but No big deal there because it'll rebuild itself and everything can go back to being happy at that point You know, these things are it removed these from the it removed this from the tank cash Uh, we lost that performance. We take a hit there But the data lives on the system moves on and we didn't have any downtime Which of course is the most important thing with these because we all know in the real world as much as we like All the hard drives last forever. They don't uh hard drives might get ejected You know by someone walking by and touching it And someone putting their fingers on it whatever may bad thing may occur This thing has a incredible amount of fault tolerance to do it So I've been really impressed overall with the performance of this with the fault tolerance and This system as a whole it's been a real pleasure Working with it and loading up all these different scenarios on it So I give a big thumbs up to the people at ChewNAS. They said they lent me this for this testing in review And they so they were more than gracious, but that really was an influence decision I already liked FreeNAS before I liked ChewNAS just as much But if you're looking for enterprise level High-grade server that will suffer through Some torture testing this box is a good one to choose. I'm definitely really happy with it All right, Felix the content here like and subscribe if you have there's some more testing or Some of the questions you had about this where I can reach out to the FreeNAS team for you or you can tweet at them They're very responsive. Hopefully this review was helpful and I catch you better understanding of just how good this system is And how well it works to fault tolerance. All right. Thanks for watching