 Now I'm here from Lauren Systems and we're going to talk about TrueNAS 12 and ZFS Replication. Now ZFS Replication has been around in the FreeNAS for a long time and with TrueNAS 12 moving to OpenZFS 2.0, they've added the modern encryption methods that are available in ZFS 2.0. Now I'm bringing this up because you're probably thinking you already know Replication and you probably do if you've used it in previous versions, but the encryption adds one more thing that you have to make sure you're keeping track of. And that's the keys for the pool. I want to cover exactly how that works and how ZFS Replication does not only what it used to do, but a little bit more with the keyed Replication pools, including the fact that you could set a destination pool that doesn't have the keys and you can actually have a server that lands that can hold all the data, hold all the snapshots, hold all the backups without being able to see them. So there's some nuance to the way they changed it. I want to do this updated tutorial to cover all of that. Before we dive into those details, let's first. If you'd like to learn more about me and my company, head over to LauranceSystems.com. If you'd like to hire a short project, there's a hires button right at the top. If you'd like to help keep this channel sponsor free and thank you to everyone who already has, there is a join button here for YouTube and a Patreon page. Your support is greatly appreciated. If you're looking for deals or discounts on products and services we offer on this channel, check out the affiliate links down below. They're in the description of all of our videos, including a link to our shirt store. We have a wide variety of shirts that we sell and new designs come out, well, randomly. So check back frequently. And finally, our forums, forums.LauranceSystems.com is where you can have a more in-depth discussion about this video and other tech topics you've seen on this channel. Now back to our content. First thing we're going to touch on is understanding how ZFS keeps your data safe, but I will not dive too deep into this topic because there's a lot of nuance to it and it's the debate that will always come up of why use ZFS replications over something like R-Sync. And being about a thousand times faster under a particular scenario that they were doing some of the testing in, there's a couple of good reasons too. I don't want to get too far off topic. I will leave these two links here where they break down the testing methodologies. As a matter of fact, this whole R-Stechnica article is really good on those of you that may not understand some of the fundamentals of ZFS. It's got a good explainer with some visuals and some animations. I don't want to get too off topic in this, but at least I'll leave you a link so you can further reading of why you should use it over something like just R-Sync. Because there are scenarios that it may make a lot more sense to do. Now let's get started. We have some test data here and the idea is we're going to take this test data. We're going to replicate it to this system right here. So this is system A and this is system B. They're both running TrueNAS, 12.0 U1.1. So there is up to date as is available on February 1st, 2021 here. And right now, this pool has no snapshots and no type of backup tasks set up. So we're going to be setting all this up from scratch that way we can show you how to do it. Of note, this pool is encrypted using keys. It's unlocked, that's why we have this right here. And this pool here, storage pool, does have its own set of key as well. Matter of fact, this is actually some of my production data as I'm doing a long-term test on this system for those wondering. All right, back over here. We're going to go and start the replication process. Now the first thing that can be confusing about replications is they're based on snapshots. I've got several videos on snapshots and the snapshots in TrueNAS 12 work the same as they did in previous versions. And replication still requires a snapshot. And bring this up because where sometimes there's a little bit of confusion is people will rerun a replication task thinking it's backing up every time the replication task runs. Not necessarily. It is re-synchronizing the last snapshot that you've attached it to. Now the good news is when we create this, it's going to create the replication task and the related periodic snapshot tasks. You can have more periodic snapshot tasks than just the one needed for replication. So if you have some other strategies to create snapshots to your data, but it is important to know that the replication is being based off of the task that is linked to it. So replications need to have a snapshot to run. Just of note when you're doing this. So we're going to go here, source location on this system. And there's our some test data to back up, recursive. This sometimes creates a little bit of confusion. This folder, the some test data, which I have right here, I've just drew a few video files in there. Matter of fact, we can create a new folder, some other folder. Actually, let's just, it looks a little less messy if we move a few of those things in there, right? We'll just drag them in, a few more in there. All right. Now we have subfolders. I'm bringing this up because this is where the recursive sometimes creates confusion. But to also replicate all snapshots contained within selected source. This is for data sets, not for folders. I bring that up because even though this has folders, those folders will be and will show the data on the destination system. They will be replicated. You actually don't need to do recursive unless you're nesting data sets. So if you just have one data set and it's a share like this one is with no sub data sets, it doesn't really matter. Data sets do show up as folders, but it will automatically without checking recursive back up those folders. All right. That's all you have to do on the source side, destination side. You have the option of on this system or on a different system. The reason you have the option for on this system is it's actually a handy way to get data synchronized between two pools that you have. So if one system has more than one pool, then you could actually duplicate the data between those pools. Now once again, it's all in one system. So it's not necessarily a good disaster recovery plan. But if you have two separate pools, if something happens to pool A and you've replicated it on pool B, now you have all the data. More ideal for disaster recovery planning is on a different system. Now it uses SSH as its transport layer. With SSH, all the connections are encrypted. They've made it really easy to build these SSH key pairs through NAS mini. This is the TrueNAS mini and we'll just use the semi-automatic TrueNAS core system for doing this. What's kind of cool is we're going to go here, just grab the URL and paste in literally the URL. And it's going to go ahead and log into that. So we have to go over here. I'm just copying the password out of it. I use Bitwarden for those wondering. Generate new private key. What this is going to do is create a secure connection so this server can talk to that server. So create connection. Once it's created it and logged in, I can actually hit this and it sees the data sets here. So here's all that data. Look over here at the pools. It's able to see it. Now we're going to drop this in and some test backup. So there's the backup that we created, some test data backup. Don't bother checking this. This is out of scope of this particular video because you'll see how the keys work next. Don't mess with the encryption options unless you know what you're doing. Go ahead and go next. Run on a schedule. We can have it set here to daily. That's fine. You have the option weekly, monthly, hourly. And if you do custom, you can get more granular for how frequently you want this to run. This is the task for both the replication task and the snapshot. The snapshot's going to occur and then replication it has to occur. So we're going to head and start replication. It's pending right now and it will run at the time that's set. But we're going to go ahead and force it to run now. Replication, small drive testing, some test data, test started. All right, great. Now it's running and going to copy all of those data. Now, because there was no snapshots on here, let's actually first, here's that snapshot test that it created will be not test, but the test data snapshot that we created here based off this replication test. If we go over here, storage, and we look at snapshots, there's the snapshot that was created along with it. We didn't have any prior and now we have one. Now it's created a snapshot. It's sending the data over, go back over to tasks, replication tasks finished, all the data is copied. You can look at the logs, download the logs if you want. But it's done, it's copied. Go over here, click back on pools just to refresh it. Hey, look, some test backup data. Out of note, this is encrypted. We got the key right here. And if we go here, we filter snapshots. Hey, cool, we can see the snapshot. But what we can't see, and this is the important part, is the data that is stored in it because we don't have the key on this destination system. So I SSH'd into it. So 2.13, 2.13, we go to the folder. There's my LTS videos, Zen, ZenLab. But you notice how the folder is not showing up. So we can LS all we want, but it's not there. That's because without the keys, there's no way to actually get in and see that data. So the sum test data is here, but it's encrypted with the keys. And we can't do anything until we decrypt it. Now, this is that important part. If you didn't back up your keys, for example, and you were sending all that data and it's landing encrypted, and that's perfectly fine, but you have a catastrophic failure of the system. Now, I have a video on how to back up keys, but if you were to not follow that and not back up the keys, that catastrophic failure would also mean you just have a pile of encrypted data on the destination system. And without those keys, you're unable to do it. The keys are not transmitted with the ZFS send. And now we can go over here, though. Go to storage, pools, and we can just go ahead and export the keys, password in, continue. And there's the key, or you can hit download, and it downloads a text version of the key. So pretty straightforward on how to get the keys. But this is a kind of a little bit of a bug right now, and I have a bug report file just so people know, because I want to cover one other scenario when you're exporting the keys. We're going to export them both ways. So we can export the dataset keys this way as well, and it created a JSON file with the keys. So here's that JSON file with that particular key in it. I'm bringing this up because let's show you how we unlock it now. We're going to go here. We want to unlock this use file. Let's grab that JSON file, submit, and it fails. This is, and I'll leave a link to this for those wondering the progress of this bug. This bug exists now. If you're watching this sometime in the future, this may have been fixed after an update. Looks like it's targeted for 1203. But FYI, if you try to pull it from the JSON file, that's broke. So let's go ahead and do it the other way. So I'm going to cancel. Alls we have to do, either pull it from that JSON file or when it popped up, just copy, paste that encryption key or even like this file here was just a text file of that key. You just need to paste it in. So you uncheck this box, instead of a file, you just drop the dataset key in here, it submit, confirms a little green checkbox, dataset unlocked, there you go. Now this actually is stored within the system now. So once you've saved it here and decrypted it, when I go to export the keys, export dataset keys here out of this system and I download this, there's two dataset keys. Here's the first one for that and here's that some test data backup and there's that key again. So it does cumulatively collect the keys. So once you've done this once, you don't really have to worry about the system over here because well, those keys are now backed up and copied over. Now, because we unlocked it, we'll bring this back over to the screen, LS and now we can see some test data backup that is now showing up and there's all that data that's in there. So we have the some other folder and roll the two folders that we had in here but you get the idea. The data is now viewable. By the way, this is important. The data on the destination system is going to be set to feed only. The reason for that is if you're trying to do a differential, every time the system does a ZFS send, it's going what changed and what needs to be sent, what's different. If you are messing with the data over here, you are going to create some problems and a bunch of errors. It by default sets us to read only because it is read only to you on this particular system so you don't delete or goof with the files. The files are visible, you can see them, you could pull some file and move it somewhere else but you are not supposed to be modifying the data on the destination system. Now, one more thing I had mentioned because this is based on snapshots. Let's go back over here to tasks, replication tasks and we see it ran, it finished and we set this to run every night. Let's actually, not that folder, this folder. There we go. Let's delete all this data. Let's just purge a bunch of it and delete. Yep. All right. The only thing we have right now is just this folder and that's it, these two folders. Actually, let's get rid of that. Let's only have one folder, even less data. Now, I've made these changes so it's gone, that data went with it. There is a snapshot. I can roll back to that snapshot and this is where sometimes the confusion has come in when people say I thought the replication task would grab it because I went ahead and made a change. It run now, it continue and nothing seemed to happen. Reason, nothing seemed to happen because there still isn't a new snapshot. So we go back over here to storage, snapshots. There's only the one snapshot here. So when we go back over to this system here, good news is data's still here. You can keep running the replication task over and over but without a new snapshot, it's not going to try to determine what's different. It's always looking at the snapshots, never looks at so to speak the active data. It always looks at the snapshots to determine if there's a change that needs to be sent over. So even though we just deleted a bunch of files which would create change, there's not really any change created. But let's go ahead and change this up a little bit more real quick and show you what does happen. We're going to go over here, tasks, replication tasks. We're just going to destroy this replication task, delete, confirm. I got this snapshot still exists but we're going to go ahead and get rid of this one too. But that does not actually get rid of the snapshots themselves. So we go back over to snapshots and we're just going to roll back to this one. All right, refresh. Hey, there's all the data packs we roll back to the snapshot. Now they roll back to it just because I don't need it anymore. I'm going to delete this snapshot because we're going to create a new task again. We'll add another replication task, this system again, go here, same thing, we're just going to back up some test data. Once again, don't need to do anything recursive on a different system. We've already established last time this SSH connection so we can just reuse it again. Demo two. I'll put a second demo we're doing here. And next, custom. And just for speed, we're going to set this to run every minute. So if you put the asterisks in for each of these, every minute it's going to run this task again. Done. Start replication. And then now let's go ahead and just skip ahead one minute. We'll start letting this replication start pushing all the data over. Now of note, if you do something this fast, make sure that the first replication, which takes the longest because it's only differentials afterwards, that you don't set it so they start overlapping each other. It'll give errors if you do. Sometimes it may be necessary, but just an FY, it will error out if you try to put too much data in there and the replication tasks have not, if the first one hasn't completed before the next one wants to begin in terms of time, because of how long it took the data to get from point A to point B, you'll end up with some problems. All right, the data has been pushed over, over here, there you go. And there's the demo two. Now, even though the encryption key is technically the same key, because we didn't change the encryption keys from when we did this, when we did delete the data set off of the destination server, it did purge the keys with it like I said. So it doesn't unlock it. So we have to kind of go through that same process again in order to unlock that data. Well, actually, let's leave it locked because I want to show you how disaster recovery works and how you don't necessarily need to have the data unlocked on the destination system. All that it's doing is holding data. It doesn't actually have to know what it's holding. It is just the data. So this will run and we'll go ahead and actually delete some files again. Go here and just delete something. If you have any data, there's some of the data in there. Purge this real quick. Eat all that. All right, we deleted. So there's some changes going on. Wait another minute, let it run again. All right, we waited a few minutes and this is the destination system and we see that a few snapshots have been copied over. Here's the reference ones. Here's the changes because we deleted some of the files in the snapshot. And so this is just running every minute. Now, what if a catastrophe strikes? This is where the question happens of how do I get the data back? There's a couple of different options. So if the catastrophe is, oops, someone deleted a bunch of files on this particular system, ideally you just go back and roll back a snapshot to bring all the data back on the local system. But let's go ahead and create a total disaster scenario which unfortunately happens occasionally. We're gonna go over here and let's delete the data set. Well, first we'll stop this from running. So I don't get a bunch of errors when I delete the data set. We'll delete the accompanying snapshots and let's go over to storage pools and what's worse than destroying a pool, right? So let's get rid of the pool. Export, disconnect. Destroy data on this pool, absolutely. Confirm. Now, it's actually gonna get rid of the SMB share as well but destroying the pool other than physically destroying the machine. Yeah, this is bad. All the data on that pool was destroyed. All right, not ever something you wanna see unless you're confident in your backup. So let's see, how good are we on the backups here? Go ahead and create a new pool. And this pool is gonna have its own encryption key. But I understand this is gonna be encrypted. New pool. All right, let's show some drives in here. Yeah, RAID Z1 is plenty for this. Contents will be erased. Don't worry, we did it in last action. So no problems there. Let's go ahead and create a new pool. All right, new pool created. So we're gonna go over here to tasks, replication tasks. Add. Now, this is under some assumption that you backed up all this system and that you have the keys installed. And I bring it up because when we say source location on a different system, it's gonna do the true and ask many. If not, we can create a new connection if you didn't back any of those up because the data is what's most important. Source, go here. And what do we call that? Demo two. There we go. And destination. New pool. Stored data. From, yeah, just restored data is fine. I don't like typing anymore. It even found six snapshots that's gonna pull back over here. So we're gonna hit next. Run once. I only need this. I don't even need to make it read only. I just need to grab that data from my backup and pull it back over. So ahead and hit start replication. Replication has started. All right, replication has finished. So let's go over storage. Look at our pools. There's all the restored data and all the restored snapshots. This is before I deleted some data. This is what happened after. So now I don't just have the data. I have the revisions of the data that were created by the snapshots. But I of course have the new challenge of how do I decrypt it? Oh, we're gonna go here to unlock. And if we were smart and we followed my previous backup video, we would have backed all this up. So let's go ahead and get that particular data out. That's that key we needed. This is why backing up dataset keys are so important. Continue. And it's decrypted and unlocked. Now this whole time, just for reference, this system here still has no idea what's in here. It just is a destination. It held all the encrypted data. We survived by destroying all the data over there and restoring it again. And this system still has no idea what's inside of that folder. This is one of the nice things about the encryption. So you don't have to have the same level of trust, so to speak, because you're encrypting all the data at rest. And there's not necessarily a reason to have it unencrypted. You can go back, completely restore it and be able to see all the data. We'll just shell into the system real quick here. Mount, new pool, restored data. Hey, look, enroll in some other folder. It's all in there restored just like it was because that's, well, we deleted all those little files. We could, of course, roll it back to a snapshot and have all the other data back if we wanted to, but you get the idea. This system is still blind to this data. It's encrypted at rest, and it doesn't have to know anything about it. As long as you back up that encryption key, which is obviously incredibly important, because if you did not, this process would have failed at that point where unable to update the key. One little quick piece of a rata before we finish this is going to be, if you go here to encryption options on this dataset that we just restored, it really doesn't matter any dataset, really. And we switch it to a passphrase instead of a keyed authentication. We're just going to set it to 12345678. Not a great password, but if you use this instead and then do the replication, this will change because now all you do is have to remember that the passwords 12345678. So if you are doing that type of replication when you replicate it over to your server, it's just going to need the passphrase. So for those of you that aren't using keys and you're using passphrase instead, same process, everything's the same, except you don't have to worry about the key, you just have to remember the passphrase. So I want to make sure I leave that in there for those of you using passwords on them. So hopefully you understand the better, the entire process by which CFS replication works and why it's so imperatively important to back up those encryption keys and also get you some better understanding of that you can have your data resting somewhere else, but the encryption key missing means it's not easily accessible. All right, and thanks. And thank you for making it to the end of the video. If you liked this video, please give it a thumbs up. If you'd like to see more content from the channel, hit the subscribe button and hit the bell icon if you'd like YouTube to notify you when new videos come out. If you'd like to hire us, head over to laurancesystems.com, fill out our contact page and let us know what we can help you with and what projects you'd like us to work together on. If you want to carry on the discussion, head over to forums.laurancesystems.com where we can carry on the discussion about this video, other videos or other tech topics in general, even suggestions for new videos that are accepted right there on our forums, which are free. Also, if you'd like to help the channel in other ways, head over to our affiliate page. We have a lot of great tech offers for you. And once again, thanks for watching and see you next time.