 In this video, we're going to dive into ZFS replication, how to get your data from your primary TrueNAS backed up to a secondary TrueNAS, which doesn't have to necessarily be on site, by the way. So this is a good way to do backups locally or even off site, provided you have two TrueNASs. That is a prerequisite for this particular video. Also worth noting, if you have more than two TrueNASs, I'll cover that too and show you how you can do with a single snapshot task and have multiple replication tasks and cover how encryption works for those of you that are curious. So let's get started. Are you an individual or forward thinking company looking for expert assistance with network engineering, storage or virtualization projects? Perhaps you're an internal IT team seeking help to proactively manage, monitor or secure your systems. We offer comprehensive consulting services tailored to meet your specific project needs. Whether you require fully managed or co-managed IT services, our experienced team is ready to step in and help. We specialize in supporting businesses that need IT administration or IT team seeking an extra layer of support to enhance their operations. To learn more about any of our services, head over to our website and fill out the hire us form at lorenzsystems.com. Let us start crafting the perfect IT solution for you. If you want to show some extra love for our channel, check out our swag store and affiliate links down below that will lead you to discounts and deals for products and services we've discussed on this channel. With the ad read out of the way, let's get you back to the content that you really came here for. Now I wanted to go over the demo today in terms of how we're setting it up and explain what ZFS replication is and really quickly what it is not. It is not ZFS synchronization. It is not a tool that synchronizes in real time two different NAS systems and two different ZFS datasets via this tool. It requires that a snapshot be created. But let's first start with we have our main share NAS which is at 172 1616 22 and our backup one is going to be destined at 172 1616 five. And of course, yes, this will work over such as a VPN. As long as these two can talk to each other over the network, this will be able to send this data. Next, the data you're sending is not the active data that you're reading and writing. It is the point in times created by snapshots. So the process works by first we set up a snapshot task. And we have a wizard that actually I will show does both of these at once. But the snapshot task is creating a snapshot in time of your data. And if you're not familiar with snapshots, they are one of the really great features in ZFS that allow you to take, let's say 100 gigs of data, create a snapshot, and then each subsequent snapshot after will only be the differential in change. So each snapshot is only the size of that differential. This becomes really, really handy. Because now once we do the initial send of the snapshot, the subsequent snapshots as it sends over to the destination system, it doesn't have to send all of the changes only the block level changes that have occurred. The difference between a block level change versus a file change with a tool such as our sinker when you're copying files is changing the name of a file or moving where that file is only changes a few kilobits of data, even if it was a 20 gig large file, because those few kilobit changes are moving it in terms of the pointer of where it may be, the replication task will run. And because it's based off of snapshot tasks that only seen those few kilobits of change, those change sinks when you maybe move a bunch of project files into an archive folder, don't have the detriment of trying to resend all that data because it's not seen as new data. It's always looking at things from the block level. Now, because it looks at the block level, you can do this with a data set snapshot or a Z vol snapshot. Z vols are block storage, data set is file storage. And if you're not really familiar with how those work, and you'd like to learn more, I do have a ZFS one on one video linked down below. But I wanted to make sure that this was very clear that everything is done off of these snapshots. And we will show using a wizard how to make this really easy insurance that you can create a snapshot and replication task all in one pretty quickly with a few steps. But there's also the advanced way I'll show how this works, where you can customize how your replication tasks run, and you can run them independent of the snapshot tasks. But you have to remember, ZFS send only sends the actual snapshots across and not the actual live running data set. So everything's going to be really keyed off of the snapshots. Both of these systems are running TrueNAS scale dragonfish 24.04 release candidate one, which should be in full release quite soon. The demo pool on our source system is called well, demo pool. And this is the 172 16 16 22 system. And I've themed it in a light blue. So you know, when I'm on this particular system, the other system, which is named dozer is at 172 16 16 five. This will be our destination system. And it is also running TrueNAS scale dragonfish 24.04. Now let's go over here and look at the data that we have, which we have your important data and your important data encrypted. I have two separate data sets with a little bit of data in each one. And we're going to start by replicating this one and run through the wizard. Once you to note, there are no snapshots on this, or this data set, neither one of them have any snapshots right now. So that'll be created as we run through the data protection wizard. And there's no snapshot task. There is no replication tasks. Now we could add a snapshot test and then tie a replication task to it. But we'll go ahead and do it as a replication task. And if you run the wizard, it's going to take care of both. The source is going to be this system. And we're going to choose that particular data set and destination is going to be on a different system. SSH connection, you can do this from here where it says add new. And this will kick off the wizard. And we'll name this one dozer, we put in the URL for the other system, the username is admin, we're going to put in the password. And then we're going to choose generate new for the private key. And we'll scroll down to the bottom and hit save brings us back to the wizard. And you can see it says, well, the misspelled dose, but that's fine, you can rename it later, now she show you how that works. destination is the dozer system. And we're going to go ahead and put it right here. And we'll put the same name, the your important data. So that's what we want to replicate. And we'll do that right there. So now that is the your important data that will land over here. We're going to choose no encryption, less secure, but faster. This is because it's going to use netcat to send the data essentially raw across. And because this data sets non encrypted, technically, someone could get through the encryption by sniffing all of the packets. If you choose this, it is much slower because it encapsulates everything within SSH. Then we're going to go and change the name down here, send data to dozer, I'm going to actually probably call it send your important data to dozer, we'll go ahead, hit next, run on a schedule or run once we want to run on schedule, you can choose daily, hourly, whichever works for you. And we'll show you how to edit this once it's created. You really want to leave this at default unless you have a very special use case, because same as source means, as we get rid of the snapshots on the system, when we set the retention for them, there will also be a mirrored retention on the other side. If not, you would potentially run out of space on the destination server. Now I'm going to go ahead and go over here before we run this job, go to credentials, backup credentials. And here's that SSH connection, you can actually add more of them here, it just brings up the same menu if you want to add another connection. But we're going to go ahead and edit this connection. So it's actually spelled properly, then we're going to save. And now this one's called dozer of note, this is also if you were to move that destination server to a new IP address, this is where you would change that, hit save, and it will automatically be and let's go back over here to our data protection, it will automatically fix the connection here. So now you can see it says dozer, it's the only connection option we have. And this is now run. So we've got a finished snapshot and a finished ZFS send. So let's go over to our other system and take a look at the data that moved across. And we'll go here to data sets. There's the your important data, it is unencrypted just like it was sent. And there are the snapshots. One is the initial that was essentially like a seed snapshot. And one is the snapshot that matches the snapshot on the other system. So there's not actually going to be any data differential between these particular snapshots, because they essentially happen at the same time. But now they will happen on each hour, every hour. And let's go back over to our main system here. And if we edit the snapshot task, we see the periodic snapshot, and we have the snapshot lifetime, we can set this maybe to one week, or even one hour, however long you want to keep these snapshots for or however many you like to keep maybe want five weeks of data, as I noted, snapshots are only differentials. So therefore they don't take up much space depending on the amount of data change that is between each snapshot. But this retention policy that we set here is then automatically reflected when we go to the replication tasks, because it has the snapshot retention policy set same as source. Now worth noting that if we go over here to this system, we're going to go to system, let's go to the shell, and we'll go to the CD mount doser your important data. And there's all the data in there, I've got my means backed up, I got a tome of backup best practices backed up. So all the data that is on each system is the same, we can do that by also doing going here. And we'll go ahead and go to shell. And we see it's the same files on both sides. And if I were to delete something here, when the next replication task runs based on the next snapshot, that would also match these right now. So if I were to delete them now, such as let's delete the backup, and that will now be also deleted when we go to the next snapshot that sends. But let me point out something here, let's go over here to our data protection. And the last one was four minutes ago, this runs every hour. But if we run this job again, replicate now, sure, let's replicate now less than a minute ago, the tome of backup is still there because we did run the job again, but we didn't tell it to create a new snapshot. So it didn't have anything new to send. So it will keep running the replication task every time you hit it. But unless we have another snapshot to send, there's not anything else to replicate. That's an important thing to understand that these are not necessarily, back to what I said earlier, real time, it is always based on when we have another snapshot task, and then this replication task run. And you can actually have, if you were to separate these, this running at a different interval than the snapshots, maybe you want snapshots running on a pretty regular basis, but you only want to send at a certain time, you can modify these to instead of queuing them off of the snapshot task, you can actually have them to run on their own task. So you can build a secondary schedule where maybe you only want this to run once a week, and then it'll send all the snapshots that occurred since the last time it's sent. That is another option you'd have for these. Let's go here to data sets. Let's look at the year important data encrypted. Let's look at the encryption it has. This is using key encryption. And if we hit export key, there's the key. This data set automatically unlocks because this particular key is stored in this TrueNAS system. But if we send this somewhere else, and that key will not be sent with it, therefore, it'll be encrypted on the destination. Let's just run through the wizard again and set that up. So we'll add a new task. Source will be this system, destination will be a different system. We already built that connection called dozer. We'll choose the source, your important data encrypted, and we'll give it that same name over here on dozer. We can still say no encryption because this is the transport encryption, but technically we're sending encrypted data so that doesn't really matter. We'll give it the name, send your important data encrypted to dozer, hit next. And we're not going to run this on a schedule, we'll just hit run once just to show you another option. We'll hit save. And it automatically starts a task when you say run once it's going to create one snapshot. And then automatically start setting the data while it's setting the data. This one's a little bigger. You'll see this in all the snapshots that are going with it. So it's sending one of one snapshots. And we'll wait for that to finish, which was pretty fast. And we'll go over here. And first, let's look here. We see your important data. You see the backups, but we don't see anything encrypted. Let's go to the data sets. And there it is. But it says locked. And if we unlock it, it's going to ask us for the key. Now, providing a key is pretty easy. So we can go here on this one. And we go over to our data sets. And we can hit export key, and we can simply copy and paste it in. But something I want to note, if you wanted to bring that data back, this system does not have to have any awareness of that key. So let's actually do that before I put the key in. So whoever's operating the system has no visibility into this particular data set, they can see how many snapshots it has, and they can see the name, but they cannot see any of the data inside of it. But if we wanted to bring that data back, we can go here to our data protection again. And we can create a poll. So if we actually wanted to go through here and see source and destination, we can actually say on this system and pull it from a source. But to make it even easier, we can actually hit this little button right here to restore it. So let's go ahead and hit restore. Bring back your data is what we'll call the task, where we want to put it back on our demo pool slash brought back data, I'm not going to send it to the same destination. I want it to go here, and we'll go ahead and hit restore. And it's going to kick off a task. As soon as I hit run this job, just say, let's bring that desk, which I did get an error, because it's not enabled. Check the little enabled box. And let's go ahead and run this. And there we go. Now it's going to start bringing back the data. And we'll see a task running right here for that job. And it's going to pull it all back. Okay, the job finished. Let's go look at our data sets again. And there's the brought data back. Now it doesn't know it has the key because the key is assigned right here. So if we go to export key, copy, close. And then we're going to hit unlock. We're just going to provide the key, paste it in. Continue. And now we've unlocked this one and stored the key that will be stored within this particular TrueNAS. And it stores it not on the ZFS pull, but it stores it in the backup for the TrueNAS. Or as I noted, you can simply go here and you can hit export key and actually download it. Now if you wanted to access that backup when it's on this side, we can do the same thing. We can click on this, we can click on lock. We're just going to paste in that same key. We're going to hit save. Continue. And now this one's unlocked here, which also means when we go back over to the system shell, you can see that it's now available for us to go into the files. And we can see what's inside there. Now let's go back over to this system. And I wanted to point out that there is still only a single snapshot tasks, because when I did these as one time tasks, it just created a snapshot so we could send the data over. But it's not creating multiple snapshots. So if we go over here to the data sets, and we look at the your important data, there's the snapshot for it, because there is no task associated with this snapshot to automatically destroy it. So let's go ahead and delete it ourselves. Delete, go to storage. And let's go ahead and get rid of these. So we don't need them anymore. But we can decide if we want to get rid of this one, I can let the snapshot test continue to run. But let's create another task, a new replication task that's done manually keying off of this. So that runs hourly. So let's go ahead and say, advance replication, name, let's call it, send your data to dozer, we'll choose dozer as an option. This is where we can choose push or poll, we want to push the data over there. And let's say SSH was netgex, it's going to be much faster. We can set manually if we need to, the net cat active list and address, but by leaving these blank, it'll let you know that it will automatically assign those. We'll choose the source, demo pool, and we'll be the your important data. We'll choose on there, the your important data. And I do recommend checking this box right here. This is replicate from scratch. If you ever make changes on the other side that don't sync up, the system will fail because it realizes there's some discrepancies that you change. If you choose this box, it will start rebuilding them if there's a discrepancy on there, use it with caution, in case there's something that you don't want changed on the other side, but of note, that is what that button does right there. And we'll set the same retention policy, same as source. But we do have to first check what snapshot task we want this related to. So we'll say, this is the snapshot task that you should queue off of, but you should run on a different schedule, running once a week here. And we'll say run automatically. And we'll go ahead and hit save. So now we have this particular replication task that's not tied directly to a snapshot because it has its own schedule, but we can still force it to run now and any snapshots that are attached to this will be sent. Now we can edit this if we wanted to change the schedule again. But as I said, that's just right here, you change it to weekly monthly. And if you choose custom, it will bring up if you have a very specific time you want it to run, you can choose that here through the advanced menu. Now, a few final notes. If you are setting these up using default TrueNAS systems, none of the steps I have will give you an error, except if you go through and change one of the defaults that I've suggested before and still think it's a good idea, which is under the settings and then into the basic GUI settings, there is an option to send HTTP to HTTPS as a redirect. That way, when you land on it as HTTP, it will redirect. If you check that box and you have a self-signed certificate and you go to set up the wizard to add another TrueNAS, you will get a self-signed certificate error. You can get around this by either having a self-signed certificate or simply unchecking that box. But then some of you that are pretty sharp at security may note that yes, you just sent your password over plain text. I have good news. You can set the password again and it will not break that connection. So once these connections are made and it establishes an SSH key between the two systems, you can change the password on either system. They're linked via that SSH key. So as long as you don't delete the SSH key, they can keep communicating despite what you may change the password to. So that's just something I think is really worth noting in terms of security. Next, let's talk about TrueNAS core or different versions of TrueNAS scale. And yes, this should actually work. As long as your source system is running a ZFS version the same as or prior to the destination system, I don't predict you'll have any issues. I bring it up like that because it may sound odd to update your backup server first, but hey, that's probably not a bad place to test things before you put it on your production server. That being said, if your production system that was sending the data has a higher ZFS version than whatever your destination server is, that could cause a problem if there's a feature flag that has some incompatibility or may prevent it from being sent over there. So generally speaking, you want to keep them at the same version to avoid any trouble at all. It's just something to keep in the back of your mind. But as of here in March of 2024, TrueNAS scale has no problem sending to TrueNAS core. So that's the status of it today. There could be a future when you're watching this video where there is some incompatibility, but I can't predict the future right now. Nonetheless, leave your thoughts and comments down below. Like and subscribe, see more content from this channel. Head over to my forums, forums.lorencestimes.com to engage me on this topic, under topics you've seen on a channel, and head over to the TrueNAS forums. There's all kinds of information, discussions, and all kinds of different use cases for ZFS replication. Maybe there's some special use case I didn't cover. Leave your comments and thoughts on that down below or head over to the forums and maybe someone else has had that issue too. And hey, this is how we figure out how to make it all better. All right, thanks. you