 Tom here from Lawrence Systems. We're going to talk about ZFS replication tasks in FreeNAS 11.3 series Which is the latest one as of May of 2020. They've really streamlined the process I want to do a new video on it and let's get started But first if you'd like to learn more about me or my company head over to Lawrence systems comm if you like to hire Short project there's a hires button right at the top if you want to support this channel in other ways There's the affiliate links down below to get you deals and discounts on products and services We talk about on this channel including a link to our patreon if you'd like to become a patreon supporter We also have a swag store where you can get shirts and other items that are for sale and that changes from time to time What's available and what's not so go ahead and check that out frequently and Finally our forums if you'd like to have a more in-depth discussion about this video suggestions for new videos or just reach out Say hi and talk tech our forums are a great place for that. All right now back to the content Now before we go all the way deep into ZFS replication if our sink is so great. Why is ZFS replication? Maybe a thing I'll leave you the link to this article over at ours Technia. They dive into some of the technical details Which really important is ZFS actually duplicates at the block level of the ZFS system The ZFS send is a function built into ZFS to replicate a data set to another ZFS pool, so it's really fast It's not just looking at files like a lot of other tools are it's working at the level below that So it's working as in sending the ZFS blocks over there, but there's a lot more technical to it And I'll leave this our technique article so you can read and see some of the advantages of how fast and Efficient it is at replicating data Now the one drawback is yes, you do need ZFS for this to work so it has to be doesn't necessarily have to be free NAS But we're gonna do all this all in free NAS, but ZFS replication Replicates not to other locations not to a file, but to another ZFS pool that is an important thing if you're doing it free NAS to free NAS no problem If you're doing it free NAS to something else running ZFS provided It's a compatible version that does support the same functionality it should work But we're gonna do this from free NAS to free NAS Replication is the process of copying a ZFS data set snapshot from one storage pool to another I wanted to make sure that this was highlighted that it's a ZFS data snapshot I've done videos before on how snapshots work and what they are is a differential of the data set at any given moment from another spot in time and It's a snapshot of the data and the only Space taken up by a snapshot and of course this also ties into how it replicates is the difference in any files Added or deleted any changes to the file system after that snapshot So think of it as a snapshot in time. That's exactly what it is now They're very important to the replication process because you can't replicate data in flight So you have to have a snapshot a point in time at which you reference to send it So the replication task itself doesn't run on a schedule of its own It ties to the schedule of a data snapshot so once we build this any adjustments we want to make to the retention policies or How long we hold the data essentially or when we snapshot the data or how frequently we snapshot the data is all controlled by the ZFS data snapshot task and we'll cover that in when we get in here Now I've set up two free NAS boxes and this ugly theme I have loaded which good news is we don't have to look at it much as the high contrast Insider here, but I just did that this to show this is our destination server. This is a Really lower end. It's got the I just did the this is my Partially spare parts free NAS build. I did a video on I created a empty pool right here Just called backup NAS really simple pool name really simple lower-end servers Usually, you know, RAID is not a backup But you can back up to another RAID system and maybe the main RAID system has all the heavy lifting to do with all the users But you see somewhere else to store the data That's no not the main system because RAID is not a backup and we chose this system for our destination This system right here is the free NAS R730 XD. I did a review of this. Thank you for tech supply Thank you to tech supply direct for sending us out to me If you're interested I have the full review I'll leave a link to on this and if you're interested in buying a server from tech supply direct We do have an offer code that gets you a 10% discount on any server equipment you'd like to purchase from them We've bought quite a few servers from their great company to deal with easy to work with And I'm not just saying it because he sent me the server. We actually started buying servers from them prior to working out the Affiliate deal that we have for them. We just like them as a company. They've been great to us. All right Back over to here. We have this storage pool and We have a data set we want to back up called SMB demo share I was doing this demo and it does say delete me later, but we're gonna pretend the important There's really important data in here that we want backed up Snapshots, there's none. We go over here to tasks Replication tasks none periodic snapshot none and we're gonna hear to system And we're gonna go SSH connection SSH key pair all blank These servers don't know each other We're gonna make an introduction and the way they've done the wizard is really slick for doing this. So Right here is 192 168 3.209 and this is 3.212 We're gonna start all through the wizard go over here to task Replication task we're gonna add one Source location this system now this is a push-pull system You can actually say the source is somewhere else and I want to pull it over to here Or I can push the data over there You can have it on either system go either way I'm choosing to do it from the main server pushing that way But this does work in both directions. So it can be done either way I just this the way I'm gonna choose to do it choose here choose here and We're gonna say SMB demo share back it up. So right there's that SMB demo share How do we know how to get to the other server on this system? Now I will know if you had multiple pools on a single system This replication will work internally on a single free NAS, but then you still have all the data physically in one box Maybe you do have a reason you want to duplicate it over to another pool That is a possibility, but ideally you're backing up to a physically different box Maybe even at a physically different location and this does use a secure transport layer But I'd still put it over a VPN if you wanted to replicate across Sites but this these two servers are in the same room with each other But it's still a backup because it's in two locations at that point just not geographically backed up So on a different system SSH connection if we already had set this up previously We could easily do that and say all right. We're gonna create an SSH connection And use an existing one But the wizard is really handy because it says create new one. This is where we get to make an introduction So we'll call this free NAS backup server We'll use the setup method of semi-automatic and then we're gonna take the Copy that And paste it here. That's it. You give it the URL for the other free NAS username root password I don't know the password, but that word and does we're gonna go here and copy the password Paste it private key if you have it generated one or not even sure what that means It'll take care of it. This uses SSH as a transport to encapsulate everything and send it across It's a really secure way to do it You can look up how SSH secure key connections work and this creates one for you automatically This is really nice the way this is built in Now the two systems have been introduced to each other Which means I can see the data set like this that says backup NAS Now you don't want to just send it to the backup NAS You want to give it a destination and we'll give it the destination of SMB demo What was it called demo share? Oh? Damn, oh Demo share. All right. Now Recursive if you're backing up and let's say we started here where it says this pool you could recursively Back up everything under that pool. It's not always how you're doing it But maybe that's how you want to do it. That is a possibility and you could do that and get everything Sub that way we're not gonna bother with that we're not gonna worry about any custom manual snapshots We're gonna leave that but says they're not found anyways because I don't have any Encryption more secure, but slower. I always go with encryption It's always better to encrypt and then err on the side of caution But maybe if you have some secure place you have an excuse not to encrypt it because you just want to send across Always encrypt that's my attitude unless there's some really good reason not to Destination snapshot lifetime and run on schedule default is running on schedule and the default to statement source I always leave the statement source. It can be adjusted later What this means is how long do you keep those snapshots for? Whatever your data retention policies are and when you adjust them you adjust them on the system doing the replication task in a Periodic snapshot task, which is this main system and like I said, they can be adjusted later Just into whatever your data retention policies are Scheduling daily at midnight is the default. We could go hourly. We can go custom Maybe you want them only backed up at certain minutes certain hours certain days They have some presets. We'll leave it a daily here because that's perfectly fine And we hit start replication which actually doesn't start the replication It starts as pending because there's no snapshots. There's no anything yet So this is built it all out, but it's not actually replicating yet It'll replicate on the schedule like it's supposed to or we're just gonna go ahead and force its hand and say run now Demo share has started now soon as we refresh this. This is starting Snapshots refresh this. Hey, there's that SMB demo share and there's already 313 megabytes of data. Let's refresh it again There's a gig of data 1.2 gigs every time refresh it a little bit more data is moving across there. It's moving pretty fast Replication run about 21% done is doing a verification As it goes making sure all the data sent and 35 will pause right here for a second and fast forward and done So now we have One snapshot In a snapshot if you dive into or my previous video snapshots, you know, there's only going to reference the data That is sent so the reference means how much data there is versus how much the snapshots using snapshots Not using any because there's not any differences and there won't unless we change any data on this particular folder It'll just keep synchronizing and verifying that everything's in sync So we look over here at the replication task and we look over here at the snapshot task What if we go well, I want to change that let's let's synchronize this faster. So we go to edit and Snapshot lifetime two weeks seems good, but why not have it? custom hourly But let's go minutes because I only want to wait minutes so I can finish my video here So if I put an asterisk here here in here every minute, it's going to replicate in sync Now the first time it synchronized they had to get all the data Every one after that is just a differential of any data That has changed so it actually ends up going really really fast So all the other ones that occur after this occur every minute really really Fast and no big deal. They're going to only look at any Block level differences and I said block not file level which means even if you do some renaming or things like that You're barely changing any blocks. So you're just changing names of files It says oh, there's not much of a block level difference between what you did in this particular folder So we're only going to send over those changes and these happen really really fast I've even seen some companies when they have massive data sets to move where they'll start with a free NAS and The local building get the first synchronization done But they don't produce that much data on a daily basis when they're doing the synchronization So then physically take the server somewhere else to replicate to it on a more ongoing basis But however you start your replication process the first one does have to send all the data and everyone after that only sends the differential data after it We'll go over here and yep another one completed Replication tasks. All right, we'll go over at the pool. So storage Snapshots now there's two so every time a new snapshot gets created it sends it over there and we look over here in snapshots And now there's two of them here. It's as simple as that for each one for each task. That's occurring There's another one now. Let's go ahead and create one more task and go over here go to task Periodic snapshot and we'll add one more. I'm sorry replication task add Source location this system and this time we'll just grab the IO cage folder and For this we do want to be recursive. There's like a whole set. There's a bunch of data sets underneath there So each each of these are data sets Now the difference between a data set and a folder the data sets an actual functional part of the ZFS versus all the Folders that are within a data set are automatically so you don't actually have to do the recursive to get folders underneath But if there's data sets underneath those are different types, but we want to back up everything. It's under the IO cage recursively Make sense. All right, and where we want to send it as a destination on a different system as the stage connections We can create a new one again, but we already have the server. So now it lets me choose it choose this Choose this and we'll put it right here and we'll put it I owe cage Back up it next Run on a schedule daily same as source. We'll let this one be daily. It's just IO cage. Maybe I want to back all that up one day Well for purposes of the video I jump back into the snapshot task. Let's actually set this to be custom like the other one Where we're just gonna run it with every minute that'll give us some data to work with here So I've set it to a minute and skipped ahead a few minutes So we have a few things going on there because one of the challenges is if you have the replication tasks, but no snapshots have been created. So you try to run now with a Replication tasks, but there's not been at least one cycle of the snapshot task running Then it doesn't really have any data to send. So for purposes of video I went and set both of these to run every minute and I waited a little while to build a whole lot of data over here So if we go back over to the system I'm gonna look at the services. I'm sorry storage snapshots Here's all those different Snapshots now because I said recursive this is what it meant So backup IO cage backup download images the pharaonics one I have on here all the jails all the logs releases It recursively grabbed every data set under there now that same thing applies because it's working off of the snapshot task that is recursive right here for The replication so this is recursive and so let's go back over here to task snapshot tasks And these are sets of recursive as well with the two week retention and set up to run every minute And then we look on the storage of this one and we see this the same thing Here's all those different shares that are on here all the IO cage ones etc etc recursively being copied over and Being copied over at speed because it's only looking at any differential So even though you do see what looks like a lot going on here It's only differential Therefore if there's not much going on in any of these it's not much data now a couple little housekeeping things and we'll go over here system SSH connections There's the SSH connection that was generated for this Freenance backup server we can edit this You can see what the host IP address is If you have some special use case where we put the host IP address but we actually have this other card in here and If each of these Freenances had make maybe a storage network you could actually change Provided they can talk to each other over that IP. They're pingable and on the same network You could change it to that IP that is the IP. It's going to be destinationed to on Assuming over port 22 and how it's going to get to that particular server So if you have some special use case you can change the host and once you edit it here because this is what SSH key was used or SSH setup was used to talk to that server It is changeable and over here SSH key pairs here is the key pair and where it was saved And if you edit you can see the private key and the public key that was sent over. I would never share your private key I'm destroying this one after this video is done But these are where those things are kept if you ever have to go back edit change or create a new key pair and start it up But that's it replication once it's up and running. It's pretty straightforward and the last thing I'll comment is like well What about all the data? I am SSH into that particular machine the one that's at 192.1683 dot 209 go back in your mount Back up nas and there's our SMB demo share all the files Are just copied over as far as what it looks like so in the event of a catastrophic failure This is that folder as replicated to this machine and we'll split the screen right here 92168 dot 3.21212 is the other server Mount you can see the same actually you got to go in the same folder. There's the same Folders across there. So if I delete something here, it'll get replicated over and delete it over here And if there was a catastrophic failure of our main system right here if something horrible happened to it all the data from the last Snapshot is over here one of two things. Well, this is free nas and if I wanted to I could just create a share and Share data back from this server out now the synchronization is not both ways So it's pushing over to this server So if you do open up a share on this server and someone edits or deletes a file and we can do that real quick You may get some weird errors out of free nas. It's not expecting that But ideally it should just be pushing the data back over and what came from the main server should overwrite back on this one That's not ideal. You probably don't want to be messing with the data on here That's the destination for it But in a catastrophic failure of the data is here and once there's no more data being sent you can just Well turn these on and share them out and be able to be back and up and running relatively quick while you rebuild your main Free nas and of course you can just reverse the process and pull all the data back from this server to your Better faster nicer server, but hopefully this clarified the replication process. It's wonderful inside of free nas It's really solid. It's really really fast because it's working at the base level Yes, it will also back up like things like ice guzzies z vols Anything that's within the data sets that you have will be copied over to the other side So that's an important aspect of it. It's really nice The wizard has made it just as you see here really quick to get it up and running But remember all the scheduling and retention all the policies are really the same as a snapshot So it's all based on snapshots. It's just the synchronization is an extension to the snapshots Not really a separate thing once you have that concept you can kind of mess around with it and go from there All right, thanks And thank you for making it to the end of the video if you like this video Please give it a thumbs up if you'd like to see more content from the channel Hit the subscribe button and hit the bell icon if you like YouTube to notify you when new videos come out If you'd like to hire us head over to Lawrence systems comm fill out our contact page and let us know what we can Help you with and what projects you'd like us to work together on if you want to carry on the discussion head Over to forums dot Lawrence systems comm where we can carry on the discussion about this video other videos or other tech topics in general Even suggestions for new videos. They're accepted right there on our forums, which are free Also, if you like to help the channel in other ways head over to our affiliate page We have a lot of great tech offers for you and once again, thanks for watching and see you next time