 Tom here from Lauren systems and ransomware was a big topic in 2018 2019 right here in 2020 And I don't really see it going away anytime soon These attacks only get better more sophisticated and harder to defend against now I'm not going to talk too much about the cybersecurity side of it in terms of defense We know that needs to be done But that is still a cat-and-mouse game because as we set up better defenses the offense gets a little bit more skilled and Away we go. We're going to talk about is how you can protect your files Using ZFS snapshots here in TrueNAS. I've done this before and I want to do an updated version of it I was just on two ransomware investigations just a week ago Unrelated to each other and both times all the volume shadow copies were completely erased as I said these attacks only get better So they go through and they methodically destroy their backups now There becomes a business question of whether or not you should pay the ransom and I'm not going to debate the merits of it But this is how the calculation goes If you have let's say a hundred terabytes of data and you did proper off-site backups How quickly can you get that data back to where it belongs? This is a big challenge. So while a lot of people just say off-site backups said and done You don't even need to do this video Tom I deal with the real world where we have to try to figure out how to put data back and put it back quickly because if The loss of time let's say it takes x amount of days That is the loss that that business will not be up and running so off-site backups Maybe a way to mitigate things but it may take too long to restore So it would be less expensive actually to pay the ransom and get the files restored faster But I want to show you here in TrueNAS with the ZFS snapshot is one how to make the shadow copies immutable as in Not accessible by any type of ransomware attack and why you need to keep your password separate When you tie TrueNAS to your domain, you know, I'm assuming a lot of these now I'm not going to have a domain tied to it in this particular one But if you do the same exact rules for the same setup apply if you have this tied to a domain The one thing I'm going to say is that you do not want to use the same password for your TrueNAS server as you do For your domain controller. This is just a bad idea. So you should have a separate login That's at least one part that we're going to talk about again in the beginning But before we dive into all these little details, let's first If you'd like to learn more about me or my company head over to laurancesystems.com If you'd like to hire a short project, there's a hires button right at the top If you'd like to help keep this channel sponsor-free and thank you to everyone who already has There is a join button here for YouTube and a patreon page. Your support is greatly appreciated If you're looking for deals or discounts on products and services we offer on this channel Check out the affiliate links down below. They're in the description of all of our videos Including a link to our shirt store. We have a wide variety of shirts that we sell and new designs come out Well, randomly. So check back frequently and finally our forums forums.laurancesystems.com is where you can have a more in-depth discussion about this video and other tech topics You've seen on this channel now back to our content. Now, let's break down how this project is set up What I have right here is just a ton of files being dumped between different folders So I'm actively dumping data to our demo that we have this way It's actually doing something and not just sitting idle kind of give you a little bit more of a real-world scenario The device of choice here is a TrueNAS Mini 3.0x provided by IAC systems and I've been doing a long-term review of this It's been working great including under heavy load like it is right now dumping lots of data now One thing I'd mentioned in the beginning and we're going to talk about a couple features here When you look at like directory services and Active Directory for this demo I did not build out an Active Directory controller to this and tie it all together you on many corporate Setups, this is the type of setup that you have where you have the files may live on the TrueNAS itself And then all the Active Directory tie into the main domain controller is done And that's perfectly fine, and it won't change this demo any what I do want to make sure is When you're looking at these systems, and this is where people Frequently do this out of convenience, and I get it when you're setting up the system accounts and we look at the users Specifically user root. Those use the same root password as they do for their Active Directory administrator password Do not do this. This is something that is Completely separate and should be separate just because you tie a system over to Active Directory will not override the root password So it is very important that you have a separate Password and don't say well just use the same because it's convenient Keep this locked down because the ransomware and threat actors will look through and figure out where things are going And if they have access to this we Completely negate all the things I tell you because they'll go in here and they will delete the things They find inside of here and of course everything is the goal of getting you to pay the ransom The way to get you to pay it is for you not to have data and even if you have those off-site backups They know how difficult they are to put back in place Now with that being qualified I just did a video the other day on permissions and I'll leave a link to that And we built out just a basic shared drive with the file system here. We dump in data actively into it there's just under a terabyte there now And this is just a standard data set created just like the video I did in youtube permissions nothing special about it We'll go here and take a look at it real quick Like edit permissions tom lts owner all really basic stuff nothing special here now the share itself There's nothing too special about that either so we're going to go over here to sharing We're going to go to window share And here is the shared drive We're going to go ahead and edit Advanced and this was all set up with default The one thing to make sure though is that enable shadow copies is turned on and what this does is we're going to be using snapshots to Pretend their shadow copies shadow copies are a feature of the microsoft ntfs system To basically have a way to restore a previous version of a file Samba which is the underlying tool inside of free nas the underlying service inside of free nas that presents window shares As if it's a windows server is going to emulate those shadows copies with the snapshots now snapshots and zfs are very clever They only take up the differential Of what data was changed and it does this at a block level And being able to have zfs snapshots has been around for a long time and been a great feature when you need to restore Data now enabling it in shadow copies makes it really convenient because now this will even allow users to do this And obviously this protection not just from ransomware, but in general when people delete files Is a great service to have that way they can restore things Now there's the share and now let's look at the tasks I set up for the snapshots We're going to go over here periodic snapshot tasks and here it is And we'll edit it to look at what we did So here is the data set that we're specifically doing the snapshots snapshot lifetime That is how long until the expire you have to make a decision based on how much storage you have Oh, should I keep them for a week two weeks a month? However long your data retention will allow and of course your storage budget will allow is probably the bigger determining factor You can come up with an aiming scheme form I usually leave the default one on there and you have a schedule begin and end When do you want these snapshots services to run? You can say only run them during business hours or run them all the time And that's essentially what this is doing here is just basically running them all the time allow empty snapshots Well, do you need them if there's no data change? And this is a Option you may need to enable for compatibility with older versions of free nas true nas if you're going back and forth between them But what they do is if there's no data change it still creates a snapshot Now this can be confusing when you're looking at the previous versions because If you are looking at the way and we'll switch to it real quick to give you an idea You go over here and you're inside of windows and you go I want to see the previous versions And it'll list every snapshot even if there wasn't any changes So you're not really restoring your previous version. You're just seeing all the different snapshots This is kind of up to you whether or not you want that in there The reason there's so many in my demo here is because I was moving data and have the snapshot set up Now specifically the way I have this scheduled is custom and I have it running every minute every hour every day The reason I'm doing this because well It's essentially A demo that I wanted to show one you can do it by the minute may not be effective It may over task your server But for purpose of this demo so we can move things around every minute means I don't have to wait as long or cut out any part of the video when I show you how we delete and restore things But let's go ahead and look at creating a new snapshot task from scratch Just to give you an idea when you're setting these up what it should look like Here's the snapshot lifetime. We have it set in weeks. It's pretty easy They have how you define them so we can even say like keep 12 weeks of it go here 12 weeks auto Daily weekly maybe you only want one a week and you can have different types of snapshot shovels You can have like an hourly one and a daily one or a weekly one And then you decide these retention times for maybe you want to keep a monthly snapshot That lasts maybe a year so you end up with only 12 of those but then the more Critical things because it depends on how long you need for data retention Maybe you want these running every hour that way you never lost an hour short the data If someone makes a big mistake or if ransomware attack does occur So it's pretty straightforward when you're setting these up now once this is all done and this is actively dumping data right here Let's look at what those snapshots look Like so we go over here to storage and we go over here snapshots And there's all the different snapshots And the reference point you see is the amount of data that the snapshots using it's only going to reference the change differential and that change differential because I started these snapshots before there was any data of course is going to be Quite big because I've dumped that much data. We've currently dumped 978 gigs of data in there So when you're going through these yes, they cumulatively get bigger and bigger as the reference data That being said it's still at the block level. So there's a really great level of efficiency With the way this is stored So it's not like it's going to just automatically take up a massive amount of dry space because snapshots as I said are Working at the block level and only seeing the differential now. Let's actually Break something delete something and go through and restore it and we'll do it first through the shadow copies so let's go here And Let's see we'll delete all this. This is a bunch of the tesla videos that I recorded and one day I'll record more But that's not today So we'll go ahead and whoops these away. Well, let's permanently delete these shift delete And now I've broken the tesla folder. There's no more folder data. It's all gone How do you restore it pretty easy go here restore previous versions And uh, let's go just before time deleted something So we'll restore that one there. Yes from 12 minutes ago And it's going to copy them in there. We can actually open these That's not going to do it while it's doing this hold on. We'll just skip ahead while this is finishing Files copy it popped open a folder now go ahead and close this Successfully restored the previous version So now we can actually go back because I copied everything back in there And you can see any of these videos if I wanted to play them that are about my tesla And there's me talking about tesla pretty straightforward in terms of how you do that And this of course didn't require any Non-user interaction or administrator interaction I should say so it's not like the user Needed a contact to admin to be able to restore previous version. It's the same Ability that you have on if this was on the windows server Now what about a faster type of restore? Well, this is where the snapshots have a couple different options Let's go over here and we're going to look at the snapshots again when we look at these And all these different snapshots are from different timestamps so we can pick which one we want And let's say we want one before I destroyed all the data There's two different ways and I haven't destroyed the data just yet But we can clone a data set from here. So we're going to click on the little three dots Clone to new data set now when you do this It creates a separate data set that is not shared This is essentially for administrator purposes going I need to know Of that snapshot what that folder structure and everything looked like so I need a point in time That I can restore to and we're just going to go restore to this one right here And here it is. This is what's going to be inside of it. We can dig around through it We can create a separate share if I need to or I can just go to the command line And I can go through and look at things exactly as they are Snapshots are not a pick and choose type thing when you say give me that snapshot of that particular instance It grabs everything within there. The snapshot itself is not particularly granular But sometimes you need to restore a snapshot to a clone data set So I can grab the one thing I needed out of there because sometimes this is a challenge users create for us administrators They delete something or overwrite something and you got to try and do some differentials and figure it out and of course this while being based on Command line like this makes it absolutely easy So you can start doing some scripting to do differentials or just open up a share and use the Standard file management tools to manage it however you want But pretty straightforward on how to do that and we'll go ahead and get out of this directory As when we're done, we don't really need this wasting any space. So we're going to go here And we're just going to go ahead and delete that data set Before you delete a data set. I actually recommend doing this you take this copy delete data set You have to type it all out. So that's why did the copy confirm that data sets deleted the snapshot itself is not deleted We restored it temporarily to a data set that was able to Go through look at it do what we want with it and then we deleted the data set But the snapshot itself is exactly as it was and it's untouched So those snapshots are still there and we didn't break anything in the snapshot system Now let's go back over here to windows And look at this shared drive. Let's go ahead and delete everything in there And i'm going to do it from the command line just because it's going to take too long in windows But we'll just go ahead and purge everything. So if we go in uh We're just going to rm rf star Just nuke everything in there Everything's gone It's all gone and we're going to wait one minute for another snapshot to occur And we're going to take a look at things and show you how to do a mass restore now deleting it or encrypting it It really doesn't matter when we do this restore We're going to revert this time in the entirety of it And if you were to suffer a ransomware attack or some type of mass deletion of files like this like someone rm rf And you wanted to restore it. We're going to show you how to very quickly restore that entire point in time right back to where it was and That's what is the next step. So After one minute goes by All right I allowed a couple snapshots to occur and I wanted to show something here shared drive file system There's still only one terabyte use because these snapshots all belong to this particular shared drive And they're still referencing data even though there's technically no data We erased all of it and as you see here everything's blank and windows go back over here We're still showing a terabyte of data. So we go over to snapshots because we waited a few minutes. We can see There's very little just a couple bytes used but we go down here and there are the Actually got to start by day created. There we go. There's our one terabyte reference that we're going to restore to So let's go and I know tom nuked it all right about here a couple minutes ago So we're going to restore this one We're just going to roll back And you will get a warning data set roll back from snapshot Please think about this of whether or not you want to do it because this can also destroy the other snapshots So we're going to just go no safety check. I know this is when it was good and we're going to do roll back And we'll look over here ls all the files are back And let's go over here the windows Refresh all the files are back that quick That is a terabyte of data restored in seconds now granted. Yes, these are fast drives And this is a fast system and there are limitations the slower the system It would take a little bit longer, but you're not talking about forever Or hours and usually it rolls back in minutes and the reason why is because ZFS snapshots are done at the block file level. They actually don't care What exactly is going on in terms of which files or directories are in there? What it does is taking advantage of the underlying ZFS copy on write transaction model And this is why ZFS is such a great file system is this works so well Because all it's doing is going back and the data is not necessarily destroyed with a copy on write file system There is a series when you have snapshots turned on there is a series of Block pointers within a file system Now i'm not going to get too much into the complexities of it You can spend some time reading and all the beautiful things of the way ZFS handles it the bottom line It is able to restore things very very fast because of this design So it's not like you're actually copying the files from one piece of it to another You're doing it at the transaction level and we're essentially rolling back the log of transactions Which are the snapshots so because we had this turned on when we hit roll back It immediately puts it back to the reference it had for this It's a lot to learn if you want to dive in deep of it But it's also one of the popularity of ZFS is based on these type of Transaction models that are underlying it that allow For us to be able to restore things very fast So go ahead and do some more reading if you want to understand it better If you just want to trust in the science of it It does work. You see me delete and remove files Rather quickly you can see how it works with shadow copies It's a really solid system for doing this and as long as you keep your root access Somewhere else other than shared password with your administrator It is not very likely that if a threat actor gets inside of your network that they'll have access to it They will do their best to destroy all the data They're going to do their best to destroy everything they find on the share Provided you have snapshots and you didn't let the snapshots expire prior to Their deletion so if you only have one hour of snapshots and well you waited two hours to go back Then you may have a problem But as long as you have a sane and rational way that you keep the data enough retention and that you notice this happen This is a great way to get you back up and running very quickly Hopefully this was helpful and hopefully this saved someone from having some of these problems appreciate it Thanks And thank you for making it to the end of the video If you like this video, please give it a thumbs up If you'd like to see more content from the channel hit the subscribe button and hit the bell icon If you like youtube to notify you when new videos come out If you'd like to hire us head over to laurancesystems.com fill out our contact page And let us know what we can help you with and what projects you'd like us to work together on If you want to carry on the discussion head over to forums.laurancesystems.com Where we can carry on the discussion about this video other videos or other tech topics in general Even suggestions for new videos. They're accepted right there on our forums, which are free Also, if you like to help the channel in other ways head over to our affiliate page We have a lot of great tech offers for you. And once again, thanks for watching and see you next time