 One of the superpowers of ZFS is snapshots. It's one of the reasons that a lot of people dive into ZFS and the absolute read-only nature of snapshots that allow you to have better confidence that if someone goofs something up or something happens, such as ransomware, there's a point, a read-only point by which we can roll back to. Now, ZFS snapshots are a function of the ZFS file system, but we're going to be doing this demo in TrueNAS because it makes managing ZFS snapshots easier. I will also note for those of you that want to know the really interesting details for how these snapshots work. There's a great talk given at FreeBSDCan 2019 that you'll find linked down below. It does a great job of covering the details, such as how do you figure out which blocks to delete when you take a snapshot out of the middle because that's completely supported. And of course, that comes with a lot of challenges. And that's a great video to watch if you're looking for that as a deep dive. I'm gonna be covering in this video the more functional side of how to set them up, how they work, how they present in Windows and how to see things from the command line. So let's get started. Are you an individual or forward-thinking company looking for expert assistance with network engineering, storage or virtualization projects? Perhaps you're an internal IT team seeking help to proactively manage, monitor or secure your systems. We offer comprehensive consulting services tailored to meet your specific project needs. Whether you require fully managed or co-managed IT services, our experienced team is ready to step in and help. We specialize in supporting businesses that need IT administration or IT team seeking an extra layer of support to enhance their operations. To learn more about any of our services, head over to our website and fill out the hire us form at lorenzsystems.com. Let us start crafting the perfect IT solution for you. If you wanna show some extra love for our channel, check out our swag store and affiliate links down below that will lead you to discounts and deals for products and services we've discussed on this channel. With the ad read out of the way, let's get you back to the content that you really came here for. I wanna start by discussing exactly how snapshots work at a very functional level inside of ZFS. They are read only points in time. So there is a time and date by which you took the snapshot and it is a frozen point in time at which that data lives. It is read only as presented to an NFS share or an SMB share. So I will show later in the video how this can be accessed and how you can see the previous versions but the end users who are attaching to that share just get a read only version of the data. That data, that snapshot is a great point in reference to be able to roll back to in case you ever need to come back to that data. It's not a backup because it's all living within the pool. More specifically, it's living attached to either the data set or Zvol by which the snapshot was taken. So if we take a snapshot and we delete that data set that the snapshot was based off of all subsequent snapshots attached to that data set go away and the same thing applies to Zvol. Now, absolutely the most common question the kids ask is how much space does a snapshot take? And it takes the difference in blocks change from one snapshot to the other. This is a very important concept that I did not say differences in files changed or differences in renamed files or way you shuffled a directory structure around. It is very specifically both for Zvol and data sets based on the block level changes. The reason this is important and to understand what block level changes are is as we know data is stored essentially as ones and zeros. The block level storage that all the mechanics of ZFS that go into it is just ones and zeros at the end of the day. Those ones and zeros are then presented to you as a file system with folders and data and structure. So it's more human readable. Not to say that binary is not readable by some humans but what we have here is if we rename a file and we do a snapshot, we can roll back to the renamed version of the file but the difference in data stored is only the few kilobits of block changes for a file having a different name. Even more specifically, if you have a large just say one terabyte file and you change 100 megabytes of a one terabyte file that snapshot will not be the entirety of that one terabyte file it will only be the 100 megabytes change. This is actually really handy with snapshots because if someone does something and I'll use ransomware as a very popular example here in 2024 and they rename a bunch of files. And I say rename because ransomware does two things. It renames and encrypts but it doesn't encrypt the entirety of the file. It only encrypts enough so the file is corrupted. So the difference from a ransomware attack would be only the blocks that are changed on the file which is gonna be the name of the file and then the blocks that were corrupted. Well, that's provided the ransomware actually renames it if it doesn't, it's then still just the same thing. So when a ransomware attack happens, if it's only corrupting a certain block or encrypting that certain pieces of the block the difference in change is not the entirety of whatever was stored, it is only those differential blocks. This is actually why ZFS works so well for rapid recovery from things because if you have a snapshot prior to a piece of data being destroyed you can snapshot right back to it. And this all happens really fast. I'll also note that databases are also something that you can take snapshots of although that can cause different issues because databases if you don't know where the transaction was in that place but conceptually if you are snapshotting something with a database it is only going to be those differential blocks from the different blocks that were written to that database. So it's always looking at the block level which means it doesn't take up that much space depending on how many blocks have changed. Now as far as taking snapshots you can have your active data than your snapshot one, snapshot two, snapshot three which of course you'd want to name them all probably the time at which they are taken. I'll show in true NASA this is done pretty much automatically it's really easy to follow but what happens if you want one of these back? Well it's easy enough to just roll back to a point in time but snapshots once again because they work at the block level will have the disadvantage so to speak of being able to roll back a single file they'll roll back everything that was within that data set or Zval to that point in time but you actually have the option if you want to fork a snapshot so you can actually restore it elsewhere. I'll also show you how to make them visible both for Linux and for a window share where they're gonna present as a volume shadow copy so you can easily let users self-manage essentially to be able to go back and find previous versions of their file. I'll go over all that set up inside of TrueNAS. One of the other thing I want to mention is what happens if you delete a snapshot out of the middle so you took one in snapshot one and then you have snapshot two that you deleted and snapshot three will allow you to do that or does it need the whole chain? Technically it needs the whole chain but as far as how it's presented to you yes, you can delete the snapshots out of the middle as well. So if you took some extra manual snapshots and then have a bunch of automatic ones around it you wanna later delete those it sorts this out automatically and if you're thinking wow that's some complicated stuff going on that makes it happen go back to that video I referenced earlier at BSDCan talking about the complexities of actually making that work. All right, this system is running DragonFish 24.04 release candidate one here in April of 2024. So it should be the same as the full release because snapshots haven't really changed much throughout the last few versions. Let's go over here to our data sets. We have a test data set and we have a fully set up YouTube snapshot demo which we'll get to shortly but I wanna talk about how we got there and cover some of the basics. So we have this test data set with 7.95 gigs of data and if we look to manage snapshots it doesn't have any right now so let's go ahead and start making them. Go to data set we're gonna click on the test data set here and we're gonna go ahead and create a snapshot. We'll leave it as default name of manual 2024, 415, 1148 it's the time date but you can choose a naming schema if you want and recursive means also snapshots sub data sets we're gonna get to next in the demo here. So first let's go ahead and save that snapshot and if we now manage the snapshots we see the one and we see there is not available under space use that's because there's not any space taken because we haven't made any changes to the file system and retention is will not be deleted automatically. This matters more for when you're setting up the automated snapshots which we'll show in a moment and when you set those up you set a retention and it's the snapshot task under data protection that actually facilitates deleting them and you can override if it's one that will automatically be deleted by putting it on hold. Now let's go here to the data set and you see that it's under dozer and then test we're just gonna move the command line and show you what the files look like in there. So we're gonna go here to mount dozer test LS and there's some data. Matter of fact, du dash sh some more data we have rounds up to the eight gigs in here. Let's go ahead and purge that eight gigs. So we RM dash RF some more data. Now we're going to back out of this directory because we've now deleted it just show that it's gone and we'll go back over to our true NAS and now we're gonna click on manage snapshots and now that it refreshed it says we've now referenced the 7.95 gigs because well we deleted all the data. Let's go back over here to our data set and our data set still takes up 7.95 gigs. Matter of fact, something important to note is that this data set takes up not just the storage of data that is active but also all of the snapshots and whatever the snapshots are holding until you delete them. So it's cumulative the snapshots plus any other data in there and the snapshots are referencing that differential. So let's go ahead and manage the snapshots again and let's just roll it back. We're not gonna do a new clone or anything like that. We're just gonna go ahead, no safety check just roll it back to the way it was cause we want all that data back in there. Hit close and we see it's still using the 7.95 gigs look to manage snapshots and this refresh is to go back to not available because there's no data used and if we go back into our test all the data is back. Let's go in here and print out some of the data. So there's a few files in here and I have something called pre-roll add. So let's go ahead and RM-RF pre-roll add maybe a little RM-RF whatever's in 2023 as well. And if we look there's only three and a half gigs of data in there now. Now the problem is what if I wanted some of that data back? Well, that's where we're gonna go in here and this will refresh in a moment, refresh this and it's only using 4.48 gigs as that's how much data we pruned out reference was the original 7.9 used that's how much it's holding on to if we roll back but what if I may change the other files and I want to roll it back but I want to get some of those files out? Well, we have two options. The easy one here is gonna be clone a new data set and we can just say clone it here and it'll create an entirely new data set. So we'll go ahead and do that. Then we'll go to the data sets and here's our test data set and here is that test manual clone that we have. Please note if you click on this data set it shows the total amount of data used versus this data set. It has an option to promote but shows no data. This is essentially a linked clone of this one so it's actually not taking up any data because it's referencing this one here but it will take up data if we promote it because then it becomes a full on data set. So it's like a temporary data set if you will where we can go inside of here and pull up files. We do an LS and we have our test dash manual. We have all that data in there, some more data and there it is and I can simply copy that over and we'll just copy it right to the root of test so it's in another location. All right, we've copied that data over. We're gonna back out of that directory and then we're gonna go ahead and delete this because I don't need it anymore. So I'll hit copy to clipboard and delete. I found the files I wanted. I'm gonna clean up the extra data set and it goes away and the snapshots are still there but now we have more data because we copied data from there and then put it in a different place. Now by doing this, we have some differentials from the different snapshots that we have from the different times. So it's actually gonna end up being a little bit more data because we have to now keep track of the changes and the extra data that I copied over into this test folder and then we can do the same thing again. We can create more snapshots and the process continues. Now let's talk about how to make them visible if I didn't wanna have to go through and have these. So let's go ahead and create another snapshot and it'll just be called manual this. Actually we'll call this another dash manual snapshot. We'll go ahead and hit save and we're gonna go here to this test data set. We're gonna go edit advanced options and we wanna change the settings here from snapshot invisible which is the default to snapshot visible. Go ahead and hit save and we're gonna go back in here to mount and those are a test and we have a new directory we can see called dot ZFS. And if we go to our dot ZFS we see snapshots and there they are. So there's our manual snapshot and there's another manual snapshot. So we go into the manual snapshot. There is our some more data and there's some of these different files are in there. Now I mentioned them being read only. Please note I'm logged in as root and it tells me it's read only even when you're root. This can only be controlled via the ZFS commands. So you're not able to actually do any data manipulation. These are, as I stated, read only so I can go through and manually see them. Now let's talk about how this is handled on a window share with volume shadow copies because not everyone's gonna be doing this from the command line or has the time to deal with a lot of end users using it and this is a really important feature. So we have our YouTube snapshot demo set up. You'll see that we have 60 snapshots and the reason is because we have an automated task and we'll cover those in a moment but this automated task basically says do a snapshot every single minute. And of course doing it every minute means there's lots of versions to roll back to. This is probably more than you need but it also makes it easier for me to demo here. So let's go look at the data set and we see that it has normal share permissions for built-in users and administrators and it has an attached SMB roll. So if we look at the SMB share, everything's pretty standard but I wanna highlight this. We're gonna go ahead and edit and we're gonna go show advanced options and we're using the default share parameters. In the default share parameters is enable shadow copies. This presents those snapshots as a volume shadow copy meaning it's easy for users to have a read only available version that they can previously roll back to inside of windows. Something else I wanna note is if you create the share prior to creating the snapshots, it won't know to share them to solve that problem. We're gonna go here to system settings and services and you just simply start and stop the SMB service and when the SMB service starts and it sees that there's snapshots available for one of the shares, it automatically converts them into shadow copies. It does this for all snapshots going forward as long as there's at least one in there it enables that feature. Now let's show what that actually looks like. So we have our test data here and we have our pre-post assets. Just a few things I threw in here just copied them out of my YouTube directory and what if we went and deleted some of the things in here or even all the things in here and hit shift delete, delete these 11 items. All right, if we go here now and that's obviously an empty folder but let's go to restore previous versions. And I've actually manipulated these a couple of times. I can look at all the different previous versions I had. Here's one from just a little bit ago and let's open it and we can see all the files are there or we could just go ahead and hit restore. Yeah, let's just restore that because we really didn't want to delete all of those and we go back in there and they're back. Now this doesn't just apply to folders of course this applies to things like the text files themselves so let's edit this file. This is some random test words. This is some more random test words and we'll just paste this all the way down. So you now change and manipulate this file. I just hit save and we'll close it. And if we want to go to properties, previous versions and let's go ahead and restore that previous version. Yes, I'd like to restore it, hit okay. And now it's back to the way it was. Something I will note again that if we wanted to look in one of these folders right here, such as pre-post asset and let's go to the properties, previous versions and let's open one of those previous versions of this. There's no way for me to delete any of these. I can only copy them. There's no way for the window share because it's all controlled by the ZFS commands on the TrueNAS itself for them to be able to delete this. So in the event of something like a ransomware attack none of the shadow copies can be deleted. There's no way to remove them because they're only emulated shadow copies that are actually based on ZFS snapshots so your data is protected. But let's talk about the nested dataset here because if we go here and we right click and do properties and previous versions, there are none. And let's explain what a nested dataset is. So we have our YouTube snapshot demo and we have the nested dataset. The nested dataset is the way you could put a dataset under another dataset. This allows you to have different permissions, parameters and even different share options or in this case specifically the data protection options as far as the snapshots. So even though this snapshot right here there are 60 of them and there's a task because in this particular task, if we edit it the recursive box is not checked. If we check the recursive box it'll automatically also snapshot that particular dataset or we can go here to data protection and we can add a snapshot task that is different because maybe we have different retention policies for different datasets. And that allows us to have a retention policy that would be for specifically this nested dataset that can be different. Maybe we need to keep the data longer. We want the snapshot lifetime to be longer, et cetera. That is all features of a dataset and we can have those different ones in there. And the same thing if we're creating things manually. So if we go back over here to our datasets if we were doing a manual snapshot doing recursive means grab all the subsequent ones underneath here as well. So if that's what recursive means just like it sounds it recursively does all the datasets that are underneath here to follow that policy. Now one thing I want to note is if we do that and let's just go ahead and edit this task real quick here and we go to our data protection snapshot task and we're just going to make it recursive. By doing so we'll give it a minute here because right now if we look at our datasets and that one runs every minute there's currently a total of zero. So I'm just going to fast forward here and wait a couple of minutes. All right, now we've jumped ahead a couple of minutes so we have a couple snapshots here for that new nested dataset. So it works and has the same retention policy as the other datasets do based on that setup but let's go back over to our window system and it still shows no previous versions available. Well, that's actually because there's not any differential that we have here. So let's go ahead and delete something out of here. Restore previous versions and there we go. Now they're showing up. Now the last piece I want to cover is actually creating these tasks. It's pretty simple but I'll cover it really quickly here. We pick a dataset and I don't recommend doing this and I've seen people do this where they'll just pick the root of the pool, save recursive and suddenly have snapshots of everything. This kind of creates a mess and I don't think you want to deal with that. I recommend having a plan where you take each one of the different datasets you have and set a retention policy that makes rational sense to you when doing this. Maybe this Nessa dataset needs two weeks of lifetime and that's fine and we can even say that. We'll call this one two dash weeks dash auto. And that way the naming scheme, these are the two week ones that we want done daily and we can exclude if we need certain filters here you can be specific about how you exclude them if you're doing it recursively but if we were gonna set a retention policy for this specific dataset, this specific lifetime and you can choose the units of our day, week, month, year. Maybe you have a really long retention policy but of note when you have these, the important part is that you have a process by which you know where they are because if you decided to keep a year of them you could run out of space. Please remember all snapshots are cumulative to that data set. So if you have only X amount of storage and you exceed that amount of storage and differential changes over a year, well you will run out of storage on there. So as much as it may be wonderful to keep a year's with the snapshots it may not be practical depending on your storage needs just something to keep in mind here that's also why it's important to have often Nessa datasets because some things you can set a really long retention policy because maybe they don't change often and they're just really archival. So having a long retention policy is not a problem and it can minimize some risk versus if you have a really frequently used folder that may not work as well so you may wanna assign that to a shorter lifetime. Once again it's gonna vary greatly based on your use case. If you're still noodling around about that nested data set and about different retention policies this is actually some projects we've completed for different government agencies and yes the government uses a lot more true nasty you may realize that store this data but then some of the data is only archived slowly like every 30 days they add there so they have a really long retention policy and the fear would be what if someone deleted something we wouldn't notice for so much time so they can have archival settings of years because essentially once the data is manipulated into there it's static. And of course there's a lot of ways you could set this data to read only after they have a lot of different policies around it but that's what having all these different datasets nested underneath each other with all different snapshot and retention policies that's what it's for is for real complicated business environments with sometimes some rather unique use cases. Now I didn't do this demo with Z-Vals but I did mention beginning this works with Z-Vals the most common use case is gonna be if you have a Z-Val that's attached to a server that you're running inside of TrueNAS as a VM this is a way you could easily manipulate the snapshots or even when you do a clone of a virtual machine that is how that's handled. The other use case of course is if you have iSCSI attached which is block device shared out to another system. Now the challenge becomes when you roll those back because ZFS doesn't necessarily have any idea what's inside there just manipulating the blocks does the OS that you've attached this to understand the differences when you roll it back or does it have some file system marker it's keeping track of. I bring that up because there are times and I've had a lot of people have challenges specifically like with Hyper-V when they've attached Hyper-V to an iSCSI Hyper-V seems to not like when you suddenly roll back to a previous state and as it's expecting something else and I've presented it that's just a few words for warning but this works much the same as far as ZFS concern but you do have to take into consideration whatever you're attaching it to will it be fine with a rollback? Just something to think about and some food for thought. Like and subscribe to see more content from this channel head over to LawrenceSystems.com to connect with you on whatever socials I'll be on when you go there. Also you'll find the ability to subscribe to my newsletter which I'm doing a monthly newsletter now so if you're interested in all the videos and some of the news articles that I share in there go ahead and sign up for that and find me over in my forums forums.lorencsystems.com to have a more in-depth discussion about this and other topics I've talked about on the channel and thanks.