 Happy World Backup Day. Yes, March 31st is World Backup Day and it's not the only day you should back up. It's just apparently some way to get people talking about backups. And I don't know if it's effective or not, but hey, why not use today to talk about backups? Specifically the immutable backups that everyone wants to achieve. This is, well, a little bit of a sticky topic because it's not well understood by a lot of the people asking it. They simply say I want my backups to be immutable or I want to have a place to store my backups where I can write the data, but it can never be deleted. The problem and the nuance that you immediately run into is okay, we will set up a repository, a place that can put the data, but you can't delete it. But at some point you can think about this and go unless you have infinite storage, well, you have to have a purge mechanism within there and that's where it gets a little bit more complicated. So mostly the way I always look at it is immutable is the word people are looking for, but specifically you're looking for hardened backups and a process to create a separation between the devices doing the backup and the ability to administer the machine, the server. In this case, we're gonna be talking about your NAS, but these same concepts can expand out to other servers and other types of backup targets, but it's all about making sure that the control plane is separate for both of these. That's often where the mistake is made, so we're gonna talk about that and dive into those details. But first, are you an individual or company looking for support on a network engineering, storage, or virtualization project? Is your company or internal IT team looking for someone to proactively monitor your system's security or offer strategic guidance to keep your IT systems operating smoothly? Not only would we love to help consulting your project, we also offer fully managed or co-managed IT service plans for businesses in need of IT administration or IT teams in need of additional support. With our expert install team, we can also assist you with all of your structured cabling and Wi-Fi planning projects. If any of this piques your interest, fill out our Hire Us form at lornsystems.com so we can start crafting a solution that works for you. If you're not interested in hiring us, but you're looking for other ways you wanna support this channel, there's affiliate links down below to get your deals and discounts on products and services we talk about on this channel. And now back to our content. And before we get into the hardening part, I wanna start with a diagram about how the data gets, the most common ways I should save for your data to get from where it is to where you want it to be in terms of duplicating it and packing it up. Those most common methodologies are gonna be over an SMB share, an NFS share, or using S3 object. Now S3 object is natively built in to TrueNAS scale and TrueNAS core, but also you can run this in a container, whether it be TrueNAS scale and Docker or TrueNAS core with a BST jail, but you can use MinIO for more advanced functionality that we'll talk about a little bit later and that is another methodology. Now this is gonna depend on your backup software and I'm keeping this relatively platform agnostic because that's not where the flaw or the problem really is. It's in the hardening of the servers. If you go through a lot of reports and wonder how threat actors were able to destroy backups, you'll frequently find that the password they had to admin their servers, admin the network was also the same password used and all on the same control plane for administering whatever target device they have. This is really where the flaw comes in. Matter of fact, the most common support question for a little while was when the update came from TrueNAS and they stopped allowing you to use root for the SMB shares, the number of people that claim that their SMB shares broke after the upgrade rose dramatically because an outstanding number of people simply use the same password that you log into your TrueNAS with, the root password for the SMB share. It was not an easy decision and people sometimes still will override that and keep using the root password. These are the really common reasons people get their backups deleted. Now, if you have a workstation and you're doing these backups, the next question, the really common one we've gotten where people ask about immutability is can't I set it to be write only? And the problem with write only, if you do it with an SMB share, NFS share, or even S3 object is, well, unless you have unlimited storage or has to be a mechanism, and of course you could probably write a cron job for this where you set these up to be write only and then set a cron job up to delete these. But generally it is the backup software that wants to control the history and revisions of these backups. So keeping it read write makes a lot of sense. So here's a few best practices though to make this very simple. When you're setting up whatever backup software we are and a lot of them will back up over like an SMB share is making sure that that username and password that you use is unique, not even unique to like the other users but unique only to the backups. This way if that password was compromised and let's say they were able to reveal that password or they have control over the software doing the backups on this particular system is sending it over, that does not give them any way to pivot so to speak to any other shares you may have on that TrueNAS. This sounds really simple but it's actually just a really common problem you run into, they just have it tied to part of the domain. Having your backups all as a separate control plane is absolutely critical to protecting them because that way if this gets taken over, yes, they will have whatever privilege your backup software may have to write these backups and then it's gonna go over to here and they're going to use that privilege to delete whatever that read write privilege is based on that but there's plenty of solutions around this to help mitigate these problems. First, when you're setting up your TrueNAS, setting up a series of snapshots and replications. I've got videos talking about snapshots and I got videos talking about replication. They are tied together and intermingled because what you wanna do is make sure whatever share that is being used and all that data that's coming here that does even include if you are using something like S3 object, you have a regular snapshot policy. Now, snapshot policies are really easy stuff in TrueNAS and only are differential to the data. So they don't take up a lot of space unless there's a lot of data to take up. And if you had, for example, a backup set to run at 10 p.m. at night, you could have a snapshot depending on how long that backup takes, let's say it finishes before 11, then you'd want that snapshot to run immediately after. That way you have a point reference in time that is not accessible over the SMB shares. One of the things with SMB is you can even have this viewed as a shadow copy but it's immutable and that term does come into play here because those volume shadow copies that could be seen from this workstation using the level of privilege it has would allow them to see the previous snapshots as shadow copies but offers no way to delete them. That is because the snapshots are handled at the dataset level of TrueNAS. The only way to delete them or purge them is A, on the schedule that you set where you set the lifetime of a snapshot. Lifetime of snapshot really depends on your data needs and your data retention policies and of course how much storage you have. But generally speaking, you may have those set for 30 days. So if you're doing the backups, you also can have the snapshot set to be 30 days and this gives you all those revisions. So even if they were to compromise this computer, go across the SMB share, get over to here and delete that last backup but you have a snapshot. The only obvious time problem might be if that snapshot occurred, well, after the backup was done but before they were able to delete it, there's that potential of timing that could come in there. That's a little bit eccentric to think that might happen but it could. Go a little bit more in-depth and set the snapshots to be constantly running so you have even more iterations of them but I'm not gonna get into those little details. That's just kind of obvious when I say it like that. Now, the next thing you wanna do is take that snapshot replication and have it on another server. You wanna make sure when you're doing things locally that you have the data replicated one more time. If you're putting it to one server, hey, servers fail. This is one of the reasons we have redundancies. Now RAID, the RAID array and the ZFS RAID is wonderful, it's great. I've talked a lot about it but RAID is not a backup. RAID is resiliency against single hard drive failures because they're the common moving component that may come into play for failures even if they're solid state, no moving parts but still chance of failure is still there. So taking these snapshots at the dataset at the block level and replicating to another TrueNAS great idea, easy enough to do and the secondary TrueNAS only has to have enough storage based on your needs but doesn't necessarily need to be the same speed. So frequently we have clients who have a server that serves up backups and other services maybe a target for their VM storage and then it replicates over to another server that hey, at least we have a copy of the data and that way it's a little bit more budget friendly to have one that's a little bit slower so you don't have to quite spend the money twice but hey, if you have the budget for two identical servers, that's even better. The next thing you wanna do is an encrypt before send I put on here because this is a good way to do your backups. Take your backups and encrypt before send this is something supported within TrueNAS when you're doing these two back plays and other services I happen to use back plays so I mentioned them, no affiliation for them just we've been using their offsite backups for a while and this allows me to have one more copy in the cloud. Now, once again, when I talk about managing control planes and keeping security separate the passwords we use to manage back plays and the usernames are not the same as the workstations and they're not the same passwords as this having each layer once again gives you that confidence to say, all right, they got in here and this is the most likely from someone clicks a phishing link or however this happens at some clients and they start working their way up to the server and then of course they're going to start destroying the backups but as long as they don't get access to the TrueNAS and then get access again to the way the cloud backups go that stops them from getting further. Now of note, yes, obviously the credentials you need the API key would be in here and allow them to destroy certain backups in here but you can also set life cycle retention policies at many cloud providers so they will have revisions themselves of data similar in the way to snapshots. Now, let's talk about this a little bit more practical about keeping things separate. Now, this is a nice write up I'm going to leave link down below I'm not going to walk through every little step because well, you can it's well documented how to set these things up and this is all solid best security practices here. One of the things that are going over in this and I actually like the how detailed they went in here they're talking about Veeam but honestly this applies to just hardening your TrueNAS system and keeping things very separate. They talk about going as far as locking down the web interface by turning it off turning on SSH when you set these up and SSHing in and starting and stopping the web service then going as far as saying, all right, restrict SSH so only one computer's IP address is in here and allowed to talk to the server configuring 2FA I've talked about this before which TrueNAS does support 2FA but right here if you have SSH to set automatically start and then you set the parameters within SSH to say we only want one user to be able to get in there these are all really solid advice when it comes to how far you want to go and lock this down. Now, having it binded the web interface itself and SSH to a different interface most TrueNAS systems when you set these up you're going to have a lot of them with multiple network cards where you can always add another network card to it but by doing this, this gives you that extra ability to say, all right, this particular network is for management this particular network is where we bind our SMB our NFS or our other shares keeping these different control planes separate means it's less likely for them to pivot over there. Now, this obviously comes with a great level of inconvenience because every time you want to log into the interface it's locked out and it's on a separate network but that's the trade-off it's also locked out in a separate network for someone who gets within your network they would have to know these extra levels of details such as the fact that you stop the web interface and start it over SSH. Now, the last thing I want to touch briefly on is S3 object locking and this does not mean using Amazon's S3 service this is a function of the S3 protocol. Now, with S3 you have the ability to do right once read many and this is where it's a little bit more complicated in your standard, well, some complexities I should say not really that much more complicated than using like an SMB or NFS share we're talking about how to handle this in a way that the system the object storage management system itself handles the locking and unlocking of the data that you put on here. And I'll leave links to this and of course the MinIO right up on object lock and immutability guide and they talk about how to set this up now, if you set up MinIO within that jail this is where you're going to get jail or in Docker if you're using Trunascale and you can go through and set some of these object storage. Now, of course, being able to use S3 is going to be very dependent on the backup software you're using whether or not it supports S3 at all. As I said, the more common support is going to be over SMB or NFS. But nonetheless, there are ways to do it but the advantage you get when this is you're now having a piece of software control the status of those data objects that you stored in there and then you can put the locks on there. So this changes the ability of the software or anyone taking over the software acquiring the API key and what they can access or delete, I should say because they would be able to read it. It's worm. So it's, you know, right once read many but it's not reading the backups. That's the problem. It's the fact that someone is able to delete them. Now, I have a few videos I'll leave linked down below including one specifically about hardening Trunascale and walking you through some of the functional steps to do that. And I know it sounds really obvious because it is really obvious that these web interfaces and the password should all be different but it needs to be said quite a few times for people to really understand it because read through any of these debriefs on a report of a company they got hacked and you'll find, well, yeah, they just tied it all together because that made administration simple or one user had privileges to all these different things and there weren't any extra steps in place. I mean, you can go further with this, of course and leave some ideas down in the comments below. Some people have even bound the web interface to a certain port and any unplug or even disable that port. So now the only way you get to the web interface would be to re-enable that port or physically plug it back in. There's also solutions like I have this little rugged sand disk in front of me and obviously, if you back up data to something like this I do recommend encrypting everything at rest as I do on this but the data backed up on here it's physically disconnected, put this in a safe. So now I have one more copy that is extremely offline. So if you were to get to all these other things I still have one more copy that you didn't get to but then you should even have multiple of these. Also, it's really hard to get people consistently and reliably backing up to external devices like this. It's a great idea to do. It's just something you have to really think about processes because automated backups are great with checks and balances but people processes when it comes to backups sometimes they forget or we've seen lots of times where the backup was plugged in and no one ever changed it out like they're supposed to. It gets skipped one day, the next day, the next day the next thing you know no one's even swapping anymore because you can see the amount of dust on it. Been to plenty of offices and seen that. Nonetheless, I'll leave links down below to the things I talked about and thank you. Hopefully you understand a little bit better about how to harden your backups how to think about separating these things leave your comments down below of course and head over to the forum for a more in-depth discussion. And thank you for making it all the way to the end of this video. If you've enjoyed the content please give us a thumbs up. If you would like to see more content from this channel hit the subscribe button and the bell icon. If you'd like to hire a short project head over to laurancesystems.com and click the hires button right at the top. To help this channel out in other ways there's a join button here for YouTube and a Patreon page where your support is greatly appreciated. For deals, discounts, and offers check out our affiliate links in the description of all of our videos including a link to our shirt store where we have a wide variety of shirts that we sell and designs come out well randomly so check back frequently. And finally our forums forums.laurancesystems.com is where you can have a more in-depth discussion about this video and other tech topics covered on this channel. Thanks again for watching and look forward to hearing from you.